Image display method and image display device

Abstract

An image display device (100) causes a monitor (110) to display a surrounding image in which the surroundings of a vehicle are captured by a plurality of cameras (211) provided to the vehicle. A visibility calculation unit (103) acquires surrounding environment information pertaining to the surroundings of the vehicle from the cameras (211) or a short-range sensor (208) provided to the vehicle, and calculates the visibility from the driver in individual imaging regions of the plurality of cameras (211) on the basis of the visual field region of the driver of the vehicle in the imaging regions and the surrounding environment information. A history storage unit (104) stores an imaging region selected on the basis of the visibility in a storage unit in association with position information indicating the position of the vehicle when the surrounding environment information is acquired. When the vehicle travels at the position indicated by the position information stored in the storage unit, a priority determination unit (105) increases the priority for displaying the surrounding image of the imaging region associated with the position information, and a display control unit (106) causes the monitor (110) to display the surrounding image selected in accordance with the priority.

Description

Image display method and image display device

[0001] The present invention relates to an image display method and an image display device.

[0002] There is a technology that displays camera images of the vehicle's surroundings, allowing the driver to visually recognize approaching vehicles, obstacles, and the like that are not directly visible to the driver (see, for example, Patent Document 1). The in-vehicle camera system described in Patent Document 1 determines the driving situation of the vehicle, and, depending on the determined driving situation, is able to display camera images associated with the driving situation from among multiple camera images as candidate images on a display device, and selectively displays a desired image selected from the candidate images by operating an operation unit on the display device. It is explained that this makes it possible to easily display camera images from among a large number of camera images according to the driving situation.

[0003] Japanese Patent Application Laid-Open No. 2004-161215

[0004] The in-vehicle camera system of Patent Document 1 determines the driving situation as when the vehicle starts moving, when moving forward at a low speed, when moving backward, or when moving in other directions, and selects and displays camera images as candidate images depending on the driving situation. However, because the visible area or the area that the driver needs to see varies depending on the surrounding environment at the time of driving, it is necessary to select an appropriate surrounding image depending on the surrounding environment, which places a heavy burden on the selection process.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image display method and an image display device that can reduce the processing required to select an appropriate surrounding image according to the surrounding environment.

[0006] In order to achieve the above object, the present invention provides an image display method that uses a processor to display, on a monitor mounted on the vehicle, surrounding images captured by multiple cameras mounted on the vehicle. The processor acquires ambient environment information about the surroundings of the vehicle from the cameras or sensors mounted on the vehicle, calculates a visibility probability from the driver in each of the imaging areas of the multiple cameras based on the driver's field of view in the imaging areas of the multiple cameras and the ambient environment information, and stores in a memory unit an imaging area selected based on the calculated visibility probability in association with position information indicating the position of the vehicle when the ambient environment information was acquired. The processor also increases the priority of displaying the surrounding image of the imaging area associated with the position information when the vehicle is traveling at a location indicated by the position information stored in the memory unit, and displays the surrounding image of the imaging area selected in accordance with the priority on the monitor.

[0007] According to the present invention, the imaging area selected based on visibility is stored in correspondence with the vehicle's location information, and the surrounding images of the stored imaging area are displayed with increased priority when traveling at the same location, thereby reducing the process of selecting appropriate surrounding images according to the surrounding environment.

[0008] Fig. 1 is a block diagram showing an example of the functional configuration of a display system including an image display device according to an embodiment of the present invention; Fig. 2 is a diagram showing an example of the hardware configuration of an image display device according to an embodiment; Fig. 3 is a diagram showing a sensor detection area and an imaging area around a vehicle; Fig. 4 is a diagram showing an example of a visual recognition area and a required visual recognition area; Fig. 5 is a diagram showing another example of a visual recognition area and a required visual recognition area; Fig. 6 is a diagram showing an example of a display that displays a surrounding image and an imaging area; Fig. 7 is a flowchart of image display processing;

[0009] An image display method and an image display device according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or equivalent parts are designated by the same reference numerals.

[0010] (Embodiment) An image display device 100 according to an embodiment of the present invention is a device that controls the display of images on a monitor 110 inside a vehicle cabin, and controls the display of surrounding images, etc., automatically selected based on visibility from among images of the vehicle's surroundings captured by multiple cameras mounted on the vehicle.

[0011] 1 is a block diagram showing an example of the functional configuration of a display system 1 including an image display device 100 according to the present embodiment. The display system 1 is a system mounted on a vehicle, and realizes a function of automatically displaying an image of the surroundings of the vehicle on a monitor 110 when an automatic display mode is enabled. The automatic display mode is a mode in which an image of the surroundings is automatically displayed on the monitor 110, for example, when the vehicle speed is equal to or lower than a predetermined speed or when the vehicle is located at a caution point such as an accident location registered in map information. The automatic display mode can be switched between enabled and disabled by a driver's setting. The following description will be given assuming that the automatic display mode is enabled.

[0012] As shown in FIG. 1, the display system 1 includes a group of sensors 200 for detecting driver operations, including a steering sensor 201, an accelerator sensor 202, a brake sensor 203, and a turn signal sensor 204, a user interface 205 for acquiring information input operations by the driver, and a GPS (Global Positioning System) device 206 and a wheel speed sensor 207 for detecting the position of the vehicle.

[0013] The display system 1 further includes a short-range sensor 208 for detecting the surrounding environment of the vehicle, a seat sensor 209 for detecting the fore-and-aft position, inclination or height of the driver's seat, a driver monitoring camera 210, a plurality of cameras 211 for capturing images of the surroundings of the vehicle, a monitor 110, and an image display device 100 that selects surrounding images with low visibility from the surrounding images captured by the plurality of cameras 211 based on the output of the various sensors and controls the display of the images on the monitor 110.

[0014] Fig. 2 is a diagram showing an example of the hardware configuration of the image display device 100. In the example of Fig. 2, the image display device 100 includes a processor 1011, a storage unit 1012, and a communication interface 1013, which are connected to each other via a bus 1010.

[0015] The processor 1011 includes, for example, one or more CPUs (Central Processing Units) and their peripheral circuits, and executes various types of arithmetic processing. The processor 1011 executes control programs stored in the storage unit 1012. The processor 1011 may include a volatile semiconductor memory such as a RAM (Random Access Memory) that functions as a working memory for the CPU. The processor 1011 may also include an arithmetic circuit such as a logical arithmetic unit or a numerical arithmetic unit.

[0016] The storage unit 1012 includes a non-volatile semiconductor memory such as an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, etc. The storage unit 1012 stores the control program executed by the processor 1011 and various data used in the arithmetic processing of the processor 1011.

[0017] The communication interface 1013 includes an interface circuit for connecting the processor 1011 to an in-vehicle network that complies with standards such as CAN (Controller Area Network). The communication interface 1013 receives signals from on-board components such as the sensor group 200, the user interface 205, the camera 211, the GPS device 206, and the wheel speed sensor 207, and passes the signals to the processor 1011. The communication interface 1013 also outputs an image selected from images of the surroundings of the vehicle acquired by the camera 211 to the monitor 110.

[0018] For example, the processor 1011 of the image display device 100 executes an image display processing program, causing the image display device 100 to realize each function shown in Fig. 1. That is, the image display device 100 includes an operation detection unit 101 that detects a driver's operation based on the output of the sensor group 200 and the user interface 205, a self-position detection unit 102 that detects the current position of the vehicle based on the output of the GPS device 206 and the wheel speed sensor 207, and a visibility calculation unit 103 that calculates visibility based on surrounding environment information from a short-range sensor 208. Note that the visibility calculation unit 103 can use the output of at least one of the seat sensor 209 and the driver monitoring camera 210.

[0019] Furthermore, the image display device 100 includes a history memory unit 104 that stores the driver's operation detected by the operation detection unit 101, the current position of the vehicle detected by the self-position detection unit 102, and the visibility calculated by the visibility calculation unit 103 in correspondence with each other, a priority determination unit 105 that determines the priority of displaying the surrounding image captured by the camera 211 based on the visibility calculated by the visibility calculation unit 103 or the information stored in the history memory unit 104, and a display control unit 106 that controls the display of the surrounding image with the highest priority among the surrounding images captured by the camera 211.

[0020] The operation detection unit 101 detects the driver's operation based on the output of a sensor group 200 including a steering sensor 201, an accelerator sensor 202, a brake sensor 203, and a turn signal sensor 204, and a user interface 205 that acquires the driver's information input operation.

[0021] The steering sensor 201 detects the steering angle, which is the rotation angle of the steering wheel, and the operation detection unit 101 detects an operation to change the course of the vehicle based on the output of the steering sensor 201. The accelerator sensor 202 detects the accelerator opening, which is the amount of depression of the accelerator pedal, and the operation detection unit 101 detects an operation to accelerate the vehicle based on the output of the accelerator sensor 202.

[0022] The brake sensor 203 detects the amount of braking operation, which is the amount of depression of the brake pedal, and the operation detection unit 101 detects the deceleration operation of the vehicle based on the output of the brake sensor 203. The turn signal sensor 204 detects the operating state of the turn signal, and the operation detection unit 101 detects the left or right turn signal operation based on the output of the turn signal sensor 204.

[0023] The user interface 205 is an interface that accepts operation inputs from the driver to various electronic devices, control devices, etc., and accepts, for example, a pressing operation of a changeover switch or the like that switches the surrounding image displayed in the automatic display mode, and outputs an operation signal to the operation detection unit 101. The changeover switch is a switch that switches the displayed surrounding image sequentially when pressed by the driver. The changeover switch may be provided in any position, and may be provided, for example, in a switch unit around the steering wheel or the monitor.

[0024] The GPS device 206 receives GPS satellite signals, acquires the latitude and longitude of the vehicle, and obtains the vehicle's position on a map. The wheel speed sensors detect the rotational speed of the wheels. The self-position detection unit 102 can detect the current position of the vehicle or the exact distance traveled based on the output of at least one of the GPS device 206 and the wheel speed sensor 207.

[0025] A plurality of short-range sensors 208 are provided in the vehicle and detect the shape and size of any object that may act as an obstruction around the vehicle, such as a building, plant, or parked vehicle, and are, for example, SONAR (Sound Navigation and Ranging), LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), LRF (Laser Range Finder), or a camera. Furthermore, a plurality of cameras 211 can be used as the short-range sensors 208. The shape and size of any object that may act as an obstruction around the vehicle, such as a building, plant, or parked vehicle, can be detected from surrounding images captured by the plurality of cameras 211.

[0026] The seat sensor 209 detects the position of the driver's seat, including the front-to-rear position, inclination (recline), and height of the seat. The driver monitoring camera 210 captures an image of the driver's face and detects the position of the driver's eyes.

[0027] A plurality of cameras 211 are provided on the vehicle and capture images in approximately all directions from the vehicle. Figure 3 shows an example of the detection area of ​​the short-range sensor 208 and the image capture area of ​​the camera 211. As shown in Figure 3, the short-range sensor 208 can detect any object present in front of, behind, or to the side of the vehicle.

[0028] Furthermore, one or more cameras 211 provided at the front end of the vehicle can capture images of a forward imaging area extending forward and to the left and right from the front end of the vehicle, and one or more cameras 211 provided at the rear end of the vehicle can capture images of a rear imaging area extending backward and to the left and right from the rear end of the vehicle. Furthermore, two or more cameras 211 provided on the sides of the vehicle can capture images of side imaging areas extending to the left and right from the sides of the vehicle. The surrounding images captured by these cameras 211 allow the driver to view blind spots that cannot be seen with the naked eye.

[0029] The visibility possibility calculation unit 103 calculates visibility based on the output of the surrounding environment information from the short-range sensor 208. The visibility is an index that indicates how much of the required visibility area that the driver can see can be occupied by the visibility area that the driver can see. An example of calculating visibility will be described with reference to FIGS. 4 and 5.

[0030] 4 shows an example of a case where a vehicle turns left at an intersection on a road where traffic is left-hand. Whether the vehicle is turning left can be determined based on the shape of the surrounding road obtained from map information, the operating state of the left turn signal detected by the operation detection unit 101 using the turn signal sensor 204, route information from a navigation system, the driving route of an autonomous driving system, etc. In this case, it is necessary to determine at least the visibility of the area to the right (hereinafter also referred to as the right front direction) from the front end of the vehicle on the road onto which the vehicle is turning left. In other words, it is necessary to determine the priority of the surrounding image captured by the camera 211 whose imaging area includes the right front direction of the vehicle.

[0031] The visibility possibility calculation unit 103 can calculate a predicted driver's visual field area from the driver's seat position and window area obtained from the vehicle specifications. Furthermore, the visibility possibility calculation unit 103 may predict the driver's eye position based on the output of at least one of the seat sensor 209 and the driver monitoring camera 210 in order to accurately obtain the driver's visual field range in advance. The visibility possibility calculation unit 103 may calculate a predicted driver's visual field area from the driver's eye position and the window area. Here, in order to calculate the visual field area accurately and in a short time, the window area may be stored in advance in the storage unit 1012. The window area may be the position, size, and shape from the driver's seat. Furthermore, the driver's visual field area may be calculated in advance, such as when starting to drive. In the example of FIG. 4 , the calculated visual field area is the driver's visual field area in the right front direction of the vehicle.

[0032] Next, in order to determine the driver's visible range according to the surrounding environment, the visibility possibility calculation unit 103 acquires surrounding environment information indicating areas obstructed by objects, such as buildings, plants, and parked vehicles, that are located around the vehicle and obstruct the driver's field of view, based on the output of the short-range sensor 208, and calculates a visible area excluding the areas obstructed by the objects from the driver's field of view. Note that the surrounding environment information indicating the areas obstructed by the obstructions may be information indicating the position and shape of the obstructions or the surface of the obstructions. In the example of FIG. 4 , the visibility possibility calculation unit 103 determines the area indicated by the straight-line frame in FIG. 4 as the visible area 2001, excluding the area obstructed by a wall 2010, which is an obstruction detected by the short-range sensor 208, from the field of view.

[0033] Furthermore, the visibility possibility calculation unit 103 may calculate a required visual confirmation area, which is an area that the driver should be able to see, based on the road shape acquired from map information. In the example of Fig. 4, in the lane into which the vehicle is turning left and entering, an area surrounded by a dashed line in Fig. 4 is set as a required visual confirmation area 2002 that the driver should be able to see in order to confirm that no other vehicles are traveling within a predetermined distance from the vehicle.

[0034] The visibility possibility calculation unit 103 calculates the proportion of the driver's visual recognition area 2001 in the required visual recognition area 2002. That is, the visibility possibility calculation unit 103 calculates the visibility from the area of ​​an overlapping area 2003, which is shown shaded in FIG. 4 and where the visual recognition area 2001 and the required visual recognition area 2002 overlap. Specifically, the visibility possibility calculation unit 103 calculates the ratio of the area of ​​the overlapping area 2003 to the area of ​​the required visual recognition area 2002 as the visibility. If this area ratio is small, it can be determined that the driver cannot see a range sufficient for safety confirmation.

[0035] 5 shows an example of a case where a vehicle turns right at an intersection on a road where traffic is left-hand. Whether the vehicle is turning right can be determined based on the shape of the surrounding road obtained from map information, the operating state of the right turn signal detected by the operation detection unit 101 using the turn signal sensor 204, route information from a navigation system, the route of an autonomous driving vehicle, and the like. In this case, it is necessary to calculate not only the visibility to the right (right front direction) from the front end of the vehicle on the road where the vehicle is turning right, but also the visibility to the left (hereinafter also referred to as the left front direction) from the front end of the vehicle. In other words, it is also necessary to determine the priority of the surrounding image captured by the camera 211 whose imaging area includes the left front direction of the vehicle.

[0036] 4, the visibility possibility calculation unit 103 predicts the position of the driver's eyes based on the output of at least one of the seat sensor 209 and the driver monitoring camera 210, and calculates in advance the driver's field of view predicted from the driver's eye position and the window area. The field of view calculated here is the driver's field of view in the left front direction of the vehicle.

[0037] Next, the visibility possibility calculation unit 103 acquires surrounding environment information including areas around the vehicle that are blocked by obstacles based on the output of the short-range sensor 208, and calculates a visibility area that excludes the areas blocked by the obstacles from the field of view. In the example of Figure 5, the visibility possibility calculation unit 103 determines the area shown by the straight line frame in Figure 5 as the visibility area 2011, excluding the area blocked by a wall 2020 detected by the short-range sensor 208 from the field of view.

[0038] Furthermore, the visibility possibility calculation unit 103 calculates a required visibility area, which is an area that the driver should be able to see, based on the road shape acquired from the map information. In the example of Fig. 5 , in a lane into which the vehicle is turning right and entering, an area surrounded by a dashed line in Fig. 5 is set as a required visibility area 2012 that the driver should be able to see in order to confirm that no other vehicles are traveling within a predetermined distance from the vehicle. The visibility possibility calculation unit 103 calculates, as the visibility possibility, the ratio of the area of ​​an overlapping area 2013, which is shown shaded in Fig. 5 and where the visibility area 2011 and the required visibility area 2012 overlap, to the area of ​​the required visibility area 2012.

[0039] If the visibility calculated in this manner is low, the driver cannot sufficiently confirm safety and cannot proceed, so the driver needs to visually confirm the required visibility areas 2002, 2012 using the surrounding image captured by the front camera 211.

[0040] That is, in order to select the surrounding images with the highest display priority from among the surrounding images captured by the multiple cameras 211, the visibility possibility calculation unit 103 calculates the visibility based on the area of ​​the overlapping area between the driver's visibility area and the required visibility area in the image capture areas of the multiple cameras 211.

[0041] The priority determination unit 105 determines the priority of displaying the surrounding image captured by the camera 211 on the monitor 110 based on the visibility calculated by the visibility calculation unit 103, position information, etc. The priority determination unit 105 may determine the priority of the imaging area of ​​the surrounding image to be displayed, and may determine priorities 1 to 4, for example, as follows: 1: front right, 2: front left, 3: right, 4: rear.

[0042] When the vehicle again travels through the position indicated by the position information stored in the storage unit 1012, the priority determination unit 105 determines that the priority of the surrounding image of the imaging area associated with the position information is high. More specifically, a high priority may be set according to the visibility associated with the position information. When the position information is not stored in the storage unit 1012, the priority determination unit 105 determines the priority of displaying the surrounding image on the monitor 110 based on the newly calculated visibility. More specifically, the priority determination unit 105 determines the priority of the imaging area in ascending order of the newly calculated visibility.

[0043] When the priority determination unit 105 determines that the priority is high based on the visibility, the history memory unit 104 stores in the memory unit 1012 the driver's operation detected by the operation detection unit 101 when the surrounding environment information was acquired to calculate the visibility, the position information of the vehicle detected by the self-position detection unit 102, the imaging area for which the visibility was calculated, and the visibility, in correspondence with each other.

[0044] 4 and 5 , when the operation detection unit 101 detects that the vehicle has stopped temporarily to turn right or left from a side road and that a turn signal has been operated, the visibility calculation unit 103 calculates the visibility of the image capture area in the left front direction or the right front direction. When the priority determination unit 105 determines that the priority is high based on the visibility, the history storage unit 104 stores the operation at that time, the vehicle position information, the image capture area, and the visibility in the storage unit 1012.

[0045] 4 and 5, if camera 211 provided at the front end of the vehicle is a wide-angle camera capable of capturing images of both necessary visual recognition areas 2002 and 2012, the forward area including the left front direction and the right front direction may be stored as the captured image area. On the other hand, if two or more cameras 211 are capable of capturing images of necessary visual recognition area 2002 in the right front direction and necessary visual recognition area 2012 in the left front direction, respectively, a left turn operation and the captured image area in the right front direction may be stored as shown in Fig. 4, and a right turn operation and the captured image areas in the right front direction and the left front direction may be stored as shown in Fig. 5.

[0046] The history storage unit 104 may also store operations performed by the driver on the user interface 205, such as pressing a switch to switch the displayed surrounding image, in association with the visibility possibility.

[0047] The display control unit 106 controls the monitor 110 to display the surrounding images captured by the camera 211 in descending order of priority determined by the priority determination unit 105. Furthermore, the display control unit 106 may cause the monitor 110 to display a schematic representation of the imaging area of ​​the displayed surrounding image.

[0048] 6 shows an example of the display on the monitor 110 when two or more cameras 211 capture images in the right front direction and the left front direction, respectively, and the image captured in the right front direction by the camera 211 provided at the front of the vehicle is displayed on the main screen 1101 as the surrounding image with the highest priority. Furthermore, the surrounding image captured in the left front direction by the camera 211 provided at the front of the vehicle is displayed on the first sub-screen 1102 as the next candidate image.

[0049] 6, the monitor 110 may display a top view in which the captured area of ​​the surrounding image on the main screen 1101 is superimposed on an overhead view or an around view on the second sub-screen 1103. The captured area is indicated by a dotted line on the second sub-screen 1103. This allows the driver to check the surrounding image capturing an area with low visibility together with the captured area.

[0050] The operation of the image display device 100 configured as above will be described in detail with reference to the flowchart in Fig. 7. Fig. 7 is a flowchart of image display processing executed by the image display device 100 of a vehicle. The processing shown in the flowchart in Fig. 7 may be executed when the operation detection unit 101 detects a predetermined driver operation such as decelerating to a certain speed or below, pressing a selector switch, or when the vehicle is located at a caution point such as an accident location registered in map information.

[0051] First, the self-position detection unit 102 acquires the current position of the vehicle based on the output of at least one of the GPS device 206 and the wheel speed sensor 207 (step S101). If the acquired current position matches the position information stored in the storage unit 1012 (step S102: Yes), the priority determination unit 105 sets a high priority to the image capture area stored in association with the position information (step S103), and the process proceeds to step S106. Here, the case where the current position matches the position information may include the case where the current position is within a range of a predetermined distance from the position indicated by the position information.

[0052] The high priority set in step S103 may be set, for example, according to the visibility stored in association with the position information in the storage unit 1012. Furthermore, the priority determination unit 105 may change the priority according to whether the driver's operation stored in association with the position information matches the operation detected by the operation detection unit 101.

[0053] If the current position does not match the position information stored in the storage unit 1012 (step S102: No), a process is performed to calculate a new visibility and select an image capture area with low visibility in order to select the optimal surrounding image for that position. Specifically, first, the visibility calculation unit 103 calculates the visibility of each image capture area around the vehicle based on the output of the surrounding environment information from the short-range sensor 208 (step S104).

[0054] In this case, the visibility possibility calculation unit 103 may limit the imaging area for which visibility is calculated in accordance with the operation detected by the operation detection unit 101. For example, when the operation detection unit 101 detects a forward operation, the visibility possibility calculation unit 103 may calculate the visibility possibility of the imaging area in the front and lateral directions, and when the operation detection unit 101 detects a reverse operation, the visibility possibility calculation unit 103 may calculate the visibility possibility in the right front direction and the left front direction. Alternatively, when the operation detection unit 101 detects the operation of a right turn signal, the visibility possibility calculation unit 103 may calculate the visibility possibility in the right front direction and the left front direction.

[0055] In step S104, first, visibility possibility calculation unit 103 calculates a predicted field of view of the driver from the driver's seat position and window area obtained from the vehicle specifications. Furthermore, visibility possibility calculation unit 103 acquires surrounding environment information indicating areas obstructed by surrounding objects that obstruct the driver's field of view, such as buildings, plants, and parked vehicles, based on the output of short-range sensor 208, and calculates visible areas 2001, 2011, which are the field of view excluding areas obstructed by objects.

[0056] Thereafter, the visibility possibility calculation unit 103 calculates the ratio of the area of ​​the overlapping areas 2003, 2013 between the predetermined necessary visual recognition areas 2002, 2012 that the driver must recognize and the visual recognition areas 2001, 2011 to the area of ​​the necessary visual recognition areas 2002, 2012 as the visibility possibility (step S104).

[0057] The priority determination unit 105 determines the priority of the imaging area based on the visibility. More specifically, the priority determination unit 105 sets the priority in ascending order of visibility (step S105).

[0058] The display control unit 106 displays the surrounding images captured by the camera according to the priorities set in steps S103 and S105 on the monitor 110 (step S106). At this time, as shown in Fig. 6 , the surrounding image of the imaging area with the highest priority may be displayed on the central main screen 1101, the surrounding image of the imaging area with the next highest priority may be displayed on the first sub-screen 1102 in the corner of the screen, and a top view in which the imaging area is represented by a frame line (dotted line) may be displayed on the second sub-screen 1103 at the edge of the screen.

[0059] When the automatic display mode switch is pressed while a high-priority surrounding image is being displayed on the main screen 1101 at the center of the monitor 110 (step S107: Yes), and if the automatic display mode switch is not pressed and held (step S108: No), the surrounding image of the imaging area with the next highest priority is displayed on the main screen 1101 (step S109), and the process returns to step S107. By repeating steps S107 to S109 in this manner, the display control unit 106 can switch and display the surrounding images in descending order of priority. At this time, all surrounding images may be switched and displayed in descending order of priority, or surrounding images associated with visibility possibilities lower than a predetermined threshold may be switched and displayed in descending order of priority.

[0060] In step S107, if the selector switch is not pressed for a certain period of time (step S107: No), the history storage unit 104 registers the currently displayed image capture area, the visibility probability, the detected operation, and the location information in association with one another (step S110). By storing the location information, visibility probability, image capture area, and operation in association with one another in the storage unit 1012, the process of automatically selecting the image capture area can be simplified the next time the vehicle travels through the same location and the same operation is detected. Furthermore, if the display of the surrounding image is switched by the driver's operation in steps S107 to S109, the image capture area and location information of the switched surrounding image can be registered, allowing the driver's selection to be reflected in the display of the surrounding image from the next time onward.

[0061] When the automatic display mode switch is pressed while any of the surrounding images is being displayed on the monitor 110 (step S107: Yes), and a long press is detected (step S108: Yes), the display control unit 106 displays a manual selection screen. On the manual selection screen, the user interface 205 accepts deletion of position information and the like stored in the storage unit 1012, or manual registration of an imaging area and position information (step S111). Then, the process ends.

[0062] After the processing is completed, the display control unit 106 may terminate the display of the surrounding image when a predetermined time has elapsed or when a predetermined operation is detected, such as acceleration to a predetermined speed or higher.

[0063] As described above, in the image display device 100 according to the present embodiment, the visibility calculation unit 103 calculates the visibility of each of the imaging areas of the multiple cameras 211 from the driver based on the driver's field of view in the imaging areas of the multiple cameras 211 and the surrounding environment information around the vehicle acquired by the short-range sensor 208. The history storage unit 104 stores the imaging area selected based on the visibility in the storage unit 1012 in association with location information indicating the location of the vehicle when the surrounding environment information was acquired. When it is detected that the vehicle is traveling at the location indicated by the location information stored in the storage unit 1012, the priority determination unit 105 increases the priority for displaying the surrounding image of the imaging area associated with the location information, and the display control unit 106 displays the surrounding image selected according to the priority on the monitor 110. This reduces the amount of processing required to select an appropriate surrounding image according to the surrounding environment at the time of traveling.

[0064] The hardware configuration and flowcharts shown in the above embodiment are merely examples and can be modified or applied as desired. For example, in the above embodiment, the imaging area includes the right front direction and the left front direction, but the imaging area can be divided in any way. A wider imaging area may be set depending on the viewing angle of the camera, or an imaging area divided into smaller parts may be set.

[0065] Furthermore, in the above embodiment, at a position where the position information is stored in the storage unit 1012, the priority of the imaging area associated with the position information is set and the visibility is not newly calculated, but this is not limited to this. The visibility calculation unit 103 may recalculate the visibility at that position every time and update the information such as the imaging area stored in the storage unit 1012. Alternatively, when a predetermined operation is detected or a predetermined surrounding environment is detected, the visibility may be calculated and the information may be updated.

[0066] In addition, in the above embodiment, pressing the automatic display mode switch switches the display of the surrounding image on the monitor 110, and pressing and holding the switch accepts manual registration, but these switches or operations are examples, and other switches or operations may also be used.

[0067] In addition, in the above embodiment, an example has been described in which each function is realized by the processor 1011 executing a control program, but the image display device 100 may also be configured using dedicated hardware that realizes each function.

[0068] Furthermore, a control program for executing the operations of the above-described embodiments may be stored and distributed on a computer-readable recording medium such as a CD-ROM (Compact Disc Read-Only Memory), a DVD (Digital Versatile Disc), an MO (Magneto Optical Disc), or a memory card, and the program may be installed on a computer to configure the image display device 100 that can realize each function. When each function is realized by sharing the work between an OS (Operating System) and an application, or by cooperation between the OS and an application, only the parts other than the OS may be stored on the recording medium.

[0069] The present invention allows various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to explain the present invention and do not limit the scope of the present invention. In other words, the scope of the present invention is defined by the claims, not by the embodiments. Various modifications made within the scope of the claims and the meaning of the disclosure equivalent thereto are considered to be within the scope of the present invention.

[0070] 1 Display system, 100 Image display device, 101 Operation detection unit, 102 Self-position detection unit, 103 Visibility calculation unit, 104 History storage unit, 105 Priority determination unit, 106 Display control unit, 110 Monitor, 200 Sensor group, 201 Steering sensor, 202 Accelerator sensor, 203 Brake sensor, 204 Turn signal sensor, 205 User interface, 206 GPS device, 207 Wheel speed sensor, 208 Short-distance sensor, 209 Seat sensor, 210 Driver monitoring camera, 211 Camera, 1010 Bus, 1011 Processor, 1012 Memory unit, 1013 Communication interface, 1101 Main screen, 1102 First sub-screen, 1103 Second sub-screen, 2001, 2011 Visibility area, 2002, 2012 Required visibility area: 2003, 2013 Overlap area: 2010, 2020 Wall.

Claims

1. An image display method in which surrounding images of the surroundings of a vehicle captured by multiple cameras mounted on the vehicle are displayed on a monitor mounted on the vehicle using a processor, wherein the processor: acquires ambient environment information about the surroundings of the vehicle from the cameras or sensors mounted on the vehicle; calculates the visibility of the imaging area of ​​each of the multiple cameras from the driver based on the field of view of the driver of the vehicle in the imaging areas of the multiple cameras and the ambient environment information; stores the imaging area selected based on the calculated visibility in a memory unit in association with position information indicating the position of the vehicle when the ambient environment information was acquired; increases the priority of displaying the surrounding image of the imaging area associated with the position information when the vehicle is traveling at a position indicated by the position information stored in the memory unit; and displays the surrounding image of the imaging area selected in accordance with the priority on the monitor.

2. The image display method of claim 1, wherein when the vehicle travels through a location where the location information is not stored in the memory unit, the processor newly calculates the visibility, sets the priority in descending order of visibility, and displays the surrounding image of the imaging area selected according to the priority on the monitor.

3. The image display method according to claim 1 or 2, wherein when the selected surrounding image is displayed and the driver switches the displayed surrounding image by pressing a changeover switch, the processor stores the imaging area and the position information of the surrounding image displayed after the switch in the memory unit.

4. The image display method according to any one of claims 1 to 3, wherein the processor sets the priority in descending order of the visibility possibility, and switches the surrounding image in descending order of the priority each time the driver presses the changeover switch.

5. An image display method according to any one of claims 1 to 4, wherein the field of view of the driver is calculated based on at least one of the window area of ​​the vehicle, the seat position, and the output of a driver monitoring camera.

6. An image display method according to any one of claims 1 to 5, wherein the visibility is calculated from the area of ​​overlap between a visibility area calculated based on the driver's field of view and the surrounding environment information, and a required visibility area, which is an area that the driver should be able to see, calculated based on road shapes obtained from map information.

7. The image display method according to claim 6, wherein the required visual recognition area is calculated based on the road shape and the operating state of a turn signal.

8. An image display method according to any one of claims 1 to 7, wherein the imaging area includes a right front direction and a left front direction of the vehicle, and the imaging area for calculating the visibility is selected based on the operating state of a turn signal.

9. An image display method according to any one of claims 1 to 8, wherein the surrounding environment information is information including areas around the vehicle that are obscured by obstructions, the information being acquired using a short-range sensor.

10. An image display device that displays surrounding images of the surroundings of a vehicle captured by multiple cameras mounted on the vehicle on a monitor mounted on the vehicle, comprising: a processor that acquires surrounding environment information of the surroundings of the vehicle from the cameras or sensors mounted on the vehicle, calculates the visibility of the imaging area of ​​each of the multiple cameras from the driver based on the field of view of the driver of the vehicle in the imaging areas of the multiple cameras and the surrounding environment information, and displays the surrounding image of the imaging area selected in accordance with the visibility on the monitor; and a memory unit that stores the imaging area selected in accordance with the visibility in association with position information indicating the position of the vehicle when the surrounding environment information was acquired, wherein the processor, when the vehicle is traveling at a position indicated by the position information stored in the memory unit, increases the priority of displaying the surrounding image of the imaging area associated with the position information, and displays the surrounding image of the imaging area selected in accordance with the priority on the monitor.

Images