Image Processing Device

Abstract

To provide an image processing device which can inform a crew member of a vehicle of approach of an emergency vehicle.SOLUTION: An information processing device 3 comprises: an information acquisition unit; a determination unit; and an image change unit. The information acquisition unit acquires information representing a position of an emergency vehicle. The determination unit determines whether or not the emergency vehicle approaches a vehicle. The image change unit changes images when the determination unit determines that the emergency vehicle approaches the vehicle when the vehicle automatically drives.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device. [Background technology]

[0002] When a vehicle is performing autonomous driving, an emergency vehicle may approach the vehicle. Patent Document 1 discloses a vehicle control device that executes control to avoid the emergency vehicle while continuing autonomous driving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-156144 A Summary of the Invention [Problem to be solved by the invention]

[0004] When an emergency vehicle approaches a vehicle, depending on the circumstances around the vehicle, the vehicle may need to switch from automatic driving to manual driving and avoid the emergency vehicle through the driving operation of the vehicle occupant. However, the occupant of a vehicle that is automatically driving may not notice the approach of the emergency vehicle.

[0005] In one aspect of the present disclosure, it is preferable to provide an image processing device that can notify a vehicle occupant of an approaching emergency vehicle. [Means for solving the problem]

[0006] One aspect of the present disclosure is an image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2). The image processing device includes an information acquisition unit (35) configured to acquire information representing the position of an emergency vehicle (73). but Based on the information obtained, before The emergency vehicle is approaching the vehicle. mosquito configured to determine whether judgmenta disconnecting unit (36) and, when the vehicle is in automatic driving mode, The emergency vehicle is notified when it is approaching the vehicle. Note judgment and an image modification unit (37) for modifying the image when the disconnection unit determines that the image is 。

[0007] An image processing device according to one aspect of the present disclosure can notify vehicle occupants of an approaching emergency vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of an in-vehicle system. [Diagram 2] FIG. 2 is a block diagram illustrating a functional configuration of an image processing device. [Diagram 3] 4 is a flowchart showing a process executed by the in-vehicle system. [Figure 4] 4 is a flowchart showing a process executed by the in-vehicle system. [Diagram 5] 5A, 5B, and 5C are explanatory diagrams each showing a normal image. [Figure 6] 6A, 6B, and 6C are explanatory diagrams showing display modes of the approach notification. [Figure 7] FIG. 7A is an explanatory diagram showing the position of an emergency vehicle, and FIG. 7B is an explanatory diagram showing the post-change image. [Figure 8] 8A, 8B, and 8C are explanatory diagrams each showing a post-change image including a planned traveling path. [Figure 9] FIG. 13 is an explanatory diagram showing a display mode of a request for preparation for manual driving. [Figure 10] 10A, 10B, and 10C are explanatory diagrams showing display modes of the vehicle control content notification, respectively. [Figure 11] 11 is an explanatory diagram showing the display manner of a manual driving request, an automatic driving suggestion, and a driving assistance suggestion when a vehicle is stopped and an emergency vehicle has passed. [Figure 12]11 is an explanatory diagram showing the display mode of the autonomous driving continuation notification and the cancellation notification when an emergency vehicle passes without stopping the vehicle. FIG. [Figure 13] FIG. 13 is an explanatory diagram showing a display mode of a manual driving request when shifting to manual driving before an emergency vehicle passes. [Figure 14] FIG. 13 is an explanatory diagram illustrating a changed image whose display range includes the position of an emergency vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Exemplary embodiments of the present disclosure will now be described with reference to the drawings. First Embodiment 1. Configuration of in-vehicle system 1 The configuration of the in-vehicle system 1 will be described with reference to Figure 1 and Figure 2. The in-vehicle system 1 is mounted on a vehicle 2.

[0010] 1, the in-vehicle system 1 includes an image processing device 3, an automatic driving system 5, a locator ECU 7, a cruise control ECU 9, a surroundings monitoring sensor 11, a meter display 15, a CID (Center Information Display) 17, a HUD 19, and a communication module 21. The components of the in-vehicle system 1 are connected to each other via a communication bus 29 so as to be able to communicate with each other.

[0011] The image processing device 3 includes a microcomputer having a CPU 31 and a semiconductor memory such as a RAM or a ROM (hereinafter, referred to as a memory 33). Each function of the image processing device 3 is realized by the CPU 31 executing a program stored in a non-transient real recording medium. In this example, the memory 33 corresponds to the non-transient real recording medium storing the program. Furthermore, the program is executed, thereby executing a method corresponding to the program. The image processing device 3 may include one microcomputer or multiple microcomputers.

[0012] 2, the image processing device 3 includes an information acquisition unit 35, a condition determination unit 36, an image change unit 37, a notification unit 38, and a request display unit 39. The function of each component included in the image processing device 3 will be described later.

[0013] The autonomous driving system 5 includes an autonomous driving ECU 41 and a driving assistance ECU 43. The autonomous driving ECU 41 is an in-vehicle ECU that realizes an autonomous driving function. Autonomous driving is a function that automatically performs driving operations in place of the occupants of the vehicle 2. The autonomous driving ECU 41 enables autonomous driving of, for example, level 3 or higher. The autonomous driving level is the autonomous driving level defined by the Society of Automotive Engineers.

[0014] The autonomous driving ECU 41 acquires locator information and map information from the locator ECU 7. The locator information and the map information will be described later. The autonomous driving ECU 41 also acquires detection information from the periphery monitoring sensor 11. The detection information will be described later.

[0015] The autonomous driving ECU 41 recognizes the driving environment of the vehicle 2 based on the locator information, map information, and detection information. Examples of the driving environment include the positions and speeds of targets present around the vehicle 2. Examples of targets include other vehicles 63, pedestrians, bicycles, animals, fixed objects, lane boundaries, and the like around the vehicle 2. Examples of the driving environment include the position and shape of the lane in which the vehicle 2 is driving.

[0016] Based on the recognized driving environment, the autonomous driving ECU 41 generates a planned driving line for the vehicle 2. In cooperation with the driving control ECU 9, the autonomous driving ECU 41 executes acceleration / deceleration control, steering control, and the like of the vehicle 2 so that the vehicle 2 drives along the generated planned driving line.

[0017] The driving assistance ECU 43 is an in-vehicle ECU that realizes driving assistance operations that assist the driving operation of an occupant of the vehicle 2. The driving assistance ECU 43 enables, for example, advanced driving assistance of about Level 2 or partial automatic driving control.

[0018] The locator ECU 7 includes a Global Navigation Satellite System (GNSS) receiver, an inertial sensor, etc. The locator ECU 7 combines the positioning signal received by the GNSS receiver, the measurement results of the inertial sensor, and vehicle speed information output to the communication bus 29, etc., to sequentially measure the position, traveling direction, etc. of the vehicle 2. The locator ECU 7 sequentially outputs locator information to the communication bus 29. The locator information is information that indicates the position, traveling direction, etc. of the vehicle 2.

[0019] The locator ECU 7 further includes a map database 51. The map database 51 is mainly composed of a large-capacity storage medium. A large amount of map data is stored in the storage medium. The map data includes three-dimensional map data and two-dimensional map data. The three-dimensional map data is so-called high-precision map data. The three-dimensional map data includes information necessary for advanced driving assistance and automated driving. The three-dimensional map data includes three-dimensional shape information of roads, detailed information of each lane, etc.

[0020] The locator ECU 7 reads out map data of the area around the current position of the vehicle 2 from the map database 51. The locator 7 ECU provides map information together with locator information to the driving assistance ECU 43, the autonomous driving ECU 41, etc. The map information is information including the map data read out from the map database 51.

[0021] In addition, instead of the locator ECU 7, the in-vehicle system 1 may acquire locator information and map information from a user terminal such as a smartphone or a navigation device, and provide the acquired locator information and map information to the driving assistance ECU 43 and the autonomous driving ECU 41, etc.

[0022] The cruise control ECU 9 is an electronic control device mainly including a microcontroller. The cruise control ECU 9 generates vehicle speed information based on detection signals from wheel speed sensors. The wheel speed sensors are provided at the hubs of each wheel of the vehicle 2. The vehicle speed information is information that indicates the current traveling speed of the vehicle 2. The cruise control ECU 9 sequentially outputs the vehicle speed information to the communication bus 29.

[0023] The driving control ECU 9 has at least the functions of a brake control ECU and a drive control ECU. The driving control ECU 9 continuously controls the braking force generated in each wheel of the vehicle 2 and the output of the power source of the vehicle 2 based on any one of an operation command based on the driving operation of the occupant of the vehicle 2, a control command from the driving assistance ECU 43, and a control command from the autonomous driving ECU 41.

[0024] The periphery monitoring sensor 11 is an autonomous sensor that monitors the environment around the vehicle 2. The periphery monitoring sensor 11 is capable of detecting targets present around the vehicle 2. The periphery monitoring sensor 11 provides detection information to the driving assistance ECU 43, the autonomous driving ECU 41, and the like via the communication bus 29. The detection information is information that indicates the position, speed, and the like of the detected target.

[0025] The periphery monitoring sensor 11 includes, for example, a camera unit 53 and a millimeter wave radar 55. The camera unit 53 may include a monocular camera or a compound eye camera. The camera unit 53 can capture, for example, a forward range, a lateral range, a rear range, and the like of the vehicle 2. The detection information includes, for example, at least one of imaging data generated by the camera unit 53 capturing an image of the periphery of the vehicle 2 and an analysis result of the imaging data.

[0026] The millimeter wave radar 55 irradiates millimeter waves or quasi-millimeter waves toward the periphery of the vehicle 2. The millimeter wave radar 55 receives waves reflected by targets. The detection information includes, for example, the detection results of the millimeter wave radar 55. The perimeter monitoring sensor 11 may further include a lidar, a sonar, and the like.

[0027] The meter display 15 and the CID 17 are each mainly composed of, for example, a liquid crystal display or an OLED (Organic Light Emitting Diode) display, etc. The meter display 15 and the CID 17 display various images on the display screen based on a control signal and video data acquired from the image processing device 3.

[0028] The meter display 15 is installed, for example, in front of the driver's seat of the vehicle 2. The CID 17 is installed, for example, above a center cluster of the vehicle 2. The CID 17 has a touch panel function. The CID 17 detects, for example, a touch operation, a swipe operation, and the like, performed on the display screen by an occupant of the vehicle 2, and the like.

[0029] The HUD 19 projects the light of an image formed in front of an occupant of the vehicle 2 onto the windshield based on a control signal and video data acquired from the image processing device 3. The light of the image reflected by the windshield into the interior of the vehicle 2 is perceived by the occupant of the vehicle 2. In this way, the HUD 19 displays a virtual image in the space in front of the windshield. The occupant of the vehicle 2 visually recognizes the virtual image displayed by the HUD 19 superimposed on the foreground of the vehicle 2.

[0030] The meter display 15, the CID 17, and the HUD 19 correspond to display devices provided in the vehicle 2. The communication module 21 is capable of wireless communication with a communication target located outside the vehicle 2. Examples of the communication target include an emergency vehicle 73 and a management center, which will be described later.

[0031] 2. Processing performed by in-vehicle system 1 The process executed by the in-vehicle system 1 will be described with reference to Fig. 3 to Fig. 13. The in-vehicle system 1 executes this process when performing autonomous driving at level 3 or higher. When performing autonomous driving at level 3 or higher, the occupants of the vehicle 2 do not need to monitor the periphery of the vehicle 2.

[0032] Until a changed image 71 is displayed in step 5 described later, the image processing device 3 displays the normal image 61 on the meter display 15. As the normal image 61, for example, there are a normal image 61A shown in Fig. 5A, a normal image 61B shown in Fig. 5B, and a normal image 61C shown in Fig. 5C.

[0033] The normal image 61A is an image including a display recommending that the occupants of the vehicle 2 relax. The normal image 61B is an overhead image. An overhead image is an image of other vehicles 63, the vehicle 2, lane boundaries, etc., around the vehicle 2, viewed from a viewpoint behind the vehicle 2. The normal image 61B is a composite image synthesized based on information acquired using the periphery monitoring sensor 11. The normal image 61B is updated according to the situation around the vehicle 2 detected using the periphery monitoring sensor 11.

[0034] The normal image 61C is an image that shows the position of the vehicle 2, like the display of a navigation device. The normal image 61C is an image that includes a map showing the surroundings of the vehicle 2. The normal image 61C is an image that shows, for example, a travel route 65 of the vehicle 2 and a position 67 of the vehicle 2 on the travel route 65. The normal image 61A, the normal image 61B, and the normal image 61C each display the speed of the vehicle 2.

[0035] 3, first, the information acquisition unit 35 of the image processing device 3 acquires information indicating the position of the emergency vehicle 73 (hereinafter referred to as emergency vehicle information) from a communication target using the communication module 21. The communication target is, for example, the emergency vehicle 73, a management center, etc. The emergency vehicle information may be, for example, information indicating the speed of the emergency vehicle 73, the traveling direction of the emergency vehicle 73, a planned driving trajectory of the emergency vehicle 73, etc. in addition to the position of the emergency vehicle 73.

[0036] Next, the condition determination unit 36 ​​of the image processing device 3 determines whether or not the emergency vehicle 73 has been detected based on the acquired emergency vehicle information. If it is determined that the emergency vehicle 73 has been detected, the process proceeds to step 2. If it is determined that the emergency vehicle 73 has not been detected, the process returns to before step 1.

[0037] In step 2, the condition judgment unit 36 ​​judges whether the emergency vehicle 73 is approaching the vehicle 2 based on the emergency vehicle information acquired in step 1. Approaching means that the distance between the vehicle 2 and the emergency vehicle 73 is getting shorter over time.

[0038] If it is determined that the emergency vehicle 73 is approaching the vehicle 2, the process proceeds to step 3. If it is determined that the emergency vehicle 73 is not approaching the vehicle 2, the process returns to before step 1. Note that the fact that the emergency vehicle 73 is approaching the vehicle 2 corresponds to the first condition being satisfied.

[0039] In step 3, the notification unit 38 of the image processing device 3 notifies the occupants of the vehicle 2 of the approach of the emergency vehicle 73. For example, as shown in FIG. 6A , when the normal image 61A is displayed on the meter display 15, the notification unit 38 first displays the approach notification 69 in a large size in the center of the normal image 61A. Next, the notification unit 38 displays the approach notification 69 in a small size in a corner of the normal image 61A. The approach notification 69 is a notification indicating that the emergency vehicle 73 is approaching the vehicle 2.

[0040] Similarly, when a normal image 61B is displayed on the meter display 15 as shown in Fig. 6B, and when a normal image 61C is displayed on the meter display 15 as shown in Fig. 6C, the notification unit 38 first displays the approach notification 69 large in the center of the normal image 61, and then displays it small in a corner of the normal image 61. The notification unit 38 may notify the approach of the emergency vehicle 73 by sound, vibration, LED illumination, or the like. When the emergency vehicle 73 is present within the range of the map displayed by the normal image 61C, the notification unit 38 displays the emergency vehicle 73 in the normal image 61C as shown in Fig. 6C.

[0041] In step 4, first, the information acquisition unit 35 acquires emergency vehicle information in the same manner as in step 1. Next, the condition determination unit 36 ​​determines whether or not the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than a preset threshold based on the acquired emergency vehicle information. If it is determined that the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than the threshold, the process proceeds to step 5. If it is determined that the distance from the emergency vehicle 73 to the vehicle 2 exceeds the threshold, the process returns to before step 4. Note that the distance from the emergency vehicle 73 to the vehicle 2 being equal to or less than the threshold corresponds to the second condition being satisfied.

[0042] In step 5, the image change unit 37 of the image processing device 3 changes the image displayed on the meter display 15 from the normal image 61 to a changed image 71 shown in Fig. 7B. The changed image 71 is an overhead image. The changed image 71 is a composite image synthesized based on information acquired using the perimeter monitoring sensor 11.

[0043] The display range in the changed image 71 is expanded in the direction of the position of the emergency vehicle 73 compared to the display range in the normal image 61B. In the case of the changed image 71 shown in Fig. 7B, the position of the emergency vehicle 73 is to the left rear with respect to the vehicle 2 as a reference, as shown in Fig. 7A. Therefore, the display range in the changed image 71 is expanded to the left rear compared to the display range in the normal image 61B. The display range in the changed image 71 may be moved compared to the display range in the normal image 61 B. For example, the changed image 71 is displayed as if the camera viewpoint has been moved compared to the normal image 61 B.

[0044] 7A and 7B, emergency vehicle 73 is traveling on road shoulder 75. The display range in changed image 71 is expanded in the direction of road shoulder 75 compared to the display range in normal image 61B, and includes road shoulder 75. Changed image 71 includes approaching notification 69 until approaching notification 69 is erased in step 8, which will be described later.

[0045] In step 6, the condition determination unit 36 ​​determines whether or not it is possible to recognize the lane in which the emergency vehicle 73 is traveling based on the emergency vehicle information. If it is determined that it is possible to recognize the lane in which the emergency vehicle 73 is traveling, the process proceeds to step 7. If it is determined that it is impossible to recognize the lane in which the emergency vehicle 73 is traveling, the process proceeds to step 8.

[0046] In step 7, the image modification unit 37 displays a planned driving path 77 on the post-modification image 71, as shown in Fig. 8A, Fig. 8B, or Fig. 8C. The planned driving path 77 represents a position where the emergency vehicle 73 will travel in the future.

[0047] The planned driving trajectory 77 shown in FIG. 8A is obtained by filling in the road shoulder 75 on which the emergency vehicle 73 is driving. When the emergency vehicle 73 is driving in any lane, the planned driving trajectory 77 is obtained by filling in the lane on which the emergency vehicle 73 is driving. The planned driving trajectory 77 shown in FIG. 8B is composed of straight lines. The planned driving trajectory 77 shown in FIG. 8B is composed of arrows. The planned driving trajectory 77 shown in FIG. 8C shows that the emergency vehicle 73 is driving across lanes. The planned driving trajectory 77 shown in FIG. 8C is composed of straight lines.

[0048] In step 8, the request display unit 39 of the image processing device 3 displays a request 79 to prepare for manual driving on the meter display 15, as shown in Fig. 9. The request 79 to prepare for manual driving is displayed superimposed on the changed image 71. The request 79 to prepare for manual driving is a display requesting the occupant of the vehicle 2 to prepare for manual driving. Preparing for manual driving means monitoring the periphery of the vehicle 2 and holding the steering wheel of the vehicle 2. The request display unit 39 may request the occupant of the vehicle 2 to prepare for manual driving by sound, vibration, LED illumination, etc.

[0049] The request display unit 39 first displays the preparation request 79 for manual driving large in the center of the meter display 15. Thereafter, when the in-vehicle system 1 detects that the occupant of the vehicle 2 has gripped the steering wheel, the request display unit 39 displays the preparation request 79 for manual driving small in a corner of the meter display 15. In step 9, the autonomous driving ECU 41 determines whether vehicle control is possible. Vehicle control is control performed by the autonomous driving ECU 41 to avoid the emergency vehicle 73 and ensure a passage for the emergency vehicle 73. Examples of vehicle control include decelerating, slowing down, stopping, moving to one side of the lane, moving to another lane or to the shoulder, etc. If it is determined that any of the vehicle controls are possible, the process proceeds to step 10. If it is determined that any of the vehicle controls are not possible, the process proceeds to step 12.

[0050] In step 10, the autonomous driving ECU 41 determines a vehicle control to be executed from among the executable vehicle controls. The notification unit 38 of the image processing device 3 acquires information indicating the vehicle control to be executed from the autonomous driving ECU 41. The notification unit 38 displays a vehicle control content notification 81 on the meter display 15 as shown in FIG. 10A, FIG. 10B, or FIG. 10C. The vehicle control content notification 81 is displayed superimposed on the changed image 71. The vehicle control content notification 81 indicates the vehicle control to be executed by the autonomous driving ECU 41. In the case shown in FIG. 10A, the vehicle control is deceleration. In the case shown in FIG. 10B, the vehicle control is stopping. In the case shown in FIG. 10C, the vehicle control is to move to the right.

[0051] In step 11, the autonomous driving ECU 41 performs the vehicle control determined in step 10. After step 11, the process proceeds to step 13. In step 12, the request display unit 39 displays a manual driving request 83 on the meter display 15, as shown in FIG. 13. The manual driving request 83 is a display that requests the occupant of the vehicle 2 to switch from automatic driving to manual driving. At this time, the request display unit 39 erases the display except for the vehicle speed and the manual driving request 83. When the display is erased, the amount of information displayed on the meter display 15 is reduced. As a result, the occupant of the vehicle 2 pays attention to the surroundings of the vehicle 2 rather than the meter display 15.

[0052] Thereafter, when the state of the vehicle 2 switches to the manual driving state and the emergency vehicle 73 passes beside the vehicle 2, the request display unit 39 erases the manual driving request 83 as shown in Fig. 13. After the manual driving request 83 is erased, the display on the meter display 15 returns to the normal display during manual driving. After step 12, the process proceeds to step 18.

[0053] In step 13 of Fig. 4, first, the information acquisition unit 35 acquires emergency vehicle information from the communication target using the communication module 21. Next, the condition determination unit 36 ​​judges whether or not the emergency vehicle 73 has already passed beside the vehicle 2 based on the acquired emergency vehicle information. If it is determined that the emergency vehicle 73 has already passed beside the vehicle 2, the process proceeds to step 14. If it is determined that the emergency vehicle 73 has not yet passed beside the vehicle 2, the process returns to before step 13.

[0054] In step 14, the notification unit 38 displays a passing notification on the meter display 15. The passing notification is a notification indicating that the emergency vehicle 73 has already passed beside the vehicle 2. In addition, the notification unit 38 erases the approaching notification 69 that had been displayed up until that point. In step 15, the condition determination unit 36 ​​determines whether or not the vehicle 2 has stopped as a result of the vehicle control performed in step 11. If it is determined that the vehicle 2 has stopped, the process proceeds to step 16. If it is determined that the vehicle 2 has not stopped, the process proceeds to step 24.

[0055] In step 16, the request display unit 39 displays a manual driving request 83 on the meter display 15 as shown in FIG. 11. The manual driving request 83 is displayed superimposed on the changed image 71. The request display unit 39 also erases unnecessary displays. An example of an unnecessary display is the request 79 to prepare for manual driving. Note that the occupant of the vehicle 2 switches from automatic driving to manual driving in response to the manual driving request 83.

[0056] In step 17, the cruise control ECU 9 starts the vehicle 2 based on the driving operation of the occupant of the vehicle 2. In step 18, the autonomous driving ECU 41 determines whether the driving conditions are stable. The driving conditions are stable means that the emergency vehicle 73 has already passed beside the vehicle 2, and the vehicles around the vehicle 2 are already in a state where they can drive. If it is determined that the driving conditions are stable, the process proceeds to step 19. If it is determined that the driving conditions are not yet stable, the process returns to before step 18.

[0057] In step 19, the autonomous driving ECU 41 determines whether or not it is necessary for the occupant to monitor the periphery of the vehicle 2. For example, if the vehicle 2 is in an area where autonomous driving of level 3 or higher is possible, it is not necessary for the occupant to monitor the periphery of the vehicle 2. Areas where autonomous driving of level 3 or higher is possible include, for example, motorways and traffic jams. If it is determined that it is not necessary to monitor the periphery of the vehicle 2, the process proceeds to step 20. If it is determined that it is necessary to monitor the periphery of the vehicle 2, the process proceeds to step 21.

[0058] In step 20, the notification unit 38 displays an autonomous driving suggestion 85 on the meter display 15, as shown in Fig. 11. The autonomous driving suggestion 85 is a notification that suggests to the occupant of the vehicle 2 to start autonomous driving. In addition, the autonomous driving suggestion 85 notifies the occupant of the vehicle 2 that it is not necessary to monitor the surroundings of the vehicle 2 and that it is not necessary to hold the steering wheel of the vehicle 2.

[0059] In step 21, the driving assistance ECU 43 determines whether driving assistance is possible. If it is determined that driving assistance is possible, the process proceeds to step 22. If it is determined that driving assistance is not possible, the process proceeds to step 23.

[0060] In step 22, the notification unit 38 displays a driving assistance suggestion 87 on the meter display 15, as shown in Fig. 11. The driving assistance suggestion 87 is displayed superimposed on the changed image 71. The driving assistance suggestion 87 is a notification that suggests to the occupant of the vehicle 2 to start driving assistance.

[0061] In step 23, the in-vehicle system 1 continues to be in manual driving mode. In step 24, the autonomous driving ECU 41 continues autonomous driving. The notification unit 38 displays an autonomous driving continuation notification 89 on the meter display 15 as shown in Fig. 12. The autonomous driving continuation notification 89 is displayed superimposed on the changed image 71. The autonomous driving continuation notification 89 indicates that autonomous driving will continue. The request display unit 39 continues to display the request 79 to prepare for manual driving on the meter display 15.

[0062] In step 25, the automatic driving ECU 41 determines whether the driving situation is stable. If it is determined that the driving situation is stable, the process proceeds to step 26. If it is determined that the driving situation is not yet stable, the process returns to before step 25.

[0063] In step 26, the request display unit 39 displays a release notification 91 on the meter display 15, as shown in Fig. 12. The release notification 91 is displayed superimposed on the changed image 71. The release notification 91 notifies the occupant of the vehicle 2 that it is no longer necessary to monitor the surroundings of the vehicle 2 and to hold the steering wheel of the vehicle 2. When the release notification 91 is displayed, the request display unit 39 erases the request 79 to prepare for manual driving.

[0064] 3. Effects of the image processing device 3 (1A) The image processing device 3 acquires emergency vehicle information. Based on the emergency vehicle information, the image processing device 3 determines whether or not the following conditions are met: (a) a first condition that the emergency vehicle 73 is approaching the vehicle 2, and (b) a second condition that the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than a preset threshold value. When the image processing device 3 determines that the first and second conditions are met while the vehicle 2 is performing autonomous driving, it changes the image displayed on the meter display 15 from the normal image 61 to the changed image 71. The changed image 71 is an overhead image.

[0065] The occupants of vehicle 2 can know that an emergency vehicle 73 is approaching vehicle 2 and that the distance from emergency vehicle 73 to vehicle 2 is below a threshold value because the image displayed on meter display 15 has changed from normal image 61 to changed image 71.

[0066] (1B) The post-change image 71 represents a planned driving path 77. By looking at the planned driving path 77, the occupants of the vehicle 2 can know the location where the emergency vehicle 73 will be traveling in the future. (1C) When the image processing device 3 determines that the first condition and the second condition are satisfied while the vehicle 2 is performing autonomous driving, it displays a request 79 for preparation for manual driving on the meter display 15. By seeing the request 79 for preparation for manual driving, the occupants of the vehicle 2 can prepare for switching from autonomous driving to manual driving.

[0067] (1D) Normal image 61B is an overhead image. The display range in changed image 71 is expanded in the direction of the position of emergency vehicle 73 compared to the display range in normal image 61B. The occupant of vehicle 2 can know the direction of the position of emergency vehicle 73 by comparing the display range in changed image 71 with the display range in normal image 61B. <Other embodiments> Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and can be implemented in various modified forms.

[0068] (1) For example, as shown in Fig. 14, the display range in the changed image 71 includes the position of the emergency vehicle 73. In this case, the occupants of the vehicle 2 can know the position of the emergency vehicle 73 in more detail by looking at the changed image 71.

[0069] (2) The determination in step 4 may be whether or not the predicted time until emergency vehicle 73 reaches vehicle 2 (hereinafter referred to as predicted arrival time) is equal to or less than a preset threshold. If it is determined that the predicted arrival time is equal to or less than the threshold, the process proceeds to step 5. If it is determined that the predicted arrival time exceeds the threshold, the process returns to before step 4. The predicted arrival time can be calculated, for example, from a change in the relative position of emergency vehicle 73 with respect to the position of vehicle 2.

[0070] (3) The display device that displays the normal image 61 and the changed image 71 may be a display device other than the meter display 15. Examples of the display device other than the meter display 15 include the CID 17, the HUD 19, and the like.

[0071] (4) A necessary condition for displaying the request 79 for preparation for manual driving on the meter display 15 in step 8 may be that the results of steps 1 and 2 are affirmative, and that the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than a specific threshold. The specific threshold may be the same as or different from the threshold used in step 4.

[0072] (5) The image processing device 3 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied in a computer program. Alternatively, the image processing device 3 and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the image processing device 3 and the method thereof described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor and a memory programmed to execute one or more functions and a processor configured with one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by a computer. The method for realizing the functions of each unit included in the image processing device 3 does not necessarily need to include software, and all of the functions may be realized using one or more hardware.

[0073] (6) Multiple functions possessed by one component in the above embodiments may be realized by multiple components, or one function possessed by one component may be realized by multiple components. Also, multiple functions possessed by multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, part of the configuration of the above embodiments may be omitted. Also, at least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments.

[0074] (7) In addition to the image processing device 3 described above, the present disclosure can also be realized in various forms, such as a system including the image processing device 3 as a component, a program for causing a computer to function as the image processing device 3, a non-transient physical recording medium such as a semiconductor memory on which the program is recorded, a display control method, etc. [Technical idea disclosed in this specification] [Item 1] An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), an information acquisition unit (35) configured to acquire information representative of a location of an emergency vehicle (73); a condition determination unit (36) configured to determine whether or not a first condition, that is, (a) the emergency vehicle is approaching the vehicle, and (b) a second condition, that the distance from the emergency vehicle to the vehicle or the predicted time for the emergency vehicle to reach the vehicle is equal to or less than a preset threshold, are satisfied based on the information acquired by the information acquisition unit; an image change unit (37) that changes the image when the condition determination unit determines that the first condition and the second condition are satisfied while the vehicle is performing automatic driving; Equipped with The image (71) after the image modification by the image modification unit is an overhead image of the vehicle and other vehicles (63) around the vehicle from a viewpoint behind the vehicle. [Item 2] Item 1. The image processing device according to item 1, The image processing device, in which the overhead image after modification by the image modification unit represents a planned travel path (77) of the emergency vehicle. [Item 3] 3. The image processing device according to claim 1, The image processing device further includes a request display unit (39) that displays, on the display device, a display (79) requesting preparation for manual driving when the condition determination unit determines that the first condition and the second condition are satisfied while the vehicle is performing autonomous driving. [Item 4] The image processing device according to any one of items 1 to 3, An image processing device in which the display range of the overhead image after the change by the image change unit includes the position of the emergency vehicle. [Item 5] The image processing device according to any one of items 1 to 4, The image (61B) before the change by the image change unit is the overhead image, An image processing device in which a display range of the overhead image after the change by the image change unit is expanded in the direction of the position of the emergency vehicle compared to a display range of the overhead image before the change by the image change unit. [Item 6] 6. The image processing device according to any one of items 1 to 5, configured to display on the display device an image including a map representative of the surroundings of the vehicle; 20. An image processing device configured to display the emergency vehicle in an image including the map if the location of the emergency vehicle is within the map. [Explanation of symbols]

[0075] 2...vehicle, 3...image processing device, 15...meter display, 35...information acquisition unit, 36...condition judgment unit, 37...image change unit, 38...notification unit, 39...request display unit, 61, 61A, 61B, 61C...normal image, 71...changed image, 73...emergency vehicle, 77...planned driving trajectory, 79...request for preparation for manual driving

Claims

1. An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), a determination unit (36) configured to determine whether an emergency vehicle (73) is approaching the vehicle based on information acquired by an information acquisition unit (35) configured to acquire information representing a position of the emergency vehicle (73); an image change unit (37) that changes the image when the determination unit determines that the emergency vehicle is approaching the vehicle while the vehicle is autonomously driving; Equipped with The determining unit is configured to further determine whether the emergency vehicle has already passed the vehicle; An image processing device configured to, when the judgment unit determines that the emergency vehicle has already passed the vehicle, further change the image (71) after modification by the image modification unit to an image including a notification indicating that the emergency vehicle has already passed beside the vehicle.

2. 2. The image processing device according to claim 1, The image (71) after the modification by the image modification unit is an overhead image of the vehicle and other vehicles (63) around the vehicle, viewed from a viewpoint behind the vehicle.

3. 3. The image processing device according to claim 1, When the vehicle is performing autonomous driving and the judgment unit determines that the emergency vehicle is approaching the vehicle, the image change unit superimposes a notification informing the driver of the approach of the emergency vehicle on the image that had been displayed up until that point.

Images