Image processing device

Abstract

To provide an image processing device which can inform a crew member of a vehicle of approach of an emergency vehicle.SOLUTION: An information processing device 3 comprises: an information acquisition unit; a determination unit; and an image change unit. The information acquisition unit acquires information representing a position of an emergency vehicle. The determination unit determines whether or not the emergency vehicle approaches a vehicle. The image change unit changes images when the determination unit determines that the emergency vehicle approaches the vehicle when the vehicle automatically drives.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device. [Background technology]

[0002] When a vehicle is operating autonomously, an emergency vehicle may approach the vehicle. Patent Document 1 discloses a vehicle control device that executes control to avoid the emergency vehicle while continuing autonomous driving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-156144 Summary of the Invention [Problem to be solved by the invention]

[0004] When an emergency vehicle approaches a vehicle, depending on the circumstances around the vehicle, it may be necessary for the vehicle's occupants to switch from automated driving to manual driving and avoid the emergency vehicle by operating the vehicle. However, the occupants of an automated driving vehicle may not notice the approaching emergency vehicle.

[0005] In one aspect of the present disclosure, it is desirable to provide an image processing device that can notify a vehicle occupant of an approaching emergency vehicle. [Means for solving the problem]

[0006] One aspect of the present disclosure is an image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2). The image processing device includes: a determination unit (36) configured to determine whether an emergency vehicle (73) is approaching the vehicle based on information acquired by an information acquisition unit (35) configured to acquire information representing the position of the emergency vehicle (73); and an image change unit (37) that changes the image when the determination unit determines that the emergency vehicle is approaching the vehicle while the vehicle is autonomously driving. a request display unit (39) that displays a message (79) requesting preparation for manual driving on the display device when the judgment unit judges that the emergency vehicle is approaching the vehicle while the vehicle is performing automatic driving; Equipped with. The determination unit is further configured to determine whether the emergency vehicle has already passed the vehicle. If the determination unit determines that the emergency vehicle has already passed the vehicle, the image modification unit is further configured to change the image (71) after modification by the image modification unit into an image including a notification indicating that the emergency vehicle has already passed beside the vehicle.

[0007] An image processing device according to one aspect of the present disclosure can notify vehicle occupants of an approaching emergency vehicle. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of an in-vehicle system. [Figure 2] FIG. 2 is a block diagram illustrating a functional configuration of the image processing device. [Figure 3] 3 is a flowchart showing a process executed by the in-vehicle system. [Figure 4] 3 is a flowchart showing a process executed by the in-vehicle system. [Figure 5] 5A, 5B, and 5C are explanatory diagrams each showing a normal image. [Figure 6] 6A, 6B, and 6C are explanatory diagrams each showing a display mode of an approach notification. [Figure 7] FIG. 7A is an explanatory diagram showing the position of an emergency vehicle, and FIG. 7B is an explanatory diagram showing the image after the change. [Figure 8] 8A, 8B, and 8C are explanatory diagrams each showing a post-change image including a planned traveling path. [Figure 9] FIG. 10 is an explanatory diagram showing a display mode of a request to prepare for manual driving. [Figure 10]10A, 10B, and 10C are explanatory diagrams each showing a display mode of the vehicle control content notification. [Figure 11] FIG. 10 is an explanatory diagram showing the display modes of a manual driving request, an automatic driving suggestion, and a driving assistance suggestion when a vehicle is stopped and an emergency vehicle has passed. [Figure 12] 10A and 10B are explanatory diagrams showing the display modes of the autonomous driving continuation notification and the cancellation notification when an emergency vehicle passes without the vehicle stopping. [Figure 13] FIG. 10 is an explanatory diagram showing a display mode of a manual driving request when shifting to manual driving before an emergency vehicle passes. [Figure 14] FIG. 10 is an explanatory diagram showing a changed image in which the display range includes the position of an emergency vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0009] Exemplary embodiments of the present disclosure will be described with reference to the drawings. First Embodiment 1. Configuration of In-Vehicle System 1 The configuration of the in-vehicle system 1 will be described with reference to Figures 1 and 2. The in-vehicle system 1 is mounted on a vehicle 2.

[0010] 1, the in-vehicle system 1 includes an image processing device 3, an automatic driving system 5, a locator ECU 7, a cruise control ECU 9, a periphery monitoring sensor 11, a meter display 15, a CID (Center Information Display) 17, an HUD 19, and a communication module 21. The components of the in-vehicle system 1 are connected to each other via a communication bus 29 so as to be able to communicate with each other.

[0011] The image processing device 3 includes a microcomputer having a CPU 31 and a semiconductor memory such as a RAM or a ROM (hereinafter referred to as a memory 33). Each function of the image processing device 3 is realized by the CPU 31 executing a program stored in a non-transitory physical recording medium. In this example, the memory 33 corresponds to the non-transitory physical recording medium storing the program. Furthermore, the execution of this program results in the execution of a method corresponding to the program. Note that the image processing device 3 may include one microcomputer or multiple microcomputers.

[0012] 2, the image processing device 3 includes an information acquisition unit 35, a condition determination unit 36, an image change unit 37, a notification unit 38, and a request display unit 39. The functions of each component included in the image processing device 3 will be described later.

[0013] The autonomous driving system 5 includes an autonomous driving ECU 41 and a driving assistance ECU 43. The autonomous driving ECU 41 is an in-vehicle ECU that realizes the autonomous driving function. Autonomous driving is a function that automatically performs driving operations on behalf of the occupants of the vehicle 2. The autonomous driving ECU 41 enables autonomous driving of, for example, level 3 or higher. The autonomous driving level is defined by the Society of Automotive Engineers.

[0014] The autonomous driving ECU 41 acquires locator information and map information from the locator ECU 7. The locator information and map information will be described later. The autonomous driving ECU 41 also acquires detection information from the periphery monitoring sensor 11. The detection information will be described later.

[0015] The autonomous driving ECU 41 recognizes the driving environment of the vehicle 2 based on the locator information, map information, and detection information. Examples of the driving environment include the positions and speeds of targets present around the vehicle 2. Examples of targets include other vehicles 63, pedestrians, bicycles, animals, fixed objects, lane boundaries, etc. around the vehicle 2. In addition, examples of the driving environment include the position and shape of the lane in which the vehicle 2 is driving.

[0016] Based on the recognized driving environment, the autonomous driving ECU 41 generates a planned driving line for the vehicle 2. In cooperation with the driving control ECU 9, the autonomous driving ECU 41 executes acceleration / deceleration control, steering control, and the like for the vehicle 2 so that the vehicle 2 travels along the generated planned driving line.

[0017] The driving assistance ECU 43 is an in-vehicle ECU that realizes driving assistance operations that assist the driving operation of an occupant of the vehicle 2. The driving assistance ECU 43 enables, for example, advanced driving assistance of about level 2 or partial automatic driving control.

[0018] The locator ECU 7 includes a GNSS (Global Navigation Satellite System) receiver, an inertial sensor, etc. The locator ECU 7 combines the positioning signal received by the GNSS receiver, the measurement results of the inertial sensor, and vehicle speed information output to the communication bus 29, etc., to sequentially measure the position, traveling direction, etc. of the vehicle 2. The locator ECU 7 sequentially outputs locator information to the communication bus 29. The locator information is information that indicates the position, traveling direction, etc. of the vehicle 2.

[0019] The locator ECU 7 further includes a map database 51. The map database 51 is mainly composed of a large-capacity storage medium. The storage medium stores a large amount of map data. The map data includes three-dimensional map data and two-dimensional map data. The three-dimensional map data is so-called high-precision map data. The three-dimensional map data includes information necessary for advanced driving assistance and autonomous driving. The three-dimensional map data includes three-dimensional shape information of roads, detailed information about each lane, etc.

[0020] The locator ECU 7 reads map data of the area around the current position of the vehicle 2 from the map database 51. The locator ECU 7 provides map information together with locator information to the driving assistance ECU 43, the autonomous driving ECU 41, etc. The map information is information that includes the map data read from the map database 51.

[0021] In addition, instead of the locator ECU 7, the in-vehicle system 1 may acquire locator information and map information from a user terminal such as a smartphone or a navigation device, and provide the acquired locator information and map information to the driving assistance ECU 43 and the autonomous driving ECU 41, etc.

[0022] The cruise control ECU 9 is an electronic control device that mainly includes a microcontroller. The cruise control ECU 9 generates vehicle speed information based on detection signals from wheel speed sensors. The wheel speed sensors are provided at the hub portions of each wheel of the vehicle 2. The vehicle speed information is information that indicates the current traveling speed of the vehicle 2. The cruise control ECU 9 sequentially outputs the vehicle speed information to the communication bus 29.

[0023] The cruise control ECU 9 has at least the functions of a brake control ECU and a drive control ECU. The cruise control ECU 9 continuously controls the braking force generated in each wheel of the vehicle 2 and the output of the power source of the vehicle 2, based on one of an operation command based on the driving operation of the occupant of the vehicle 2, a control command from the driving assistance ECU 43, and a control command from the autonomous driving ECU 41.

[0024] The periphery monitoring sensor 11 is an autonomous sensor that monitors the environment around the vehicle 2. The periphery monitoring sensor 11 is capable of detecting targets present around the vehicle 2. The periphery monitoring sensor 11 provides detection information to the driving assistance ECU 43, the autonomous driving ECU 41, etc. via the communication bus 29. The detection information is information that indicates the position, speed, etc. of the detected target.

[0025] The perimeter monitoring sensor 11 includes, for example, a camera unit 53 and a millimeter-wave radar 55. The camera unit 53 may be configured to include a monocular camera or a compound eye camera. The camera unit 53 is capable of capturing images of, for example, the front range, the side range, and the rear range of the vehicle 2. The detection information includes, for example, at least one of image data generated by the camera unit 53 capturing images of the periphery of the vehicle 2 and an analysis result of the image data.

[0026] The millimeter-wave radar 55 emits millimeter waves or quasi-millimeter waves toward the periphery of the vehicle 2. The millimeter-wave radar 55 receives the waves reflected by targets. The detection information includes, for example, the detection results of the millimeter-wave radar 55. The perimeter monitoring sensor 11 may further include a lidar, a sonar, etc.

[0027] The meter display 15 and the CID 17 are each mainly configured with, for example, a liquid crystal display or an OLED (Organic Light Emitting Diode) display, etc. The meter display 15 and the CID 17 display various images on their display screens based on control signals and video data acquired from the image processing device 3.

[0028] The meter display 15 is installed, for example, in front of the driver's seat of the vehicle 2. The CID 17 is installed, for example, above a center cluster of the vehicle 2. The CID 17 has a touch panel function. The CID 17 detects, for example, touch operations and swipe operations on the display screen by an occupant of the vehicle 2 or the like.

[0029] Based on the control signal and video data acquired from the image processing device 3, the HUD 19 projects onto the windshield the light of an image formed in front of the occupants of the vehicle 2. The light of the image reflected by the windshield toward the interior of the vehicle 2 is perceived by the occupants of the vehicle 2. In this way, the HUD 19 displays a virtual image in the space ahead of the windshield. The occupants of the vehicle 2 visually recognize the virtual image displayed by the HUD 19 superimposed on the view ahead of the vehicle 2.

[0030] The meter display 15, the CID 17, and the HUD 19 correspond to display devices provided in the vehicle 2. The communication module 21 is capable of wireless communication with a communication target located outside the vehicle 2. Examples of the communication target include an emergency vehicle 73, a management center, and the like, which will be described later.

[0031] 2. Processing performed by in-vehicle system 1 The processing executed by the in-vehicle system 1 will be described with reference to Fig. 3 to Fig. 13. The in-vehicle system 1 executes this processing when performing autonomous driving at level 3 or higher. When performing autonomous driving at level 3 or higher, the occupants of the vehicle 2 do not need to monitor the periphery of the vehicle 2.

[0032] Until the changed image 71 is displayed in step 5 described later, the image processing device 3 displays the normal image 61 on the meter display 15. Examples of the normal image 61 include a normal image 61A shown in Fig. 5A, a normal image 61B shown in Fig. 5B, and a normal image 61C shown in Fig. 5C.

[0033] The normal image 61A is an image that includes a message recommending that the occupants of the vehicle 2 relax. The normal image 61B is an overhead image. An overhead image is an image of other vehicles 63, the vehicle 2, lane boundaries, etc., around the vehicle 2, viewed from a viewpoint behind the vehicle 2. The normal image 61B is a composite image that is synthesized based on information acquired using the periphery monitoring sensor 11. The normal image 61B is updated according to the situation around the vehicle 2 detected using the periphery monitoring sensor 11.

[0034] The normal image 61C is an image that shows the position of the vehicle 2, like the display on a navigation device. The normal image 61C is an image that includes a map showing the surroundings of the vehicle 2. The normal image 61C is an image that shows, for example, a driving route 65 of the vehicle 2 and a position 67 of the vehicle 2 on the driving route 65. The normal image 61A, the normal image 61B, and the normal image 61C each display the speed of the vehicle 2.

[0035] 3, first, the information acquisition unit 35 of the image processing device 3 acquires information indicating the position of the emergency vehicle 73 (hereinafter referred to as emergency vehicle information) from a communication target using the communication module 21. The communication target is, for example, the emergency vehicle 73, a management center, etc. The emergency vehicle information may be, for example, information indicating the speed of the emergency vehicle 73, the traveling direction of the emergency vehicle 73, the planned traveling trajectory of the emergency vehicle 73, etc. in addition to the position of the emergency vehicle 73.

[0036] Next, the condition determination unit 36 ​​of the image processing device 3 determines whether or not the emergency vehicle 73 has been detected based on the acquired emergency vehicle information. If it is determined that the emergency vehicle 73 has been detected, the process proceeds to step 2. If it is determined that the emergency vehicle 73 has not been detected, the process returns to before step 1.

[0037] In step 2, the condition determination unit 36 ​​determines whether the emergency vehicle 73 is approaching the vehicle 2 based on the emergency vehicle information acquired in step 1. Approaching means that the distance between the vehicle 2 and the emergency vehicle 73 is getting shorter over time.

[0038] If it is determined that the emergency vehicle 73 is approaching the vehicle 2, the process proceeds to step 3. If it is determined that the emergency vehicle 73 is not approaching the vehicle 2, the process returns to before step 1. Note that the fact that the emergency vehicle 73 is approaching the vehicle 2 corresponds to the first condition being met.

[0039] In step 3, the notification unit 38 of the image processing device 3 notifies the occupants of the vehicle 2 of the approaching emergency vehicle 73. For example, as shown in FIG. 6A , when a normal image 61A is displayed on the meter display 15, the notification unit 38 first displays an approaching notification 69 in a large size in the center of the normal image 61A. Next, the notification unit 38 displays the approaching notification 69 in a small size in a corner of the normal image 61A. The approaching notification 69 is a notification indicating that the emergency vehicle 73 is approaching the vehicle 2.

[0040] Similarly, when a normal image 61B is displayed on the meter display 15 as shown in FIG. 6B, and when a normal image 61C is displayed on the meter display 15 as shown in FIG. 6C, the notification unit 38 first displays the approach notification 69 large in the center of the normal image 61, and then displays it small in a corner of the normal image 61. The notification unit 38 may notify the driver of the approach of an emergency vehicle 73 by sound, vibration, LED illumination, or the like. When the emergency vehicle 73 is present within the range of the map displayed by the normal image 61C, the notification unit 38 displays the emergency vehicle 73 on the normal image 61C as shown in FIG. 6C.

[0041] In step 4, first, the information acquisition unit 35 acquires emergency vehicle information in the same manner as in step 1. Next, the condition determination unit 36 ​​determines whether the distance from the emergency vehicle 73 to vehicle 2 is equal to or less than a preset threshold based on the acquired emergency vehicle information. If it is determined that the distance from the emergency vehicle 73 to vehicle 2 is equal to or less than the threshold, the process proceeds to step 5. If it is determined that the distance from the emergency vehicle 73 to vehicle 2 exceeds the threshold, the process returns to before step 4. Note that the distance from the emergency vehicle 73 to vehicle 2 being equal to or less than the threshold corresponds to the second condition being met.

[0042] In step 5, the image modification unit 37 of the image processing device 3 modifies the image displayed on the meter display 15 from the normal image 61 to the modified image 71 shown in FIG. 7B. The modified image 71 is an overhead image. The modified image 71 is a composite image synthesized based on information acquired using the perimeter monitoring sensor 11.

[0043] The display range in changed image 71 is expanded in the direction of the position of emergency vehicle 73 compared to the display range in normal image 61B. In the case of changed image 71 shown in Fig. 7B, as shown in Fig. 7A, the position of emergency vehicle 73 is to the left rear of vehicle 2. Therefore, the display range in changed image 71 is expanded to the left rear compared to the display range in normal image 61B. The display range in the changed image 71 may be shifted compared to the display range in the normal image 61 B. For example, the changed image 71 is displayed as if the camera viewpoint has shifted compared to the normal image 61 B.

[0044] 7A and 7B, emergency vehicle 73 is traveling on road shoulder 75. The display range in changed image 71 is expanded in the direction of road shoulder 75 compared to the display range in normal image 61B, and includes road shoulder 75. Changed image 71 includes approaching notification 69 until approaching notification 69 is erased in step 8, which will be described later.

[0045] In step 6, the condition determination unit 36 ​​determines whether it is possible to recognize the lane in which the emergency vehicle 73 is traveling based on the emergency vehicle information. If it is determined that it is possible to recognize the lane in which the emergency vehicle 73 is traveling, the process proceeds to step 7. If it is determined that it is impossible to recognize the lane in which the emergency vehicle 73 is traveling, the process proceeds to step 8.

[0046] In step 7, the image modification unit 37 displays a planned driving path 77 on the modified image 71, as shown in Fig. 8A, 8B, or 8C. The planned driving path 77 represents the position where the emergency vehicle 73 will travel in the future.

[0047] The planned driving trajectory 77 shown in FIG. 8A is obtained by filling in the shoulder 75 on which the emergency vehicle 73 is driving. When the emergency vehicle 73 is driving in one of the lanes, the planned driving trajectory 77 is obtained by filling in the lane on which the emergency vehicle 73 is driving. The planned driving trajectory 77 shown in FIG. 8B is made up of straight lines. The planned driving trajectory 77 shown in FIG. 8B is made up of arrows. The planned driving trajectory 77 shown in FIG. 8C shows that the emergency vehicle 73 is driving across lanes. The planned driving trajectory 77 shown in FIG. 8C is made up of straight lines.

[0048] In step 8, the request display unit 39 of the image processing device 3 displays a request 79 to prepare for manual driving on the meter display 15, as shown in FIG. 9. The request 79 to prepare for manual driving is displayed superimposed on the changed image 71. The request 79 to prepare for manual driving is a display requesting the occupants of the vehicle 2 to prepare for manual driving. Preparing for manual driving means monitoring the surroundings of the vehicle 2 and holding the steering wheel of the vehicle 2. The request display unit 39 may request the occupants of the vehicle 2 to prepare for manual driving by sound, vibration, LED illumination, etc.

[0049] The request display unit 39 initially displays the request 79 to prepare for manual driving in a large font in the center of the meter display 15. Thereafter, when the in-vehicle system 1 detects that an occupant of the vehicle 2 has gripped the steering wheel, the request display unit 39 displays the request 79 to prepare for manual driving in a small font in the corner of the meter display 15. In step 9, the autonomous driving ECU 41 determines whether vehicle control is possible. Vehicle control is control performed by the autonomous driving ECU 41 to avoid the emergency vehicle 73 and ensure passage for the emergency vehicle 73. Examples of vehicle control include decelerating, slowing down, stopping, moving to one side of the lane, moving to another lane or onto the shoulder, etc. If it is determined that any of the vehicle controls is possible, the process proceeds to step 10. If it is determined that any of the vehicle controls is impossible, the process proceeds to step 12.

[0050] In step 10, the autonomous driving ECU 41 determines a vehicle control to be executed from among the executable vehicle controls. The notification unit 38 of the image processing device 3 acquires information indicating the vehicle control to be executed from the autonomous driving ECU 41. The notification unit 38 displays a vehicle control content notification 81 on the meter display 15, as shown in FIG. 10A, FIG. 10B, or FIG. 10C. The vehicle control content notification 81 is displayed superimposed on the changed image 71. The vehicle control content notification 81 indicates the vehicle control that the autonomous driving ECU 41 will execute. In the example shown in FIG. 10A, the vehicle control is deceleration. In the example shown in FIG. 10B, the vehicle control is stopping. In the example shown in FIG. 10C, the vehicle control is to move to the right.

[0051] In step 11, the autonomous driving ECU 41 performs the vehicle control determined in step 10. After step 11, the process proceeds to step 13. In step 12, the request display unit 39 displays a manual driving request 83 on the meter display 15, as shown in FIG. 13. The manual driving request 83 is a display that requests the occupant of the vehicle 2 to switch from automatic driving to manual driving. At this time, the request display unit 39 erases the display except for the vehicle speed and the manual driving request 83. When the display is erased, the amount of information displayed on the meter display 15 is reduced. As a result, the occupant of the vehicle 2 focuses on the surroundings of the vehicle 2 rather than the meter display 15.

[0052] Thereafter, when the state of vehicle 2 switches to manual driving and an emergency vehicle 73 passes beside vehicle 2, request display unit 39 erases manual driving request 83 as shown in Fig. 13. After manual driving request 83 is erased, the display on meter display 15 returns to the normal display during manual driving. After step 12, this process proceeds to step 18.

[0053] In step 13 of Figure 4, first, the information acquisition unit 35 acquires emergency vehicle information from the communication target using the communication module 21. Next, the condition determination unit 36 ​​determines whether or not the emergency vehicle 73 has already passed beside the vehicle 2 based on the acquired emergency vehicle information. If it is determined that the emergency vehicle 73 has already passed beside the vehicle 2, the process proceeds to step 14. If it is determined that the emergency vehicle 73 has not yet passed beside the vehicle 2, the process returns to before step 13.

[0054] In step 14, the notification unit 38 displays a passing notification on the meter display 15. The passing notification is a notification indicating that the emergency vehicle 73 has already passed beside the vehicle 2. The notification unit 38 also erases the approaching notification 69 that had been displayed up until then. In step 15, the condition determination unit 36 ​​determines whether the vehicle 2 has stopped as a result of the vehicle control performed in step 11. If it is determined that the vehicle 2 has stopped, the process proceeds to step 16. If it is determined that the vehicle 2 has not stopped, the process proceeds to step 24.

[0055] In step 16, the request display unit 39 displays a manual driving request 83 on the meter display 15, as shown in FIG. 11. The manual driving request 83 is displayed superimposed on the changed image 71. The request display unit 39 also erases unnecessary displays. An example of an unnecessary display is the request 79 to prepare for manual driving. In response to the manual driving request 83, the occupant of the vehicle 2 switches from automatic driving to manual driving.

[0056] In step 17, the driving control ECU 9 starts the vehicle 2 based on the driving operation of the occupant of the vehicle 2. In step 18, the autonomous driving ECU 41 determines whether the driving situation has stabilized. A stable driving situation means that the emergency vehicle 73 has already passed beside vehicle 2 and the vehicles around vehicle 2 are already in a state where they can drive. If it is determined that the driving situation has stabilized, the process proceeds to step 19. If it is determined that the driving situation is not yet stable, the process returns to before step 18.

[0057] In step 19, the autonomous driving ECU 41 determines whether it is unnecessary for the occupant to monitor the surroundings of the vehicle 2. For example, if the vehicle 2 is in an area where autonomous driving of level 3 or higher is possible, it is unnecessary for the occupant to monitor the surroundings of the vehicle 2. Examples of areas where autonomous driving of level 3 or higher is possible include expressways and traffic jams. If it is determined that it is unnecessary to monitor the surroundings of the vehicle 2, the process proceeds to step 20. If it is determined that it is necessary to monitor the surroundings of the vehicle 2, the process proceeds to step 21.

[0058] In step 20, the notification unit 38 displays an automated driving suggestion 85 on the meter display 15, as shown in Fig. 11. The automated driving suggestion 85 is a notification suggesting to the occupant of the vehicle 2 that they start automated driving. The automated driving suggestion 85 also notifies the occupant of the vehicle 2 that they do not need to monitor the surroundings of the vehicle 2 and that they do not need to hold the steering wheel of the vehicle 2.

[0059] In step 21, the driving assistance ECU 43 determines whether driving assistance is possible. If it is determined that driving assistance is possible, the process proceeds to step 22. If it is determined that driving assistance is not possible, the process proceeds to step 23.

[0060] In step 22, the notification unit 38 displays a driving assistance suggestion 87 on the meter display 15, as shown in FIG. 11. The driving assistance suggestion 87 is displayed superimposed on the changed image 71. The driving assistance suggestion 87 is a notification suggesting to the occupant of the vehicle 2 that driving assistance be started.

[0061] In step 23, the in-vehicle system 1 continues to be in manual driving mode. In step 24, the autonomous driving ECU 41 continues autonomous driving. The notification unit 38 displays an autonomous driving continuation notification 89 on the meter display 15, as shown in FIG. 12 . The autonomous driving continuation notification 89 is displayed superimposed on the changed image 71. The autonomous driving continuation notification 89 indicates that autonomous driving will continue. The request display unit 39 continues to display the request 79 to prepare for manual driving on the meter display 15.

[0062] In step 25, the automatic driving ECU 41 determines whether the driving situation has stabilized. If it is determined that the driving situation has stabilized, the process proceeds to step 26. If it is determined that the driving situation has not yet stabilized, the process returns to before step 25.

[0063] In step 26, the request display unit 39 displays a release notification 91 on the meter display 15, as shown in FIG. 12. The release notification 91 is displayed superimposed on the changed image 71. The release notification 91 notifies the occupants of the vehicle 2 that it is no longer necessary to monitor the surroundings of the vehicle 2 and to hold the steering wheel of the vehicle 2. When the release notification 91 is displayed, the request display unit 39 erases the request 79 to prepare for manual driving.

[0064] 3. Effects of the image processing device 3 (1A) The image processing device 3 acquires emergency vehicle information. Based on the emergency vehicle information, the image processing device 3 determines whether or not the following conditions are met: (a) a first condition that the emergency vehicle 73 is approaching the vehicle 2; and (b) a second condition that the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than a preset threshold. When the image processing device 3 determines that the first and second conditions are met while the vehicle 2 is performing autonomous driving, it changes the image displayed on the meter display 15 from the normal image 61 to the changed image 71. The changed image 71 is an overhead image.

[0065] The occupants of vehicle 2 can see that the image displayed on meter display 15 has changed from normal image 61 to changed image 71, indicating that emergency vehicle 73 is approaching vehicle 2 and that the distance from emergency vehicle 73 to vehicle 2 is below a threshold.

[0066] (1B) The changed image 71 shows a planned driving path 77. By looking at the planned driving path 77, the occupants of the vehicle 2 can know the location where the emergency vehicle 73 will be traveling from now on. (1C) When the image processing device 3 determines that the first condition and the second condition are met while the vehicle 2 is performing autonomous driving, it displays a request 79 for preparation for manual driving on the meter display 15. By seeing the request 79 for preparation for manual driving, the occupants of the vehicle 2 can prepare for switching from autonomous driving to manual driving.

[0067] (1D) Normal image 61B is an overhead image. The display range in changed image 71 is expanded in the direction of the position of emergency vehicle 73 compared to the display range in normal image 61B. The occupants of vehicle 2 can know the direction of the position of emergency vehicle 73 by comparing the display range in changed image 71 with the display range in normal image 61B. <Other embodiments> Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and can be implemented in various modified forms.

[0068] (1) For example, as shown in Fig. 14, the display range of the changed image 71 includes the position of the emergency vehicle 73. In this case, the occupants of the vehicle 2 can know the position of the emergency vehicle 73 in more detail by looking at the changed image 71.

[0069] (2) The determination in step 4 may be made as to whether the predicted time it takes for the emergency vehicle 73 to arrive at vehicle 2 (hereinafter referred to as the predicted arrival time) is equal to or less than a preset threshold. If it is determined that the predicted arrival time is equal to or less than the threshold, the process proceeds to step 5. If it is determined that the predicted arrival time exceeds the threshold, the process returns to before step 4. The predicted arrival time can be calculated, for example, from a change in the relative position of the emergency vehicle 73 with respect to the position of vehicle 2.

[0070] (3) The display device that displays the normal image 61 and the changed image 71 may be a display device other than the meter display 15. Examples of the display device other than the meter display 15 include the CID 17, the HUD 19, and the like.

[0071] (4) The necessary conditions for displaying the manual driving preparation request 79 on the meter display 15 in step 8 may be that the results of steps 1 and 2 are affirmative and that the distance from the emergency vehicle 73 to the vehicle 2 is equal to or less than a specific threshold. The specific threshold may be the same as or different from the threshold used in step 4.

[0072] (5) The image processing device 3 and the method described herein may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to execute one or more functions embodied in a computer program. Alternatively, the image processing device 3 and the method described herein may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the image processing device 3 and the method described herein may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to execute one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium. The method for implementing the functions of each unit included in the image processing device 3 does not necessarily need to include software; all of the functions may be implemented using one or more hardware devices.

[0073] (6) Multiple functions of one component in the above embodiments may be realized by multiple components, or one function of one component may be realized by multiple components. Also, multiple functions of multiple components may be realized by one component, or one function realized by multiple components may be realized by one component. Also, part of the configuration of the above embodiments may be omitted. Also, at least part of the configuration of the above embodiments may be added to or substituted for the configuration of another of the above embodiments.

[0074] (7) In addition to the image processing device 3 described above, the present disclosure can also be realized in various forms, such as a system including the image processing device 3 as a component, a program for causing a computer to function as the image processing device 3, a non-transient physical recording medium such as a semiconductor memory on which this program is recorded, a display control method, etc. [Technical idea disclosed in this specification] [Item 1] An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), an information acquisition unit (35) configured to acquire information representative of the location of an emergency vehicle (73); a condition determination unit (36) configured to determine whether or not a first condition, that is, (a) the emergency vehicle is approaching the vehicle, and (b) a second condition, that is, the distance from the emergency vehicle to the vehicle or the predicted time until the emergency vehicle reaches the vehicle is equal to or less than a predetermined threshold, are satisfied based on the information acquired by the information acquisition unit; an image change unit (37) that changes the image when the condition determination unit determines that the first condition and the second condition are satisfied while the vehicle is performing automatic driving; Equipped with The image (71) after the change by the image change unit is an overhead image of the vehicle and other vehicles (63) around the vehicle from a viewpoint behind the vehicle. [Item 2] Item 1, the image processing device according to item 1, The image processing device, wherein the overhead image after modification by the image modification unit represents a planned travel path (77) of the emergency vehicle. [Item 3] Item 1 or 2, the image processing device The image processing device further includes a request display unit (39) that displays, on the display device, a message (79) requesting preparation for manual driving when the condition determination unit determines that the first condition and the second condition are met while the vehicle is performing automatic driving. [Item 4] The image processing device according to any one of items 1 to 3, An image processing device in which the display range of the overhead image after the change by the image change unit includes the position of the emergency vehicle. [Item 5] The image processing device according to any one of items 1 to 4, The image (61B) before the change by the image change unit is the overhead image, An image processing device in which the display range of the overhead image after the change by the image change unit is expanded in the direction of the position of the emergency vehicle compared to the display range of the overhead image before the change by the image change unit. [Item 6] The image processing device according to any one of items 1 to 5, configured to display an image including a map representing the surroundings of the vehicle on the display device; an image processing device configured to display the emergency vehicle in an image including the map if the location of the emergency vehicle is within the map; [Explanation of symbols]

[0075] 2...vehicle, 3...image processing device, 15...meter display, 35...information acquisition unit, 36...condition judgment unit, 37...image change unit, 38...notification unit, 39...request display unit, 61, 61A, 61B, 61C...normal image, 71...changed image, 73...emergency vehicle, 77...planned driving trajectory, 79...request for preparation for manual driving

Claims

1. An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), an information acquisition unit (35) configured to acquire information representing the location of an emergency vehicle (73); and a determination unit (36) configured to determine whether the emergency vehicle is approaching the vehicle based on the information acquired by the information acquisition unit (35); an image change unit (37) that changes the image when the determination unit determines that the emergency vehicle is approaching the vehicle while the vehicle is automatically driving; a request display unit (39) that displays, on the display device, a message (79) requesting preparation for manual driving when the judgment unit judges that the emergency vehicle is approaching the vehicle while the vehicle is performing automatic driving; Equipped with The determining unit is configured to further determine whether the emergency vehicle has already passed the vehicle; An image processing device configured such that, if the judgment unit determines that the emergency vehicle has already passed the vehicle, the image modification unit further changes the image (71) after modification by the image modification unit into an image including a notification indicating that the emergency vehicle has already passed beside the vehicle.

2. An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), an information acquisition unit (35) configured to acquire information representing the location of an emergency vehicle (73); and a determination unit (36) configured to determine whether the emergency vehicle is approaching the vehicle based on the information acquired by the information acquisition unit (35); an image change unit (37) that changes the image when the determination unit determines that the emergency vehicle is approaching the vehicle while the vehicle is automatically driving; Equipped with the image (71) after being changed by the image change unit includes an indication of the approach of the emergency vehicle and an indication (79) requesting preparation for manual driving; The determining unit is configured to further determine whether the emergency vehicle has already passed the vehicle; An image processing device configured such that, if the judgment unit determines that the emergency vehicle has already passed the vehicle, the image modification unit further changes the image (71) after modification by the image modification unit into an image including a notification indicating that the emergency vehicle has already passed beside the vehicle.

3. An image processing device (3) that displays an image (61, 61A, 61B, 61C, 71) on a display device (15) provided in a vehicle (2), an information acquisition unit (35) configured to acquire information representing the location of an emergency vehicle (73); and a determination unit (36) configured to determine whether the emergency vehicle is approaching the vehicle based on the information acquired by the information acquisition unit (35); an image change unit (37) that changes the image when the determination unit determines that the emergency vehicle is approaching the vehicle while the vehicle is automatically driving; Equipped with the image (71) after modification by the image modification unit includes an indication (79) requesting the driver to hold the steering wheel; when detecting that an occupant of the vehicle has gripped the steering wheel while the display requesting that the occupant grip the steering wheel is being displayed, the image change unit changes the display mode of the display requesting that the occupant grip the steering wheel and displays a display indicating that the emergency vehicle is approaching; The determining unit is configured to further determine whether the emergency vehicle has already passed the vehicle; An image processing device configured such that, if the judgment unit determines that the emergency vehicle has already passed the vehicle, the image modification unit further changes the image (71) after modification by the image modification unit into an image including a notification indicating that the emergency vehicle has already passed beside the vehicle.

Images