Vehicle surrounding environment display system, control method for vehicle surrounding environment display system, and program

Abstract

To improve user convenience regarding the display of projected signs from the vehicle. [Solution] A vehicle surrounding environment display system that generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from the vehicle's external sensors and displays an image of the virtual space as seen from a virtual viewpoint operated by the vehicle's user on a display device, comprising: a projection display unit that projects projection signs around the vehicle using the vehicle's lighting equipment; and a projection instruction recognition unit that recognizes the projection signs selected by the user and the projection position specified by the user on a display device displaying the virtual space, based on the user's operation, wherein the projection display unit projects the projection signs selected by the user to a position around the vehicle corresponding to the projection position on the display device recognized by the projection instruction recognition unit.

Description

Technical Field

[0004] , , ,

[0005] , , , ,

[0001] The present invention relates to a vehicle surrounding environment display system, a control method for the vehicle surrounding environment display system, and a program.

Background Art

[0002] Conventionally, as a technical document related to a vehicle surrounding environment display system, Japanese Unexamined Patent Application Publication No. 2020-088697 is known. This publication discloses a peripheral monitoring device that generates a virtual space including an image of the host vehicle and projects the surrounding environment of the vehicle as a three-dimensional video into the virtual space, and a user can freely view the environment around the vehicle by operating a virtual viewpoint in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, there is a known technique of projecting a predetermined sign onto a road surface or the like using a projection lamp or the like of the host vehicle to transmit information such as the traveling direction of the host vehicle to pedestrians or other vehicles around the host vehicle. It is required to improve the convenience for users with respect to such a technique.

Means for Solving the Problems

[0005] One aspect of the present invention is a vehicle surrounding environment display system that generates a virtual space corresponding to the surrounding environment of a vehicle based on detection information from the vehicle's external sensors and displays an image of the virtual space as seen from a virtual viewpoint operated by the vehicle's user on a display device, comprising: a projection display unit that projects a projection sign around the vehicle using the vehicle's lighting equipment; and a projection instruction recognition unit that recognizes a projection sign selected by the user and a projection position specified by the user on a display device displaying the virtual space, based on user operation, wherein the projection display unit projects the projection sign selected by the user to a position around the vehicle corresponding to the projection position on the display device recognized by the projection instruction recognition unit.

[0006] According to one aspect of the present invention, a vehicle surrounding environment display system can project a user-selected projection sign onto a location around the vehicle corresponding to a projection position specified in a virtual space based on user operation. This allows the user to intuitively display necessary signs at any location around their vehicle and appropriately communicate information about their vehicle to surrounding pedestrians and other vehicles, thereby improving user convenience regarding the display of projection signs from the vehicle.

[0007] In a vehicle surrounding environment display system according to one aspect of the present invention, the projected sign may include at least one of the following: a pedestrian priority sign to indicate that the vehicle should yield to pedestrians crossing in front of it; an emergency stop sign to indicate a vehicle malfunction; a safety area sign to indicate a safety area around a parked vehicle; a pick-up sign to indicate that the vehicle has arrived to pick up a vehicle; a parking location notification sign to help users outside the vehicle find the parked vehicle; a direction of travel sign to indicate the direction of travel of the vehicle in an environment with poor visibility; a vehicle width projection sign for oncoming vehicles to indicate the width of the vehicle to oncoming vehicles passing by the vehicle; and a vehicle width projection sign for the vehicle to indicate the width of the vehicle on a narrow road. This vehicle surrounding environment display system allows users to select appropriate projected signs according to various situations, effectively communicating their vehicle's intentions and status to surrounding pedestrians and other vehicles.

[0008] In a vehicle surrounding environment display system according to one aspect of the present invention, the projection display unit may change the position of the projection sign projected by the vehicle's lights when it recognizes a user operation to change the position of the projection sign being projected. This allows users to intuitively adjust the position of the projected sign, enabling flexible responses depending on the situation. For example, by displaying the projected sign in the appropriate position in various situations, such as when indicating an intention to yield to pedestrians or when informing others of a vehicle malfunction, communication with surrounding pedestrians and other vehicles can be facilitated.

[0009] In a vehicle surrounding environment display system according to one aspect of the present invention, the projection display unit may change the size of the projection sign projected by the vehicle's lighting equipment when it recognizes a user operation to change the size of the projection sign being projected. This allows users to freely adjust the size of projected signs according to the situation, improving visibility. For example, projected signs can be made larger for easier viewing from a distance, or smaller when detailed information is needed at close range. This allows for effective communication of information to surrounding pedestrians and other vehicles.

[0010] A vehicle surrounding environment display system according to one aspect of the present invention may further include: a situation determination unit that determines whether the situation of the vehicle is a pre-set sign projection suggestion scene based on the detection results of the vehicle's external sensors; and a projection suggestion unit that, when the situation determination unit determines that the situation of the vehicle is a sign projection suggestion scene, proposes to the user the projection of a sign by the vehicle's lighting equipment by displaying a projection sign at a projection position on a display device displaying a virtual space, using a projection sign and projection position pre-associated with the sign projection suggestion scene. This vehicle surrounding environment display system can automatically determine whether the vehicle's situation is a sign projection suggestion scene based on the detection results of the vehicle's external sensors. This eliminates the need for the user to select the appropriate projection sign in a given situation. Furthermore, if the situation determination unit determines that it is a sign projection suggestion scene, the projection suggestion unit can display the projection sign on the display device, suggesting the projection sign to the user. This allows the user to quickly and appropriately select the projection sign and clearly communicate the vehicle's intentions and situation to surrounding pedestrians and other vehicles. In addition, the projection position of the projection sign is also automatically suggested, allowing the user to operate it intuitively and enabling the effective display of the projection sign.

[0011] In a vehicle surrounding environment display system according to one aspect of the present invention, the display device is a touch panel display of the user's mobile information terminal, and the projection display unit may project a parking location notification sign as a projected sign around the vehicle in response to the user's operation of the mobile information terminal, so that users outside the vehicle can find the parked vehicle. This vehicle surrounding environment display system allows users to easily project parking location notification signs by operating the touch panel display of their mobile device. This makes it possible for users outside the vehicle to quickly and easily find the location of their vehicle in a parking lot.

[0012] Another aspect of the present invention is a control method for a vehicle surrounding environment display system that generates a virtual space corresponding to the surrounding environment of a vehicle based on detection information from the vehicle's external sensors, and displays an image of the virtual space as seen from a virtual viewpoint operated by the vehicle's user on a display device, wherein, based on user operation, the system recognizes a projection sign selected by the user and a projection position specified by the user on the display device displaying the virtual space, and uses the vehicle's lighting equipment to project the projection sign selected by the user onto a position around the vehicle corresponding to the projection position on the display device.

[0013] According to another aspect of the present invention, a control method for a vehicle surrounding environment display system allows a user-selected projection sign to be projected onto a position around the vehicle corresponding to a projection position specified in a virtual space based on user operation. This allows the user to intuitively display necessary signs at any position around their vehicle and appropriately communicate information about their vehicle to surrounding pedestrians and other vehicles, thereby improving user convenience regarding the display of projection signs from the vehicle.

[0014] A further aspect of the present invention is a program that operates the ECU of a vehicle as a vehicle surrounding environment display system, which generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from the vehicle's external sensors and displays an image of the virtual space as seen from a virtual viewpoint operated by the user of the vehicle on a display device, wherein the ECU is operated as a projection display unit that projects a projection sign around the vehicle using the vehicle's lighting equipment, and a projection instruction recognition unit that recognizes the projection sign selected by the user and the projection position specified by the user on a display device displaying the virtual space, and the projection display unit is instructed to project the projection sign selected by the user to a position around the vehicle corresponding to the projection position on the display device recognized by the projection instruction recognition unit.

[0015] According to the program according to another aspect of the present invention, the projection sign selected by the user can be projected onto a position around the host vehicle corresponding to the projection position specified in the virtual space based on the user's operation. As a result, the user can display the necessary sign at an arbitrary position around the host vehicle with an intuitive operation, and can appropriately transmit the information of the host vehicle to pedestrians and other vehicles around. Therefore, the convenience of the user regarding the display of the projection sign from the host vehicle can be improved.

Effects of the Invention

[0016] According to each aspect of the present invention, the convenience of the user regarding the display of the projection sign from the host vehicle can be improved.

Brief Description of the Drawings

[0017] [Figure 1] It is a block diagram showing a vehicle surrounding environment display system according to an embodiment. [Figure 2] It is a diagram for explaining an example of a user's operation in a scene of yielding the road to a pedestrian. [Figure 3] It is a diagram for explaining an example of projection processing in a scene of yielding the road to a pedestrian. [Figure 4] It is a diagram for explaining an example of a user's operation in a scene where the host vehicle has made an emergency stop on the road shoulder or the like. [Figure 5] It is a diagram for explaining an example of projection processing in a scene where the host vehicle has made an emergency stop on the road shoulder or the like. [[ID=D28]] [Figure 6] It is a diagram for explaining another example of a user's operation in a scene where the host vehicle has made an emergency stop on the road shoulder or the like. [Figure 7] It is a diagram for explaining another example of projection processing in a scene where the host vehicle has made an emergency stop on the road shoulder or the like. [Figure 8] It is a diagram for explaining an example of a user's operation in a scene where the host vehicle has come to pick up a child. [Figure 9] It is a diagram for explaining an example of projection processing in a scene where the host vehicle has come to pick up a child. [Figure 10] This is a diagram for explaining an example of a user operation in a scene where an outside user finds their parked vehicle. [Figure 11] This is a diagram for explaining an example of projection processing in a scene where an outside user finds their parked vehicle. [Figure 12] This is a diagram for explaining an example of projection processing in a road environment with poor visibility. [Figure 13] This is a diagram for explaining an example of a user operation in a scene where there is an oncoming vehicle passing by the own vehicle. [Figure 14] This is a diagram for explaining an example of projection processing in a scene where there is an oncoming vehicle passing by the own vehicle. [[ID=1G]] [Figure 15] This is a flowchart showing an example of the projection proposal process of a vehicle surrounding environment display system. [Figure 16] (a) This is a flowchart showing an example of a projection sign correction process. (b) This is a flowchart showing an example of a first proposed scene determination process. [Figure 17] (a) This is a flowchart showing an example of a second proposed scene determination process. (b) This is a flowchart showing an example of a third proposed scene determination process. [Figure 18] (a) This is a flowchart showing an example of a fourth proposed scene determination process. (b) This is a flowchart showing an example of a fifth proposed scene determination process.

Embodiments for Carrying out the Invention

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0019] [Control Method of Vehicle Surrounding Environment Display System] Figure 1 is a block diagram showing a vehicle surrounding environment display system 100 according to one embodiment. The vehicle surrounding environment display system 100 shown in Figure 1 is a system installed in a vehicle such as a passenger car or cargo truck (hereinafter referred to as "the vehicle") to support the user's recognition of the vehicle's surrounding environment. The vehicle surrounding environment display system 100 generates a virtual space that reflects the surrounding environment of the vehicle and displays images of the virtual space on a display from a virtual viewpoint operated by the user. The vehicle surrounding environment display system 100 displays the surrounding environment of the vehicle on the display as a so-called 3D view.

[0020] The user may be the driver of the vehicle, a passenger in the vehicle, or the owner of the vehicle. The user may also be an operator who provides remote support to the vehicle using a remote support system. In the remote support system, the operator can make decisions about the vehicle's movement (decisions such as going, turning left or right, stopping, etc.) or perform driving operations on the vehicle through remote support equipment installed outside the vehicle and capable of communicating with the vehicle. The vehicle is not limited to vehicles that can be remotely supported by the remote support system. The vehicle may be a vehicle with autonomous driving capabilities or a vehicle without autonomous driving capabilities.

[0021] As shown in Figure 1, the vehicle surrounding environment display system 100 is equipped with an ECU (Electronic Control Unit) 20 that comprehensively manages the system. The ECU 20 is an electronic control unit having a CPU (Central Processing Unit) and a memory unit. The memory unit consists of, for example, ROM (Read-Only Memory), RAM (Random Access Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), etc. The ECU 20 realizes various functions by, for example, executing a program stored in the memory unit with the CPU. The ECU 20 may be composed of multiple electronic units. The ECU 20 is connected to an external camera 1 (external sensor), a radar sensor 2 (external sensor), a vehicle speed sensor 3, a user operation reception unit 4, a driver monitor camera 5, a GNSS receiver 6, a map database 7, a display device 8, a communication unit 9, and a lighting control unit 10.

[0022] External camera 1 is an imaging device that captures images of the external conditions of the vehicle. External camera 1 is composed of, for example, a front camera that captures images of the area in front of the vehicle, a rear camera that captures images of the area behind the vehicle, and side cameras that capture images of the left and right sides of the vehicle. The number of cameras in external camera 1 is not particularly limited and may be just one. External camera 1 transmits the captured image information to ECU 20.

[0023] The radar sensor 2 is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle. The radar sensor 2 can be configured to include a millimeter-wave radar or LiDAR (Light Detection and Ranging). The radar sensor 2 transmits object detection information about the detected objects to the ECU 20. The radar sensor 2 and the external camera 1 constitute an external sensor for detecting the surrounding environment of the vehicle. The object detection information from the radar sensor 2 or the image information captured by the external camera 1 corresponds to the detection information of the external sensor.

[0024] The vehicle speed sensor 3 is a detector that detects the speed of the vehicle. For example, the vehicle speed sensor 3 can be a wheel speed sensor that detects the rotational speed of the vehicle's wheels or a drive shaft that rotates integrally with the wheels. The vehicle speed sensor 3 transmits the detected vehicle speed information to the ECU 20.

[0025] The user operation reception unit 4 is a device that receives user input for virtual viewpoint control. The user operation reception unit 4 can, for example, be an input unit for an HMI (Human Machine Interface) installed in the vehicle. The input unit may include, for example, a touch panel display, buttons, levers, switches, etc. Furthermore, the user operation reception unit 4 may also be capable of receiving input via voice recognition or gestures.

[0026] The user operation reception unit 4 may utilize a portable information terminal or computer input device connected to the vehicle via communication. Alternatively, the operator terminal for a remote support system may be used as the user operation reception unit 4.

[0027] The driver monitoring camera 5 is a camera that captures images of the driver of the vehicle. The driver monitoring camera 5 is mounted, for example, on the cover of the steering column of the vehicle and is positioned to capture images of the driver's head. The driver monitoring camera 5 may be embedded in the steering wheel instead of the steering column, or it may be integrated with the steering wheel. The driver monitoring camera 5 transmits the captured images of the driver to the ECU 20.

[0028] The GNSS receiver 6 measures the vehicle's position (e.g., the vehicle's latitude and longitude) by receiving signals from positioning satellites. The GNSS receiver 6 may also be part of a GPS (Global Positioning System) system. The GNSS receiver 6 transmits the measured vehicle position information to the ECU 20.

[0029] Map database 7 is a database that stores map information. Map database 7 is formed in a storage device such as an HDD (Hard Disk Drive) installed in the vehicle. Map information includes road location information, road shape information (e.g., types of curves and straight sections, curvature of curves, etc.), intersection and junction location information, and structure location information. Map information may also include information such as intersections with poor visibility. Map database 7 may also be formed on a server that can communicate with the vehicle.

[0030] Display device 8 is, for example, a center display mounted on the dashboard of the vehicle. Display device 8 may also be the display of a tablet computer that can be installed in the vehicle, or it may be a HUD (Head Up Display). Display device 8 is not limited to those mounted in the vehicle.

[0031] The display device 8 may be an operator's display for a remote support system located at a facility away from the vehicle. The display device 8 may be the display portion of a portable information terminal carried by the user, or the display portion of the user's tablet computer or desktop computer. In this embodiment, the display device 8, which is a touch panel display, also functions as a user operation reception unit 4.

[0032] The communication unit 9 is a communication device that controls wireless communication with the outside world. The communication unit 9 communicates with the user's mobile device or computer to obtain information about the user's operations.

[0033] The lighting control unit 10 is the part that controls the vehicle's lighting. In this embodiment, the lighting control unit 10 functions as an external information notification device that notifies information to the outside of the vehicle by projecting a projection sign, described later, onto the road surface or a structure. The structure may include walls or guardrails, or it may include the side of other vehicles such as trucks.

[0034] The lighting control unit 10 is configured to project a projection sign around the vehicle. The lighting control unit 10 may also be configured to freely project the projection sign in all 360° around the vehicle. The vehicle's lighting may be equipped with a dedicated projection lamp, or standard lighting such as turn signal lamps or headlights may be configured to be usable for projecting the projection sign. The configuration of the vehicle's lighting and the lighting control unit 10 is not particularly limited. Various configurations can be adopted as long as projection of the projection sign is possible.

[0035] Next, the functional configuration of the ECU20 will be described. As shown in Figure 1, the ECU20 includes a virtual space generation unit 21, an image display unit 22, a projection instruction recognition unit 23, a projection display unit 24, a status determination unit 25, and a projection proposal unit 26.

[0036] Some of the functions of the ECU20 described below may be executed on a server (e.g., a server for a remote support system), a mobile terminal, or a computer (e.g., a tablet computer or a desktop computer) that can communicate with the vehicle.

[0037] The virtual space generation unit 21 generates a virtual space corresponding to the surrounding environment of the vehicle, for example, based on the image information captured by the external camera 1. The surrounding environment of the vehicle includes, for example, the position of the white lines of the lane in which the vehicle is traveling. The surrounding environment of the vehicle may also include the status (position, direction of travel, etc.) of other vehicles, such as preceding vehicles and vehicles traveling alongside adjacent vehicles.

[0038] The virtual space is generated, for example, as a 3D image created by combining multiple images. The method of image synthesis is not particularly limited. The virtual space generation unit 21 generates the virtual space as a 3D image by, for example, projecting each image onto a global coordinate system that serves as the reference for the virtual space and associating overlapping pixels with each other.

[0039] The virtual space generation unit 21 places a vehicle icon corresponding to the vehicle in the virtual space. The vehicle icon is placed as a three-dimensional icon. The vehicle icon can be formed using polygons, voxels, or other CG processing. The virtual space generation unit 21 may generate a vehicle icon that reflects the state of the vehicle. The virtual space generation unit 21 may reflect the lighting status of the vehicle's lights (headlights, turn signals, brake lights, etc.) in the lighting status of the vehicle's lights on the vehicle icon, and may also reflect the steering angle of the vehicle's tires in the tires of the vehicle icon.

[0040] The virtual space generation unit 21 may, when it recognizes another vehicle based on the image information captured by the external camera 1, place a three-dimensional icon of that other vehicle in the virtual space. Similarly, the virtual space generation unit 21 may place a three-dimensional pedestrian icon in the virtual space. The virtual space generation unit 21 may recognize other vehicles, etc., based on object detection information from the radar sensor 2 rather than the image information captured by the external camera 1, or it may recognize other vehicles, etc., using both the external camera 1 and the radar sensor 2.

[0041] The virtual space generation unit 21 may also recognize other vehicles around its own vehicle using information about the surrounding environment recognized by other vehicles through vehicle-to-vehicle communication. The virtual space generation unit 21 may also acquire image information from cameras installed on the road and various traffic information by communicating with, for example, a traffic information management server managed by the government, and use this to recognize other vehicles.

[0042] Furthermore, the virtual space generation unit 21 may predict the behavior of other vehicles based on the image information captured by the external camera 1 or the object detection information of the radar sensor 2, and may display the predicted behavior of other vehicles in association with other vehicle icons. The virtual space generation unit 21 may, for example, display the predicted path of other vehicles using arrow icons, or display the predicted stopping position of other vehicles that are decelerating using block-shaped icons that extend in the direction of the lane width. Similarly, the virtual space generation unit 21 may display the predicted behavior of pedestrians in association with pedestrian icons.

[0043] The method for generating the virtual space is not limited to combining multiple images from the external camera 1. The virtual space generation unit 21 does not need to generate the virtual space as a 3D image as long as the user can recognize the surrounding environment of the vehicle. The virtual space generation unit 21 may also generate the virtual space by arranging the vehicle icon, white lines, and other vehicle icons so that the positional relationship between the vehicle and the white lines and other vehicles can be seen.

[0044] The image display unit 22 displays images of the virtual space generated by the virtual space generation unit 21 on the display device 8, as seen from a virtual viewpoint operated by the user. The image display unit 22 moves the virtual viewpoint in response to user operations input to the user operation reception unit 4.

[0045] The projection instruction recognition unit 23 recognizes the user's projection instruction for a projection sign based on the user's operation input to the user operation reception unit 4. The projection instruction includes the type of projection sign and the projection position.

[0046] Projected signs are visual displays projected onto the road surface or structures surrounding a vehicle using the vehicle's lights, intended to communicate the vehicle's intentions and status to pedestrians and other vehicles. Projected signs are selected based on user operation or automatic system detection. Projected signs may include any of the following: pedestrian priority signs, emergency stop signs, safety area signs, pick-up signs, parking location notification signs, direction of travel signs, oncoming vehicle width projection signs, and vehicle width projection signs. Each sign will be described in more detail later.

[0047] Here, Figure 2 illustrates an example of user action in a scene where a character yields the right of way to a pedestrian. Figure 3 illustrates an example of projection processing in a scene where a character yields the right of way to a pedestrian.

[0048] Figure 2 shows an example of a display device 8 as the center display of the vehicle. The display device 8 has a first screen D1 that freely displays images in a virtual space as seen from a virtual viewpoint operated by the user, and a second screen D2 that displays a plan view image of the vehicle seen from above. A pedestrian crossing is displayed on the first screen D1 as a surrounding element. A pedestrian crossing can also be displayed on the second screen D2. In this embodiment, the second screen D2 is not essential.

[0049] As shown in Figure 2, on the display device 8, various icons 71-78 corresponding to the projection sign are displayed below the first screen D1 and the second screen D2. The display device 8 is a touch panel display. The user's hand is indicated by symbol 50.

[0050] In Figure 2, the user drags and drops the pedestrian priority icon 71 in front of the vehicle icon Ma on the first screen D1. The icon display after dragging and dropping is shown as code 71A. The pedestrian priority icon 71 is the icon corresponding to the pedestrian priority sign 81.

[0051] The pedestrian priority sign 81 is a projected sign indicating that the vehicle M will yield the right of way to pedestrians crossing in front of it. For example, the pedestrian priority sign 81 may project a sign combining the text "AFTER YOU" with a hand gesture inviting the pedestrian to cross. The shape of the pedestrian priority sign 81 is not particularly limited, as long as it conveys the intention to yield to the pedestrian. In Figure 3, pedestrian P can see the pedestrian priority sign 81 and understand that the user of vehicle M intends to yield the right of way.

[0052] The user can drag and drop the pedestrian priority icon 71 to any position within a 360° radius around their vehicle icon Ma, in combination with changing the position of the virtual viewpoint on the first screen D1. Note that the operation to indicate the projection position of the projection sign is not limited to drag-and-drop using a finger.

[0053] The user may activate any icon by touching it, and then indicate the projection position of the projection sign by touching any location on the first screen D1 or the second screen D2. This is an example of changing the position of the projection sign. The user may also indicate the projection position using an operating lever, or by using voice recognition or gesture recognition. Voice recognition can be performed, for example, by the voice recognition function of the display device 8. Gesture recognition may be performed from the image captured by the driver monitor camera 5, or the display device 8 may be equipped with a camera. The operating lever may be a virtual lever on the touch panel.

[0054] Furthermore, users can freely adjust the size of the projected pedestrian priority sign by pinching out or pinching in with two fingers (for example, the thumb and index finger) to change the size of the pedestrian priority icon 71. The operation to change the size is not limited to pinching out or pinching in, and various operations can be used. Users may change the size of the icon by touching and activating the icon on the first screen D1 or the second screen D2, and then operating buttons or other means, or the size of the icon may be changed by voice recognition or gesture recognition. The buttons may also be virtual buttons on the touch panel.

[0055] Furthermore, even after the projected sign has been projected, users can change the projection position and size of the projected sign by manipulating icons on the display device 8. This allows users to adjust the position and size of the projected sign while observing the situation so that pedestrians and drivers of other vehicles can notice the projected sign.

[0056] The projection display unit 24 uses the vehicle's lighting control unit 10 to project a projection sign around the vehicle. The projection display unit 24 projects a projection sign corresponding to the icon selected by the user to a position around the vehicle corresponding to the projection position on the display device 8 recognized by the projection instruction recognition unit 23. Specifically, when the user operation shown in Figure 2 is performed, the projection display unit 24 projects a pedestrian priority sign 81 to the front of the vehicle M, as shown in Figure 3. As shown in Figure 3, the projection display unit 24 may also automatically adjust the orientation of the pedestrian priority sign 81 based on the pedestrian's position. The projection display unit 24 projects so that the pedestrian priority sign 81 is facing forward from the pedestrian's perspective.

[0057] The vehicle surrounding environment display system 100 may also be configured to automatically determine the status of its own vehicle and propose the projection of a projected sign. For example, in a scene where the vehicle stops before a crosswalk to yield to pedestrians, the vehicle surrounding environment display system 100 may propose to the user the projection of a pedestrian priority sign 81.

[0058] The status determination unit 25 and projection suggestion unit 26 of the ECU 20 are used to suggest projections. The status determination unit 25 determines whether the status of the vehicle is a sign projection suggestion scene. A sign projection suggestion scene is a scene that is pre-set as a scene in which a projected sign should be suggested to the user.

[0059] Specifically, the situation determination unit 25 determines, for example, whether the vehicle is stopping before a crosswalk to yield to pedestrians, based on the image captured by the external camera 1 and / or the detection results of the radar sensor 2 and the detection results of the vehicle speed sensor 3. Note that detection of a crosswalk is not mandatory; it is sufficient to determine whether the vehicle is yielding to pedestrians.

[0060] If the situation determination unit 25 determines that the vehicle is stopping before a crosswalk to yield to pedestrians, the projection suggestion unit 26 suggests to the user the projection of a pedestrian priority sign 81 by displaying an image or outputting sound. For example, the projection suggestion unit 26 displays a pedestrian priority icon 71 in front of the vehicle icon Ma on the first screen D1 of the display device 8, and also suggests projection to the user by voice. An approve button and a cancel button may also be displayed on the first screen D1.

[0061] The projection proposal unit 26 determines whether the user has performed an operation to approve the proposal. For example, if the user touches the approval button displayed on the first screen D1, the projection proposal unit 26 recognizes that the user has approved the proposal. The projection proposal unit 26 may also recognize that the user has approved the proposal through voice recognition or gesture recognition.

[0062] The projection display unit 24 projects a projection sign according to the proposal content of the projection proposal unit 26 approved by the user. If the user approves the projection of a pedestrian priority sign 81, the projection display unit 24 projects the pedestrian priority sign 81 around the vehicle.

[0063] Furthermore, the user may modify the projection position and size of the projection sign proposed by the projection proposal unit 26 by manipulating the icon of the projection sign before approving the proposal. In that case, the projection display unit 24 will project the projection sign according to the projection position and size of the projection sign approved by the user. This is the same in other scenes as well.

[0064] Next, we will explain an example of a scenario in which your vehicle makes an emergency stop on the shoulder of the road. Figure 4 is a diagram illustrating an example of user operation in a scenario in which your vehicle makes an emergency stop on the shoulder of the road. Figure 5 is a diagram illustrating an example of projection processing in a scenario in which your vehicle makes an emergency stop on the shoulder of the road.

[0065] In a scenario where the vehicle makes an emergency stop on the shoulder of the road, the user drags and drops the emergency stop icon 76 to the right of the vehicle icon Ma, as shown in Figure 4. The emergency stop icon 76 after being dragged and dropped is shown as code 76A. The emergency stop icon 76 is the icon corresponding to the emergency stop sign 86 (see Figure 5). The emergency stop sign 86 is a projected sign used to inform those around the vehicle of a malfunction. For example, the emergency stop sign 86 can be a triangular sign including an exclamation mark "!".

[0066] In the situation shown in Figure 4, the projection instruction recognition unit 23 recognizes the emergency stop sign 86 (emergency stop icon 76) selected by the user and the projection position of the emergency stop sign 86. Also in Figure 4, the user enlarges the size of the projected sign by pinching out the emergency stop icon 76.

[0067] As shown in Figure 5, the projection display unit 24 uses the lighting control unit 10 to project an emergency stop sign 86 onto the road surface to the right of the vehicle M. Because the user has performed a pinch-out operation to enlarge the size of the projected sign, the projection display unit 24 projects the emergency stop sign 86 at a larger size than its initial size. As a result, drivers of other vehicles N traveling around the vehicle M can notice the enlarged emergency stop sign 86 and understand that the vehicle M has stopped on the shoulder of the road due to a breakdown.

[0068] The vehicle surrounding environment display system 100 may suggest to the user the projection of an emergency stop sign 86 when the vehicle makes an emergency stop on the shoulder of the road or elsewhere.

[0069] Specifically, the situation determination unit 25 may determine, for example, whether the vehicle has made an emergency stop on the shoulder of the road or elsewhere, based on the fault detection result of the ECU 20 and the detection result of the vehicle speed sensor 3. Alternatively, instead of fault detection results, the system may determine that the vehicle has stopped due to a malfunction or an impediment to the driver's operation if it detects that the vehicle has stopped on the shoulder of the road in the middle of an expressway, based on the vehicle's position information from the GNSS receiver 6 and the map information from the map database 7. The situation determination unit 25 may also detect the emergency stop of the vehicle using other well-known methods. The situation determination unit 25 may also initiate a proposal by the projection proposal unit 26 after a certain period of time has elapsed while the vehicle has made an emergency stop.

[0070] If the situation determination unit 25 determines that the vehicle has made an emergency stop on the shoulder of the road or elsewhere, the projection suggestion unit 26 suggests to the user the projection of an emergency stop sign 86 by displaying an image or outputting sound. For example, the projection suggestion unit 26 displays an emergency stop icon 76 on the first screen D1 of the display device 8, to the right rear of the vehicle icon Ma (towards the rear on the lane opposite the wall), and also suggests projection to the user by voice. The projection suggestion unit 26 may also suggest projecting emergency stop icons 76 to both the left rear and the right rear of the vehicle. The projection suggestion unit 26 may project the emergency stop sign 86 not only onto the road surface but also onto walls to make it easier to inform drivers of other vehicles in the vicinity of the vehicle's status.

[0071] If the user approves the proposal, the projection proposal unit 26 uses the lighting control unit 10 to project the emergency stop sign 86. If the user changes the projection position or size of the proposed emergency stop sign 86, the projection proposal unit 26 will perform the projection according to the projection position and size of the emergency stop sign 86 that were approved by the user.

[0072] Next, Figure 6 illustrates another example of user actions in a scenario where the vehicle makes an emergency stop on the roadside or similar location. Figure 7 illustrates another example of projection processing in a scenario where the vehicle makes an emergency stop on the roadside or similar location.

[0073] In a scenario where the vehicle makes an emergency stop on the shoulder of the road, the user activates the safety area drawing icon 78, as shown in Figure 6, and then draws a safety area line L on the first screen D1 by tracing around the vehicle icon Ma with their finger 50. The user can draw the safety area line L freehand.

[0074] In the situation shown in Figure 6, the projection instruction recognition unit 23 recognizes the safety area sign 88 (safety area drawing icon 78) selected by the user and the range of the safety area sign 88. As shown in Figure 7, the projection display unit 24 uses the lighting control unit 10 to project the safety area sign 88 around the vehicle M.

[0075] A safety area sign 88 is a sign used to indicate a safety area around a vehicle while it is stopped. The safety area sign 88 is projected as an L-shaped sign at the four corners around the vehicle M, for example, to form an area. The safety area sign 88 may be projected as a line enclosing the entire circumference of the vehicle M, or it may be projected as an area that includes the vehicle M and is illuminated in a predetermined color. This ensures that drivers of other vehicles N around the vehicle M are careful not to enter the area inside the safety area sign 88 of the vehicle M, making it easier for the user of the vehicle M to act outside the vehicle.

[0076] Furthermore, the projection area of ​​the safety area sign 88 does not necessarily need to be drawn freehand. The projection area can be set simply by touching the icon corresponding to the safety area sign 88 to activate it. In this case, the projection area of ​​the safety area sign 88 may be expandable by operations such as pinching out.

[0077] Next, we will explain an example of a scenario where the vehicle comes to pick up a child. Note that the person being picked up is not limited to a child; it can also be an adult. In the scenario where the vehicle comes to pick up a child, the vehicle surrounding environment display system 100 projects a pick-up sign at the user's request. The pick-up sign is a projected sign that indicates the vehicle has come to pick up the child.

[0078] Figure 8 illustrates an example of user actions in a scene where the vehicle is picking up a child. Figure 9 illustrates an example of projection processing in a scene where the vehicle is picking up a child.

[0079] In a scene where the user's vehicle is picking up a child, the user drags and drops the pick-up icon 77 between the vehicle icon Ma and the school entrance, as shown in Figure 8. The pick-up icon 77 after being dragged and dropped is shown as code 77A. The pick-up icon 77 is the icon corresponding to the pick-up sign 87 (see Figure 9). The pick-up sign 87 is a sign that indicates the user's vehicle has come to pick up the child. For example, the pick-up sign 87 can be shaped like a pedestrian crossing. The pick-up sign 87 may also include the words "Pick-up" and an arrow.

[0080] Users can freely adjust the length of the projected welcome sign by pinching out or in with two fingers to change the size of the welcome icon 77. For example, a user can pinch out on the welcome icon 77 to make it the length from their vehicle M to the school entrance.

[0081] The projection instruction recognition unit 23 recognizes, based on the user's operation described above, that the user has selected the welcome sign 87 (welcome icon 77) as the projection sign, and the projection position of the welcome sign 87 relative to its own vehicle. The projection display unit 24 projects the welcome sign 87 selected by the user onto the road surface position corresponding to the user's projection position. The projection display unit 24 performs the projection of the welcome sign 87 by controlling the lighting control unit 10 (see Figure 9).

[0082] Next, we will describe an example of a scenario in which a user outside the vehicle finds their parked vehicle. In a scenario in which a user outside the vehicle finds their parked vehicle, the vehicle surrounding environment display system 100 projects a parking location notification sign at the user's request. The parking location notification sign is a projected sign that helps users outside the vehicle find their parked vehicle.

[0083] Figure 10 illustrates an example of user actions in a scenario where a user outside the vehicle finds their parked vehicle. Figure 11 illustrates an example of projection processing in a scenario where a user outside the vehicle finds their parked vehicle.

[0084] In Figure 10, a user outside the vehicle operates a portable information terminal 90 that functions as a touch panel display (display device). The portable information terminal 90 is wirelessly connected to the ECU 20 via a communication unit 9, and can view the screen corresponding to the first screen D1 or the second screen D2.

[0085] In a scenario where a user outside the vehicle finds their parked vehicle, the user drags and drops the parking location notification icon 91 onto the screen of the mobile information terminal 90 in front of the vehicle icon Ma, as shown in Figure 10. The parking location notification icon 91 after being dragged and dropped is shown as code 91A.

[0086] The projection instruction recognition unit 23 recognizes, based on the user's operation described above, that the user has selected a parking location notification sign as the projection sign, and recognizes the projection position of the parking location notification sign 92 relative to the vehicle itself.

[0087] The projection display unit 24 projects the parking location notification sign 92 selected by the user onto the road surface location corresponding to the user's projection position (see Figure 11). The projection display unit 24 projects the parking location notification sign 92 by controlling the lighting control unit 10. This allows the user to easily find the location of their vehicle in the parking lot.

[0088] The vehicle surrounding environment display system 100 may also automate the projection of the parking location notification sign 92. When the user has turned on automatic projection of the parking location notification sign 92, the projection display unit 24 automatically projects the parking location notification sign 92 after the user's mobile device 90 has been parked and the distance between the mobile device 90 and the vehicle has exceeded a certain distance, and then when the distance between the mobile device 90 and the vehicle falls below that certain distance. After that, the user can adjust the position and size of the parking location notification sign 92 by operating the parking location notification icon 91 on the mobile device 90.

[0089] Next, we will explain an example of a road environment with poor visibility. The vehicle surrounding environment display system 100 projects a direction of travel sign at the user's operation in a road environment with poor visibility. A direction of travel sign is a projected sign that informs those around the vehicle of its direction of travel in a road environment with poor visibility.

[0090] Figure 12 illustrates an example of projection processing in a road environment with poor visibility. Figure 12 shows a poor-visibility intersection BI and a direction of travel sign 93 projected in front of the vehicle M. An example of a poor-visibility intersection BI is an intersection where the view is obstructed by obstacles such as walls on either side of the vehicle M. More details about road environments with poor visibility will be described later.

[0091] The user operation will be omitted here. The user drags and drops the direction of travel icon to any position in front of the vehicle icon Ma, for example, similar to the pedestrian priority icon 71 in Figure 2. The direction of travel icon corresponds to the direction of travel sign. The direction of travel sign is a sign that informs those around the vehicle of its direction of travel in road environments with poor visibility. For example, an arrow-shaped sign is projected. The direction of travel sign does not need to be a sign that can inform those around the vehicle of its presence, and its shape is not particularly limited.

[0092] The projection instruction recognition unit 23 recognizes, based on user operation, that the user has selected a direction of travel sign as the projection sign, and recognizes the projection position of the direction of travel sign 93 relative to the vehicle M. It should be noted that projection of the direction of travel sign does not necessarily require drag-and-drop or other operations to indicate the projection position; the direction of travel sign may be projected at a predetermined position in front of the vehicle M simply by touching and activating the direction of travel icon. Even in this case, the size of the direction of travel sign can be changed by pinching out or similar gestures.

[0093] The projection display unit 24 projects the user-selected direction of travel sign 93 onto the road surface corresponding to the user's projection position (see Figure 12). The projection display unit 24 projects the direction of travel sign 93 by controlling the lighting control unit 10. This makes it easier for other vehicles and pedestrians to notice the presence of the vehicle and understand the direction of travel in road environments with poor visibility, thereby reducing risks caused by poor visibility. It should be noted that the sign does not necessarily have to be related to the direction of travel of the vehicle M; any sign that can make the presence of the vehicle M noticeable to those around it in environments with poor visibility is acceptable.

[0094] The vehicle surrounding environment display system 100 may suggest to the user the projection of a direction of travel sign 93 in road environments with poor visibility.

[0095] Specifically, the situation determination unit 25 determines whether the vehicle's driving environment is a road environment with poor visibility based on the vehicle's position information obtained from the GNSS receiver 6 and the information on intersections with poor visibility included in the map database 7. The situation determination unit 25 may also determine that the vehicle's driving environment is a road environment with poor visibility when the driver's view is significantly obstructed, as shown in Figure 12, based on the image captured by the external camera 1 and the detection results of the radar sensor 2. If an intersection is the target of the determination, the situation determination unit 25 may also determine the presence of road signs with poor visibility when the intersection detection distance by the vehicle falls below a predetermined distance (e.g., 5 m).

[0096] The situation determination unit 25 may determine that the vehicle's driving environment is a road environment with poor visibility if it detects a road sign indicating an intersection with poor visibility based on the image captured by the external camera 1. The situation determination unit 25 may also determine whether or not the road environment has poor visibility based on weather conditions such as rain or fog.

[0097] If the situation determination unit 25 determines that the vehicle's driving environment is a road environment with poor visibility, the projection suggestion unit 26 suggests to the user the projection of the direction of travel sign 93 by displaying an image or outputting sound. For example, the projection suggestion unit 26 displays the direction of travel sign 93 in front of the vehicle icon Ma on the first screen D1 of the display device 8, and also suggests the projection to the user by voice.

[0098] If the user approves the proposal, the projection proposal unit 26 uses the lighting control unit 10 to project the direction of travel sign 93. If the user changes the projection position or size of the proposed direction of travel sign 93, the projection proposal unit 26 will perform projection according to the projection position and size of the direction of travel sign 93 that were approved by the user.

[0099] The vehicle surrounding environment display system 100 may also automate the projection of the direction of travel sign 93. When the user has turned on automatic projection of the direction of travel sign 93, the projection display unit 24 automatically projects the direction of travel sign 93 when the situation determination unit 25 determines that the vehicle's driving environment is a road environment with poor visibility. After that, the user can adjust the position and size of the direction of travel sign 93 by operating the direction of travel icon on the display device 8.

[0100] Next, we will explain an example of a scenario in which there is an oncoming vehicle passing the vehicle. In a scenario in which there is an oncoming vehicle passing the vehicle, the vehicle surrounding environment display system 100 projects a vehicle width projection sign for oncoming vehicles at the user's operation. The vehicle width projection sign for oncoming vehicles is a sign that indicates the vehicle width of the vehicle to oncoming vehicles passing the vehicle. The vehicle width projection sign for oncoming vehicles is projected, for example, as two lines indicating the vehicle width of the vehicle.

[0101] Figure 13 illustrates an example of user operation in a scene where an oncoming vehicle is passing by the user's vehicle. Figure 14 illustrates an example of projection processing in a scene where an oncoming vehicle is passing by the user's vehicle.

[0102] In a scene where an oncoming vehicle is passing the user's vehicle, the user drags and drops the oncoming vehicle width projection icon 79, which corresponds to the oncoming vehicle width projection sign, in front of the user's vehicle icon Ma on the first screen D1, as shown in Figure 13. The oncoming vehicle width projection icons 79 after being dragged and dropped are shown as symbols 79A and 79B. The user can also freely adjust the size of the oncoming vehicle width projection sign that is actually projected by changing the length of the oncoming vehicle width projection icon 79 by pinching out or pinching in with two fingers.

[0103] The projection instruction recognition unit 23 recognizes, based on the user's operation described above, that the user has selected a vehicle width projection sign for oncoming vehicles as the projection sign, and recognizes the projection position of the vehicle width projection sign for oncoming vehicles relative to the user's own vehicle. It should be noted that projection of the vehicle width projection sign for oncoming vehicles does not necessarily require drag-and-drop or other operations to indicate the projection position; simply touching the vehicle width projection icon for oncoming vehicles to activate it may cause the sign to be projected at a predetermined position in front of the user's own vehicle M. Even in this case, the length of the vehicle width projection sign for oncoming vehicles can be changed by pinching out or similar gestures.

[0104] The projection display unit 24 projects the vehicle width projection signs 89A and 89B for oncoming vehicles, selected by the user, onto the road surface position corresponding to the user's projection position (see Figure 14). The projection display unit 24 projects the vehicle width projection signs 89A and 89B for oncoming vehicles by controlling the lighting control unit 10. This allows drivers of oncoming vehicles passing the vehicle to easily perceive the vehicle's width, thereby reducing the risk of collision when passing.

[0105] The vehicle surrounding environment display system 100 may suggest to the user the projection of vehicle width projection signs 89A and 89B for oncoming vehicles in scenes where there is an oncoming vehicle passing the vehicle.

[0106] Specifically, the situation determination unit 25 may determine whether or not there is an oncoming vehicle passing its own vehicle in a given scene, based on the image captured by the external camera 1 and the detection results of the radar sensor 2. For example, if the situation determination unit 25 is driving on a road without a center line and an oncoming vehicle is detected, it may determine that there is an oncoming vehicle passing its own vehicle in a given scene. Furthermore, the situation determination unit 25 may determine that there is an oncoming vehicle passing its own vehicle in a given scene only when the road width is less than a certain value.

[0107] If the situation determination unit 25 determines that there is an oncoming vehicle passing the vehicle, the projection suggestion unit 26 suggests to the user the projection of oncoming vehicle width projection signs 89A and 89B by displaying an image or outputting sound. For example, the projection suggestion unit 26 displays the oncoming vehicle width projection signs 89A and 89B in front of the vehicle icon Ma on the first screen D1 of the display device 8, and also suggests the projection to the user by voice.

[0108] If the user approves the proposal, the projection display unit 24 uses the lighting control unit 10 to project the vehicle width projection signs 89A and 89B for oncoming vehicles. If the user changes the size of the proposed vehicle width projection signs 89A and 89B for oncoming vehicles, the projection display unit 24 will project them according to the size of the vehicle width projection signs 89A and 89B that the user approved.

[0109] The vehicle surrounding environment display system 100 may also automate the projection of the oncoming vehicle width projection signs 89A and 89B. When the user has turned on automatic projection of the oncoming vehicle width projection signs 89A and 89B, the projection display unit 24 automatically projects the oncoming vehicle width projection signs 89A and 89B when the situation determination unit 25 determines that there is an oncoming vehicle passing the vehicle. After that, the user can adjust the size of the oncoming vehicle width projection signs 89A and 89B by operating the oncoming vehicle width projection icon on the display device 8.

[0110] Next, an example of a scenario in which the vehicle is driving on a narrow road will be described. In a scenario in which the vehicle is driving on a narrow road, the vehicle surrounding environment display system 100 may project a vehicle width projection sign for the vehicle itself. The vehicle width projection sign for the vehicle itself can be two lines corresponding to the width of the vehicle, similar to the vehicle width projection signs 89A and 89B for oncoming vehicles.

[0111] In scenes where the vehicle is driving on a narrow road, the user drags and drops the vehicle width projection icon corresponding to the vehicle width projection sign to the vehicle icon Ma on the first screen D1. Furthermore, the user can freely adjust the size of the projected vehicle width projection sign by pinching out or in with two fingers to change the length of the vehicle width projection icon.

[0112] The projection instruction recognition unit 23 recognizes, based on the user's operation described above, that the user has selected a vehicle width projection sign for their own vehicle as the projection sign, and recognizes the projection position of the vehicle width projection sign for their own vehicle relative to their own vehicle. It should be noted that projection of the vehicle width projection sign for their own vehicle does not necessarily require drag-and-drop or other operations to indicate the projection position; simply touching the vehicle width projection icon to activate it may cause the vehicle width projection sign for their own vehicle to be projected at a predetermined position in front of their own vehicle M. Even in this case, the length of the vehicle width projection sign for their own vehicle can be changed by pinching out or similar gestures.

[0113] The projection display unit 24 projects the vehicle width projection sign selected by the user onto the road surface position corresponding to the user's projection position. This makes it easier for the vehicle driver to understand the relationship between the road width and the vehicle width, thereby reducing the risk of the vehicle deviating from the road on narrow roads. This is also useful when the vehicle is operating autonomously.

[0114] The vehicle surrounding environment display system 100 may suggest projecting a vehicle width projection sign for the vehicle when the vehicle is traveling on a narrow road. Specifically, the situation determination unit 25 may determine whether or not the vehicle is traveling on a narrow road based on the image captured by the external camera 1 and the detection results of the radar sensor 2.

[0115] The situation determination unit 25 determines, for example, that the vehicle is traveling on a narrow road when the road width is less than a certain value. When the road width is limited by a gutter or wall, the narrowest point is used for determination. The road width may also be narrowed by structures placed in the construction zone or parked vehicles. The situation determination unit 25 may also determine whether or not the vehicle is traveling on a narrow road based on the vehicle's position information obtained from the GNSS receiver unit 6 and the road width information included in the map information of the map database 7.

[0116] If the situation determination unit 25 determines that the vehicle is driving on a narrow road, the projection suggestion unit 26 suggests to the user the projection of a vehicle width projection sign by displaying an image or outputting sound. For example, the projection suggestion unit 26 displays the vehicle width projection sign in front of the vehicle icon Ma on the first screen D1 of the display device 8, and also suggests projection to the user by voice.

[0117] If the user approves the proposal, the projection proposal unit 26 uses the lighting control unit 10 to project the vehicle width projection sign for the vehicle. If the user changes the size of the proposed vehicle width projection sign for the vehicle, the projection proposal unit 26 will perform projection according to the size of the vehicle width projection sign approved by the user.

[0118] The vehicle surrounding environment display system 100 may also automate the projection of the vehicle width projection sign for the vehicle itself. When the user has turned on automatic projection of the vehicle width projection sign for the vehicle itself, the projection display unit 24 automatically projects the vehicle width projection sign for the vehicle itself when the situation determination unit 25 determines that the vehicle is driving on a narrow road. After that, the user can adjust the size of the vehicle width projection sign for the vehicle itself by operating the vehicle width projection icon on the display device 8.

[0119] [program] The program causes the ECU 20 to operate (function) as the virtual space generation unit 21, image display unit 22, projection instruction recognition unit 23, projection display unit 24, status determination unit 25, and projection proposal unit 26 described above. The program is provided, for example, by a non-temporary recording medium such as ROM or semiconductor memory. Alternatively, the program may be provided via communication such as a network.

[0120] [Control method for the vehicle surrounding environment display system] Next, the control method of the vehicle surrounding environment display system 100 according to this embodiment will be described with reference to the drawings. Figure 15 is a flowchart showing an example of the projection suggestion process of the vehicle surrounding environment display system 100.

[0121] As shown in Figure 15, first, in step S10, the ECU 20 uses a situation determination unit 25 to determine whether the situation of its own vehicle is a suggested sign projection scene. The situation determination unit 25 makes this determination based, for example, on the image captured by the external camera 1 or the detection result of the radar sensor 2.

[0122] In step S10, if the ECU20 determines that the current situation is a sign projection suggestion scene (S10: YES), the ECU20 proceeds to step S11. On the other hand, in step S10, if the ECU20 determines that the current situation is not a sign projection suggestion scene (S10: NO), the ECU20 terminates the projection suggestion process. In this case, no sign projection suggestion is made to the user.

[0123] In step S11, the ECU 20 proposes a sign projection to the user via the projection proposal unit 26. This proposal may be made visually using the display device 8 or by voice. The projection proposal unit 26 proposes to the user a type of projection sign that is pre-associated with the scene determined by the situation determination unit 25.

[0124] Next, after the sign projection is proposed to the user in step S11, the ECU 20 proceeds to step S12. In step S12, the ECU 20 determines whether the user has performed an operation to approve the proposed sign projection. The user can approve the proposed sign projection, for example, by operating the touch panel display.

[0125] If the ECU20 determines in step S12 that the user has approved the proposed sign projection (S12: YES), it proceeds to step S13. On the other hand, if the ECU20 determines in step S12 that the user has not approved the proposed sign projection (S12: NO), it terminates the projection proposal process. In this case, since the user did not approve, the projected sign is not actually projected.

[0126] In step S13, the ECU 20 actually projects the projection sign approved by the user. Specifically, by controlling the lighting control unit 10, it projects the user-approved projection sign onto the road surface and structures around the vehicle.

[0127] Next, Figure 16(a) is a flowchart showing an example of the projection sign correction process.

[0128] As shown in Figure 16(a), in step S20, the ECU20 determines whether or not the user has made any changes to the projected sign. User changes refer to operations that change the position or size of the projected sign.

[0129] In step S20, if the ECU20 determines that a user change operation has been performed (S20: YES), the ECU20 proceeds to step S21. On the other hand, in step S20, if the ECU20 determines that no user change operation has been performed (S20: NO), the ECU20 terminates the projected sign modification process. In this case, the position and size of the projected sign are not changed.

[0130] In step S21, the ECU20 modifies the projected sign to reflect the user's changes. Specifically, if the user performs operations such as drag and drop or pinch out / pinch in, the position and size of the projected sign are adjusted based on those operations.

[0131] Next, a specific example of the projection proposed scene determination process in step S10 of Figure 15 will be described. Figure 16(b) is a flowchart showing an example of the first proposed scene determination process.

[0132] As shown in Figure 16(b), in step S30, the ECU 20 uses the situation determination unit 25 to determine whether or not the vehicle is in a situation where it should yield to a pedestrian. This determination is made, for example, based on the image captured by the external camera 1 or the detection result of the radar sensor 2.

[0133] If, in step S30, it is determined that the vehicle should yield to a pedestrian (S30: YES), the ECU20 proceeds to step S31. On the other hand, if, in step S30, it is determined that the vehicle should not yield to a pedestrian (S30: NO), the ECU20 terminates the first proposed scene determination process. In this case, no sign projection is proposed.

[0134] In step S31, the ECU20 determines that the current situation is a sign projection suggestion scene. Specifically, it determines that it is a scene in which the projection of the pedestrian priority sign 81 is suggested to the user. After that, the ECU20 terminates the first suggestion scene determination process.

[0135] Figure 17(a) is a flowchart showing an example of the second proposed scene determination process. As shown in Figure 17(a), in step S40, the ECU 20 determines whether a certain amount of time has elapsed since the vehicle was in an emergency stop state using the situation determination unit 25. An emergency stop refers to a state in which the vehicle is stopped due to a malfunction or emergency.

[0136] In step S40, if it is determined that a certain period of time has elapsed while the vehicle is in an emergency stop (S40: YES), the ECU20 proceeds to step S41. On the other hand, if it is determined in step S40 that a certain period of time has not elapsed while the vehicle is in an emergency stop (S40: NO), the ECU20 terminates the second proposed scene determination process. In this case, no sign projection is proposed.

[0137] In step S41, the ECU20 determines that the current situation is a sign projection suggestion scene. Specifically, it determines that the scene is one in which the projection of the emergency stop sign 86 is suggested to the user. After that, the ECU20 terminates the second suggestion scene determination process.

[0138] Figure 17(b) is a flowchart showing an example of the third proposed scene determination process. As shown in Figure 17(b), in step S50, the ECU 20 determines whether or not the road environment has poor visibility using the situation determination unit 25.

[0139] If, in step S50, the ECU20 determines that the road environment has poor visibility (S50: YES), the ECU20 proceeds to step S51. On the other hand, if, in step S50, the ECU20 determines that the road environment does not have poor visibility (S50: NO), the ECU20 terminates the third proposed scene determination process. In this case, no sign projection is proposed.

[0140] In step S51, the ECU20 determines that the current situation is a sign projection suggestion scene. Specifically, it determines that it is a scene in which the projection of the direction of travel sign 93 is suggested to the user. After that, the ECU20 terminates the third suggestion scene determination process.

[0141] Next, Figure 18(a) is a flowchart showing an example of the fourth proposed scene determination process. In step S60, the ECU 20 determines whether or not it has detected an oncoming vehicle that is a vehicle to be passed on a road without a center line, using the situation determination unit 25.

[0142] If the ECU20 determines in step S60 that an oncoming vehicle has been detected (S60: YES), it proceeds to step S61. On the other hand, if the ECU20 determines in step S60 that no oncoming vehicle has been detected (S60: NO), it terminates the fourth proposed scene determination process. In this case, no sign projection is proposed.

[0143] In step S61, the ECU20 determines that the current situation is a sign projection suggestion scene. Specifically, it determines that the projection of the oncoming vehicle width projection signs 89A and 89B is suggested to the user. After that, the ECU20 terminates the fourth suggestion scene determination process.

[0144] Figure 18(b) is a flowchart showing an example of the fifth proposed scene determination process. As shown in Figure 18(b), in step S70, the ECU 20 determines whether or not its vehicle is traveling on a narrow road using the situation determination unit 25.

[0145] If the determination in step S70 is that the vehicle is traveling on a narrow road (S70: YES), the ECU20 proceeds to step S71. On the other hand, if the determination in step S70 is that the vehicle is not traveling on a narrow road (S70: NO), the ECU20 terminates the fifth proposed scene determination process. In this case, no sign projection is proposed.

[0146] In step S71, the ECU20 determines that the current situation is a sign projection suggestion scene. Specifically, it determines that it is a scene in which the projection of a vehicle width projection sign for the vehicle itself is suggested to the user. After that, the ECU20 terminates the fifth suggestion scene determination process.

[0147] According to the vehicle surrounding environment display system 100 and its control method as described above, a projection sign selected by the user can be projected onto a position around the vehicle corresponding to a projection position specified in a virtual space based on the user's operation. This allows the user to intuitively display the necessary sign at any position around the vehicle and appropriately communicate information about the vehicle to surrounding pedestrians and other vehicles, thereby improving user convenience regarding the display of projection signs from the vehicle.

[0148] Furthermore, the projected signs of the vehicle surrounding environment display system 100 include at least one of the following: a pedestrian priority sign to indicate that the vehicle should yield to pedestrians crossing in front of it; an emergency stop sign to notify those around the vehicle of a malfunction; a safety area sign to indicate the safety area around the parked vehicle; a pick-up sign to indicate that the vehicle has come to pick up the vehicle; a parking location notification sign to help users outside the vehicle find the parked vehicle; a direction of travel sign to inform those around the vehicle of its direction of travel in environments with poor visibility; a vehicle width projection sign for oncoming vehicles to indicate the width of the vehicle to oncoming vehicles passing by the vehicle; and a vehicle width projection sign for the vehicle to indicate the width of the vehicle on narrow roads. This allows the user to select the appropriate projected sign according to various situations and appropriately communicate the intentions and status of their vehicle to pedestrians and other vehicles around them.

[0149] Furthermore, according to the vehicle surrounding environment display system 100, if a user wishes to change the position of the projected sign, the system can recognize this operation and change the position of the projected sign projected by the vehicle's lights. This allows users to intuitively adjust the position of the projected sign, enabling flexible responses according to the situation. For example, by displaying the projected sign in an appropriate position in various situations, such as when indicating an intention to yield to pedestrians or when informing others of a vehicle malfunction, communication with surrounding pedestrians and other vehicles can be facilitated.

[0150] Similarly, according to the vehicle surrounding environment display system 100, when a user performs an operation to change the size of the projected sign, the size of the projected sign projected by the vehicle's lights can be appropriately changed in response to that operation. This allows the user to freely adjust the size of the projected sign according to the situation, thereby improving visibility. For example, it is possible to enlarge the projected sign so that it is easy to see from a distance, or to reduce the projected sign when detailed display is needed at close range. This makes it possible to effectively convey information to surrounding pedestrians and other vehicles.

[0151] Furthermore, the vehicle surrounding environment display system 100 can automatically determine whether the vehicle's situation is a sign projection suggestion scene based on the detection results of the vehicle's external and internal sensors. This eliminates the need for the user to select an appropriate projection sign in a given situation. When the situation determination unit determines that it is a sign projection suggestion scene, the projection suggestion unit can display the projection sign on the display device, suggesting the projection sign to the user. This allows the user to quickly and appropriately select a projection sign and clearly communicate the vehicle's intentions and situation to surrounding pedestrians and other vehicles. In addition, the projection position of the projection sign is also automatically suggested, allowing the user to operate it intuitively and enabling effective display of the projection sign.

[0152] Furthermore, according to the vehicle surrounding environment display system 100, users can easily project a parking location notification sign by operating the touch panel display of their mobile device. This makes it possible for users outside the vehicle to quickly and easily find the location of their vehicle in the parking lot.

[0153] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. The present invention can be implemented in various forms, starting with the embodiments described above, by making various changes and improvements based on the knowledge of those skilled in the art.

[0154] The vehicle surrounding environment display system 100 does not necessarily need to propose projections. In this case, the ECU 20 does not need to include the situation determination unit 25 and the projection proposal unit 26. Also, if the system does not determine whether a sign projection proposal scene is to be performed, the vehicle speed sensor 3, driver monitor camera 5, GNSS receiver 6, and map database 7 are not required.

[0155] The display device 8 does not necessarily have to be a touch panel display; it may also be configured to allow icon operation through button operations, lever operations, gesture recognition, or voice recognition.

[0156] The vehicle surrounding environment display system 100 does not necessarily need to be equipped with a communication unit 9 and be able to communicate with the user's portable information terminal 90. The vehicle surrounding environment display system 100 does not necessarily need to use the portable information terminal 90 as the system's display device. The vehicle surrounding environment display system 100 does not necessarily need to be able to adjust the projection position and size of the projected sign after projection. [Explanation of Symbols]

[0157] 1...External camera, 2...Radar sensor, 3...Vehicle speed sensor, 4...User operation reception unit, 5...Driver monitor camera, 6...GNSS receiver unit, 7...Map database, 8...Display device, 9...Communication unit, 10...Lighting control unit, 20...ECU, 21...Virtual space generation unit, 22...Image display unit, 23...Projection instruction recognition unit, 24...Projection display unit, 25...Status determination unit, 26...Projection proposal unit, 100...Vehicle surrounding environment display system.

Claims

1. A vehicle surrounding environment display system that generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from the vehicle's external sensors, and displays an image of the virtual space on a display device from a virtual viewpoint operated by the vehicle's user, A projection display unit that projects a projection sign onto the area around the vehicle using the vehicle's lighting equipment, The system includes a projection instruction recognition unit that recognizes the projection sign selected by the user and the projection position specified by the user on the display device displaying the virtual space, based on the user's operation, The projection display unit projects the user-selected projection sign onto a position around the vehicle corresponding to the projection position on the display device recognized by the projection instruction recognition unit, as part of a vehicle surrounding environment display system.

2. The vehicle surrounding environment display system according to claim 1, wherein the projected signs include at least one of the following: a pedestrian priority sign for the vehicle to yield to pedestrians crossing in front of it; an emergency stop sign for informing those around of a vehicle malfunction; a safety area sign for indicating a safety area around a parked vehicle; a pick-up sign for informing those around that the vehicle has come to pick up a vehicle; a parking location notification sign for the user outside the vehicle to find the parked vehicle; a direction of travel sign for informing those around of the vehicle's direction of travel in an environment with poor visibility; a vehicle width projection sign for oncoming vehicles to show the vehicle's width to oncoming vehicles passing by the vehicle; and a vehicle width projection sign for showing the vehicle's width on a narrow road.

3. The vehicle surrounding environment display system according to claim 1 or 2, wherein the projection display unit changes the position of the projection sign projected by the vehicle's lighting device when it recognizes the user's operation to change the position of the projection sign that is being projected.

4. The vehicle surrounding environment display system according to claim 1 or 2, wherein the projection display unit changes the size of the projection sign projected by the vehicle's lighting device when it recognizes the user's operation to change the size of the projection sign that is being projected.

5. A status determination unit that determines whether the status of the vehicle is a pre-set sign projection suggestion scene based on the detection results of the vehicle's external sensors, The vehicle surrounding environment display system according to claim 2, further comprising: a projection proposal unit that, when the situation determination unit determines that the situation of the vehicle is the sign projection proposal scene, uses the projection sign and projection position pre-associated with the sign projection proposal scene to display the projection sign at the projection position on the display device displaying the virtual space, thereby proposing to the user the projection of the projection sign by the vehicle's lights.

6. The display device is the touch panel display of the user's mobile information terminal. The vehicle surrounding environment display system according to claim 1, wherein the projection display unit projects a parking location notification sign as the projection sign around the vehicle, in response to the user's operation of the mobile information terminal, so that the user outside the vehicle can find the parked vehicle.

7. A control method for a vehicle surrounding environment display system that generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from the vehicle's external sensors, and displays an image of the virtual space on a display device from a virtual viewpoint operated by the vehicle's user, Based on the user's actions, the display device that is displaying the projected sign selected by the user and the virtual space recognizes the projection position specified by the user. A control method for a vehicle surrounding environment display system, which uses the vehicle's lighting equipment to project the user-selected projection sign onto a position around the vehicle corresponding to the projection position on the display device.

8. A program that operates the ECU of a vehicle as a vehicle surrounding environment display system, which generates a virtual space corresponding to the surrounding environment of the vehicle based on detection information from the vehicle's external sensors, and displays an image of the virtual space on a display device from a virtual viewpoint operated by the vehicle's user, The ECU is operated as a projection display unit that projects a projection sign around the vehicle using the vehicle's lighting equipment, and a projection instruction recognition unit that recognizes the projection sign selected by the user and the projection position specified by the user on the display device displaying the virtual space, based on the user's operation. A program that causes the projection display unit to project the user-selected projection sign onto a position around the vehicle corresponding to the projection position on the display device recognized by the projection instruction recognition unit.

Images