Information projection method and information projection device

Abstract

Provided is an information projection method in which a projection device mounted on a vehicle displays information, wherein if it is determined that the vehicle is in an automatic driving state during automatic driving, the project unit displays information related to automatic driving on the basis of a display condition during the automatic driving.

Description

Information projection method and information projection device

[0001] The present invention relates to an information projection method and an information projection device.

[0002] As disclosed in Japanese Patent Application Laid-Open No. 2003-122299 and Japanese Patent Application Laid-Open No. 2003-122299, there are known techniques for projecting information onto a surface.

[0003] Patent Literature 1 discloses a projection device for a vehicle that projects a route guidance image onto a road surface ahead of the vehicle. This projection device for a vehicle includes a route search means that searches for the current position of the vehicle and a route from the current position to a set destination, and a projection means that, when the vehicle approaches a branch point on the route to a visible extent for a passenger, projects a route guidance image onto the road surface ahead of the vehicle based on route information searched by the route search means, guiding the vehicle in the branch direction.

[0004] Patent Literature 2 discloses a technology that can clearly alert moving objects such as pedestrians, and as an example, discloses a road surface projection method. This road surface projection method includes the steps of pre-storing image data of an image to be projected onto the road surface, monitoring the area around the vehicle, identifying the moving object based on the monitoring results, setting a discrimination area in the monitored area, and when the identified moving object is present in the discrimination area, reading out the pre-stored image data and projecting the image onto a road surface position near the moving object.

[0005] JP 2012-247369 A JP 2008-7079 A

[0006] There is a need to provide technology that enables more appropriate display during both manual and automatic driving.

[0007] According to a first aspect of the present invention, there is provided an information projection method as follows. This information projection method is a method in which a projection device mounted on a vehicle displays information. In this method, when it is determined that the vehicle is in an autonomous driving state, the projection unit displays information related to autonomous driving based on display conditions for autonomous driving.

[0008] According to a second aspect of the present invention, there is provided an information projection device as follows. This information projection device is mounted on a vehicle. This information projection device includes a projection unit that displays an image including information. When the control unit determines that the vehicle is in an autonomous driving state, the projection unit displays information related to autonomous driving based on display conditions for autonomous driving.

[0009] According to the present invention, a technology is provided that enables more appropriate display during both manual and automatic driving. Note that problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention.

[0010] 1 is a diagram illustrating an example of an overview of a vehicle equipped with an information projection device. FIG. 1 is a diagram illustrating an example of a configuration provided in a vehicle. FIG. 2 is a diagram illustrating an example of a configuration provided in a vehicle. FIG. 3 is a diagram illustrating an example of an image display when information is projected onto a road surface in front of a vehicle. FIG. 4 is a diagram illustrating an example of an image display when information is projected onto a road surface behind a vehicle. FIG. 5 is a diagram illustrating an example of a configuration of a projection device. FIG. 6 is a diagram illustrating an example of a configuration of a projection device. FIG. 7 is a diagram illustrating an example of a configuration of a projection device. FIG. 8 is a diagram illustrating an example of an arrangement of optical components and the like. FIG. 9 is a diagram illustrating an example of an arrangement of optical components and the like. FIG. 10 is a diagram illustrating an example of a display area of ​​an image viewed from the driver. FIG. 11 is a diagram illustrating an example of a display area onto which image light is projected. FIG. 12 is a diagram illustrating an example of switching between manual driving and autonomous driving. FIG. 13 is a diagram illustrating an example of information display for the driver. FIG. 14 is a diagram illustrating an example of information display for surroundings. FIG. 15 is a flowchart illustrating an example of an image display process. FIG. 16 is a flowchart illustrating an example of an image display process. FIG. 17 is a diagram illustrating an example of information displayed for the driver. FIG. 18 is a diagram illustrating an example of information displayed for surroundings. FIG. 19 is a diagram illustrating an example of information displayed in a special case. 17 is a diagram for explaining an example of information to be displayed in a special case; FIG. 18 is a diagram for explaining an example of information to be displayed in a special case; FIG. 19 is a diagram for explaining an example of information to be displayed in a special case; FIG. 20 is a diagram for explaining a specific example of the start determination (S11) of the projection device; FIG. 21 is a diagram for explaining a specific example of the display start condition determination 1 (S14); FIG. 22 is a diagram for explaining a specific example of the display start condition determination 1 (S14) related to information to be displayed only during autonomous driving; FIG. 23 is a diagram for explaining a specific example of the display start condition determination 1 (S14) related to information to be displayed only during autonomous driving; FIG. 24 is a diagram for explaining a specific example of the display start condition determination 2 (S15); FIG. 25 is a diagram for explaining a specific example of the display start operation (S2); FIG. 26 is a diagram for explaining a specific example of the display end condition determination (S3); FIG. 27 is a diagram for explaining a specific example of the display end operation (S4);1 is a diagram illustrating an example of a display by a projection device. FIG. 1 is a diagram illustrating an example of a display by a projection device. FIG. 2 is a diagram illustrating an example of a display by a projection device. FIG. 3 is a diagram illustrating an example of a display by a projection device. FIG. 4 is a diagram illustrating an example of a process of image display taking into account the direction of the driver. FIG. 5 is a diagram illustrating an example of a process of image display taking into account the direction of the driver. FIG. 6 is a flowchart relating to an example of a process of image display taking into account the direction of the driver. FIG. 7 is a diagram illustrating an example of a process of image display taking into account the direction of people around the vehicle. FIG. 8 is a diagram illustrating an example of a process of image display taking into account the direction of people around the vehicle. FIG. 9 is a diagram illustrating an example of a process of image display taking into account the direction of people around the vehicle. FIG. 10 is a diagram illustrating an example of a process of image display taking into account vehicles around the vehicle.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0013] Although various types of information may be described using expressions such as "table," "list," and "queue" as examples, the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0014] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0015] In some embodiments, processing performed by executing a program will be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0016] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0017] In the embodiment, an example of a technology for displaying information on a road surface using a projection device mounted on a vehicle will be described. Note that in the embodiment, with respect to the vehicle and the driver, the horizontal direction is the left-right direction, the lateral direction of the vehicle, or the width direction of the vehicle, the vertical direction is the up-down direction or longitudinal direction of the vehicle, and the vertical direction perpendicular to the lateral direction of the vehicle is the front-rear direction of the vehicle or the direction of travel of the vehicle.

[0018] FIG. 1 is a diagram illustrating an overview of a vehicle. As shown in FIG. 1, the vehicle 2 includes, for example, a projection device 11 (information projection device), an in-vehicle image display device 12, a car navigation system 150, headlamps 13 (i.e., headlights 13), tail lamps (not shown), turn signals 14, and a controller 100. The vehicle 2 also includes an in-vehicle system 300 including these components. An in-vehicle network is implemented in the in-vehicle system 300, and the controller 100 can transmit and receive data or information to and from these components, as well as components described below. The in-vehicle system 300 includes, for example, a controller area network (CAN), an in-vehicle Ethernet, a local interconnect network (LIN), and the like.

[0019] The in-vehicle system 300 can also communicate with the outside of the vehicle via a communication device. Examples of communication with the outside of the vehicle include a dedicated direct communication system and an indirect communication system. The direct communication system uses the 760 MHz band, 5.9 GHz band, etc., which are internationally used for ITS (Intelligent Transport System) communication, and performs direct communication between vehicles, roads, vehicles, and pedestrians. In contrast, the indirect communication system uses a mobile phone band other than 5.9 GHz to perform indirect communication via a mobile carrier network. The in-vehicle system 300 can transmit and receive data or information to, for example, a server 24 connected to the network 21 via, for example, an access point 22 or a relay station 23 on the network 21. The in-vehicle system 300 may communicate with information terminals carried by other vehicles or pedestrians, or with infrastructure such as terminals installed on the road where the vehicle is traveling. An example of road-to-vehicle communication is receiving traffic congestion information and weather information using radio beacons or optical beacons.

[0020] Vehicle information 4, which is an example of data or information acquired by the in-vehicle system 300, includes, for example, speed information, gear information, steering angle information, lamp illumination information, external light information, distance information, infrared information, engine ON / OFF information, camera image information, acceleration gyro information, GPS information, navigation information, vehicle-to-vehicle communication information, road-to-vehicle communication information, pedestrian-to-vehicle communication information, and in-vehicle sensor information such as Lidar (Light Detection and Ranging). The camera image information includes in-vehicle camera image information and outside-vehicle camera image information. The GPS information includes current time information, latitude, and longitude information. This vehicle information can be acquired, for example, from sensors described below.

[0021] The vehicle information 4 also includes information input by the driver. The driver can input the information using an appropriate device used for inputting information. This device may be a device that is pre-installed in the vehicle. Alternatively, this device may be an external device that can be connected to the in-vehicle system 300 via a wired or wireless connection, such as an input device, a tablet, a smartphone, a wearable device such as AR (Augmented Reality) glasses or an HMD (Head Mounted Display), or a personal computer.

[0022] The in-vehicle system 300 can execute various types of control, such as driving control and display control, using the acquired data or information.

[0023] The projection device 11 projects image light for displaying information. A specific configuration example of the projection device 11 will be described later. The image projected by the projection device 11 can be seen by the driver and people around the vehicle 2, i.e., pedestrians walking near the vehicle, and drivers and passengers of other vehicles traveling near the vehicle.

[0024] The in-vehicle image display device 12 generates image light for displaying information and projects the image light toward a predetermined display area 5 on the windshield 3. As a result, the in-vehicle image display device 12 superimposes a virtual image corresponding to the displayed image on the scenery, for example, so that the driver of the vehicle (from the driver's viewpoint) can view it. Note that in this example, the image light is projected onto the display area 5 on the windshield 3, but the projection unit that projects the image light may be a projection member such as a combiner. The in-vehicle image display device 12 may be, for example, a known HUD (Head-Up Display) that includes a light source, a display panel that forms a display image, a control unit, etc.

[0025] The car navigation system 150 is an electronic device called a car navigation system. The car navigation system 150 is a device that uses, for example, map information and GPS location information to display the current location and directions to the destination. The car navigation system 150 can present an efficient route to the destination using traffic information such as VICS (Vehicle Information and Communication System, registered trademark).

[0026] In this example, a pair of headlights 13 are provided on the left and right at the front of the vehicle. A lamp, which is an illuminant, is built into the headlights 13. The turn indicators 14 are devices that indicate the direction to those around when turning right or left, changing lanes, etc., and are provided in a pair on the left and right at the front of the vehicle, just like the headlights 13.

[0027] The controller 100 is an electronic control unit (ECU) mounted in the vehicle 2, and includes, for example, a processing unit (e.g., a central processing unit), a storage unit, and an input / output unit (I / O unit). The storage unit can be configured using, for example, a main storage unit and an auxiliary storage unit. The main storage unit is a work area for the processing unit, and the processing unit stores data in the main storage unit and executes data processing. The main storage unit is, for example, a random access memory (RAM). The auxiliary storage unit is a non-volatile storage device that stores data in a non-volatile manner. The auxiliary storage unit is, for example, a read-only memory (ROM).

[0028] Data or information is input to the controller 100 via an input / output device and an in-vehicle network 300. The controller 100 can also control various devices connected to the in-vehicle network via the input / output device and the in-vehicle network.

[0029] The controller 100 receives, for example, vehicle information 4 and information acquired from the server 24 via an input / output device. Based on the acquired information, the controller 100 may control the driving of the headlights 13, the driving of the turn signals 14, the operation of the projection device 11, the operation of the in-vehicle image display device 12, the operation of the car navigation system 150, and the like.

[0030] The projection device 11 may be connected to various sensors mounted on the vehicle 2, devices mounted on the vehicle 2 (e.g., the car navigation system 150), communication devices used for communication with the outside of the vehicle, and the like, to acquire data or information. Then, the projection device 11 may use the acquired data or information to generate image light for displaying information and project the image light.

[0031] The projection device 11 can omit communication with the controller 100. Here, the projection device 11 can acquire data or information from a configuration different from that of the controller 100 through communication based on, for example, a CAN or an in-vehicle Ethernet. The same can be said for the in-vehicle image display device 12.

[0032] Alternatively, the controller 100 may generate video data using the acquired data or information and transmit the generated video data to the projection device 11, and the projection device 11 may generate video light for displaying information based on the video data generated by the controller 100 and project the video light. Alternatively, the controller 100 may not generate video data, but rather a video generation unit may generate the video data and transmit the generated video data to the projection device 11. Here, the controller 100 and the projection device 11 may communicate based on, for example, FPD-Link III or GMSL (Gigabit Multimedia Serial Link), and the projection device 11 may acquire the video data from the controller 100. The projection device 11 may perform any necessary video processing on the acquired video data, and the video processing may include processing related to image distortion correction, color correction, brightness correction, contrast correction, conversion (e.g., decoding), and the like. The video data may be stored in advance in a storage device of the controller or the projection device, or may be processed in real time without being stored in a storage device. When the video data is stored in advance in a storage device of the controller or the projection device, the stored video data may be updated successively by system updates or user operation, via wired or wireless methods. As described above, the projection device 11 may be connected to various sensors mounted on the vehicle, devices mounted on the vehicle 2 (e.g., the car navigation system 150), and communication devices used for communication with the outside of the vehicle, and may acquire data or information via communication based on a CAN, in-vehicle Ethernet, or the like. The projection device 11 may then generate video light for displaying information based on the acquired information and project the video light. The same can be applied to the in-vehicle video display device 12.

[0033] Furthermore, the controller 100 may control the headlights 13 and the projection device 11 in a coordinated manner. For example, the controller 100 may turn on the headlights 13 and cause the projection device 11 to project image light ahead of the vehicle. For example, the controller 100 may turn off the headlights 13 and cause the projection device 11 to generate an image of information to be displayed and project the image light ahead of the vehicle. In other words, the headlights 13 and the projection device 11 may operate in a coordinated manner via the controller 100.

[0034] On the other hand, the headlights 13 and the projection device 11 may operate without the intervention of the controller 100. For example, the headlights 13 and the projection device 11 may be connected, and the operation of the headlights 13 may be controlled by the projection device 11 (more specifically, a control unit of the projection device 11). The projection device 11 may, for example, turn on the headlights 13, generate an image of the information to be displayed, and project the image light in front of the vehicle. Alternatively, the projection device 11 may, for example, turn off the headlights 13, generate an image of the information to be displayed, and project the image light in front of the vehicle.

[0035] For example, the projection device 11 may be mounted on the front of the vehicle body, and the image light from the projection device 11 may be projected onto the road surface in front of the vehicle. The projection device 11 may be mounted, for example, near the roof of the vehicle body 2. The projection device 11 may also be mounted, for example, in the side mirror area, on the roof, or on the side or bottom of the vehicle body 2. However, the present invention is not limited to these.

[0036] One or more projection devices 11 may be mounted. For example, a pair of projection devices 11 may be mounted at the front end of the vehicle 2. Furthermore, the projection device 11 may be integrated into the headlight 13, for example. When the projection device 11 is integrated into the headlight 13, the light source of the headlight 13 can also be used as a light source for projection.

[0037] For example, the projection device 11 may be mounted on the rear of the vehicle body, and the image light from the projection device 11 may be projected onto the road surface behind the vehicle. Alternatively, a pair of projection devices 11 may be mounted, for example, at the rear end of the vehicle 2. Alternatively, the projection device 11 may be integrated into, for example, a tail lamp. When the projection device 11 is integrated into, the tail lamp's light source can also be used as a light source for projection. The above-described tail lamp may also be a stop lamp or a backup lamp, and hereinafter, the term "tail lamp" may be replaced with "stop lamp" or "back lamp."

[0038] Furthermore, the controller 100 may control the tail lamps and the projection device 11 in a coordinated manner. For example, the controller 100 may turn on the tail lamps and cause the projection device 11 to generate an image of the information to be displayed and project the image light toward the rear of the vehicle. For example, the controller 100 may turn off the tail lamps and cause the projection device 11 to generate an image of the information to be displayed and project the image light toward the rear of the vehicle. In other words, the tail lamps and the projection device 11 may operate in a coordinated manner via the controller 100.

[0039] On the other hand, the tail lamps and the projection device 11 may operate without the intervention of the controller 100. For example, the tail lamps and the projection device 11 may be connected, and the operation of the tail lamps may be controlled by the projection device 11 (more specifically, a control unit of the projection device 11). The projection device 11 may, for example, turn on the tail lamps, generate an image of the information to be displayed, and project the image light toward the rear of the vehicle. Alternatively, the projection device 11 may, for example, turn off the tail lamps, generate an image of the information to be displayed, and project the image light toward the rear of the vehicle.

[0040] The controller 100 may also control the turn indicators and the projection device 11 in a coordinated manner. For example, the controller 100 may turn on the turn indicators and cause the projection device 11 to generate an image of the information to be displayed and project the image light forward or backward on the side of the vehicle. For example, the controller 100 may turn off the turn indicators and cause the projection device 11 to generate an image of the information to be displayed and project the image light forward or backward on the side of the vehicle. That is, the turn indicators and the projection device 11 may operate in a coordinated manner via the controller 100. The turning on and off of the turn indicators may also be controlled by the driver's operation. For example, the projection device may be configured to project image light toward the road surface on the side of the vehicle 2.

[0041] Information such as the vehicle information 4 is acquired using devices such as cameras and various sensors. FIG. 2A shows an example in which the controller 100 is connected to various devices. The controller 100 is configured to generate images. FIG. 2B shows an example in which the controller 100 is connected to various devices. FIG. 2B includes an image generation unit that generates images. FIG. 2C shows an example in which various devices are connected to the controller 100. The controller 100 acquires signals from the various devices and transmits them to the projection device 11. However, control units such as the projection device 11, the in-vehicle image display device 12, and the car navigation system 150 can be connected to the devices shown in FIGS. 2A and 2B and acquire information directly without going through the controller 100. The various devices shown in FIGS. 2A, 2B, and 2C can be deleted, or other types of devices can be added or replaced with other types of devices as appropriate.

[0042] The vehicle speed sensor 501 detects the speed of the vehicle 2 and is used to generate speed information as the detection result. The shift position sensor 502 detects the current gear and is used to generate gear information as the detection result. The steering wheel angle sensor 503 detects the current steering wheel angle and is used to generate steering wheel angle information as the detection result.

[0043] The headlight sensor 504 detects, for example, the ON / OFF state of the headlights 13. The headlight sensor 504 may also detect the luminance of the headlights 13 when they are turned on. The headlight sensor 504 is used to generate lamp lighting information, which is the detection result. The vehicle 2 may also be provided with a high / low sensor that detects the high beam or low beam state of the headlights 13, and the high / low sensor is used to generate information indicating the high beam or low beam. The vehicle 2 may also be provided with a tail lamp sensor (not shown). The tail lamp sensor detects, for example, the ON / OFF state of the tail lamps. The tail lamp sensor may also detect the luminance of the tail lamps when they are turned on. The tail lamp sensor is used to generate lamp lighting information, which is the detection result. Similarly, although not shown, sensors that detect the ON / OFF state and lighting luminance of stop lamps, backup lamps, hazard lamps, and turn signals may also be provided.

[0044] The illuminance sensor 505 and the chromaticity sensor 506 detect external light from the vehicle 2 and are used to generate external light information as the detection result. The chromaticity sensor 506 may also be used to detect the color of the road surface around the vehicle 2 and generate projection surface color information as the detection result. The distance measurement sensor 507 detects the distance between the vehicle 2 and an external object or the distance between two external objects and is used to generate distance information as the detection result. The infrared sensor 508 detects the presence or absence of an object in the vicinity of the vehicle and the distance thereto and is used to generate infrared information as the detection result. The engine start sensor 509 detects whether the engine is on or off and is used to generate on / off information as the detection result.

[0045] The vehicle operation switches 510 are various switches operated by the driver or the like, and are used to generate operation information such as ON / OFF of these switches. The vehicle operation switches 510 include steering switches, switches on the dashboard, vehicle door switches, switches on the armrest, switches on the center console, etc.

[0046] The communication unit 511 is a component used for communication, and includes, for example, a first communication unit 5111 , a second communication unit 5112 , a third communication unit 5113 , an in-vehicle wireless communication unit 5114 , and an exterior-vehicle wireless communication unit 5115 .

[0047] The first communication unit 5111 is configured to perform communication by, for example, FPD-Link III, and includes a communication line and a communication device used for communication by FPD-Link III.

[0048] The second communication unit 5112 implements a communication protocol (CAN communication protocol) and includes a communication line and a communication device used for communication via CAN.

[0049] The third communication unit 5113 is configured to perform communication via in-vehicle Ethernet, has an in-vehicle Ethernet communication protocol implemented, and includes a communication line and a communication device used for communication via in-vehicle Ethernet. The third communication unit 5113 may include a USB (Universal Serial Bus) port, and the driver or the like may connect a device (e.g., a USB memory or a device having computer functionality) to the port as appropriate. The third communication unit 5113 may then perform communication via in-vehicle Ethernet between the device connected to the port and the connected configuration.

[0050] The in-vehicle wireless communication unit 5114 is configured to communicate with information devices in the vehicle, has a communication protocol implemented, and includes a wireless device. The in-vehicle wireless communication unit 5114 performs wireless communication using, for example, Wi-Fi (registered trademark) or Bluetooth (registered trademark). Note that the in-vehicle wireless communication unit 5114 may also perform short-range wireless communication such as NFC (Near Field Communication).

[0051] The exterior wireless communication unit 5115 is configured to communicate with the outside of the vehicle 2, has a communication protocol implemented therein, and includes a wireless device. The exterior wireless communication unit 5115 performs wireless communication using, for example, Long Term Evolution (LTE), 5G, or Wi-Fi.

[0052] The configuration of the communication unit 511 may be changed as appropriate. For example, the communication unit 511 may be configured to be able to communicate using LIN. Furthermore, the communication unit 511 may be configured to be able to communicate using GMSL.

[0053] The acceleration sensor 512 and the gyro sensor 513 detect the acceleration and angular velocity of the vehicle 2 and are used to generate acceleration gyro information that indicates the attitude and behavior of the vehicle 2. The temperature sensor 514 detects the temperatures inside and outside the vehicle and on the road surface, and is used to generate temperature information that is the detection result.

[0054] The road-to-vehicle communication wireless transceiver 515 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless transceiver 516 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other surrounding vehicles. The terminal-to-vehicle communication wired and wireless communication unit 517 is a device that acquires information from a device connected to the LTE network (e.g., a Wi-Fi device) through wired or wireless communication. The controller 100 or the control unit can acquire information transmitted and received over the LTE network via the terminal-to-vehicle communication wired and wireless communication unit 517.

[0055] The GPS receiver 518 generates GPS information by receiving GPS signals from GPS satellites. For example, the GPS receiver 518 can acquire the current time, latitude, and longitude. The VICS receiver 519 generates VICS information by receiving VICS signals. The VICS signals include traffic congestion information and weather information provided by radio beacons and optical beacons. The GPS receiver 518 and the VICS receiver 519 may be provided as part of a navigation system.

[0056] In-vehicle camera 520 and exterior camera 521 capture images of the interior and exterior of the vehicle and are used to generate in-vehicle camera image information and exterior camera image information. Specifically, in-vehicle camera 520 is, for example, a camera for a DMS (Driver Monitoring System) that captures the driver's posture, eye position, movement, etc. In this case, the driver's fatigue level, line of sight position, etc. can be determined by analyzing the captured images.

[0057] The voice of the driver is input to the voice input device 522, which is used to generate voice information. By emitting voice, the driver can input operation details via the voice input device 522. The voice output device 523 is, for example, a device that outputs voice processed by the controller 100 or the control unit.

[0058] The humidity sensor 524 detects humidity and is used to generate humidity information. The humidity sensor 524 may detect humidity outside the vehicle 2 or on the road surface, and humidity information about the outside of the vehicle 2 may be generated. The humidity sensor 524 may also detect humidity inside the vehicle 2, and humidity information about the interior of the vehicle 2 may be generated.

[0059] A rain sensor 525 detects raindrops and is used to generate rainfall information. A wiper switch 526 detects whether the wipers are on or off and is used to generate wiper on / off information. A turn signal switch (not shown) detects whether the driver operates a turn signal and is used to generate turn signal information.

[0060] The image generating unit 527 may generate image information based on information acquired from each sensor, information from a mobile terminal, information from the Internet, or the like, or may generate image information based on information acquired by the controller 100. Information other than that related to the vehicle includes information from a mobile terminal, information from the Internet, and the like.

[0061] The controller 100 or the processor may, for example, transmit and receive data or information via wireless communication and acquire information necessary for driving. The controller 100 or the processor may also acquire information necessary for autonomous driving. The controller 100 or the processor may also, for example, transmit and receive data or information via wireless communication and perform update processing of the data or information. The controller 100 or the processor may, for example, update various data or information (map data, data used for video processing, software, etc.) as update processing. Such technology is sometimes referred to as OTA (Over the Air) technology.

[0062] Next, an example of information projection by a projection device will be described with reference to FIG. 3 . As shown in FIG. 3A , image light is projected from the projection device onto the road surface ahead. The figure shows a projection area 14a projected from the projection device 11 through a window 13a on the front right side of the vehicle 2, and a projection area 14b projected from the projection device 11 through a window 13b on the front left side of the vehicle. The projection images of the respective projection areas (14a, 14b) are combined to project an image (in this example, an arrow 15 indicating that the vehicle 2 is moving straight ahead) onto the road surface ahead of the vehicle 2. Note that, although the projection areas (14a, 14b) are divided into left and right areas in this example, the projection areas may be divided into areas near and far from the vehicle. Furthermore, only one of the projection devices 11 installed on the left and right sides of the vehicle may be used to project image light. In this example, the projection device 11 is incorporated inside the headlight 13, and the light source of the headlight 13 is used as the light source for projection. However, as described above, the location of the projection device 11 can be changed as appropriate. The projection device 11 may have a light-emitting structure different from that of the headlight 13, and may project image light by utilizing the light emitted by this structure.

[0063] As shown in FIG. 3B , image light is projected from the projection devices 11 onto the road surface behind the vehicle. The figure shows a projection area 17a projected from the projection device 11 on the rear right side of the vehicle and a projection area 17b projected from the projection device 11 on the rear left side of the vehicle. The projection images of the respective projection areas (17a, 17b) are combined to project an image (in this example, an arrow 18 indicating that the vehicle is moving straight backward) onto the road surface behind the vehicle. While the projection areas (17a, 17b) are divided into left and right areas in this example, the projection areas may be divided into areas near and far from the vehicle. Furthermore, only one of the projection devices 11 installed on the left and right sides of the vehicle may be used to project image light. In this example, the projection devices 11 are installed so that the light source of the tail lamp can be used as the light source for projection. However, as described above, the location of the projection devices 11 can be changed as appropriate. The projection device 11 may have a light-emitting structure different from a tail lamp, and may project image light by utilizing the light emitted from this structure.

[0064] An example configuration of the projection device 11 will be described with reference to FIG. 4 . The projection device 11 generates an image of information to be displayed using, for example, data or information acquired via various sensors and a communication unit, and projects the image light. For example, data or information may be input to the projection device 11 from the controller 100. For example, signal data, video data, etc. may be input to the projection device 11 from the controller 100. The projection device 11 may then perform processing using the input information. Alternatively, for example, the controller 100 may control the projection device 11 instead of the control unit of the projection device 11. Alternatively, the control unit of the projection device 11 and the controller 100 may share processing responsibilities. For example, the controller 100 may control adjustment of the brightness of the headlights 13 and the taillights, and the control unit of the projection device 11 may control adjustment of the brightness of the projected image.

[0065] As shown in FIG. 4A , the projection device 11 includes a projection optical system 701 and a light source device 702. The projection device 11 also includes a power supply 703, a cooling unit 704, an operation input unit 705, a video signal input unit 706, an audio signal input unit 707, an audio output unit 708, a communication unit 709, a non-volatile memory 710, a memory 711, a storage unit 712, an adjustment unit 713, and a control unit 714. The control unit 714 may be configured as a processing device. The control unit 714 can control the operation of the projection device 11. On the other hand, if the projection device 11 does not include a control unit, the controller 100 can also control the projection device 11.

[0066] The projection optical system 701 is a component used for projecting light, and includes optical components such as lenses and / or mirrors.

[0067] The light source device 702 is a device capable of generating image light. The light source device may use, for example, a high-pressure mercury lamp, a xenon lamp, an LED light source, a laser light source, or the like. The light source device may also include an optical element used for condensing and homogenizing light.

[0068] The power supply 703 supplies power to, for example, the light source, and also supplies the necessary power to each of the other components.

[0069] The cooling unit 704 cools down the various components that reach high temperatures, such as the light source, the power supply 703, or the light source device 702, by air cooling or liquid cooling as needed.

[0070] The operation input unit 705 is, for example, an operation button or a light receiving unit of a remote control, and receives an operation signal from a user. The input of the operation signal from the user switches on / off a switch (operation switch) that activates the projection device 11.

[0071] The video signal input unit 706 is an interface device that acquires video data from an external device. The audio signal input unit 707 is an interface device that acquires audio data from an external device. The audio output unit 708 can output audio based on the audio data input to the audio signal input unit 707. The audio output unit 708 may output, for example, an operation sound or an error warning sound.

[0072] The communication unit 709 is an interface device used for communication with the outside. The communication unit 709 is connected to, for example, an external information processing device (e.g., the controller 100) and inputs and outputs various control signals. The communication unit 709 may be connected to various sensors, communication devices provided in the vehicle 2, and the like and inputs and outputs various data or information.

[0073] The nonvolatile memory 710 stores various data used in the projector function, for example. The data stored in the nonvolatile memory 710 includes image data and video data prepared in advance for projecting an image.

[0074] The memory 711 stores image data to be projected, control parameters for each part of the device, and the like.

[0075] The storage unit 712 is a device that records video, images, audio, various data, etc. For example, video, images, audio, various data, etc. may be recorded in advance at the time of product shipment, or video, images, audio, various data, etc. obtained from an external device or external server via the communication unit 709 may be recorded. The control unit 714 may also obtain updated data or information from the outside via the communication unit 709 and update the recorded data or information with new data or information. Furthermore, a portion or all of the recorded data or information may be updated with new data or information at the user's option. The video, images, various data, etc. recorded in the storage unit 712 may be output as projected video. The audio recorded in the storage unit 712 may be output as audio from the audio output unit 708.

[0076] The adjustment unit 713 is capable of adjusting the image light and includes, for example, an image adjustment unit and a polarization adjustment unit. The image adjustment unit performs image processing on the image data input via the image signal input unit, the image data stored in the non-volatile memory 710, and the image data. Examples of the image processing include image distortion correction, scaling processing for enlarging, reducing, or transforming the image, brightness adjustment processing for changing the image brightness, contrast adjustment processing for changing the image contrast curve (including adjustment of the brightness gradation linearity characteristics), color correction processing for changing the image chromaticity, and Retinex processing for decomposing the image into light components (illumination light component, reflected light component, and ambient light component) and changing the weighting of each component. The image adjustment unit is realized by the control unit 714 storing data in the memory 711 and executing image processing.

[0077] The polarization adjustment unit adjusts the degree of polarization of the image light to be projected. Here, the degree of polarization refers to the ratio of P-polarized light components to S-polarized light components contained in the light (P-polarized light components and S-polarized light components are defined relative to the projection surface). For example, the projection device 11 is provided with a configuration capable of adjusting the degree of polarization of the emitted light, such as a polarization separation element and a polarization conversion element described below, and the control unit 714 adjusts the degree of polarization of the image light by controlling this configuration. Note that the polarization control unit is realized by the control unit 714 storing data (parameters used for control, etc.) in the memory 711 and controlling this configuration. Alternatively, multiple projection devices 11 with different degrees of polarization may be provided and switched between.

[0078] In the example of FIG. 4B , the light source device 702a includes a light source 7021, a display element 7022, an optical element 7023, a polarization separation element 7024, and a polarization conversion element 7025. The light source 7021 generates light for image projection. The optical element 7023 is used for condensing and homogenizing light. Note that descriptions similar to those above may be omitted. The polarization separation element 7024 is an element that separates incident light into S-polarized light and P-polarized light. For example, the polarization separation element 7024 may be a polarizing beam splitter, which separates light by reflecting or transmitting light of a specific polarization state. The polarization conversion element 7025 is an element that converts the degree of polarization. For example, the polarization conversion element 7025 may be a waveplate, which delays the phase of light to change the polarization state. Here, for example, the polarization conversion element 7025 may be connected to an actuator (e.g., a motor) used to control the incident angle of transmission, and the adjustment unit 713 or the polarization adjustment unit 722 may adjust the polarization degree of the incident light according to the incident angle.

[0079] The projection device 11 also includes a display element, a display element driver, and the like. The display element is an element that generates an image by modulating transmitted or reflected light, and may be, for example, a transmissive LCD (Liquid Crystal Display) panel, a reflective LCD panel, or a DMD (Digital Micromirror Device) (registered trademark) panel. The display element driver sends a drive signal to the display element, causing the display element to generate an image. Display elements include those that can display multiple images using control signals, such as DMDs and LCDs, as well as those that can only display a fixed image, such as mask types.

[0080] 4C, the light source device 702b does not have a light source that serves as a backlight for the display element. In this example, the display element itself emits light. Note that the same description as above may be omitted.

[0081] The light source device 702b includes a display light-emitting element 7026, an optical element 7023, a polarization separation element 7024, and a polarization conversion element 7025. The display light-emitting element 7026 is configured as a self-luminous display such as an LED array or an OLED (organic light-emitting diode) display. The optical element 7023 is an optical system used for collecting and homogenizing light. The optical element 7023, the polarization separation element 7024, and the polarization conversion element 7025 are the same as those described in FIG. 4B .

[0082] Next, an example of the arrangement of optical components and the like will be described with reference to FIG. 5 . FIG. 5A shows an example using a reflective display element, in which an image is projected using the DMD system. The first optical element 901 is an optical element that condenses light generated by a light source 900, such as a collimator. The second optical element 902 is a mirror that projects the condensed light onto a display element 907. In this example, the display element 907 is configured as a DMD panel, and the light projection is adjusted or controlled for each display pixel. The third optical element 903 is configured to project image light and is configured using optical components such as lenses and mirrors. A reflective LCD panel may also be used as the reflective display element.

[0083] 5B shows an example in which the display light-emitting element 908 itself emits light and projects an image. The display light-emitting element 908 is configured as an LED array panel or an LED matrix panel, and an image is generated by combining the lighting positions and lighting colors of the LEDs on the panel. The fourth optical element 904 is configured to project image light, and is configured using optical components such as lenses and mirrors. In addition to the above panels, an OLED display may also be used as the display light-emitting element.

[0084] FIG. 5C shows an example in which light is transmitted through a display element 909 to display an image. The fifth optical element 905 is similar to the first optical element 901 described above. The display element 909 is, for example, a transmissive LCD panel, and an image is generated on the display element 909. The sixth optical element 906 is configured to project image light and is configured using optical components such as lenses and mirrors. The display element 909 may be configured such that a mask having a predetermined pattern is disposed on the light source side. In this case, a fixed image based on the pattern formed on the mask can be displayed by light that is not blocked by the mask serving as the display element and that passes through the mask. The display element 909 may also be configured such that a lens having a predetermined pattern is disposed on the light source side. In this case, the display element 909 can display a fixed image based on the pattern formed on the lens. The predetermined pattern formed on the lens is formed on the lens surface or inside the lens, and the formed pattern partially blocks or partially transmits light. The lens used in the display element 909 may be a microlens array. In this case, the pattern formed on each lens is different.

[0085] 5A and 5C can be configured by the light source of the projection device 11, but may also be the light source of the headlight 13 or a tail lamp. In this case, the light source of the projection device 11 may be omitted. On the other hand, the light source shown in FIGS. 5A and 5C may be configured by combining the light source of the projection device 11 and the light source of the headlight 13 or a tail lamp. Furthermore, the projection device 11 may be configured with one light source or multiple light sources. Similarly, the display element corresponding to the light source may be one or multiple.

[0086] Next, an example of the display area of ​​an image viewed by the driver will be described with reference to FIG. 6 . FIG. 6 shows the high beam illumination area A1 of the headlight 13, the low beam illumination area A2 of the headlight 13, and the image area A3 for the road surface image projected by the projection device 11. The high beam is illuminated at a high position in the upper illumination area of ​​the high beam. That is, this illumination area can be illuminated with the high beam, which brightly illuminates a distant area. The low beam is illuminated at a low position in the lower illumination area of ​​the high beam. Since this overlaps with the low beam projection area, the low beam may be formed by blocking a portion of the illumination area of ​​the headlight 13 that can project the high beam. Furthermore, the image area A3 projected by the projection device 11 is included in the illumination area below the low beam illumination area A2 and the high beam illumination area A1. Therefore, the projection device 11 can also project an image using the low beam and high beam of the headlight 13, or the projection device 11 may be provided separately from the headlight 13.

[0087] An example of a projection area around a vehicle will be described with reference to FIG. 7 . FIG. 7 shows a road surface display area A11 in front of the vehicle, a road surface display area A12 behind the vehicle, and a road surface display area A13 on the left and right sides of the vehicle. The road surface display area A11 in front of the vehicle is a display area formed on the road surface in front of the vehicle based on the position of the vehicle. For example, the projection device 11 may be provided to project image light onto this area to display information. The road surface display area A12 behind the vehicle is a display area formed on the road surface behind the vehicle based on the position of the vehicle. For example, the projection device 11 may be provided to project image light onto this area to display information. The road surface display area A13 on the left and / or right sides of the vehicle is a display area formed on the road surface on the left and / or right sides of the vehicle based on the position of the vehicle. For example, the projection device 11 may be provided to project image light onto this area to display information. This area A13 can be an area where, for example, vehicles and people (pedestrians, etc.) around the vehicle can see the displayed information.

[0088] Next, an example of switching between manual driving and automatic driving will be described with reference to FIG. 8 . Switching between manual driving and automatic driving is performed appropriately based on a user's operation. For example, a vehicle may be provided with a button used to switch between manual driving and automatic driving. As shown in the figure, an automatic driving switch button 1000, a manual driving switch button 1001, etc., may be provided, for example, on the steering wheel. When the ignition is turned on, the vehicle starts operating in manual driving or automatic driving. For example, starting operation in manual driving may be set as the initial setting, and automatic driving may be started according to individual user settings. However, starting operation in automatic driving is only performed when automatic driving is possible. Furthermore, if the previous driving was completed in manual driving, the vehicle may start operating in manual driving, or if automatic driving was completed, the vehicle may start operating in automatic driving. Here, if the vehicle starts operating in manual driving, the user can operate the manual driving switch button 1001 to, for example, start manual driving or switch from automatic driving to manual driving. Note that the user can perform this operation before the vehicle starts, while the vehicle is stopped, or while the vehicle is traveling.

[0089] In this example, an automatic driving switch button 1000 and a manual driving switch button 1001 are provided, but a configuration in which only one of the buttons is provided may also be used. For example, in a configuration in which only the automatic driving switch button 1000 is provided, the vehicle generally performs driving control by manual driving, and the user can switch from manual driving to automatic driving by operating the automatic driving switch button 1000. Then, the user can return from automatic driving to manual driving by operating the automatic driving switch button 1000 again. For example, in a configuration in which only the manual driving switch button 1001 is provided, the vehicle generally performs driving control by automatic driving, and the user can switch from automatic driving to manual driving by operating the manual driving switch button 1001. Then, the user can return from manual driving to automatic driving by operating the manual driving switch button 1001 again.

[0090] Switching between manual driving and autonomous driving may also be performed by the user's voice. Here, a microphone that picks up the user's voice may be provided inside the vehicle, and manual driving or switching to manual driving may be started by the user vocalizing a command to start manual driving or switch to manual driving. Also, automatic driving or switching to automatic driving may be started by the user vocalizing a command to start automatic driving or switch to automatic driving.

[0091] The vehicle controller 100 can control switching between manual driving and automatic driving based on information regarding the switching (for example, the operation contents of the automatic driving switching button 1000, manual driving switching button 1001, etc., as described above, the user's voice operation contents, etc.).

[0092] The autonomous driving switching button 1000 may be configured to be able to set the autonomous driving level. When switching from manual driving to autonomous driving, the vehicle controller 100 may start autonomous driving based on the autonomous driving level information and taking into account the autonomous driving level information. The vehicle controller 100 may also start autonomous driving based on the user's voice instructions regarding the autonomous driving level and taking into account the autonomous driving level information. The vehicle controller 100 stores the current driving mode setting (manual driving or autonomous driving) and the current autonomous driving level in memory 711 or the like, and stores the driving mode setting history and the autonomous driving level setting history in non-volatile memory 710 or the like.

[0093] The following describes information display for the driver and information display for surroundings according to differences in driving conditions and driving roads. Here, the display of the timing for switching between manual driving and autonomous driving, or the display of the timing for changing the autonomous driving level, may be performed after a predetermined transition period or a predetermined time has elapsed. For example, when switching from manual driving to autonomous driving, the display of autonomous driving may be performed after a predetermined time (a pre-set transition period) or a predetermined transition period, such as one minute, has elapsed. In this embodiment, information for manual driving is displayed during manual driving (before switching), and information for autonomous driving is displayed during autonomous driving (after switching). The transition period indicates the timing for switching from manual driving to autonomous driving, or the timing for switching from autonomous driving to manual driving. During the transition period, information for the transition period may be displayed, or information for the transition period may not be displayed. For example, before switching, information related to manual driving is displayed for the driver or surroundings, and no information is displayed during or at the time of switching, and after switching, information related to autonomous driving is displayed for the driver or surroundings. Here, the predetermined time or predetermined transition period when changing the autonomous driving level may be different depending on whether the level is increased (reducing the driver's driving workload) or decreased (increasing the driver's driving workload), and may not be the same for each level. As an example, when the autonomous driving level is decreased, the predetermined time or predetermined transition period is made longer than when the autonomous driving level is increased. This allows the driver to have time to respond to the increase in driving workload.

[0094] Next, an example of information display will be described with reference to Figures 9 and 10. Figure 9 shows an example of information display for the driver, where (a) shows the manual driving state, (b) shows the state of automated driving at a predetermined level of automated driving, for example, level 3, and (c) shows the state of automated driving at a predetermined level of automated driving, for example, level 4. As shown in Figure 9, the display content changes as the driving state changes. Note that information for the driver may be referred to as "Category 1." Also, the projection distance, projection position, and projection range onto the road surface shown in the figure are examples, and the projection distance, projection position, and projection range may differ from those shown in the figure.

[0095] In detail, (a) shows a state in which the vehicle is driving on a road (driving lane) that is not reserved for autonomous vehicles and is being driven manually. In this state, the projection device 11 displays information for the driver in a combination of manual driving and driving on a road other than a road reserved for autonomous vehicles. (b) shows a state in which the vehicle is driving on a road that is not reserved for autonomous vehicles and is being driven autonomously at autonomous driving level 3. In this state, the projection device 11 displays information for the driver in a combination of autonomous driving level 3 and driving on a road other than a road reserved for autonomous vehicles. (c) shows a state in which the vehicle is driving on a road reserved for autonomous vehicles and is being driven autonomously at autonomous driving level 4. In this state, the projection device 11 displays information for the driver in a combination of autonomous driving level 4 and driving on a road reserved for autonomous vehicles. If the driving conditions are different for the same driving road, the display for the driver may be different. If the driving conditions are the same for different driving roads, the display for the driver may be different. That is, the display to the driver can change depending on the driving conditions and the road.

[0096] FIG. 10 shows an example of information display for surroundings, where (a) shows a manual driving state, (b) shows an automatic driving state, and (c) shows an automatic driving state. The display content changes as the driving state changes. Note that information for surroundings may be referred to as "Category 2." Also, the projection distance and range onto the road surface shown in the figure are examples, and the projection distance, projection position, and projection range may differ from those shown in the figure.

[0097] In detail, (a) shows a state in which the vehicle is driving on a road (driving lane) that is not reserved for autonomous vehicles and is being driven manually. In this state, the projection device 11 displays information to those around it in a combined condition of manual driving and driving on a road other than a road reserved for autonomous vehicles. (b) shows a state in which the vehicle is driving on a road that is not reserved for autonomous vehicles and is being driven automatically. In this state, the projection device 11 displays information to those around it in a combined condition of autonomous driving and driving on a road other than a road reserved for autonomous vehicles. (c) shows a state in which the vehicle is driving on a road reserved for autonomous vehicles and is being driven automatically. In this state, the projection device 11 displays information to those around it in a combined condition of autonomous driving and driving on a road reserved for autonomous vehicles.

[0098] In addition, it may be determined which of the multiple projection devices 11 installed in the vehicle to use to display information to the surroundings, taking into consideration the target object to be displayed and the information projection position of the projection device 11. In the situations (a) and (b) of the same figure, if a pedestrian is on the front left side of the vehicle, display may be performed using the projection device 11 on the front left side of the vehicle. Also, in the situation (c) of the same figure, if there is another vehicle on the front right side of the vehicle, display may be performed using the projection device 11 on the front right side of the vehicle. If the driving conditions are different on the same driving road, the display to the surroundings may be different. If the driving conditions are the same but the driving road is different, the display to the surroundings may be different. In other words, the display to the surroundings can change depending on the driving conditions, the driving road, and the surrounding conditions.

[0099] Here, the controller 100 may, for example, detect surrounding objects from images acquired by an exterior camera and control the headlights and projection device based on the detection results. The control unit of the projection device may use data acquired via communication to control the projection device 11 to perform display. Furthermore, when surrounding objects (objects to whom the display is desired) view the information, information displayed by a projection device 11 that can project the information at a position that is easier to see may be displayed by another projection device 11, and different information may be displayed or displayed at different brightness.

[0100] Next, an example of the image display process will be described with reference to Figures 11 and 12. In this embodiment, the projection device 11 may be controlled by the controller 100 or a processor, or a control unit of the projection device 11 may control the projection device 11 based on information from the vehicle or an external device, or the vehicle and the projection device 11 may share control. For example, when control is performed by a processor, image information based on information from a sensor is sent to the projection device 11, and the image is projected. Alternatively, control related to the process is performed by the controller 100 or the processor, and the projection device 11 only performs projection and does not perform control. In the following embodiments, the control method related to the image display process is not particularly limited.

[0101] For example, as shown in Fig. 11, step S1 is a display preparation step in which a determination is made as to whether the projection device should start displaying. Step S2 is an image display step in which image displaying is started. Thereafter, step S3 determines whether the display should be ended. Then, step S4 ends the display in accordance with the display end operation in accordance with the result of the display end condition determination S3.

[0102] 12, steps S11-S15 are display preparation steps in which it is determined whether to start the projection device 11 and start display. Step S11 determines whether to start the projection device. Step S12 determines the roadway. Step S13 determines the driving control state. Step S14 determines display start condition 1. Step S15 determines display start condition 2. Steps S2 to S4 are the same as those in FIG. 11.

[0103] Here, in the driving path determination in step S12, it is determined whether the vehicle's driving path is a dedicated lane for autonomous driving or a non-autonomous road. For example, this determination may be made based on information acquired by the exterior camera 521 or information about the driving path from a navigation system or the like. In addition, in the driving control status determination in step S13, it is determined whether the vehicle is being driven manually or autonomously. If the vehicle is being driven autonomously, it is also determined which level of control corresponds to the autonomous driving. For example, this determination may be made based on information acquired by the vehicle. On the other hand, the driving path determination may be made after the driving control status determination is made.

[0104] The order of the determination processes from step S11 to step S15 may be adjusted as appropriate depending on the projection, etc. Alternatively, the processes may be performed in a predetermined order.

[0105] 11 and 12 are executed by the controller 100, the control unit 714 of the projection device 11, another processing device provided in the vehicle, or a processing device outside the vehicle. Here, the processing may be executed by one of the controllers, the control unit, etc., or may be executed by a plurality of components in a distributed manner. That is, the processing in FIGS. 11 and 12 may be executed by one of these components, or may be executed by a plurality of components. In controlling the projection device 11, the controller 100 may send a signal to the projection device 11 to control the projection device 11. Furthermore, the control unit 714 of the projection device 11 may control the display of the projection device 11 based on information acquired from the vehicle.

[0106] Next, with reference to FIG. 13 , examples of information displayed to the driver (i.e., Category 1) will be described. The projection device 11 displays, for example, a frozen road warning, a speeding warning, a lane departure warning, a wrong-way driving warning, a collision risk warning, a driving speed display, a navigation display, a vehicle width display, a predicted trajectory display, a driving operation request display, a driving state switching display, a driving operation switching request availability display, a control state display, an object recognition display, a driving path information display, and an abnormality occurrence notification display. Here, the frozen road warning display is information that notifies the driver of frozen road conditions. For example, when a sensor detects frozen road conditions, information about the frozen road conditions is displayed. The speeding warning display is information that notifies the driver that the driving speed exceeds a predetermined speed. When the vehicle's driving speed is faster than the predetermined speed, information about a speeding warning is displayed. The lane departure warning display is information that notifies the driver that the vehicle is traveling outside its lane. The wrong-way driving warning display indicates that if a vehicle enters a road where the vehicle's travel direction is restricted on the current driving path, it will be considered wrong-way driving, and in such cases, information regarding a wrong-way driving warning is displayed. The collision risk warning display is information that notifies the driver of the risk of collision with surrounding objects such as vehicles or people such as pedestrians. The driving speed display is information that notifies the driver of the vehicle's driving speed. The navigation display is information that notifies the driver of navigation information such as the destination set for the vehicle. The vehicle width display is information that notifies the driver of the vehicle's width. The predicted trajectory display is information that notifies the driver of the vehicle's predicted trajectory.

[0107] Furthermore, the driving operation request display is information requesting the driver to perform a driving operation (e.g., manual driving). The driving state switching display is information informing the driver of switching between manual driving and automated driving. The driving operation switching request availability display is information informing the user of whether or not switching between manual driving and automated driving is possible. The control state display is information informing the vehicle's control state, etc. The control state display may, for example, display information regarding abnormalities in the control state, display information regarding the automated driving level during automated driving, and display information regarding changes in the automated driving level. The object recognition display is information informing the driver of information regarding objects around the vehicle. The driving path information display is information regarding the path the vehicle is traveling on. The abnormality occurrence notification display is information informing the driver that some abnormality has occurred in the vehicle. The display format is not particularly limited as long as it can appropriately convey the information, and it is possible to use symbols, letters, etc.

[0108] The projection device 11 may select and determine information to be displayed to the driver depending on the vehicle's control state (manual driving or automated driving, or several levels of automated driving) and the road (whether a road dedicated to automated driving or not). For example, the projection device 11 may display a frozen road warning, an excessive speed warning, a lane departure warning, a wrong-way driving warning, and a collision risk warning for the driver when the vehicle's control state is automated at level 4 or 5 and the vehicle is on a road dedicated to automated driving. Similarly, the projection device 11 may display a frozen road warning, an excessive speed warning, a lane departure warning, a wrong-way driving warning, and a collision risk warning for the driver when the vehicle's control state is automated at level 1 or 2 and the vehicle is on a public road (i.e., when the vehicle is on a public road with a low level of automated driving).

[0109] The projection device 11 may display the driving speed, navigation, vehicle width, predicted trajectory, driving state switching, control state, object recognition, driving route information, and abnormality occurrence notification for the driver when the vehicle's control state is at autonomous driving level 4 or 5 and the vehicle is located on an autonomous driving-only road. Similarly, the projection device 11 may display the driving speed, navigation, vehicle width, predicted trajectory, driving state switching, control state, object recognition, driving route information, and abnormality occurrence notification for the driver when the vehicle's control state is at any of autonomous driving levels 1 to 5 and the vehicle is located on a public road.

[0110] The projection device 11 may display a driving operation request and a driving operation switching request to the driver when the vehicle is in an autonomous driving control state at autonomous driving level 4 or 5 and is located on an autonomous driving-only road. Similarly, the projection device 11 may display a driving operation request and a driving operation switching request to the driver when the vehicle is in an autonomous driving control state at any of autonomous driving levels 3 to 5 and is located on a public road (i.e., when the vehicle is located on a public road with a high autonomous driving level).

[0111] In this specification and drawings, "general roads" refer to roads on which vehicles can travel. A general road is, for example, a road other than a road exclusively for autonomous driving, such as an expressway, a highway, a freeway, or the like, and is not particularly limited as long as it is a road on which vehicles can travel. Furthermore, in this specification and drawings, "roads exclusively for autonomous driving" refer to roads on which autonomous vehicles can travel. Furthermore, by limiting roads to those that can only be traveled by autonomous vehicles, the safety of autonomous vehicles can be expected to be improved by eliminating the unpredictable movements of people and manually driven vehicles. There may be cases where the entire road is exclusively for autonomous driving, or where some lanes of a road are exclusively for autonomous driving, or where lanes are set as exclusively for autonomous driving only during certain times of the day, such as late at night.

[0112] Next, with reference to FIG. 14 , examples of information displayed to those around (i.e., Category 2) will be described. The projection device 11 displays, for example, a right turn indication, a left turn indication, a hazard indication, a reversing indication, a lane change indication, a parking / stopping indication, a starting indication, a door open / close indication, a crossing indication, a driving status indication, an object recognition indication, an abnormality occurrence notification indication, and the like to those around. Here, the right turn indication is information informing people that the vehicle is turning right. The left turn indication is information informing people that the vehicle is turning left. The hazard information is information informing people that the vehicle is making an emergency stop. The reversing indication is information informing people that the vehicle is reversing. The lane change indication is information informing people that the vehicle is changing lanes. The parking / stopping indication is information informing people that the vehicle is parking / stopping. The starting indication is information informing people that the vehicle is starting. The door open / close indication is information informing people that the vehicle doors are opening and closing. The crossing indication is information informing people around that they can cross the road (such as a pedestrian crossing indication).

[0113] Furthermore, the driving status display is information that conveys the driving status of the vehicle. The driving status display includes, for example, a display indicating that the vehicle is driving autonomously, a display indicating the level of autonomous driving of the vehicle, and other displays related to the autonomous driving of the vehicle. The object recognition display is information that conveys information about objects around the vehicle. The abnormality notification display is information that conveys that some kind of abnormality has occurred in the vehicle. Note that the display format is not particularly limited as long as it can convey information appropriately, and displays using symbols, letters, etc. are possible.

[0114] Images for the surroundings can be displayed in front, on the left and right sides, or behind the vehicle. Images for the driver can be displayed, for example, in front or behind the vehicle. Projected images can be classified into images aimed at the driver and images aimed at the surroundings of the vehicle, or into two or more categories. The classified images are then adjusted for display position, polarization, geometric processing, etc.

[0115] The projection device 11 may select and determine information to be displayed to those around the vehicle depending on the vehicle's control state (manual driving or autonomous driving, or several levels of autonomous driving) and the road (whether a road dedicated to autonomous driving or not). For example, when the vehicle's control state is autonomous driving level 4 or 5 and the vehicle is on an autonomous driving road, the projection device 11 may display right turn indications, left turn indications, hazard indications, reversing indications, lane change indications, parking and stopping indications, starting indications, door opening and closing indications, crossing indications, driving status indications, object recognition indications, and abnormality occurrence notification indications to those around the vehicle. Similarly, when the vehicle's control state is autonomous driving level 1 to 5 and the vehicle is on a public road, the projection device 11 may display right turn indications, left turn indications, hazard indications, reversing indications, lane change indications, parking and stopping indications, starting indications, door opening and closing indications, crossing indications, driving status indications, object recognition indications, and abnormality occurrence notification indications to those around the vehicle.

[0116] Next, with reference to FIGS. 15A to 15C, examples of information display in special cases will be described. If an abnormality occurs during autonomous driving, the projection device 11 may display information regarding the control state to the driver. The information regarding the control state includes information indicating the abnormality in the control state. Furthermore, if the user does not respond to the abnormality in the control state, the projection device 11 may display a display prompting the user to take action. When an abnormality in the control state is detected and it is determined that the control state needs to be switched from autonomous driving to manual driving, a driving operation request display (shown as display (A) in the figure) is displayed to the driver, as shown in FIG. 15B(a), for example. If the user does not perform a driving operation after a predetermined time has elapsed, i.e., does not start manual driving, a driving operation request display (shown as display (B) in the figure) different from display (A) may be displayed, as shown in FIG. 15B(b), for example. If the user still does not perform a driving operation after a predetermined time has elapsed, i.e., does not start manual driving, a driving operation request display (shown as display (C) in the figure) different from display (A) and display (B) may be displayed, as shown in FIG. 15B(c), for example. In (a), (b), and (c) of FIG. 15B , an abnormality in the control state occurs, and the driver needs to switch to user control (manual driving) as soon as possible. The urgency of the situation increases in the order of (a) < (b) < (c). Therefore, the display (a) may be designated as a driving operation request display level 1, and the display (b) may be designated as a driving operation request display level 2, with the notification intensity and seriousness being higher than that of (a) (level 1). The display (c) may be designated as a driving operation request display level 3, with the notification intensity and seriousness being even higher than that of (b) (level 2). Examples of displays that increase the intensity and seriousness include changing the brightness, color, or size of the display, or flashing the display. Possible means for notifying the driver when it is determined that the control state needs to be switched from automated driving to manual driving include an in-vehicle audio notification or a notification displayed on an in-vehicle display device. These notifications may also be combined with the driving operation request display.

[0117] For example, if the in-vehicle system 300, an autonomous driving control system, or the like detects an abnormality during autonomous driving, it displays control information related to the abnormality. For example, the color of the currently displayed image information may be changed, or an icon related to the abnormality may be added to the currently displayed image information. If the user does not respond to the abnormality, the projection device 11 may display information related to a driving operation request for the driver. Alternatively, the projection device 11 may simultaneously display information related to the abnormality and the driving operation request. For example, the projection device 11 may notify the driver of the abnormality information by voice and then display the driving operation request information, or may notify the driver of the abnormality information and the driving operation request information by voice while displaying the abnormality information. Here, the projection device 11 may continue to display the driving operation request information until the user responds to the abnormality. Furthermore, if the user does not respond to the abnormality or the driving operation request within a predetermined time after the abnormality information or the driving operation request display is displayed, the projection device 11 may change the display of the driving operation request display and display the changed driving operation request display. That is, the projection device 11 may change the display color, brightness, and / or display size of the driving operation request display, for example. Furthermore, the notification of the driving request to the user may be performed using hardware other than the projection device 11. The other hardware includes a speaker, a mobile terminal connected to the vehicle, and the like.

[0118] The projection device 11 may display a control status when an abnormality occurs during autonomous driving. Here, the projection device 11 may display a control status indication to the driver indicating that an abnormality has occurred. Furthermore, when the in-vehicle system 300 or a system related to autonomous driving performs a vehicle stopping action from the viewpoint of safety, the projection device 11 may display (notify) information regarding the system's vehicle stopping action to the driver. The information regarding the vehicle stopping action includes, for example, a notification of the start of the vehicle stopping action, a notification of the planned stopping location, and / or a notification of the end of the vehicle stopping action. Furthermore, the projection device 11 may display, to those around it, an indication of the occurrence of an abnormality or a control status indication, for example, a vehicle parking / stopping indication.

[0119] As shown by the case of driving state change 1, for example, when the vehicle changes its driving route to an autonomous driving-only road or an autonomous driving-only lane and the driving state is changed by the in-vehicle system 300 or a system related to autonomous driving control, the projection device 11 may display driving route information, may display driving state switching, or may display driving route information and driving state switching simultaneously. When the vehicle changes its driving route between a general road and an autonomous driving-only road or an autonomous driving-only lane, the projection device 11 displays driving route information after the turn signal 14 of the vehicle is operated for a predetermined time (or the turn signal turns on for a predetermined time). For example, when a vehicle is traveling manually or autonomously on a public road and the vehicle changes lane to the adjacent autonomous driving road or lane on the right or left, the projection device 11 displays information about the current road and activates the turn signal 14 for a predetermined period of time to notify surrounding vehicles that the vehicle will be changing lanes (roads) to the right or left. The projection device 11 then displays information indicating that the vehicle will be on an autonomous driving road or lane after the lane change. Meanwhile, depending on the timing of the change in road and driving state, the projection device 11 may first display information indicating the road change and then display driving state switching information. For example, the projection device 11 may display information about the road change to those around the vehicle at a predetermined time, activate the turn signal, and change the road. Furthermore, when the system changes the autonomous driving level, the projection device 11 may also display information about the change in autonomous driving level. For example, when the vehicle changes road to the autonomous driving road and the autonomous driving level is changed from one of levels 1 to 3 to level 4 or 5, the projection device 11 may also display information about the changed autonomous driving level. In addition, when changing from manual driving to automatic driving, since the user operation after changing the driving state is reduced, a simple notification (display) may be provided.

[0120] As shown in the case of driving state change 2, for example, when a vehicle changes from an autonomous driving road to a driving state changed by the in-vehicle system 300 or a system related to autonomous driving control, the projection device 11 may display driving state information to the driver, display a driving state switch, or simultaneously display both the driving state information and the driving state switch. The projection device 11 may also display a driving state switch to those around the vehicle. Here, when the autonomous driving level is changed by the system, the projection device 11 may display information regarding the change in autonomous driving level. For example, when the vehicle changes from an autonomous driving road to a general road, the autonomous driving level is changed from level 4 or 5 to any one of levels 1 to 3. In this case, since the number of user operations after the driving state change increases, for example, more detailed (careful) notifications (displays) may be provided than in the above case (driving state change 1). For example, in the case of driving state change 2, the display time may be longer, a more noticeable display color may be used, or the display may be brighter.

[0121] FIG. 15C illustrates examples of display changes for driving state changes 1 and 2. As shown in FIG. 15C(a), when it is determined from navigation information or the like that a change in driving path, i.e., a change from a general road to a road / lane dedicated for autonomous driving, or a change from a road / lane dedicated for autonomous driving to a general road, is about to occur, driving path information is displayed (shown as display (a) in the figure) to notify the user of the change in driving path in advance. Thereafter, when the change in driving path is completed, driving path information is displayed (shown as display (b) in the figure) to notify the user that the change has been completed. Furthermore, when the driving control state is switched in conjunction with the change in driving path, i.e., from manual driving to automated driving, or from automated driving to manual driving, a driving control state switch notification (shown as display (c) in the figure) is issued.

[0122] As shown in the case of driving state change 3, for example, when a user requests the in-vehicle system 300 or a system related to automatic driving control to switch from manual driving to automatic driving, the system determines whether the switch is possible. In such a case, the projection device 11 may notify the driver or display a message indicating the switch. Furthermore, if the system determines that the switch is possible, the projection device 11 may notify the driver or display a message indicating the switch. Furthermore, a driving state switch display may be displayed. Furthermore, in this case, the projection device 11 may display a driving state to those around the vehicle. Note that the notification of whether the switch request is possible can be, for example, a road surface display, but notification (display) by other means may also be used. In this case, since the user's operations after the driving state change are reduced, for example, a simple notification (display) may be displayed to the driver.

[0123] As shown in the case of driving state change 4, for example, when a user requests a system, such as the in-vehicle system 300 or a system related to automatic driving control, to switch from automatic driving to manual driving, the system determines whether the switch is possible. The projection device 11 may notify the driver of whether the switch is possible or display a notification indicating the switch. Furthermore, if the system determines that the switch is possible, the projection device 11 may notify (display) the driver that the switch is possible or display a driving state switch. Note that the notification of whether the switch request is possible can be, for example, a road surface display, but notification (display) by other means may also be used. Note that in this case, since the number of user operations after the driving state change increases, for example, a more detailed (careful) notification (display) may be used. For example, in the case of driving state change 4, the display time may be longer than in the case of driving state change 3. Furthermore, a more noticeable display color may be used. Furthermore, a brighter display may be used. However, since the request is originally a user request, a simple display may also be used.

[0124] For example, if the autonomous driving level is lowered based on the judgment of the in-vehicle system 300, an autonomous driving control system, or other system, rather than through user operation, the projection device 11 may display a different display to the driver than the normal display. Furthermore, if an unusual operation or control, such as a sudden change (e.g., a decrease) in the autonomous driving level, is performed based on the judgment of the system, rather than through user operation, the projection device 11 may display a different display to the driver than the normal display. For example, the driver can recognize the sudden decrease in the autonomous driving level and respond to the increased load (i.e., an increase in the amount of operation). The display different from the normal display is displayed for a predetermined time or a preset period after the autonomous driving level change. For example, Figure 26(e) shows the display content for a predetermined time before the level change and the display content for a predetermined time after the level change, with arrows indicating the meaning of the level change. In the normal display, no arrow is displayed.

[0125] When a special condition occurs, such as a sudden drop in the autonomous driving level, the projection device 11 performs display control to change the display content and the predetermined display time from normal, thereby increasing the reliability of information notification to the user. Here, the projection device 11 may perform control such as lengthening the display time, making the display clearer, enlarging the display, or increasing the amount of information to be displayed, for example.

[0126] 16, a specific example of the startup determination (S11) of the projection device 11 will be described. In S11, the startup of the projection device 11 is determined according to the preset information of whether the projection device 11 is in use or not.

[0127] In Example 1-1, whether or not to start the projection device 11 is determined based on the installation of the projection device 11. For example, if the installation of the projection device 11 is confirmed, it is determined to start the projection device 11, and if the installation of the projection device 11 is not confirmed, it is determined not to start the projection device 11.

[0128] In Example 1-2, whether to start the projection device 11 is determined based on the setting for use of the projection device 11. For example, if the use of the projection device 11 is set, it is determined to start the projection device 11, and if the use of the projection device 11 is not set, it is determined not to start the projection device 11. However, even if the use of the projection device 11 is not set, if it is determined that there is a risk of danger to the driver or surrounding vehicles and people, the projection device 11 displays an image. For example, if a risk of collision with surrounding vehicles or people is detected, a collision risk warning display may be displayed even if the use of the projection device 11 is not set.

[0129] In Example 1-3, whether or not to start the projection device 11 is determined based on the reception status of a fault signal for the projection device 11. For example, if a fault signal for the projection device 11 is not received, or if the projection device 11 has not acquired a fault signal from another source, it is determined to start the projection device 11. If a fault signal for the projection device 11 is received, it is determined not to start the projection device 11. In other words, it is determined that the projection device 11 is not faulty before starting the projection device 11. On the other hand, if a fault signal is not received, or if no signal is received within a predetermined time, it is determined not to start the projection device 11.

[0130] In Example 1-4, when a failure signal is received for the projection device 11, whether or not to start the projection device 11 is determined by checking whether or not there is an area on the display element where normal display is possible. For example, even when a failure signal for the projection device 11 is received, if it is confirmed that there is an area on the display element where normal display is possible, it is possible to display using the normal portion, and therefore it is determined to start the projection device 11.

[0131] In Example 1-5, a failure signal for the projection device 11 is received, and whether or not to start the projection device 11 is determined by checking whether another alternative projection device 11 is installed. Even if a failure signal for the projection device 11 is received, if it is confirmed that an alternative projection device 11 is installed, the alternative projection device 11 is started. For example, if a projection device 11 is incorporated in each of the left and right headlights, and only one of the projection devices 11 is malfunctioning, the other projection device is started.

[0132] In Example 1-6, whether or not to activate the projection device 11 is determined based on whether or not the projection device 11 is set to be unused. For example, if the projection device 11 is not set to be unused, it is determined to activate the projection device 11, and if the projection device is set to be unused, it is determined not to activate the projection device 11. However, even if the projection device 11 is set to be unused, if it is determined that there is a risk of danger to the driver or surrounding vehicles and people, the projection device 11 can display an image.

[0133] In Example 1-7, it is determined whether the projection device 11 is started based on the reception status of a start signal from the start switch or operation switch of the projection device 11. For example, it is determined that the projection device is started when there is a start signal from the operation switch of the projection device 11. For example, when the user uses the start switch or operation switch to set the projection device 11 to ON / OFF, and it is confirmed that the operation switch of the projection device 11 is ON, it is determined that the projection device 11 is started.

[0134] Example 1-8 determines whether or not a voice input is to activate the projection device 11. If there is data or information indicating that a voice input is to activate the projection device 11, it is determined that the projection device 11 is to be activated.

[0135] In Example 1-9, the activation of the projection device 11 is determined based on whether or not a signal indicating activation of the projection device 11 by an information terminal has been received. When a signal or information indicating activation of the projection device 11 is received from the information terminal, the determination is made to activate the projection device 11. For example, when an activation signal is received from a smartphone or wearable device operated by a driver or passenger, the determination is made to activate the projection device 11. Furthermore, as shown in FIG. 16 , the conditions in each example may occur simultaneously. For example, even if the projection device 11 is set to be unused in Example 1-6, activation of the projection device 11 by voice input in Example 1-8 may be received. In such a case, the determination is made based on the priority of the determination conditions for activation of the projection device. Alternatively, as shown in FIG. 16 , the projection device 11 may be activated when multiple conditions in each example are met simultaneously.

[0136] Specific examples of display start condition determination 1 (S14) will be described with reference to FIGS. 17 to 20. FIG. 17 relates to an image display for the driver. FIG. 18 relates to an image display for vehicles and people around the vehicle. FIG. 19 relates to an image display for the driver (i.e., Category 1) that is displayed only during autonomous driving. FIG. 20 relates to an image display for the surroundings (i.e., Category 2) that is displayed only during autonomous driving. In other words, the projection device 11 displays information based on different display conditions during manual driving and autonomous driving. In this example, the image display shown in FIGS. 19 and 20 is information that is displayed during autonomous driving. The projection device 11 displays information of a type that is not displayed during manual driving, based on display conditions that are not present during manual driving and that are valid during autonomous driving (FIGS. 19 and 20).

[0137] As shown in FIG. 17 , when one or more of the following conditions (2-1A) to (2-1D) are met, the projection device 11 determines whether to display a frozen road warning (Example 2-1). Here, the condition (2-1A) determines whether to display a frozen road warning based on temperature information. For example, when a temperature detection unit or sensor detects or senses that the temperature is below a set temperature, the frozen road warning display is initiated. Roads are considered to freeze when the temperature outside the vehicle drops below 3 to 5 degrees Celsius, and this temperature may be used as the set temperature. Alternatively, a road temperature lower than the temperature outside the vehicle may be used as the set temperature, and whether the road temperature is below freezing may be detected. Similarly to the set temperature, a humidity condition may be set. The condition (2-1B) determines whether to display a frozen road warning based on frozen road information. For example, if frozen roads are determined based on road surface information captured by a camera, the frozen road warning display is initiated. Frozen roads may be determined based on changes in the amount of light reflected, or by capturing images using a polarizing filter attached to a camera. Furthermore, when information about frozen roads is received via communication from outside the vehicle, a frozen road warning may be displayed. The condition (2-1C) determines whether or not a frozen road warning needs to be displayed based on the operation of the anti-lock brake system. For example, when a signal or information indicating that the anti-lock brake system has been activated is detected, the frozen road warning is displayed. The condition (2-1D) determines whether or not a frozen road warning needs to be displayed based on the operation of the vehicle's anti-skid system. For example, when a signal or information indicating that the vehicle's anti-skid system has been activated is detected, the frozen road warning is displayed. The operation of the anti-lock brake system or the vehicle's anti-skid system indicates a decrease in friction between the vehicle and the road surface, such as wheel spin, and indicates the possibility of frozen roads.

[0138] When one or more of the following conditions (2-2A) to (2-2B) are satisfied, the projection device 11 determines whether to display a speeding warning (Example 2-2). Here, the condition (2-2A) determines whether to display a speeding warning based on the vehicle's traveling speed. When the vehicle speed sensor 501 detects a signal or information indicating that the vehicle's current traveling speed exceeds the maximum speed sign, the speeding warning is displayed. Note that the speed information may be acquired from the acceleration sensor 512, the gyro sensor 513, the GPS receiver 518, or the like. The maximum speed sign may also be acquired by any appropriate method. For example, information on the speed set by the maximum speed sign may be acquired based on an image acquired by the external camera 521. The condition (2-2B) determines whether to display a speeding warning when the vehicle speed sensor 501 detects a signal or information indicating that the vehicle's current traveling speed exceeds the maximum speed indicated on the map information for the current location. The maximum speed on the map information at the current location may be obtained from a device such as the car navigation system 150, based on the speed limit sign on the road on which the vehicle is traveling.

[0139] When one or more of the following conditions (2-3A), or (2-3A) and (2-3B) to (2-3C) are met, the projection device 11 is determined to display a lane departure warning (Example 2-3). Here, condition (2-3A) indicates that a lane departure warning is to be displayed when a sensor installed in the vehicle detects lane departure. For example, the lane information of an image acquired by the exterior camera 521 is used to determine whether the host vehicle is traveling within the driving lane. Then, when it is detected that the vehicle is deviating from the lane, the lane departure warning is displayed. Condition (2-3B) indicates that a lane departure warning is to be displayed when the illumination of the turn signal cannot be detected. Furthermore, a driver may intentionally deviate from a lane in order to change lanes. To determine whether a lane departure is intentional, the illumination of the turn signal is checked. The exterior camera 521 may also be used to check whether there are multiple lanes into which a lane change is possible, and then determine whether the lane change is intentional. In the condition (2-3C), if the driving lane cannot be detected on either side of the vehicle, a lane departure warning is displayed. The driving lane may be detected by the external camera 521, for example.

[0140] When one or more of the conditions (2-4A) to (2-4C) are satisfied, it is determined that the projection device 11 should display a wrong-way driving warning (Example 2-4). Here, the condition (2-4A) indicates that the wrong-way driving warning should be displayed when a no entry sign is detected in the lane in which the vehicle is traveling. The condition (2-4B) indicates that the wrong-way driving warning should be displayed when a one-way sign is detected in the lane in which the vehicle is traveling. The condition (2-4C) indicates that the wrong-way driving warning should be displayed when wrong-way driving is detected based on map information and sensors (such as the GPS receiver 518, the gyro sensor 513, the acceleration sensor 512, and the vehicle speed sensor 501).

[0141] If one or more of the conditions (2-5A) to (2-5C) are met, the projection device 11 is determined to display a collision risk warning (Example 2-5). Here, the condition (2-5A) determines whether or not to display a collision risk warning based on the inter-vehicle distance. If the inter-vehicle distance between the host vehicle and a vehicle traveling ahead is less than a set distance, a collision risk warning is displayed. For example, a collision mitigation braking system is required to avoid collision when approaching a stopped vehicle ahead at a predetermined speed, e.g., 50 km / h. The stopping distance at 50 km / h is said to be approximately 24 m. However, if no human intervention is involved in the stopping operation, no free-running distance occurs, so only the braking distance may be considered. In this case, the stopping distance is approximately 14 m. Furthermore, a warning is issued to the driver at least a predetermined time before the collision mitigation braking system is activated, e.g., 0.8 seconds before activation, and the free-running distance when traveling at 50 km / h for 0.8 seconds is approximately 11 m. It is desirable that the collision risk warning be displayed before the collision mitigation brake system activates. Therefore, assuming, for example, that there is no free-travel distance when traveling at 50 km / h, the collision risk warning should be displayed at least when the inter-vehicle distance is between 25 m and 30 m. The free-travel distance is the distance traveled before a person recognizes, judges, and takes action against danger, and the braking distance is the distance from when the brakes are applied until the vehicle stops. Furthermore, since the free-travel distance is proportional to the traveling speed and the braking distance is proportional to the square of the traveling speed, the inter-vehicle distance setting at which the display starts may be changed according to the traveling speed of the vehicle, or the display may start at a greater distance to alert the driver. Furthermore, since the stopping distance varies depending on the road surface condition in inverse proportion to the road surface friction coefficient, the inter-vehicle distance setting at which the display starts may be changed according to the road surface condition. The collision risk warning can also be displayed when the inter-vehicle distance between the host vehicle and the vehicle traveling behind the host vehicle is equal to or less than the set distance.

[0142] The condition (2-5B) determines whether or not to display a collision risk warning based on the relative speed. A collision risk warning is displayed when the relative speed between the host vehicle and a vehicle traveling ahead of the host vehicle is equal to or greater than a set speed. A collision risk warning can also be displayed when the relative speed between the host vehicle and a vehicle traveling behind the host vehicle is equal to or greater than a set speed. If the vehicle traveling ahead of the host vehicle is traveling at a higher speed than the host vehicle, a collision will not occur. However, if the vehicle traveling ahead of the host vehicle is traveling at a lower speed than the host vehicle, there is a risk of a collision. However, since the vehicle traveling ahead is moving forward, the inter-vehicle distance at which a collision with the vehicle traveling ahead occurs is shorter than when the vehicle traveling ahead is stopped. Therefore, the inter-vehicle distance required to start displaying the collision risk warning described in (2-5A) needs to be set shorter than when the vehicle traveling ahead is stopped. For example, in a collision mitigation braking system, a collision is required to be avoided when the vehicle traveling ahead is traveling at 20 km / h and the host vehicle is traveling at 50 km / h. In this case, assuming that there is no free-running distance when applying the collision mitigation brake, the vehicle will travel approximately 14 meters before stopping, requiring approximately 2 seconds. During this time, the vehicle in front will travel approximately 11 meters. Therefore, a 3-meter inter-vehicle distance will allow the vehicle to avoid a collision and stop. Even in this case, an alarm is issued to the driver at least 0.8 seconds before the collision mitigation brake is activated. Since the host vehicle travels 11 meters between the time the alarm is issued and the time the collision mitigation brake is activated, the collision danger warning should be displayed when the inter-vehicle distance is between 11 meters and 15 meters. As mentioned above, the inter-vehicle distance required to avoid a collision varies depending not only on the vehicle's speed but also on the relative speed with respect to surrounding vehicles. Therefore, the relative speed must also be taken into consideration when displaying the collision danger warning. Furthermore, the collision danger warning can be displayed even when the relative speed between the host vehicle and the vehicle behind it is below a set speed.

[0143] The condition (2-5C) is that a collision danger warning is displayed when the predicted time until a collision between the host vehicle and a vehicle traveling ahead is equal to or shorter than a set time. Furthermore, a collision danger warning is displayed when the predicted time until a collision between the host vehicle and a vehicle traveling behind the host vehicle is equal to or shorter than a set time. The stopping distance during braking can be calculated from the vehicle's traveling speed and the friction coefficient of the road surface. The speeds of the vehicles traveling ahead and behind the host vehicle may be obtained, for example, via appropriate communication, or may be calculated from the host vehicle's traveling speed and the relative position of surrounding vehicles captured by the external camera 521. The friction coefficient of the road surface may be calculated from the brake pedal depression amount and the vehicle speed deceleration amount or the time required for deceleration during braking. Furthermore, a collision danger warning may be displayed in front of or behind the vehicle in conjunction with activation of the collision mitigation brake. Alternatively, the collision danger warning may be displayed in conjunction with a collision mitigation brake warning, a predetermined time before activation of the collision mitigation brake, for example, 0.8 seconds or more, or may be displayed before the collision mitigation brake warning. Furthermore, when the vehicle's sensors detect a risk of collision if the vehicle is traveling at the current speed, the collision risk warning can be displayed at least a specified time before the expected time of collision without being linked to the collision mitigation brake warning. For example, if the vehicle is traveling at 50 km / h, the warning can be displayed 5 to 10 seconds before the expected time of collision to prompt the driver to apply the brakes and prevent a collision. This specified or set time can also be changed according to the speed and road conditions.

[0144] In Example 2-6, when a predetermined condition is met, the projection device 11 determines whether to display the driving speed. The condition (2-6) is that if the driving speed display setting of the projection device 11 is ON, the driving speed is displayed. Furthermore, if the driving speed display setting of the projection device 11 is OFF, the driving speed is not displayed. The vehicle's current driving speed is detected by the vehicle speed sensor 501. The color and size of the displayed speed may be changed according to the vehicle's driving speed, or a speed range for the displayed speed may be preset. By displaying the speed only when the speed limit of the road being traveled is approached, it is possible to maintain a safe driving speed while preventing distraction during driving due to a constant display. For example, the display is displayed when the driving speed exceeds 90% of the speed limit or when the speed is within a predetermined speed, e.g., 5 km / h, of the speed limit. Note that if the driving speed changes rapidly, the display will be turned ON / OFF continuously. Therefore, the display may be determined based on the time average of the driving speed, or hysteresis may be provided between the display start speed condition and the display end speed condition. In addition, the color and size can be adjusted to make the display less noticeable unless necessary.

[0145] In Example 2-7, when a predetermined condition is satisfied, it is determined whether or not to cause the projection device 11 to display navigation. Regarding condition (2-7), when the navigation display setting of the projection device 11 is ON, navigation display is enabled. When the navigation display setting of the projection device 11 is OFF, navigation display is not enabled. The navigation display may be limited to content that can be superimposed on the road to improve recognition. For example, information such as right and left turns included in navigation information to a destination can be displayed on the actual road rather than on the navigation screen, thereby preventing misrecognition. Furthermore, a portion of the HUD display may be displayed, or only content that does not overlap with the HUD may be displayed. For example, an image of the vehicle's lane may be displayed on the HUD, while information about adjacent lanes, sidewalks, and other areas outside the lane may be displayed by the projection device. In order to display an image at a wide angle, a HUD requires a large optical system to be installed in the device, which poses a problem when installing it in a vehicle. The projection device can display images outside the HUD's display area and can supplement the display area, thereby avoiding problems associated with vehicle installation and displaying information to the driver and objects around the vehicle over a wide area. Furthermore, since the HUD's display position is adjusted according to the driver's seating position, the projection device's display position can also be adjusted to match the driver's seating position. The display position of the image displayed by the projection device may be adjusted in conjunction with the HUD's display position, or the HUD's display position and the display position of the image displayed by the projection device may be adjusted independently. This prevents the images from the HUD and the projection device from overlapping.

[0146] In Example 2-8, when one or more conditions from (2-8A) to (2-8C) are satisfied, it is determined whether or not to cause the projection device 11 to display the host vehicle width. The condition in (2-8A) is that the host vehicle width is displayed when the display setting for the host vehicle width display is ON. The display setting for the host vehicle width display may be set by a user operation or may be set by default. For example, when traveling on a road with a narrow roadway width or a road that has been narrowed due to construction or the like, the host vehicle width may be displayed by a user operation to visually confirm whether the host vehicle is passable.

[0147] The condition (2-8B) indicates the vehicle's vehicle width if the lane width on which the vehicle is traveling is less than a set value. For example, if the lane width is close to the vehicle width, traveling off-center in the lane could result in the vehicle's wheels skidding or contact with obstacles such as walls or road signs. Displaying the vehicle's vehicle width in such cases allows the driver to confirm the vehicle's position within the lane and avoid danger. While vehicle width generally refers to the width excluding the door mirrors, the vehicle's vehicle width display can display the vehicle's maximum width including the door mirrors. For example, the vehicle's vehicle width is displayed at a position offset by 25 cm to the left and right of the vehicle's width. Furthermore, if a pedestrian is present, the vehicle's vehicle width is displayed at a position offset by 50 cm to the left and right of the vehicle's width to avoid contact with the pedestrian. Furthermore, the required vehicle width for passing vehicles relative to the road width is (roadway width - 0.5 m) ÷ 2 on general urban roads, and (roadway width - 1.5 m) ÷ 2 on roads with many pedestrians and no sidewalks. The displayed vehicle width may be changed depending on the width of the surrounding roads and the presence of other vehicles.

[0148] The condition (2-8C) is that if the lane on which the vehicle is traveling cannot be detected, the vehicle width is displayed. Alternatively, a value slightly larger than the vehicle width may be displayed. For example, if there are no lanes on the road, or if the lanes are partially or completely missing and cannot be detected, the vehicle width is displayed. Alternatively, if there are obstacles such as walls on the left and right sides of the vehicle, or if pedestrians are detected, the vehicle width may be displayed. In such cases, only the vehicle width on the side where the obstacles or pedestrians are present may be displayed.

[0149] In Example 2-9, when one or more conditions from (2-9A) to (2-9C) are satisfied, it is determined whether or not to cause the projection device 11 to display a predicted trajectory. Here, the condition (2-9A) is that when the display setting for the predicted trajectory display is ON, the predicted trajectory is displayed. Furthermore, the display setting for the predicted trajectory display may be set by a user operation, or setting information from the previous driving may be displayed. For example, the predicted trajectory indicates the trajectory that the vehicle will travel in the future, and the driving trajectory may be displayed based on information from the steering angle sensor 503, or may be displayed based on map information or navigation information to the destination.

[0150] Regarding condition (2-9B), if the turn signal is detected to be on, the predicted trajectory is displayed. If the turn signal is not on for a predetermined period of time or longer, the display of the predicted trajectory disappears. On the other hand, if the turn signal is detected to be on for a predetermined period of time or longer, the predicted trajectory may be displayed. Alternatively, if the steering angle of the steering wheel is detected, the predicted trajectory may be displayed. Regarding condition (2-9C), if the inter-vehicle distance is less than a predetermined value, the predicted trajectory is displayed. By displaying the predicted trajectory to the driver, the driver can visually recognize that the predicted trajectory is interfering with surrounding vehicles, thereby drawing their attention. Note that the inter-vehicle distance may be acquired, for example, from the exterior camera 521 and the distance sensor 507. Note that, similar to the vehicle width display described above, the maximum width of the vehicle, including the door mirrors, may also be displayed in the predicted trajectory display. For example, the predicted trajectory is displayed at a position offset by a predetermined distance, for example, 25 cm, to the left and right of the vehicle width. Furthermore, if a pedestrian is present, the predicted trajectory is displayed at a position offset by a predetermined distance, for example, 50 cm, to the left and right of the vehicle width to avoid contact with the pedestrian.

[0151] 17, the information projected onto the road surface may be different between manual driving and automatic driving. For example, the image display during manual driving and automatic driving may differ in color, bordering, or the like.

[0152] As shown in FIG. 18 , Example 2-10 determines whether to cause the projection device 11 to display a right turn when one or more of the conditions (2-10A) or (2-10A) and (2-10B) to (2-10C) are met. Here, the condition (2-10A) indicates that a right turn is to be displayed when the right turn indicator is detected. Furthermore, in consideration of possible driver error, the right turn is displayed when the indicator is on for a predetermined time or longer, and the right turn is not displayed when the indicator is on for less than the predetermined time. The state of the indicator may be obtained, for example, from the indicator switch. Alternatively, the state may be obtained from a system such as the in-vehicle system 300 or a system related to automatic driving control. The condition (2-10B) indicates that a right turn is to be displayed when a signal or information indicating a right turn is detected. For example, the actual steering angle of the tires may be used to determine whether to display a right turn. For example, a right turn is displayed when the actual steering angle of the tires is greater than or equal to a predetermined angle. The right turn display may also be determined based on the steering angle of the steering wheel. For example, if the steering angle of the steering wheel is equal to or greater than a predetermined angle, a right turn is displayed. Information about steering wheel rotation may be acquired from the steering wheel angle sensor 503. There is a difference between the steering angle of the steering wheel and the resulting actual steering angle of the tires. Therefore, the steering angle of the steering wheel can be converted and displayed as the actual steering angle of the tires. Alternatively, the actual steering angle of the tires may be sensed and displayed instead of the steering angle of the steering wheel. The condition (2-10C) is that a right turn is displayed when the right turn indicator is illuminated and the vehicle is moving forward or the vehicle's stop status is released. For example, when the vehicle is stopped at an intersection, the vehicle moves forward or the vehicle's stop status is released in order to enter the intersection and turn right. By starting the display when the vehicle is moving forward or the vehicle is stopped, a display linked to the vehicle's behavior can be performed. For example, signal information such as vehicle movement forward or stop during autonomous driving may be acquired and displayed.

[0153] In Example 2-11, when one or more conditions (2-11A) or (2-11A) and one or more of (2-11B) to (2-11C) are satisfied, the projection device 11 determines whether to display a left turn. Here, the condition (2-11A) is to display a left turn when the left turn indicator is detected as illuminated. Furthermore, to prevent driver error, the left turn indicator is displayed when the indicator is illuminated for a predetermined time or longer, and the left turn indicator is not displayed when the indicator is illuminated for less than the predetermined time. The state of the indicator may be acquired, for example, from the indicator switch. The state of the indicator may also be acquired, for example, from a system such as the in-vehicle system 300 or a system related to autonomous driving control. The condition (2-11B) is to display a left turn when a signal or information for a left turn is detected. For example, the left turn indicator may be determined based on the actual steering angle of the tires. For example, a left turn indicator is displayed when the actual steering angle of the tires is greater than or equal to a predetermined angle. Furthermore, the left turn display may be determined based on the steering angle of the steering wheel. For example, if the steering angle is equal to or greater than a predetermined angle, a left turn display is displayed. The condition (2-11C) is that a left turn display is displayed when the left turn indicator is illuminated and the vehicle is moving forward or the vehicle has been released from a stopped state. This allows a display linked to the vehicle's behavior. For example, when a vehicle is stopped at an intersection, such as when waiting for a traffic light, if the vehicle is moving forward or the vehicle has been released from a stopped state in order to enter the intersection and turn left, the display can be started, allowing a display linked to the vehicle's behavior.

[0154] Example 2-12 determines whether to cause the projection device 11 to display hazard warning lights when one or more of the following conditions are met: (2-12A) or (2-12A) and (2-12B) through (2-12C). Here, condition (2-12A) indicates that the hazard warning lights are activated when the hazard warning lights are turned on. Condition (2-12B) indicates that the hazard warning lights are activated when the vehicle's deceleration or stop is detected. For example, signals indicating vehicle deceleration or stoppage during autonomous driving may also be used. Condition (2-12C) indicates that the handbrake is engaged when the handbrake is activated. Hazard warning lights are primarily used to inform surrounding vehicles of the vehicle's actions when parking or stopping, but they are also used to inform following vehicles of traffic congestion ahead or to express gratitude when given the right of way. By considering conditions (2-12B) through (2-12C), it is possible to determine whether the vehicle is parked or stopped and switch between displaying and not displaying the hazard warning lights.

[0155] In Example 2-13, when one or more conditions (2-13A) or (2-13A) and one or more of (2-13B) to (2-13D) are satisfied, the projection device 11 determines whether to display a reverse indication. Here, the condition (2-13A) is that the reverse indication is displayed when the vehicle is detected to be reversing due to the reverse gear being engaged, for example. Information about the reverse gear may be acquired, for example, from the shift position sensor 502. Information about the vehicle reversing may also be acquired, for example, from the in-vehicle system 300 or a system related to automatic driving control. The condition (2-13B) is that when the vehicle is reversing and the vehicle is moving forward or the stop release is detected, the projection device 11 determines that the vehicle has begun reversing, and displays a reverse indication. Information about the vehicle speed sensor 501 may also be used to determine whether the vehicle is reversing. Information about the vehicle's moving forward and the vehicle's stop release may also be acquired, for example, from the in-vehicle system 300 or a system related to automatic driving control. The condition (2-13C) is that the vehicle is reversing and no obstacle is detected behind the vehicle, and the reversing display is performed. If an obstacle is present in the displayable area behind the vehicle, it is determined that there is no display area or that there is no area for the vehicle to reversing, and the reversing display is not performed. Note that information about obstacles behind the vehicle may be obtained, for example, from the exterior camera 521, the distance sensor 507, the infrared sensor 508, etc. The condition (2-13D) is that the vehicle is reversing and no obstacle is detected within a specified distance behind the vehicle, and the reversing display is performed. An obstacle is an object that affects the display of the vehicle and its surroundings. For example, the specified distance behind the vehicle is an area up to 3 meters behind the vehicle for a standard vehicle. For a large vehicle, the specified distance behind the vehicle may be the same as for a standard vehicle, up to 3 meters, or a wider area.

[0156] In example 2-14, when one or more of the conditions (2-14A) or (2-14A) and (2-14B) to (2-14C) are met, the projection device 11 determines whether to display a lane change. Here, the condition (2-14A) indicates that a lane change is to be displayed when the turn signal is detected as being on. Furthermore, because of possible driver error, the lane change is displayed when the turn signal is on for a predetermined period of time or longer, and the lane change is not displayed when the turn signal is on for less than the predetermined period of time. The condition (2-14B) indicates that a lane change is to be displayed when the turn signal is on and the approach of a following vehicle in an adjacent lane is detected. Note that this information about the approach of a vehicle may be acquired, for example, from the external camera 521, the distance measurement sensor 507, the infrared sensor 508, or the like. The condition (2-14C) is that when the turn signal is on and a person or vehicle is detected around the vehicle, a lane change display is performed. Note that information on the presence of a person or vehicle may be acquired from, for example, the external camera 521, the distance measurement sensor 507, the infrared sensor 508, etc.

[0157] In Example 2-15, when the conditions (2-15A) and (2-15B) are met, it is determined whether or not to cause the projection device 11 to display a parking / stopping display. Here, when it is detected that the vehicle is in a vehicle start preparation state, such as when the vehicle's drive source (engine, motor, etc.) is ON, and the parking brake is set, the parking / stopping display is displayed. Also, instead of detecting the setting of the parking brake, it may be determined that the vehicle is stopped using information from the vehicle speed sensor 501.

[0158] In example 2-16, when the conditions (2-16A) and (2-16B) are satisfied, it is determined whether or not to cause the projection device 11 to display a start display. Here, when the vehicle is released from a stopped state and the vehicle is detected moving forward, the start display is performed. In addition, information from the vehicle speed sensor 501 may be used to determine whether or not the vehicle is starting.

[0159] In Example 2-17, when the condition (2-17A) is satisfied, it is determined whether or not to cause the projection device 11 to display a door open / closed display. Here, the condition (2-17A) is that when a door unlock signal is detected, a door open / closed display is performed. For example, when a door lock signal is detected, a door closed display may be performed. Also, when a door lock signal is not detected within a predetermined time after the vehicle starts, a door unlocked display may be set.

[0160] In Example 2-18, when the conditions (2-18A) and (2-18B) are met, it is determined whether or not to have the projection device 11 display a crossing display. When a vehicle is stopped and a person is detected near the lane, a crossing display is displayed. However, there are cases where the detected person has no intention of crossing the road. Therefore, the display is displayed for a certain period of time, and if it is not confirmed that the detected person is crossing within that period, the display may be terminated.

[0161] 18 (Category 2), the information projected onto the road surface may be different between manual driving and autonomous driving. For example, the image displayed during manual driving and autonomous driving may differ in color, whether or not there is a border, whether or not there is a mark (such as a square) surrounding the image displayed.

[0162] As shown in FIG. 19 , Example 3-1 is an example of a driving operation request display. When the following condition (3-1A) that prevents autonomous driving from continuing is met, the projection device 11 displays a driving operation request display. Here, the condition (3-1A) is information indicating the occurrence of an event that prevents autonomous driving from continuing from the in-vehicle system 300 or a system related to autonomous driving control. Examples of events that prevent autonomous driving from continuing include failure of various sensors related to autonomous driving or problems with the connection between the sensors and the controller 100, which result in sensors not providing values, or values ​​being provided but being abnormal. Other examples include communication failures or abnormalities outside or inside the vehicle, which prevent information that should be available via communication from outside the vehicle from being obtained, or prevent various information that should be available via communication from being transmitted or received within the vehicle.

[0163] Example 3-2 is an example showing a driving state switching display. When the following condition (3-2A) for completing switching between manual driving and automatic driving is met, the projection device 11 is caused to display the driving state switching. For the condition (3-2A), information indicating that switching between manual driving and automatic driving has been completed is acquired from the in-vehicle system 300 or a system related to automatic driving control. For example, a signal to press a manual / automatic changeover switch provided in the vehicle is sent to the in-vehicle system 300.

[0164] Example 3-3 is an example showing a display of whether a driving operation switching request is possible. When the following conditions (3-3A) and (3-3B) are met, the projection device 11 is caused to display whether a driving operation switching request is possible. Here, the condition (3-3A) is information on a request to switch between manual driving and automatic driving by a user's operation. The condition (3-3B) is that a system such as the in-vehicle system 300 or a system related to automatic driving control completes a determination of whether or not to respond to the user's request.

[0165] Example 3-4 is an example showing a control status display. When the following condition (3-4A) or (3-4B) is met, the projection device 11 displays the control status. Here, the condition (3-4A) is information that a change has occurred in the control status between normal and abnormal. The condition (3-4B) is information that a change has occurred in the autonomous driving level.

[0166] Example 3-5 is an example illustrating an object recognition display. When the following condition (3-5A) is satisfied, the projection device 11 is caused to display the object recognition. Here, the condition (3-5A) is information indicating that a change has occurred in the recognition state. For example, when a new object is detected in an image acquired from the exterior camera 521, the projection device 11 may display the object recognition. Furthermore, when a new object is recognized within a predetermined distance from the vehicle, the projection device 11 may display the object recognition. Note that the distance to the object may be acquired, for example, from the exterior camera 521, the distance sensor 507, the infrared sensor 508, etc.

[0167] Example 3-6 is an example showing a roadway information display. When one or more of the following conditions (3-6A) to (3-6C) are met, the projection device 11 is caused to display the roadway information. Here, the condition (3-6A) is information indicating that the vehicle has not yet changed roads. The condition (3-6B) is information indicating that the vehicle has completed changing roads. The condition (3-6C) is information indicating that the vehicle is currently traveling. Note that the condition (3-6C) may be determined by acquiring information every predetermined time period and acquiring information indicating that the vehicle is currently traveling a predetermined number of times in succession.

[0168] Example 3-7 is an example showing an abnormality notification display. When the following condition (3-7A) is met, the projection device 11 displays an abnormality notification display. Here, the condition (3-7A) is information that an abnormality has occurred in the vehicle.

[0169] As shown in FIG. 20, Example 3-8 is an example showing a driving status display. When the following conditions (3-8A) and (3-8B) or (3-8C) are met, the projection device 11 displays the driving status. Here, the condition (3-8A) is information that a person and / or a vehicle is detected around the vehicle. The condition (3-8B) is information that the vehicle is in a stopped state. The condition (3-8C) is information that the vehicle has resumed its stopped state.

[0170] Example 3-9 is an example showing an object recognition display. When the following condition (3-9A) is met, the projection device 11 is caused to perform object recognition display. Here, the condition (3-9A) is information that an object (such as a person or a vehicle including a motorcycle) has been detected near the vehicle.

[0171] Example 3-9 is an example showing an abnormality notification display. When the following conditions (3-10A) and (3-10B) are met, the projection device 11 is caused to display an abnormality notification display. Here, the condition (3-10A) is information that an object (such as a person or a vehicle including a motorcycle) has been detected near the vehicle. The condition (3-10B) is information that an abnormality has occurred in the vehicle.

[0172] Next, a specific example of the display start condition determination 2 (step S15) will be described with reference to Fig. 21. For example, even if the conditions in Fig. 17 and Fig. 18 are satisfied, it is possible that the determination as to whether or not to display is based on other conditions. A specific example is shown below.

[0173] In Example 4-1, if the condition (4A) is satisfied, it is determined whether or not to allow the projection device 11 to project an image. For example, when the ambient illuminance is high, such as during the daytime, it is difficult to increase the difference in brightness between the image light projected by the projection device 11 and the ambient light. As a result, the contrast of the image light decreases, making it impossible to distinguish the projected image. Therefore, in order to recognize the image, the ambient illuminance condition must be below a specified value. Here, the condition (4A) determines that the image can be displayed if the illuminance around the vehicle is below a specified value. As a result, the driver, surrounding vehicles, and people can recognize the projected image. Note that information about the illuminance around the vehicle may be acquired, for example, from the illuminance sensor 505.

[0174] In Example 4-2, if the condition (4B) is satisfied, it is determined whether or not to cause the projection device 11 to project an image. Here, the condition (4B) determines that an image can be displayed if the headlights are ON. Headlights are required to automatically turn on within two seconds when the illuminance around the vehicle is 1,000 lux or less, and to turn off within five to 300 seconds when the illuminance around the vehicle is 7,000 lux or more. Therefore, the ON / OFF information of the headlights 13 reflects the illuminance around the vehicle. By linking this information with the ON information of the headlights 13, unnecessary display can be prevented and the image projected by the projection device 11 can be recognized. The ON / OFF information of the headlights 13 may be obtained, for example, from the headlight sensor 504.

[0175] In Example 4-3, if the condition (4C) is satisfied, it is determined whether or not to cause the projection device 11 to project an image. Here, the condition (4C) is that if the vehicle speed is equal to or greater than a set predetermined speed (in the figure, display lower limit speed 1), it is determined that the image can be displayed. The image for the driver is information intended for the driver and contains information unnecessary for surrounding vehicles and people. Therefore, it is desirable that only the target of the image to be displayed is visible. For example, if the vehicle speed decreases, the relative speed with respect to objects other than the vehicle decreases, and the number of objects that can recognize the image projected by the vehicle increases. Therefore, by displaying the image only when the vehicle is traveling at a certain speed or above, distraction by others is prevented.

[0176] In Example 4-4, if the condition (4D) is met, it is determined whether or not to cause the projection device 11 to project an image. Here, the condition (4D) is that if the vehicle speed is equal to or less than a set predetermined speed (in the figure, upper display speed limit 1), it is determined that the image can be displayed. The image for the surroundings is information that should be recognized by surrounding objects, and a low relative speed with respect to surrounding objects is necessary for the recognition of the image projected by the vehicle itself. Therefore, by displaying the image only when the vehicle is traveling at a certain speed or less, the image is encouraged to be recognized by others.

[0177] In Example 4-5, if condition (4E) is met, it is determined whether to have the projection device 11 project an image. Here, condition (4E) determines that an image can be displayed if it is detected that the wipers are not operating continuously. On the other hand, when the wipers operate continuously for a short period of time, for example, to remove dirt from the windshield or front glass, the wipers may operate continuously for a certain period of time. Therefore, if it is determined that the wipers are not operating continuously for a predetermined period of time or longer, it may be determined that an image should be displayed. During rainfall, the reflective characteristics of the road surface change, and the directionality of the reflected light shifts. In some directions, light is strongly reflected, causing glare, while in other directions, light reflection is weakened, resulting in a dark image. Furthermore, when the vehicle's windshield 3 or the projection device 11 becomes wet, it becomes difficult to see the surrounding conditions, and adhering water droplets refract light, potentially preventing the intended image from being displayed and resulting in erroneous recognition. Therefore, it may be set not to project an image during rainy weather, etc. The wiper operation information may be acquired from, for example, the wiper switch 526. Furthermore, if the rain sensor 525 does not detect raindrops, it may be determined that the wipers are not operating continuously. Furthermore, if the amount of raindrops is small or if it is light rain, it may be determined that the wipers are not operating continuously. For example, if the number of times the wipers move is less than a preset number, it may be determined that the wipers are not operating. While an example of linking the wiper operation with the rain sensor has been described here, it may also be linked to the rain sensor. A typical rain sensor detects the amount of rainfall by detecting changes in the intensity of infrared reflection caused by raindrops adhering to the windshield. Depending on the detection level of the rain sensor, if the detection level is low, it may be determined that the amount of rainfall is small and will not affect the image display of the projection device, and if the detection level is high, it may be determined that the amount of rainfall is large and will not affect the image display of the projection device.

[0178] In Example 4-6, if the condition (4F) is met, it is determined whether or not to cause the projection device 11 to project an image. Here, the condition (4F) determines that an image can be displayed if a specified level of wetness or more is not detected on the road surface. Even if it is not currently raining, it may have rained in the past and the road surface may be wet. Even in such a case, the reflective characteristics of the road surface may change, and the directionality of the reflected light may change, thereby changing the appearance of the image. Therefore, it may be set not to project an image when the road surface is highly wet. Note that information about the wetness of the road surface may be obtained, for example, from an external camera. Furthermore, the information about the wetness of the road surface may be information obtained via communication and provided from an external system.

[0179] In Example 4-7, if the condition (4G) is satisfied, it is determined that the projection device 11 should project an image. Here, the condition (4G) is that if it is detected that no image is being projected onto the windshield 3, it is determined that an image can be displayed. When an image is displayed on the windshield 3 by an image display device such as a HUD, the image of the road surface displayed by the projection device 11 and the image displayed on the windshield 3 may overlap in the driver's field of view, causing the image to be incorrectly recognized. Furthermore, there is a risk that the image will occupy the driver's field of view, obscuring information about the real space. Therefore, when an image is displayed on the windshield 3, the projection device 11 does not project an image. Alternatively, it is desirable to project an image onto the area where the image on the windshield 3 is displayed so that the image from the projection device 11 does not overlap in the driver's field of view. Whether or not an image is being projected onto the windshield 3 may be acquired, for example, from the in-vehicle image display device 12.

[0180] In Example 4-8, if the condition (4H) is satisfied, the processor determines whether to cause the projection device 11 to project an image. Here, the condition (4H) is determined to not display an image if the information display setting satisfied in S14 is previously set to OFF. The information display may be set to OFF by the user. The user may set it on the operation screen, the setting screen, or by using an operation switch. Alternatively, the setting may be made using voice input.

[0181] In Example 4-9, if the condition (4J) is satisfied, it is determined whether to cause the projection device 11 to project an image. Here, the condition (4J) is that it is determined that an image can be displayed if the inter-vehicle distance from the surrounding vehicle is equal to or greater than a specified value. If the inter-vehicle distance from the surrounding vehicle is not sufficiently secured, part or all of the image may interfere with the surrounding vehicle, preventing proper recognition. Therefore, it is determined whether there is a sufficient inter-vehicle distance to secure the necessary display area for the image, and whether to display it is determined accordingly. The required inter-vehicle distance may vary depending on the image. For example, an image displayed for the driver cannot be seen unless a predetermined inter-vehicle distance, e.g., 5 m or more, is secured on the road surface in front of the vehicle due to blind spots caused by the vehicle body. Furthermore, an image displayed for the surroundings does not require driver visibility, so it can be displayed if there is an inter-vehicle distance of approximately the size of the image display. Here, the inter-vehicle distance from the surrounding vehicle is used, but the distance from surrounding walls or obstacles may also be determined to be equal to or greater than a specified value.

[0182] In Example 4-10, if the (4K) condition is met, it is determined whether or not to cause the projection device 11 to project an image. Here, the (4K) condition is that if it is detected that no other vehicles are present in the display area of ​​the road surface, it is determined that an image can be displayed. If there are no surrounding vehicles, it is determined that a sufficient area for displaying the image is secured, and the image can be displayed. Note that the presence of other vehicles may be acquired, for example, from the exterior camera 521, the distance measurement sensor 507, the infrared sensor 508, etc. Furthermore, if surrounding vehicles or people cannot be detected, the projection of the image toward the surroundings may be stopped.

[0183] Next, a specific example of the display start operation (step S2) will be described with reference to Fig. 22. For example, video display starts with one of the following operations (5A) to (5H).

[0184] Based on the result of the determination regarding the start of display, the projection device 11 may display an image (5A). The time until the start of display and the time for which the display continues may be set to a preset value or may be changed depending on the content of the display. The driver or another user may also change the settings.

[0185] The projection device 11 may display an image based on the result of the display start determination. In (5B), image display is started at a brightness set relative to the ambient illuminance. Alternatively, the image display is adjusted so that the image to be displayed has a brightness set relative to the ambient illuminance. If the brightness of one projection device 11 is insufficient, the necessary brightness may be adjusted by superimposing images from multiple projection devices 11. Furthermore, brightness adjustment may be performed by adjusting the current of the light source, by adjusting the lighting time ratio of the light source, or by adjusting the image display element. Furthermore, when the projection device starts displaying an image, the brightness may be set high and then decreased, or the brightness may be gradually decreased until the display is turned off.

[0186] The projection device 11 may display an image based on the result of the display start determination. Here, the projection device 11 displays an image so that image display starts at a specified cycle (5C). For example, the lighting cycle is set to between 60 and 120 times per minute, similar to that of a turn signal. Furthermore, if a danger is determined to exist around the vehicle based on detection information from a sensor or the like, the lighting cycle may be shortened to warn the driver. The lighting cycle may also be changed depending on the distance to the object detected by the sensor. This makes it possible to make the driver and vehicles and people around the vehicle aware of an approaching danger.

[0187] The projection device 11 may display an image based on the result of the determination regarding the start of display. In (5D), the color of the image is corrected to match the color of the display area, and image display is started. Alternatively, the image display of the projection device 11 is adjusted so that the color of the image is corrected to match the color of the road surface display area. The color of the image displayed on the road surface is determined by both the color of the image light and the color of the road surface. Therefore, even if the color of the image light is the same on an asphalt-paved road surface, a concrete-paved road surface, and an unpaved road surface, the color of the image displayed on the road surface will differ. When the road surface color is close to black, the image light has a strong absorbing property, and when the road surface color is close to white, the image light has a strong reflecting property. Therefore, when the road surface color is close to black, the image may be corrected to an achromatic color and projected, and when the road surface color is white or gray, the image may be corrected to a chromatic color and projected.

[0188] This allows the effects of light absorption to be mitigated on roads with high light absorption, and enables color-enhanced display on roads with high light reflectivity. If the image permitted for display is achromatic, the number of gradations of the image to be displayed may be reduced on roads with high light absorption, and the number of gradations of the image may be displayed in the state of the image before adjustment or increased on roads with high light reflectivity. Furthermore, if the road surface is a specific color, the spectral reflectivity characteristics of the road surface may cause the specific color to be strongly reflected, making it impossible to display the image in the intended color. In such a case, the image may be adjusted to a state where the luminance of the specific color is reduced, and then projected. This enables image projection that takes into account the spectral reflectivity characteristics of the road surface. In addition, corrections such as increasing the line width or size of letters, symbols, etc. in the image may be performed on road surfaces with low reflectivity.

[0189] Based on the result of the display start determination, the projection device 11 may display an image. In (5E), the image contrast is corrected to match the brightness around the display image and image display is started. Alternatively, the image display of the projection device 11 is adjusted to correct the image contrast to match the brightness around the display area. The brightness around the display image may be the ambient illuminance around the vehicle, or it may be road surface illuminance influenced by the ambient illuminance, lighting from the vehicle itself and surrounding vehicles, or the reflectivity of the road surface. If the brightness of the image displayed by the projection device 11 is equivalent to the brightness around the display image, the contrast between the displayed image and the road surface background may not be sufficient, and the information may not be correctly recognized. For example, contrast correction may be performed by increasing the luminance of the outline of the image. Furthermore, it is known that, based on human visual characteristics, images with high spatial frequencies require higher frequencies for recognition than images with low spatial frequencies. Therefore, to lower the spatial frequency of the image, corrections such as increasing the line width of characters, symbols, etc. in the image or enlarging part or all of the image size may be performed.

[0190] Based on the result of the determination regarding the start of display, the projection device 11 may display an image. In (5F), the display image is edged in black and then image display is started. Alternatively, the image display of the projection device 11 is adjusted so that the display image is edged in black. For example, if the projection device 11 is integrally incorporated inside the headlight 13, the projection device 11 can project an image edged in black. This emphasizes the image portion, making the information easier to recognize. Furthermore, if the projection device 11 is not incorporated inside the headlight 13, the headlight 13 may display an image by blocking light from the area corresponding to the outer edge of the image displayed by the projection device 11.

[0191] Based on the result of the determination regarding the start of display, the projection device 11 may display an image. Here, in (5G), the brightness level of the display image is inverted before the image display begins. That is, the image display of the projection device 11 is adjusted so as to invert the brightness level of the display image. In a typical information display, the area displaying the information is small relative to the background area. In such a display in which the information display area is small relative to the background area, the image is small, and therefore the information may not be correctly recognized due to the influence of factors such as the brightness around the displayed image and the reflection characteristics of the road surface. In this example, by inverting the information display area and the background area, i.e., by inverting the brightness level, the image is enlarged and the information can be correctly recognized. Furthermore, the brightness level of the display image may be inverted for the entire display image or for a portion of the image.

[0192] Based on the result of the display start determination, the projection device 11 may display an image. In (5H), the brightness of the headlights 13 that illuminate the area overlapping the display image is partially dimmed, and then image display is started. While only the headlights 13 are described in this example, other lighting provided on the vehicle may also be dimmed in the same manner. For example, if the display area of ​​the projection device 11 overlaps with lighting provided on the vehicle, such as the headlights 13, the brightness of all lighting provided on the vehicle other than the projection device 11 may be dimmed, or the brightness of a portion of the area overlapping the display area may be dimmed or blocked, thereby ensuring the contrast of the display image. It is desirable for the contrast between the background and the image displayed by the projection device to be 30% or higher.

[0193] Next, a specific example of the display termination condition determination (step S3) will be described with reference to FIG.

[0194] When the projection device 11 is displaying an image and the ambient illuminance is equal to or greater than a specified value according to the illuminance conditions from the illuminance sensor, the projection device 11 determines to end the image display (Example 6-1). Although not shown in the figure in this example, the process may return to the display start operation determination (S2) and determine whether it is possible to reduce the ambient illuminance to a specified value or less by dimming the vehicle's lights as in (5H), and consider continuing the display.

[0195] When the projection device 11 detects that the headlights are turned off while the projection device 11 is displaying an image, it determines to end the image display on the projection device 11 (Example 6-2). When the headlights are turned off, it is considered that the ambient illuminance is equal to or greater than a specified value, or that the driver has stopped driving. Therefore, when the headlights are turned off and the ambient illuminance is equal to or less than a specified value, it is possible to end only the image for the driver.

[0196] When the projection device 11 is displaying an image and the vehicle speed is equal to or lower than the display lower limit speed 2, which is slower than the display lower limit speed 1, the projection device 11 determines to end the image display (Example 6-3). In the display start condition determination 2 (S15), the illumination start condition is described as being equal to or higher than the display lower limit speed 1. However, the illumination start condition may fall below the display lower limit speed 1 during image display. In such a case, the illumination end speed during image display may be set to the display lower limit speed 2, and hysteresis may be provided between the speed conditions for illumination start and illumination end. This allows the display to continue up to the display lower limit speed 2 after the display start even if the vehicle speed falls below the display lower limit speed 1. This prevents flicker caused by repeated image display and image end when the vehicle is traveling near the display lower limit speed 1. Although not described in this example, flicker caused by repeated image display and image end can also be prevented by ending the display when the vehicle speed falls below the display lower limit speed 1 and setting a pause period until the next illumination.

[0197] When the projection device 11 is displaying an image and the vehicle speed is equal to or greater than the display upper limit speed 2, which is faster than the display upper limit speed 1, the projection device 11 determines to end the image display (Example 6-4). In the display start condition determination 2 (S15), the illumination start condition is described as being equal to or less than the display upper limit speed 1. However, the illumination start condition may exceed the display upper limit speed 1 during image display. In such a case, the illumination end speed during image display may be set to the display upper limit speed 2, and hysteresis may be provided between the speed conditions for illumination start and illumination end. This allows the display to continue up to the display upper limit speed 2 after the display start even if the vehicle speed exceeds the display upper limit speed 1. This prevents flicker caused by repeated image display and image end when the vehicle is traveling near the display upper limit speed 1. Furthermore, although not shown in this example, flicker caused by repeated image display and image end can also be prevented by ending the display when the display upper limit speed 1 is exceeded and setting a pause period until the next illumination.

[0198] If the projection device 11 detects continuous wiper operation while displaying an image, it determines to end the image display on the projection device 11 (Example 6-5). Continuous wiper operation indirectly indicates rainy or snowy weather. In such a state, the image displayed on the road surface may not be properly recognized due to rain or snow adhering to the windshield, and changes in road surface conditions may prevent the image from being displayed properly, so the image display is ended. Also, although not shown in this example, even if continuous wiper operation is detected while displaying an image, if the operation is intermittent, the image display does not need to be ended.

[0199] When the projection device 11 is displaying an image and detects that the road surface is wetter than a specified level, it is determined that the projection device 11 should end the image display (Example 6-6). When the road surface is wetter than a specified level, the light from the projection device 11 may be totally reflected depending on the angle of incidence to the road surface. Therefore, when the road surface is wetter than a specified level, it is possible to end the image display.

[0200] When the projection device 11 is displaying an image and detects information display by an in-vehicle image display device, it determines to terminate the image display on the projection device 11 (Example 6-7). For example, if an image display device such as a HUD starts displaying an image on the windshield 3 while the projection device 11 is displaying an image, the image from the projection device 11 and the image from the image display device may overlap from the driver's viewpoint. In this case, the information from each display device cannot be correctly recognized. Therefore, when the projection device 11 is displaying an image and detects information display by an in-vehicle image display device, it is possible to terminate either display. In this example, the image display on the projection device 11 is terminated. Although not shown in this example, if the image from the projection device 11 and the image from the image display device do not overlap from the driver's viewpoint, the image display on the projection device 11 may be continued. Alternatively, even if the image from the projection device 11 and the image from the image display device overlap from the driver's viewpoint, or if the display position of the image from the projection device 11 or the image from the image display device can be changed, the image display from the projection device 11 may be continued.

[0201] The image display position may be adjusted by the driver, or by the projection device without the driver's intervention. Furthermore, when the driver adjusts the display position of the image displayed by the projection device, it is desirable that the adjustment be performed when the vehicle is stopped. The vehicle's stop may be determined, for example, by a speed sensor or by the on / off state of the parking brake. It may also be determined by the image displayed by the projection device. For example, displays intended for surrounding people or vehicles are often displayed in positions that are not visible from the driver's viewpoint because they are not designed to be visible to the driver. In such cases, there is no problem with simultaneously displaying information from the in-vehicle image display device and information from the projection device, since there is no risk of interference in the driver's field of view.

[0202] Furthermore, if any of the conditions in S14 is not met, it is determined whether to cause the projection device 11 to end the image display (Example 6-8).

[0203] Furthermore, when the projection device 11 that is displaying an image outputs a failure signal and the processor acquires this failure signal, it determines that the projection device 11 should end the image display (Example 6-9). Note that, in a state where an image is being displayed based on the determination result based on the above (Example 1-4), if the processor cannot confirm that the area of ​​the display element used for the image display is operating normally, it determines that the projection device 11 should end the image display.

[0204] Furthermore, when a signal to turn off the display of the projection device 11 is received, the projection device 11 is controlled to end the image display (Example 6-10). For example, when the projection device 11 is turned off by a user operation, the projection device 11 determines to end the image display. If the driver does not need the image display by the projection device 11 while the vehicle is running, the image display is ended while the image is being displayed. However, if the conditions set on the device side are set to take priority over the user's operation, the image display continues. For example, if there is a risk of collision with a nearby vehicle or person, a collision risk warning display is displayed. While this display is being displayed, the image display continues even if the projection device 11 is turned off by a user operation.

[0205] Furthermore, when the presence of another vehicle is detected in the display area of ​​the road surface, it is determined that the projection device 11 should end the image display (Example 6-11). When another vehicle intrudes into the display area of ​​the road surface, the image of the road surface may not be displayed correctly. The area into which the other vehicle has intruded is compared with the image display position to determine whether the image display overlaps with the other vehicle, and if it is determined that they overlap, the image display is ended. Alternatively, when it is determined that the image display overlaps with the other vehicle, if it is determined that the image display position can be changed, the display position is adjusted and the image display continues, but if it is determined that the image display position cannot be changed, the image display is ended. Note that the driver cannot change the image display position while driving.

[0206] Furthermore, when it is detected that the distance to the surrounding vehicle is equal to or less than a predetermined value, it is determined that the projection device 11 should terminate image display (Example 6-12). If the distance between the vehicle and the surrounding vehicle is short, the image of the road surface may not be displayed correctly. Therefore, the distance between the vehicle and the surrounding vehicle is compared with a predetermined value, and if it is determined that the distance is equal to or less than the predetermined value, the image display is terminated. At this time, the required distance between the vehicle and the surrounding vehicle may be specified for each image displayed by the projection device. For example, because of the presence of blind spots caused by the vehicle body, image display for the driver cannot be seen unless a predetermined distance, e.g., 5 meters or more, is maintained on the road surface in front of the vehicle. Furthermore, image display for the surrounding vehicle does not require driver visibility, so it can be displayed if there is a distance between the vehicle and the surrounding vehicle of approximately the size of the image display.

[0207] Next, a specific example of the display termination operation (step S4) will be described with reference to Fig. 24. For example, the operation of the projection device 11 at the time of display termination is determined based on any one of the following (7A) to (7D), and control is performed to cause the projection device 11 to terminate the display.

[0208] By stopping the image display on the projection device 11, control is performed to cause the projection device 11 to end the display (7A). In this example, the image display from the projection device 11 is immediately stopped.

[0209] The projection device 11 is controlled to execute one image display cycle and then stop the image display of the projection device 11, thereby causing the projection device 11 to terminate the display (7B). When displaying images using the projection device 11, an image of the road surface may be displayed that changes over time. For example, a driving direction may be indicated by sequentially displaying multiple images. When such a display is performed, terminating the display midway may result in misinterpretation of the image information. To prevent the image from being stopped midway during the image display from the projection device 11, the image display may be terminated after executing one image display cycle when a display end signal is received from the projection device.

[0210] The projection device 11 is caused to display an image for a specified time, and then the image display on the projection device 11 is stopped, thereby controlling the projection device 11 to end the display (7C).

[0211] The brightness of the image display is lowered for a specified time, thereby causing the projection device 11 to end the display (7D). Note that the brightness of the image display may also be controlled to be gradually lowered.

[0212] Next, with reference to Figures 25 to 28, an example of display by a projection device during autonomous driving will be described. Information projection for the driver during autonomous driving may be controlled differently from information projection for the driver during manual driving. Furthermore, the display mode may vary depending on the control state of the autonomous driving and the type of information displayed during autonomous driving. For example, when autonomous driving is operating normally, the main purpose of information such as what the vehicle has detected and what action it will take (e.g., turn right, turn left, change lanes, stop, etc.) is to allow the driver to recognize that the autonomous driving is operating normally. Therefore, immediacy is not necessarily required; it is sufficient to correctly convey the information to the driver. Therefore, the frequency (cycle) of the determination of whether or not display start, end, or display update is necessary, and the period until the determination that display start, end, or update is "necessary" and the actual reflection in the execution of information projection control may be longer than when manual driving or autonomous driving is not operating normally (when a fault, etc. occurs).

[0213] Furthermore, it is desirable that displays that request a driving operation or decision from the driver or that notify the driver of the occurrence of an abnormality be displayed immediately when it is determined that such a display is necessary, but if the display simply communicates the control status or operation of autonomous driving to the driver, the timing of the transmission may be immediate or may vary as long as it is within a predetermined time from the operation executed by autonomous driving control. For example, as long as the information projection display begins within 20 seconds after the autonomous driving level changes from 3 to 4, it may be 5 seconds, 10 seconds, or 15 seconds later. Furthermore, since during autonomous driving, the driver often does not pay attention to the area ahead of the vehicle, i.e., the area where the information projection display is performed, or pays attention for a shorter period of time than during manual driving, control may be performed to lengthen the display time of each piece of information compared to information projection for the driver during manual driving, thereby increasing the likelihood that the driver will see the information for the driver even during autonomous driving.

[0214] Fig. 25 shows a typical display example of Category 1. Fig. 25(a) shows an example of a driving speed display, and Fig. 25(b) shows an example of a navigation display. As shown in Fig. 25(a), the driving speed display may display information related to the vehicle's driving speed. As shown in Fig. 25(b), the navigation display may display information related to travel guidance, etc.

[0215] FIG. 26 shows a representative display example of Category 1 related to autonomous driving. FIG. 26(c) shows an example of a driving operation request display, FIG. 26(d) shows an example of a driving state switching display, and FIG. 26(e) shows an example of a control state display. As shown in FIG. 26(c), the driving operation request display may display, for example, information requesting manual driving (in this example, a symbol). As shown in FIG. 26(d), the driving state switching display may display information regarding switching from manual driving to autonomous driving. As shown in FIG. 26(e), the control state display may display information regarding changes in the autonomous driving level.

[0216] FIG. 27 shows a representative display example of Category 1 related to autonomous driving. (f) of FIG. 27 shows an example of an object recognition display, and (g) of FIG. 27 shows an example of a roadway information display. As shown in (f) of FIG. 27, the object recognition display displays information about objects such as people, bicycles, and cars. Here, information about the object's position, the object's movement direction, etc. may be displayed using an image such as an arrow. As shown in (g) of FIG. 27, the roadway information display may display information about changes to the roadway. Here, "dedicated" in the figure means a road dedicated to autonomous driving, and "general" means a general road that is different from a road dedicated to autonomous driving.

[0217] FIG. 28 shows a representative display example of Category 2. FIG. 28(a) shows an example of a right turn display, FIG. 28(b) shows an example of an abnormality notification display, and FIG. 28(c) and FIG. 28(d) show examples of a driving status display. Note that (a) is a general display, (b) and (c) are displays related to automatic driving, and (d) is a modified example. As shown in FIG. 28(a), the right turn display displays information to inform the surrounding area that the vehicle will turn right. As shown in FIG. 28(b), the abnormality notification display displays information to inform the surrounding area that an abnormality has occurred in the vehicle. For example, a caution symbol may be displayed. As shown in FIG. 28(c), the driving status display displays information to inform the surrounding area whether the vehicle is being driven manually or automatically. Note that as shown in FIG. 28(d), information on whether the vehicle is being driven manually or automatically may be displayed to the surrounding area using a lighting color rather than a road surface display. In (d) of Figure 28, as an example, the lights can be turned on in at least one location among the front of the vehicle, the side of the vehicle (e.g., the door mirror area), and the rear of the vehicle, and the lights are shown by hatching for convenience.

[0218] The display for the surroundings during autonomous driving serves as a substitute for the eye contact that is used to communicate between the driver and surrounding people during manual driving. For example, even if the driver (who is not actually driving) makes eye contact with surrounding people during autonomous driving, there is no guarantee that the vehicle will operate according to the driver's intentions, because it is the vehicle controller 100, not the driver, that controls the vehicle. To solve this problem, it is preferable to control the timing of the display for the surroundings during autonomous driving from the time it is determined that a display for the surroundings is necessary to the time it starts to be displayed as soon as possible compared to manual driving, so that surrounding people can notice the display earlier, reliably see the display, and understand how the vehicle will operate.

[0219] Next, an example of image display processing that takes into account the driver's orientation will be described with reference to FIGS. 29 and 30. As shown in FIG. 29A, when a vehicle is traveling, the driver basically looks ahead of the vehicle. However, as shown in FIG. 29B, during autonomous driving, it is considered that the driver does not need to face the front of the vehicle. Therefore, with regard to the display of Category 1 information, if the driver is not facing the front of the vehicle, it may be possible to not project the information. In other words, even if the display conditions for the driver are met, if the driver is not looking in the direction of the display, it may not be necessary to display the information. Alternatively, if the driver is not looking in the direction of the display, it may not be necessary to check whether the display conditions for the driver are met.

[0220] The controller 100 or the projection device 11 may acquire information from a camera 520a installed inside the vehicle (for example, on the ceiling of the vehicle) that photographs the driver, an in-vehicle camera 520, a gaze detection sensor, etc., and may check the driver's orientation, gaze direction, etc. to determine whether or not display is necessary. However, if the driver's gaze shifts and / or temporarily moves away from the display direction (i.e., the front of the vehicle), a display for the driver is performed.

[0221] Next, an example of processing related to display for the driver (Category 1) will be described with reference to FIG. 30 . As described above, step S21 checks whether the display start condition for Category 1 is met. Step S22 checks the driver's orientation. If it is confirmed that the driver is facing forward of the vehicle, step S23 starts display related to Category 1. On the other hand, if it is confirmed that the driver is not facing forward of the vehicle, step S24 does not display related to Category 1.

[0222] Next, an example of a process for displaying an image taking into consideration people in the surroundings will be described with reference to FIG. 31 . As shown in FIG. 31A , when driving a vehicle, if people around the vehicle or other vehicles around the vehicle are facing the image display direction, a display for the surroundings (i.e., a display related to Category 2) needs to be performed. The projection position of the image in the projection area may differ depending on the locations of the people around the vehicle and other vehicles around the vehicle relative to the vehicle. On the other hand, as shown in FIG. 31B , if people around the vehicle or other vehicles around the vehicle are not facing the image display direction, a display for the surroundings does not need to be performed.

[0223] However, as shown in Figure 31C, if multiple objects are confirmed on the image display direction side of the vehicle and at least one of the objects is facing the image display direction side (in this example, the image display direction side in front of the vehicle), it is necessary to display an image directed toward the surroundings. On the other hand, if all of the objects confirmed on the image display direction side of the vehicle are not facing the image display direction side of the vehicle (in this example, the image display direction in front of the vehicle), it is not necessary to display an image directed toward the surroundings. For example, if person A and person B are confirmed on the front side of the vehicle, and person A is not facing the image display direction, but person B is facing the image display direction, the projection device 11 displays an image directed toward the surroundings.

[0224] Here, the image display direction in front of the vehicle has been described as an example, but the same applies when displaying images rearward or to the side of the vehicle. That is, when multiple objects are identified behind the vehicle and at least one of the objects faces the image display direction (in this example, the image display direction behind the vehicle), it is necessary to display images directed toward the surroundings toward the rear of the vehicle. Also, when multiple objects are identified to the side of the vehicle and at least one of the objects faces the image display direction (in this example, the image display direction to the side of the vehicle), it is necessary to display images directed toward the surroundings toward the side of the vehicle. On the other hand, when all of the identified objects are not facing the image display direction, it is not necessary to display images directed toward the surroundings.

[0225] Next, with reference to FIG. 32 , an example of image display processing that takes surrounding vehicles into consideration will be described. When a vehicle is traveling and a nearby vehicle is detected, if the vehicle is being driven manually or semi-automatically, the driver of the vehicle may view the image display. Therefore, as shown in FIG. 32A , the controller 100 or the projection device 11 acquires information on the vehicle's control status, and if the other vehicle is being driven manually or semi-automatically, the projection device 11 displays a display for the surrounding vehicle. The control status of the other vehicle can be acquired, for example, through vehicle-to-vehicle communication. Here, information on the driver's status (such as line of sight) may be acquired in addition to the control status, and the controller 100 or the projection device 11 may use the acquired information to determine the driving status of the other vehicle. For example, as shown in FIG. 32A , the host vehicle acquires information on an oncoming vehicle through vehicle-to-vehicle communication, and the controller 100 or the projection device 11 determines that the oncoming vehicle is being driven manually. Then, the controller 100 or the projection device 11 may determine to display an image and display a message to the surroundings (in this case, a right turn message) according to the display conditions.

[0226] On the other hand, even if a vehicle is detected in the vicinity, if the vehicle is operating autonomously (e.g., fully autonomously or nearly fully autonomously), the driver of the vehicle may not see the video display, and therefore the video display may not be performed. Therefore, as shown in FIG. 32B , the controller 100 or the projection device 11 acquires information about the vehicle's control state. If the other vehicle is operating autonomously (e.g., fully autonomously or nearly fully autonomously), the projection device 11 may not display a display for the other vehicle. Similarly to the above, the control state of the other vehicle may be acquired, for example, through vehicle-to-vehicle communication. Furthermore, information about the driver's state (such as line of sight) may be acquired in addition to the control state to determine the driving state of the other vehicle. For example, as shown in the same figure, the host vehicle acquires information about an oncoming vehicle through vehicle-to-vehicle communication, and the controller 100 or the projection device 11 determines that the oncoming vehicle is operating autonomously. The projection device 11 may not display a display for the oncoming vehicle.

[0227] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, for example, other configurations may be added, deleted, or replaced with part of the configuration of the embodiment.

[0228] When a pair of devices such as lights used to display images is provided on the left and right sides of a vehicle, images may be displayed using one of the devices on one side or both of the devices on both sides. Here, the controller 100 or the projection device 11 may determine the device to use for information projection based on the positions of surrounding people, the orientation of surrounding people, the positions of surrounding vehicles, etc. Then, the projection device 11 may display images using the determined device.

[0229] For example, the processor may perform processing that takes into consideration standby time management, such as determining that the signal is ON when a signal continues for a certain period of time and OFF when the signal no longer continues for a certain period of time, and then performing processing.

[0230] Furthermore, by using the technology according to the above-described embodiment, it is possible to appropriately display necessary information, such as images for the driver and images for people around the vehicle. Furthermore, it is possible to appropriately display necessary information during both manual and autonomous driving. This makes it possible to provide an information projection device that contributes to safe driving, etc. Accordingly, it is possible to reduce traffic accidents. Furthermore, it is possible to contribute to "3. Good health and well-being for all" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0231] 2 Vehicle 11 Projection device 13 Headlamp 14 Turn indicator 100 Controller

Claims

1. An information projection method for displaying information, comprising: a determination step for determining the driving state of a vehicle based on the driving state of the vehicle; and a display step, mounted on the vehicle, for projecting or displaying information about the vehicle based on the determination result determined in the determination step, wherein, in the display step, if it is determined that the vehicle is in an autonomous driving state, information about autonomous driving is displayed based on the display conditions for autonomous driving.

2. An information projection method according to claim 1, wherein the display step displays information relating to the vehicle switching between automatic driving and manual driving.

3. An information projection method according to claim 1, wherein in the display step, when the vehicle is switching between automatic driving and manual driving, the information is not displayed for a predetermined time.

4. An information projection method according to claim 1, wherein in the display step, a display relating to an autonomous driving level is performed based on the display conditions during autonomous driving.

5. An information projection method according to claim 1, wherein in the display step, when the level of autonomous driving is changed from a high level to a low level, a display different from a normal display is performed.

6. An information projection method as described in claim 1, wherein in the display step, when an abnormality occurs during the automated driving, a driving operation request is displayed, and if the driver of the vehicle does not respond to the driving operation request within a predetermined time, information regarding a change in the display of the driving operation request is displayed.

7. An information projection method according to claim 1, wherein in the display step, a display relating to the route along which the vehicle is traveling is performed based on the display conditions.

8. An information projection method as described in claim 1, wherein, in the display step, when the vehicle changes course between a general road and a road exclusively for autonomous driving, the direction indicators of the vehicle turn on for a predetermined period of time, and then information relating to the road exclusively for autonomous driving is displayed.

9. An information projection method according to claim 1, wherein, in the display step, if an abnormality occurs in the control state of the vehicle, information relating to the abnormality is displayed based on the display conditions.

10. An information projection method according to claim 1, wherein in the display step, it is determined whether the driver of the vehicle is facing a display direction in which information is to be displayed, and if the driver is not facing the display direction, the information is not displayed.

11. An information projection method as described in claim 2, wherein in the display step, the state in which the vehicle displays information about switching the driving state between autonomous driving and manual driving is performed when a request to change the autonomous driving level is received from the vehicle, or when the vehicle determines that the autonomous driving level can be changed.

12. An information projection method according to claim 5, wherein the display different from the normal display is performed at a predetermined time when the autonomous driving level is changed.

13. An information projection device that displays information, comprising: an acquisition unit that acquires vehicle information; and a projection unit that displays an image including the vehicle information, wherein, when the control unit determines that the vehicle is in an autonomous driving state, the projection unit displays information related to autonomous driving based on display conditions for autonomous driving.

Images