Information display device

Abstract

This information display device is worn on the head, and includes a processor and a display for displaying images. The processor displays images in a normal display mode in which normal display is performed or in a driving mode used when driving a vehicle. In the driving mode, the processor restricts the display of images as compared to the normal display mode.

Description

information display device

[0001] The present invention relates to an information display device.

[0002] 2. Description of the Related Art Techniques for using a wearable device while driving a vehicle are known.

[0003] For example, Patent Document 1 states, "Wearable device 1 is used together with in-vehicle equipment when a user is in a vehicle. Wearable device 1 is a glasses-type wearable device. Control unit 30 of wearable device 1 controls display unit 27 to display information from display unit 27 in the user's field of vision. The in-vehicle equipment is equipped with a head-up display that uses the windshield as a display surface and displays information on the windshield. Control unit 30 controls display unit 27 so that the head-up display and display unit 27 do not display the same type of information overlapping in the user's field of vision."

[0004] Furthermore, for example, Patent Document 2 states that "The display system of the present invention comprises a wearable device 120 that is worn on the user's body and displays information in the user's field of view P, a detection means for detecting the direction of the user's field of view P, and a control means for controlling the display of the wearable device based on the positional relationship between the direction of the user's field of view P and a pillar Q1 of the vehicle. When the pillar Q1 is included in the field of view P, the control means displays information J and K in the field of view P, and when the pillar 110 is not included, the control means displays only information J in the field of view P."

[0005] JP 2015-210580 A JP 2019-119357 A

[0006] When using an information display device worn on a user's head while driving a vehicle, it is necessary to provide a technology that enables more appropriate display.

[0007] According to a first aspect of the present invention, there is provided an information display device as follows. This information display device is a head-mounted device. The information display device includes a processor and a display that displays images. The processor displays images in a normal display mode that provides normal display, or in a driving mode that is used when driving a vehicle, and in the driving mode, the processor imposes restrictions on the image display in the normal display mode.

[0008] According to a second aspect of the present invention, there is provided an information display device as follows. The information display device is a device worn on the head. The information display device includes a processor and a display that displays an image. The processor is capable of executing a process of determining a driving state, a process of restricting display when the driving state is determined, a process of displaying an image, and a process of determining an end of the display restriction while driving.

[0009] According to the present invention, a technology is provided that enables more appropriate display when using an information display device that is worn on the user's head while driving a vehicle. Note that problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention.

[0010] 1 is a block diagram showing an example of the configuration of an information display device. A diagram showing an example of the appearance of the information display device. A diagram showing an example of image display by the information display device. A diagram showing an example of image display by an information projection device. A diagram for explaining an example of display control. A flowchart showing an example of processing by the information display device. A flowchart showing an example of processing for determining whether to activate a driving mode. A diagram showing a specific example of determination in S11. A diagram showing specific examples of determinations in S13, S14, and S15. A flowchart showing an example of processing for display area control. A diagram showing a specific example of S21. An example of a display area based on the processing shown in FIG. 11. A diagram showing a specific example of S22. A diagram showing an example of display based on the processing by S22. A diagram for explaining an example of the type of display image, whether it can be displayed in the display area, etc. A diagram showing an example of processing with display restriction in S22. A diagram for explaining an example of display with display restriction in S22. A diagram showing an example of image processing in S2. A diagram showing a specific example of S23. A diagram for explaining a display coordinate system. A diagram for explaining a display coordinate system. A diagram for explaining an example of display in each different display coordinate system. A diagram for explaining correction of image movement amount relative to head movement amount. A flowchart showing an example of processing for display adjustment control. FIG. 1 is a diagram showing a processing example in S31. FIG. 2 is a diagram showing a specific example in S32. FIG. 3 is a flowchart showing a processing example of driving mode termination determination. FIG. 4 is a diagram showing a specific example in S51. FIG. 5 is a diagram showing a specific example in S52. FIG. 6 is a diagram showing an example of divided areas and display. FIG. 7 is a diagram showing an example of divided areas and display. FIG. 8 is a diagram showing an example of divided areas and display. FIG. 9 is a flowchart showing processing of a modified example.

[0011] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are merely examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural. The position, size, shape, and range of each component shown in the drawings may not represent the actual position, size, shape, and range in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, and range disclosed in the drawings. While various types of information may be described using terms such as "table," "list," and "queue," various types of information may also be expressed using other data structures. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be referred to as "XX information." When describing identification information, terms such as "identification information," "identifier," "name," "ID," and "number" are used, but these terms are interchangeable. When there are multiple components having the same or similar functions, they may be described using the same reference numeral with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. In the embodiments, processing performed by executing a program may be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., a memory) and interface devices (e.g., a communication port). Therefore, the entity that executes the program and performs the processing may be the processor. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. Similarly, the entity that executes the program and performs the processing may be a controller, device, system, computer, or node that has a processor. The entity that executes the program and performs the processing may be any processing unit, and may include a dedicated circuit that performs specific processing.Here, the dedicated circuit may be, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a complex programmable logic device (CPLD). A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Furthermore, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0012] When driving a vehicle, if an information display device that is worn on the head and provides an augmented display and an information projection device mounted on the vehicle are used to display information, the image displayed by the information display device may interfere with driving. Therefore, in the embodiments, a technology that enables more appropriate display when using an information display device while driving is described. This makes it possible, for example, to display images that do not interfere with driving.

[0013] The information display device is, for example, a head-mounted display, AR glasses, etc. The information projection device is a device that is mounted on a vehicle and displays images, for example, a head-up display, an information projection headlight, etc.

[0014] First, an example of the configuration of an information display device will be described with reference to Fig. 1. The information display device 1 is a device capable of displaying information based on AR (Augmented Reality). As shown in Fig. 1, the information display device 1 includes a processor 101, a storage device 110, an input I / F 120, a video input / output device 130, an audio input / output device 140, a sensor group 150, a communication I / F 160, an expansion I / F 171, a timer 172, and an actuator 173.

[0015] The processor 101 is configured using a CPU (Central Processing Unit) and the like, and is connected to various components via a bus 102 .

[0016] The storage device 110 includes a volatile memory 111 and a non-volatile memory 112. The volatile memory 111 is a main storage device and is configured using, for example, RAM (Random Access Memory). The processor 101 temporarily stores data such as programs in the volatile memory 111 and executes data processing. The non-volatile memory 112 is an auxiliary storage device that stores data in a non-volatile manner. The non-volatile memory 112 is configured using a non-volatile storage medium and stores data such as programs. The non-volatile memory 112 stores, for example, a basic operation PRG 112a, an AR viewing PRG 112b, and an approach transmission PRG 112c. The basic operation PRG 112a is, for example, a program related to an OS (Operating System). The AR viewing PRG 112b is a program used for viewing AR content. The proximity transmission program 112c is a program for transmitting proximity information to the outside using near-field communication. These programs may be separate programs. However, this is not a limitation, and one program may include a basic operation function, an AR viewing function, and a proximity transmission function and provide these functions. Alternatively, different separate programs may provide the above functions. Furthermore, these programs and data stored in the storage device 110 may be modified or newly added using the communication I / F 160 described below.

[0017] The input I / F 120 is an interface used by the user for inputting operations, and information desired by the user is input via the input I / F 120. The input I / F 120 may include, for example, button switches 121 including a power button, a volume button, etc., and may be configured to accept user operations when the user operates a predetermined button switch 121. The input I / F 120 may also be configured to accept user operations based on detection of the user's gaze, detection of the user's hand, detection of the user's gesture, detection of a predetermined sound, etc. The input I / F 120 may also be configured to accept user operations by the user operating a pointer on a display. Furthermore, the input I / F 120 may also be configured to accept user operations by operating an operating device connected via an expansion I / F 171, which will be described later.

[0018] The video input / output device 130 includes a display 131, an image signal processing unit 132, and an outer camera 133. The display 131 is configured to output video. The display 131 is a transmissive display, and a user wearing the information display device 1 can view the outside world through the display 131. The display 131 can display, for example, an image (AR image) superimposed on the outside world, a pointer used for user operation, and the like. The image signal processing unit 132 is configured to process image signals, and can be configured using, for example, a VRAM (video RAM), a dedicated image processing circuit (LSI), an image (video) signal processor, software, and the like. The outer camera 133 is configured to capture images of the outside world and is a camera unit that inputs image data of an object in the outside world by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The outer camera 133 captures the user's hands, gestures, etc., and the information display device 1 may acquire the image data captured by the outer camera 133 as instruction information such as input operations and execute predetermined processing. The display 131 may be a non-transparent display that displays images from a camera capturing the outside world, such as the outer camera 133, allowing the user to view the outside world. Although not shown in the figure, the device may also be equipped with an inner camera that captures the user's eyes.

[0019] The audio input / output device 140 includes a speaker 141, an audio signal processing unit 142, and a microphone 143. The speaker 141 outputs, for example, audio (sound) of content played back when experiencing the information display device 1. The speaker 141 can also notify the user of various notification information by audio. The audio signal processing unit 142 is configured to process audio signals and can be configured using, for example, a circuit board, an audio signal processor, software, etc. The microphone 143 collects the user's own voice, external sounds, etc., and converts them into audio data. The user may speak voices related to instructions such as input operations, and the information display device 1 may acquire the audio data collected by the microphone as instruction information for input operations, etc., and execute predetermined processing.

[0020] The sensor group 150 may include, for example, a positioning sensor 151, a geomagnetic sensor 152, a distance measurement sensor 153, an acceleration sensor 154, a gyro sensor 155, and a gaze detection sensor 156. The sensor group 150 may include sensors of different types than those described above. The sensor group 150 may also be configured such that the above sensors are appropriately omitted. The sensor group 150 may also be omitted from the information display device 1, and sensors used for processing may be appropriately provided.

[0021] The positioning sensor 151 is a GNSS sensor. The positioning sensor 151 is a device that receives signals from GNSS (Global Navigation Satellite System) satellites in the sky and is used to detect the current position of the information display device 1. Using the positioning sensor 151, the information display device 1 can detect the position of the information display device 1 (in other words, the position of the user wearing the information display device 1).

[0022] The geomagnetic sensor 152 is a sensor that detects the magnetic force of the Earth and the direction in which the information display device 1 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions can be used, and by capturing changes in geomagnetism in response to the movement of the information display device 1, it is possible to detect the movement of the information display device 1. This makes it possible to detect the posture state of the user wearing the information display device 1.

[0023] The ranging sensor 153 is a sensor that can measure the distance from the information display device 1 to an object, the object's position, and capture the three-dimensional shape of the object. Examples of the ranging sensor 153 include a LiDAR (Light Detection and Ranging) sensor that irradiates an object with laser light such as infrared light and measures the scattered light that bounces back; a TOF (Time of Flight) sensor that measures distance by measuring the reflection time of pulsed light irradiated on the object for each pixel; and a millimeter-wave radar that emits millimeter-wave radio waves and captures the reflected waves. The ranging sensor 153 may also be a sensor that performs measurements based on the angle at which light reflected by the object is received. That is, the ranging sensor 153 may be a triangulation sensor. Furthermore, in the information display device 1, the ranging sensor 153 may be configured as a stereo camera that performs measurements based on parallax images.

[0024] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The acceleration sensor 154 can detect the tilt and orientation of the information display device 1 worn by the user. The gyro sensor 155 is a sensor that detects angular velocity in the rotation direction, and can detect the vertical, horizontal, and diagonal orientation states. Therefore, the acceleration sensor 154 and the gyro sensor 155 can be used to detect the orientation, such as the tilt and orientation, of the information display device 1.

[0025] The gaze detection sensor 156 may include a left eye gaze sensor that detects the gaze of the left eye and a right eye gaze sensor that detects the gaze of the right eye, and can detect the movement and direction of the left and right eyes to capture the user's gaze point. The information display device 1 may perform eye tracking using the gaze detection sensor 156 and, for example, acquire the detection result as input information. The information display device 1 may then execute predetermined processing based on this input information. For example, the information display device 1 may detect the user's gaze toward a predetermined display on the display and acquire the detection result as input information to execute predetermined processing corresponding to the display. The gaze may also be detected from an image of the user's eyes acquired by an in-camera. The in-camera may be a camera that captures images in the visible light range or a camera that captures images in the infrared light range. Note that, if the camera captures images in the infrared light range, an infrared LED may be provided as illumination.

[0026] The communication I / F 160 is an interface used for communication, and includes, for example, a LAN communication I / F 161 , a short-range wireless communication I / F 162 , and a telephone network communication I / F 163 .

[0027] The LAN communication I / F 161 is an interface used for communication via a LAN (Local Area Network). The LAN communication I / F 161 is connected to a network via an access point (AP) device by a wireless connection such as Wi-Fi (registered trademark), and transmits and receives data to and from other devices on the network.

[0028] The short-range wireless communication I / F 162 is a communication interface that performs short-range wireless communication with a device within a range where short-range wireless communication is possible. The short-range wireless communication is performed using, for example, but not limited to, an electronic tag. If the information display device 1 is located near a device that is capable of at least wireless communication, the short-range wireless communication may be performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or a wireless LAN (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g).

[0029] The telephone network communication I / F 163 is a third-generation mobile communication system (hereinafter referred to as "3G") such as the GSM (registered trademark) (Global System for Mobile Communications), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, or UMTS (Universal Mobile Telecommunications System), or a communication system called the LTE (Long Term Evolution), fourth generation (4G), or fifth generation (5G), and is connected to a communication network through a base station using a mobile communication network to send and receive information to and from a server on the communication network.

[0030] The expansion I / F 171 is an interface for expanding the functions of the information display device 1, and is, for example, a USB device connection terminal. Various devices can be connected to the expansion I / F 171, and the expansion I / F 171 is used, for example, for transmission / reception, charging, etc. For example, an operation device separate from the information display device 1, such as a keyboard, key buttons, or touch keys, may be connected to the expansion I / F 171. The information display device 1 may then acquire content input to the operation device as instruction information for input operations and execute predetermined processing. The expansion I / F 171 may be configured to connect devices via wired or wireless connections.

[0031] The timer 172 is a timer that holds the current time in the real world, and holds a time such as Coordinated Universal Time (UTC) as the current time. The timer may be configured as software, or may be configured using an RTC (Real Time Clock). The actuator 173 conveys physical movement such as vibration to the user. The actuator may be electrically operated, or may use a piezoelectric element, magnetic force, or hydraulic pressure, as long as it generates a physical force.

[0032] Next, an example of the configuration of an information display device will be described with reference to Fig. 2. In this example, the information display device 1 is an eyeglass-type HMD (head-mounted display) and has the same configuration as that described above.

[0033] 2, an outer camera is disposed in front of the eyeglass-type frame 193. That is, outer camera 133L is disposed on the front left side of frame 193, and outer camera 133R is disposed on the front right side of frame 193. Furthermore, outer camera 133F and distance measurement sensor 153 are disposed in the front center of the frame. Furthermore, a left display 131L, which is a transmissive display on the left side, and a right display 131R, which is a transmissive display on the right side, are disposed in front of frame 193.

[0034] Furthermore, a microphone 143 and a left speaker 141L are arranged on the left side of the frame 193, and a right speaker 141R, a sensor group 150, a communication I / F 160, and a control device 192 are arranged on the right side of the frame 193. Here, the control device 192 is a device that implements the processor 101, the storage device 110, the input I / F 120, the expansion I / F 171, the timer 172, and the actuator 173. Furthermore, a gaze detection sensor 156 is arranged on the inside of the front part of the frame 193.

[0035] 2, the outer cameras (133L, 133R, 133F) are components related to the outer camera 133. The left display 131L and the right display 131R are components related to the display 131. The speakers (141L, 141R) are components related to the speaker 141.

[0036] Next, an example of video display within the user's field of view will be described with reference to FIGS. 3 and 4. FIG. 3 shows an example of video display by an information display device. A user wearing the information display device 1 on their head can view the scenery in front of the user via the display 131 of the information display device 1. Furthermore, the user can view AR video that is displayed on the display 131 of the information display device 1 and superimposed on the scenery in front of the user. Here, the information display device 1 displays, for example, navigation information to a destination set by the user. Furthermore, the information display device 1 displays, for example, video of music content to be viewed using the information display device 1, and video used to operate the music content (play, rewind, fast forward, stop, etc.).

[0037] 4 shows an example of an image displayed by an information projection device (in this example, a head-up display). As shown in (a) and (b) of FIG. 4, a user 13 can view an image I (virtual image) through the windshield 12 of the vehicle 11, the image I being based on image light emitted by the information projection device P and reflected by the windshield 12 of the vehicle 11. Here, as shown in (b) of the same figure, a display range PA in which the information projection device P can display an image is limited to a portion of the field of view FA of the user 13.

[0038] In order to widen the display range PA of an information projection device P, such as a head-up display or an information projection light, the information projection device P must be enlarged, which makes it difficult to install the information projection device P in the vehicle 11. This limits the display range PA of the image projected by the information projection device P. Here, the driver cannot see the image projected by the information projection device P unless, for example, the driver is looking at a limited range ahead of the vehicle. This is because the information projection device P is fixed to the vehicle 11 and does not move in conjunction with the driver's viewpoint. On the other hand, the information display device 1 is, for example, a head-mounted display or AR glasses, and the positional relationship between the display 131 and the user is fixed, making it possible to display an image in conjunction with the user's field of view. Therefore, by using the information display device 1, the problem of the limited image display area of ​​the information projection device P is solved, and a wide range of information can be displayed to the driver.

[0039] However, using the information display device 1 and the information projection device P may, for example, interfere with driving. For example, if the image from the information display device 1 is displayed in a portion of the field of view that the driver is focused on while driving, it may be difficult to see the surroundings. Furthermore, if the image from the information display device 1 and the image from the information projection device P interfere with the driver's field of view, the driver may not be able to recognize both images or may misrecognize them. Furthermore, compared to normal use, when driving, not only the driver himself or herself moves, but also surrounding vehicles and pedestrians. Therefore, for example, the driver is required to judge the surroundings more quickly than during normal use. However, if the driver stares at the image from the information display device 1 that is not related to driving for a long period of time, the driver may be distracted, which may delay judgment and impair safety. Furthermore, for example, images other than those related to driving, such as emails and the time, displayed on the information display device 1 may be linked to the user's head movement, but if the image related to driving displayed on the information display device 1 is linked to the user's head movement, the driving task may be impaired. Therefore, a technology for solving such problems will be described in detail with reference to Figures 5 and subsequent figures.

[0040] FIG. 5 is a diagram illustrating an example of information display control. Here, FIG. 5 shows a vehicle window from the driver's seat from the driver's viewpoint. In this example, the information display device 1 is an AR glass. In this example, an image through the AR glass seen through one eye is displayed. The AR glass may be AR glass worn on one eye, or AR glass worn on both eyes. The user can view the external environment while driving through the AR glass.

[0041] When driving a vehicle, the information display device 1 divides the display area of ​​the information display device 1 (head-mounted display terminal) into multiple areas, separating the area necessary for the user to know the external situation from other areas. The information display device 1 also controls the display of the areas to restrict the areas. As shown in FIG. 5 , the information display device 1 divides the display area of ​​the information display device 1 into, for example, Zone A, which is the area necessary for knowing the external situation, and Zone B, which is the other area. In the example shown in the figure, the information display device 1 is set not to display any images in Zone A to prevent the image display from obscuring the surrounding environment. In Zone B, the information display device 1 displays images, but only images that are permitted to be displayed. The permitted images here refer to images that are permitted to be displayed in advance, or images that are determined by the information display device 1 or a server connected to the information display device 1 via a network to meet display requirements.

[0042] The above control allows the image display to continue without interfering with the user's driving operation, for example, even when the user is driving. In another example, the image display in Zone A is permitted, but the displayed image is changed to one for driving, or the number of images displayed is limited, so that the image display does not limit the user's driving field of view, which allows the user to grasp the external situation, and the occupancy rate of the image per unit area is set to limit the display.

[0043] Next, an example of processing of the information display device will be described with reference to Fig. 6. Processing starts (S1000), and the information display device 1 performs a driving mode activation determination, which is a determination to start control related to limiting the display area (S1). Note that, in contrast to the normal display mode in which normal display is performed, display control in which the display area is limited is sometimes called the driving mode.

[0044] For example, the information display device 1 determines whether the user is driving the vehicle 11, and if the user is driving, determines whether to activate the driving mode.

[0045] Furthermore, the information display device 1 may determine whether to activate the driving mode, for example, when the information display device 1 has a driving mode function and can be switched to the driving mode. Furthermore, the information display device 1 may check, for example, the usage setting for the driving mode and determine whether to activate the driving mode in accordance with the usage setting. Furthermore, the information display device 1 may determine whether to activate the driving mode in accordance with, for example, a user's ON operation.

[0046] An example of the process of determining whether to start the operation mode (S1) will be described in detail later with reference to the drawings.

[0047] When the information display device 1 determines that the driving mode should be activated, the information display device 1 controls the display area in accordance with the activation of the driving mode (S230). The information display device executes, for example, display area control (S2) and display adjustment (S3). Note that the order of the processes in S230 may be reversed.

[0048] The information display device 1 resets the display area in the processing of S2. For example, the information display device 1 divides the area horizontally (left and right on the display screen) and resets the display area extending in the up and down directions of the display screen. Note that while driving, the user often moves their line of sight horizontally (left and right on the display screen) to look around. Therefore, the area may be divided vertically (up and down on the display screen) to form an area extending in the left and right directions of the display screen. Furthermore, the information display device 1 may divide the area concentrically, for example, and reset a display area for a central area of ​​the field of view and a display area for a peripheral area. Furthermore, the information display device 1 can perform area division taking into account the relationship with the vehicle 11.

[0049] The information display device 1 sets a display restriction in the processing of S2. For example, the information display device 1 may impose a display restriction on an area in which the surrounding environment is visually recognized while driving, with the aim of ensuring a clear field of view. Note that the display restriction may be, for example, a setting that prohibits display within the area, or a setting that prohibits display of a specific object within the area. Furthermore, the information display device 1 may change (reset) the display coordinate system in the processing of S2.

[0050] An example of the display area control process (S2) will be described in detail later with reference to the drawings.

[0051] The information display device 1 performs a display adjustment process in the process of S3. The information display device 1 adjusts the display by the application during standard (normal) operation to correspond to the driving mode. For example, the information display device 1 ends the display or adjusts the display position according to the area of ​​the image used during standard (normal) operation and the display content of the image. In addition, the information display device 1 adjusts, for example, by moving a display that is not related to driving within the area in which the surrounding environment is viewed while driving to another area.

[0052] Furthermore, the information display device 1 adjusts with an in-vehicle information projection device P that displays information using, for example, the windshield 12 of the vehicle 11. When a display by an in-vehicle information projection device P, such as a head-up display or an information projection headlight, is within the user's field of view, the information display device 1 adjusts the display position, stops the display, or the like.

[0053] An example of the display adjustment process (S3) will be described in detail later with reference to the drawings.

[0054] The information display device 1 starts displaying in the driving mode based on the settings made in S230 (S2 and S3 in this example) (S4). Then, the information display device 1 determines whether to terminate the driving mode (S5). If the termination determination is not satisfied, the information display device 1 continues displaying in the driving mode. However, the information display device 1 may temporarily suspend the driving mode depending on the situation. If the termination determination is satisfied, the information display device 1 terminates displaying in the driving mode (S6), and the process ends (S1001).

[0055] An example of the process of determining whether the operation mode is finished (S5) will be described in detail later with reference to the drawings.

[0056] Next, an example of the process of determining whether to start the operation mode (S1) will be described in detail with reference to FIGS.

[0057] 7 is a flowchart showing an example of the process of the drive mode activation determination (S1). When the process transitions from the previous step (S100) to the drive mode activation determination (S1), for example, as shown in (a) of the same figure, the information display device 1 performs the drive mode activation determination (S1), and, for example, performs a vehicle driving determination (S11) and a device-side determination (S12) as the drive mode activation determination. Then, depending on the determination result, the process transitions to the next step (S101).

[0058] FIG. 8 shows a specific example of the processing of S11. Example 1-1-1 is an example in which a change to driving mode is made by a user operation. In Example 1-1-1, the information processing device 1 determines whether condition (1A) and condition (1B) are met. Condition (1A) is that the information display device 1 (head-mounted information display terminal) is activated. Condition (1B) is that the information processing device 1 confirms that the driving mode has been changed by a user operation. The user can change to driving mode by an appropriate operation, such as operating the button switch 121 of the information display device 1 or selecting from a menu screen on the display. Alternatively, the user's voice may be detected by the microphone 143 and the operation to change to driving mode may be detected by the outer camera 133 and the operation to change to driving mode may be performed. If the information processing device 1 determines that condition (1A) and condition (1B) are met, the information processing device 1 can proceed to the next step.

[0059] Example 1-1-2 is an example in which the driving mode is changed without user operation. Therefore, when Example 1-1-2 is satisfied, the driving mode can be changed automatically. In Example 1-1-2, the information processing device 1 determines whether condition (1A) and condition (1C) are met. Condition (1A) is the same as above. Condition (1C) is that the information display device 1 communicates with a vehicle or an in-vehicle device and detects that a user is in the vehicle. When the information processing device 1 determines that condition (1A) and condition (1C) are met, it can proceed to the next step.

[0060] Here, the information display device 1 may detect that the user is in the vehicle by being connected to an in-vehicle network (Wi-Fi, Bluetooth, etc.) or by a wired connection with the vehicle. The information display device 1 may also be connected to the in-vehicle network via a mobile information terminal such as a smartphone, or the vehicle 11 and the information display device 1 may be connected via the same server. The information display device 1 may detect that the user is in the vehicle by detecting a connection to software (OS) for linking with the vehicle's in-vehicle devices. The information display device 1 may also detect that the user is in the vehicle based on information acquired from an OBD (On Board Diagnosis). The information display device 1 may also acquire information detected by devices such as sensors installed in the vehicle (e.g., pressure sensors installed in the driver's seat, the floor of the driver's seat, a seat belt sensor, an accelerator sensor, a sensor for detecting steering wheel holding (i.e., steering wheel grip), an in-vehicle camera, a driver monitoring system), and so on.

[0061] Example 1-1-3 is an example in which a change to the driving mode is made without user operation. Therefore, when Example 1-1-3 is satisfied, a change to the driving mode can be made automatically. In Example 1-1-3, the information display device 1 determines whether condition (1A) and condition (1D) are met. Condition (1A) is the same as above. Condition (1D) is that the information display device 1 (head-mounted information display terminal) detects that the vehicle is being driven. When the information processing device 1 determines that conditions (1A) and (1D) are met, it can proceed to the next step. In an example in which a change to the driving mode is made without user operation, the determination of whether the user is in the driver's seat and the determination of whether the user is performing driving tasks can be made separately. As a result, when the user is in the driver's seat but not performing driving tasks, the display of the information display device 1 can be unrestricted and functions can be used.

[0062] Here, the information display device 1 may detect that the vehicle is being driven using a device included in the information display device 1. The information display device 1 may detect that the vehicle is being driven, for example, based on the detection result of the acceleration sensor 154. Furthermore, the information display device 1 may detect that the vehicle is being driven, for example, based on a change per unit time in the detection result (i.e., location information) of the positioning sensor 151. The information display device 1 may detect that the vehicle is being driven, for example, based on a change in the external scenery captured by the outer camera 133. Furthermore, the information display device 1 may detect that the vehicle is being driven by recognizing that the user is sitting in the driver's seat from, for example, an image captured by the outer camera 133 or a distance image (point cloud data) obtained from a ranging sensor. Furthermore, the information display device 1 may detect that the vehicle is being driven by combining the detection results of these multiple sensors. Furthermore, activation may be performed in multiple stages by combining the examples described in S11. For example, in the first step, it is determined whether the driving mode can be switched to without user operation, and if it is determined that the mode can be switched to, in the second step, the driving mode is changed to by user operation, etc. This makes it possible to prevent inadvertent mode switching due to user operation or the like.

[0063] 9 shows specific examples of the processes of S13, S14, and S15. Example 1-3-1 is an example of S13 in which a transition to a driving mode occurs when a driving mode is available. In Example 1-3-1, the information display device 1 determines whether a condition (1E) is met. The condition (1E) is that the information display device 1 has (exists) a driving mode. If the information processing device 1 determines that the condition (1E) is met, it can proceed to the next step.

[0064] If it is determined that the information display device 1 does not have a driving mode, the image displayed on the information display device 1 may occupy the user's field of view and interfere with driving. Therefore, in order to prioritize driving, the information display device 1 forcibly performs a process of not displaying anything and does not display anything until it detects that the user is not driving. Furthermore, for example, if it is determined that the user is driving, the information display device 1 may forcibly transition to a process of not displaying anything, thereby determining that there is no driving mode even when there is no display mode that can be used while driving. Here, the process of not displaying anything may be, for example, any of hiding, ending, and temporarily saving.

[0065] Example 1-4-1 is an example of S14 in which a determination is made as to whether or not the driving mode is being used. Here, it is determined whether or not a function is provided that allows the user to select whether or not to use the driving mode, and whether or not the system will automatically activate the driving mode. In Example 1-4-1, the information display device 1 determines condition (1F). Condition (1F) is whether or not a setting for using the driving mode exists. If the information processing device 1 determines that condition (1F) is satisfied, it can proceed to the next step. If it is determined that there is no determination as to whether or not the driving mode is being used, S14 is terminated and the process proceeds to S230.

[0066] Example 1-5-1 is an example of determining the use setting of the driving mode in S15. Here, it is determined whether the user has selected to use or not use the driving mode when driving. Some users may not want to restrict the image display of the information display device 1 when driving. In that case, the use setting of the driving mode may be set to non-use, and no image display restriction may be imposed, prohibiting image display altogether. Alternatively, during autonomous driving, the display restriction imposed by the driving mode may be lifted and image display may be allowed. In Example 1-5-1, the information display device 1 determines condition (1G). Condition (1G) is that the use setting of the driving mode is ON. If the information processing device 1 determines that condition (1G) is met, it can proceed to the next step.

[0067] In the above example 1-4-1, if an operation mode is available (exists), the process proceeds to the next step, realizing automatic processing.In contrast, in the example 1-5-1, the process proceeds to the next step when the ON / OFF that can be set by the user is referenced and the ON setting by the user is confirmed, realizing processing based on the user's intention.

[0068] Example 1-5-2 is an example of determining the use setting of the driving mode in S15. In Example 1-5-2, the information display device 1 determines whether condition (1H) and condition (1J) are met. Condition (1H) is that the use setting of the driving mode is OFF. Condition (1J) is that warning information is received. If the information processing device 1 determines that condition (1H) and condition (1J) are met, it can proceed to the next step.

[0069] In this example, even if the user's operation to turn off the device is confirmed, the information display device 1 receives warning information, and the process proceeds to the next step. This warning information may be received from an external device such as a vehicle or a mobile information terminal, or the information display device 1 may determine that a warning is to be issued based on information detected by a camera or sensor. In other words, if a serious danger is imminent for the user, the information display device 1 may activate a driving mode to display a warning, regardless of the setting state of the information display device 1, from the viewpoint that a warning needs to be displayed to the user. Furthermore, the warning displayed on the information display device 1 may be a flashing display, or may display a larger image than other display content. Furthermore, the display position of other images may be moved to a peripheral area, or the other images may be erased before the warning display is displayed. These effects can be achieved, making the warning display more noticeable to the user.

[0070] Here, the information display device 1 may activate the driving mode and display a warning when, for example, it receives a warning / attention alert regarding a risk of an accident around the vehicle as warning information. Furthermore, the information display device 1 may activate the driving mode and display a warning when, for example, it receives information indicating a risk of a pedestrian, bicycle, or other object jumping out into the road or a collision as warning information. Furthermore, the information display device 1 may activate the driving mode and display a warning when, for example, it receives information indicating that the distance between the vehicle and a surrounding vehicle or the distance between the vehicle and a pedestrian is below a specified value as warning information. Furthermore, the information display device 1 may activate the driving mode and display a warning when, for example, it receives information indicating that the vehicle is running out of its lane or driving in the wrong direction as warning information. When displaying the warning, audio information may be output from the speaker 141 or vibrations may be output by the actuator 173 in addition to the video display. In this case, the audio information or vibrations may be output to the user simultaneously with or before the video display, thereby making it easier for the user to recognize that a warning display has been displayed.

[0071] The information display device 1 may acquire the warning information using, for example, a sensor, the outer camera 133, or the communication I / F 160 provided in the information display device 1. Here, the information display device 1 may acquire the warning information based on information acquired by a sensor provided in the vehicle.

[0072] In the process (S1) of determining whether to start the driving mode, if, for example, at least one of the driving determinations (S11) (here, at least one of Example 1-1-1, Example 1-1-2, and Example 1-1-3) is true and at least one of the equipment determinations (S12) (here, at least one of Example 1-3-1, Example 1-4-1, Example 1-5-1, and Example 1-5-2) is true, the information display device 1 may proceed to the next step (S101). On the other hand, for example, if each example of the operation judgment (S11) (here, Example 1-1-1, Example 1-1-2, Example 1-1-3) is not established, or if at least one of the examples of the operation judgment is established and all of the examples of the equipment judgment (S12) (here, Example 1-3-1, Example 1-4-1, Example 1-5-1, Example 1-5-2) are not established, the process does not proceed to the next step (S101) and ends.

[0073] The information display device 1 may be configured to use any one of S13, S14, and S15 alone or a combination of them in the device determination (S12). Also, in the operation mode activation determination (S1), the order of the processes of S11 and S13 to S15 may be reversed.

[0074] The process is not limited to the above, and can be modified as appropriate. For example, as shown in FIG. 7B, the information display device 1 may be configured to perform only the determination of the driving state (S11) in the process of determining whether to activate the driving mode (S1). In this case, if at least one of the driving determinations (S11) (here, at least one of Examples 1-1-1, 1-1-2, and 1-1-3) is satisfied, the information display device 1 may proceed to the next step (S101).

[0075] For example, in S11, an external device such as an in-vehicle device of the vehicle 11 or a mobile information terminal may determine whether the user is in the vehicle or driving the vehicle, and may transmit information indicating that the user is in the vehicle or driving the vehicle to the information display device 1. Then, based on the acquired information, the information display device 1 may provide the user with guidance regarding permission to change the driving mode. Here, the manner of the guidance is not particularly limited, and may be, for example, guidance using a display, a voice, or the like. Then, for example, as described in Example 1-1-1, the driving mode may be changed by the user performing an operation.

[0076] Furthermore, in S11, it may be determined that the user is in the vehicle, that the vehicle is being driven, etc., using both the equipment included in the information display device 1 and external equipment such as an in-vehicle device or a mobile information terminal of the vehicle 11. For example, a sensor on the vehicle may detect that the user is in the vehicle, and a camera on the information display device 1 (head-mounted information display terminal) may detect that the user is seated in the driver's seat. Then, the information display device 1 may acquire the detection result and automatically switch to the driving mode as described in Examples 1-1-2 and 1-1-3.

[0077] Next, an example of the display area control (S2) will be described in detail with reference to FIGS.

[0078] 10 is a flowchart showing an example of processing in the display area control (S2). When the driving mode is activated and processing transitions from the previous step (S200) to the display area control (S2), for example, as shown in (a) of the same figure, the information display device 1 resets the display area (S21) and sets display restrictions on the display area (S22). Then, processing transitions to the next step (S201). Note that, as shown in (b) of the same figure, the information display device 1 may also set the coordinates of the display area (S23).

[0079] 11 and 12 show a specific example of the process of S21. In S21, when the driving mode is executed, the information display device 1 divides the display area into at least two areas. For example, it divides the display area into an area with a high priority for driving operations and another area.

[0080] FIG. 11 shows a processing example of an information display device. In Example 2-1-1, the information display device 1 divides the display area vertically (up and down on the display screen) into two or more areas and proceeds to the next step (2A). Dividing the area vertically allows for a wider area to be set horizontally (left and right on the display screen). In particular, during driving, users tend to move their faces horizontally more frequently than vertically, so it is desirable for the area to be wider horizontally. This separates the area used for driving from other areas. During driving, the user is required to focus on the environment outside the vehicle. Therefore, by setting areas important for driving, the images that can be displayed for each displayable area are clarified, allowing the user to simultaneously perform driving tasks and check other information. In other words, the necessary information can be provided to the user while prioritizing driving tasks.

[0081] In Example 2-1-2, the information display device 1 vertically divides the display area into two or more areas so that at least one area includes the vehicle's axis of travel and the driver's horizontal viewing position, and then proceeds to the next step (2B). Here, the vehicle's axis of travel is a direction that includes the velocity vector in a coordinate system based on the vehicle. When the vehicle is moving forward, it is the forward direction of the vehicle, and when the vehicle is moving backward, it is the rearward direction of the vehicle. Furthermore, when the vehicle is turning, it is a direction that includes the tangent direction of the curve's radius of curvature. This example adds limitations to the area setting compared to Example 2-1-1 above. In this example, the information display device 1 sets a portion of the user's field of view as an area to be used for driving tasks with high priority relative to driving, taking into consideration ensuring visibility in the vehicle's direction of travel. Specifically, this area includes the user's horizontal viewing position when seated, is in the direction of the vehicle's axis of travel, and is an area (range) that includes objects sharing the road, such as other vehicles and pedestrians, and road signs such as traffic lights. Typically, drivers drive by directing their gaze into the distance in the vehicle's direction of travel. Therefore, this area including the horizontal viewing position is the area where the driver frequently fixes their gaze to recognize the surrounding environment while driving, and is the most important area while driving. Ensuring a field of view in this area so as not to interfere with driving is important for safety. In the above, the line of sight is set to the direction of the vehicle's axis of travel, but since the user's line of sight moves in the turning direction when the vehicle turns, the position may be shifted from the direction of the vehicle's axis of travel to the turning direction. In this case, a process of returning the line of sight to the original position is also performed after the turn is completed.

[0082] In Example 2-1-3, the information display device 1 divides the display area into two or more concentric regions and proceeds to the next step (2C). The central region of the visual field has higher resolution than the surrounding regions (in other words, it is an area where the user can more easily see objects, etc. than the surrounding regions), and is an important area for, for example, monitoring danger. Therefore, in this example, the information display device 1 divides the display area into a central vision region (i.e., an area including the center of the display area) and a peripheral vision region (i.e., an area around the central vision region). Here, as an example, the central vision region is an area where driving is prioritized, and the information display device 1 displays information less relevant to driving in the peripheral vision region. In addition, the widths of the regions in the horizontal and vertical directions may be changed. As mentioned above, horizontal gaze movement is frequent when driving a vehicle. Therefore, by setting the horizontal region to a large size, the field of view can be secured during gaze movement.

[0083] In Example 2-1-4, the information display device 1 divides the display area into two or more concentric regions so that at least one region includes the vehicle's axis of travel and the driver's horizontal line of sight, and then proceeds to the next step (2D). This example adds limitations to the region setting compared to Example 2-1-3 above. In this example, the information display device 1 takes the vehicle's direction of travel into account and sets a portion of the user's field of view as a region to be used for driving tasks with high priority. Specifically, this region includes the user's horizontal line of sight when seated, is in the direction of the vehicle's axis of travel, and is an area (range) that includes objects sharing the road, such as other vehicles and pedestrians, and road signs such as traffic lights. Typically, drivers drive by directing their gaze toward a distant point in the vehicle's direction of travel. Therefore, this region, including the horizontal line of sight, is the region where the driver frequently focuses their gaze to recognize the surrounding environment while driving, making it the most important region during driving. Ensuring a clear field of view in this region without interfering with driving tasks is important for safety.

[0084] The horizontal viewing position is the boundary between the ground and the user's view of a distant object through the windshield. The information display device 1 may recognize the horizontal viewing position based on, for example, an image captured by the outer camera 133. The horizontal viewing position is set taking into account the vehicle's inclination. However, if the road the vehicle is entering has a slope or other road surface inclination, the vertical position may be adjusted. The road surface inclination may be detected using a sensor such as the gyro sensor 155 of the information display device 1, or may be detected from the image captured by the outer camera 133. The adjustment may also be performed when the road surface inclination exceeds a certain threshold. In another example, the device may be provided with a function to adjust the variation in the gaze position of each user while driving. Some users may normally use a gaze position several degrees lower than the water-level viewing position while driving. Such differences among users may be adjusted by providing a function to adjust the horizontal viewing position.

[0085] Fig. 12 shows an example of a display area based on each process described using Fig. 11. In the figure, an area with a high usage priority during driving is designated as Zone A, and the other area is designated as Zone B. Here, as shown in the figure, the information display device 1 may set an area including the center side of the display area (center side area) as Zone A.

[0086] (a) in the figure corresponds to Example 2-1-1. As shown in (a), the area is set based on the information display device 1 (head-mounted display terminal). In the example in the figure, a line crossing the center of the display area of ​​the information display device 1 is represented by a dashed-dotted line, and Zone A is set as the area including the dashed-dotted line.

[0087] (b) in the same figure corresponds to Example 2-1-2. As shown in (b), the area is set taking into consideration both the information display device 1 (head-mounted display terminal) and the vehicle. Here, in (b) in the same figure, the horizontal viewing position when the user is seated is indicated by the dashed-dotted line in the figure. In the example in the figure, the dashed-dotted line that crosses the center of the display area of ​​the information display device 1 overlaps with the horizon in the distance of the vehicle, and the area in which the external field of view above the dashboard is viewed is set as Zone A.

[0088] (c) in the same figure corresponds to Example 2-1-3. As shown in (c), the area is set based on the information display device 1 (head-mounted display terminal). In the example in the figure, a line crossing the center of the display area of ​​the information display device 1 is represented by a dashed-dotted line, and Zone A is set as a concentric area including the center of the display area indicated by the dashed-dotted line.

[0089] (d) in the same figure corresponds to Example 2-1-4. As shown in (d), the area is set taking into consideration both the information display device 1 (head-mounted display terminal) and the vehicle. Here, in (d) in the same figure, the horizontal viewing position when the user is seated is indicated by the dashed-dotted line in the figure. In the example in the figure, the dashed-dotted line represents a line that crosses the center of the display area of ​​the information display device 1, and Zone A is set as a concentric area that includes the central part of the display area indicated by the dashed-dotted line. Furthermore, the center of the concentric circle that sets the area of ​​Zone A is offset from the center of the display area of ​​the information display device 1.

[0090] As an example, the information display device 1 can be configured to be able to perform reconfiguration of the display area in at least one of examples 2-1-1, 2-1-2, 2-1-3, and 2-1-4 in S21.

[0091] 13 to 18 show specific examples of the processing in S22. In S22, the information display device 1 performs processing to set display restrictions on the display area. That is, when the driving mode is executed in contrast to the normal mode in which normal display is performed, the user is required to prioritize driving tasks, so the information display device 1 operates the information display as an auxiliary function. Therefore, in S22, the information display device 1 performs processing to achieve both safety and the display of information required by the user by eliminating in advance any display that would interfere with the user's driving tasks.

[0092] FIG. 13 shows a processing example of an information display device. In Example 2-2-1, the information display device 1 sets at least one of the divided areas as a non-display area and proceeds to the next step (2E). The information display device 1, for example, imposes a restriction on not displaying images in areas that have a high priority for driving tasks. The information display device 1 then displays the display desired by the user and information necessary for driving assistance in areas other than the non-display area, thereby enabling compatibility with driving. Alternatively, a restriction on not displaying images in all areas may be imposed to prioritize driving tasks. The restriction on not displaying images in these multiple areas may be set in advance, or the user may arbitrarily set multiple non-display areas.

[0093] In Example 2-2-2, the information display device 1 sets at least one of the divided areas as a display restriction area and proceeds to the next step. Areas with high priority are areas that are more effective in drawing the user's attention while driving. Therefore, in order to notify the user of danger, it is preferable to set a display restriction on this area.

[0094] Here, instead of setting a non-display area, the information display device 1 may impose a display restriction to ensure safety, while displaying images necessary for driving in the display-restricted area. Furthermore, the information display device 1 may change the range of the display-restricted area in accordance with the vehicle's traveling speed. For example, when the traveling speed is high, the user is required to gaze at a greater distance, and therefore is less likely to shake their head horizontally. Therefore, when the traveling speed is higher than a predetermined speed, the information display device 1 may perform control such as narrowing the display-restricted area from the outer edge or the left and right ends by an amount corresponding to the traveling speed.

[0095] The information display device 1 can acquire the vehicle's traveling speed using an appropriate method. For example, the information display device 1 may acquire the vehicle's traveling speed from a sensor provided in the information display device 1. For example, the information display device 1 may acquire the vehicle's traveling speed based on a change per unit time in the detection results of the positioning sensor 151. Furthermore, the information display device 1 may estimate the speed based on an image acquired by the outer camera 133 and acquire the vehicle's traveling speed. On the other hand, the information display device 1 may acquire the vehicle's traveling speed through communication. For example, the information display device 1 may acquire the vehicle's traveling speed by communicating with the vehicle.

[0096] Examples of display restrictions include the following. For example, the types of display that can be displayed may be restricted. For example, in an area where a display restriction is set, the information display device 1 may restrict displays unrelated to driving operations and warnings, and display only permitted images.

[0097] Furthermore, for example, the number of images that can be displayed may be limited. If an area is filled with many images, the images may obscure vehicles and pedestrians, reducing visibility around the vehicle. Therefore, the information display device 1 may limit the number of images to be displayed so that the image occupancy rate within the field of view does not exceed a certain value. Furthermore, when limiting the image display, the information display device 1 may set a smaller number of images that can be displayed in areas that have a high priority for driving tasks than in other areas.

[0098] Furthermore, for example, the spatial density of the displayed images may be limited. Even when the number of images displayed is small, if the images are large, they may occupy surrounding objects. Therefore, the information display device 1 may limit the size of the displayed images so that the image occupancy rate per unit area does not exceed a certain value. Furthermore, when limiting the image display, the information display device 1 may set a lower occupancy rate per unit area of ​​the displayed images in areas with a high priority for driving tasks than in other areas.

[0099] Furthermore, for example, a restriction may be imposed to change the displayed image for the driving mode. In the area where the display restriction is set, the information display device 1 may limit the images to images prepared in advance for the driving mode and display only the permitted images. Alternatively, the image may be converted for the driving mode and only the converted image may be displayed.

[0100] FIG. 14 relates to an example of a display based on the processing of S22. That is, FIG. 14 shows an example of a display on the information display device 1 (in this example, AR glasses) in driving mode. Here, in FIG. 14, an area including the horizontal viewing position when the user is seated while driving is defined as Zone A, and the dashed-dotted line in the same figure represents the horizontal viewing position when the user is seated. Note that in this example, Zone B is set in addition to Zone A. Here, the number of Zone Bs may be set to more than the number shown in the figure (one or two in this example). Furthermore, if there are multiple Zone Bs, one (or more) of these areas may be set as Zone C, which has a different setting to change the role of Zone A and Zone B.

[0101] In (a) of FIG. 14 , the information display device 1 performs control so as not to display an image (in this example, an AR image) in the Zone A area. In other words, if an image were displayed in Zone A, it could potentially obscure surrounding traffic conditions, such as vehicles and pedestrians, that require attention. Therefore, in this example, no image is displayed in Zone A. Furthermore, images (such as navigation images) from driving assistance functions provided by information projection devices, such as head-up displays and driving assistance projections that display images from headlights on the road surface, also transmit information to the user through the windshield. Therefore, if the information display device 1 displays an image in Zone A, there is a possibility that images from each device will overlap within the user's field of view. Therefore, by preventing images from being displayed in advance, concerns about information overlapping with other image display devices can be prevented.

[0102] In (b) of FIG. 14 , the information display device 1, like the above example, performs control so as not to display an image (in this example, an AR image) in the area of ​​Zone A. In this example, Zone B is set above and below Zone A, and the information display device 1 displays various images separately in the areas above and below Zone B. Here, there is a concern that the upper area of ​​the display area may overlap with a safety component, such as a rearview mirror, which is an object that the user should be able to see inside the vehicle. Therefore, when the image overlaps with a safety component, the information display device 1 erases the AR display of the overlapping portion or the entire area. Alternatively, when the image overlaps with a safety component, the information display device 1 moves the image display position of the AR display of the overlapping portion or the entire area. Furthermore, when the information display device 1 overlaps (is overlapping with) a speedometer, gauges, switches, or the like, which are objects that the user should be able to see inside the vehicle, in the lower region of the display area, it erases or moves the image display position of the AR display of the overlapping portion or the entire region, as described above.

[0103] In (c) of Figure 14, the information display device 1 restricts the image (in this example, AR image) displayed in Zone A by restricting the display in Zone A. From the same perspective as described above, Zone A reflects the surrounding traffic conditions, so if an image is displayed, it may block objects that should be focused on, such as vehicles and pedestrians. Therefore, the information display device 1 restricts the image to be displayed. Details regarding the display restriction will be described using another figure.

[0104] 14(d), the information display device 1 restricts the image (in this example, the AR image) displayed in Zone A by restricting the display in Zone A, as in the above example. In this example, Zone B is set above and below Zone A, and the information display device 1 displays various images separately in the upper and lower areas of Zone B. Here, as in the explanation of (b) above, when images overlap (when they overlap), the information display device 1 erases or moves the image display position of the AR display of the overlapping portion or the entire area.

[0105] Note that it is desirable for the information display device 1 to preset a displayable angle range of the display image corresponding to the user's head tilt angle range so that the image displayed in Zone A or Zone B when the user shakes their head up, down, left, or right does not overlap with safety devices such as a rearview mirror, a speedometer, gauges, switches, etc. Furthermore, the information display device 1 may be controlled so as not to display the display image if the user's head tilts beyond this angle range. Here, the head tilt angle may be detected using the outer camera 133 or gyro sensor 155 of the information display device 1, or the user may appropriately set the head tilt angle range using the input I / F 120 or the like. Furthermore, the information display device 1 may have this head tilt angle range preset. Alternatively, the display area of ​​the information display device 1 may be fixed to the direction of the user's face when the driving mode is activated, and Zone A may always be displayed following the direction of the user's face, or the display area of ​​the information display device 1 may be set so that Zone A is always fixed to the direction of the vehicle's travel within a coordinate system centered on the vehicle.

[0106] Furthermore, the information display device 1 may limit the number of images (e.g., AR images of icons) to be displayed in an area where display restriction is performed. Note that, when the vehicle is stopped or parked, the information display device 1 may remove or relax the display restriction and temporarily increase the number of images (e.g., AR images of icons) to be displayed. Here, the information display device 1 may acquire the vehicle's traveling speed using the same method as above, and determine the vehicle's status, such as being stopped or parked, based on the vehicle's traveling speed. On the other hand, the information display device 1 may determine the vehicle's status, such as being parked, based on communication with the vehicle.

[0107] Furthermore, in the display restriction, priorities may be set for images to be displayed (e.g., AR images of icons), and the information display device 1 may preferentially display images with higher priorities based on the set priorities in the display restriction. Furthermore, the information display device 1 may constantly display images with higher priorities, and in the display restriction, perform control to make images with lower priorities less noticeable. Here, in the display restriction, the information display device 1 may perform control such as temporarily displaying images with lower priorities and then erasing or reducing their size. In the display restriction, the information display device 1 may also perform control such as increasing the transmittance of images with lower priorities or moving images with lower priorities toward the edge of the display area.

[0108] Furthermore, the information display device 1 may impose different restrictions on the first and second display restriction areas, for example. Furthermore, the information display device 1 may impose the same restrictions on all display restriction areas. Taking into account the obstruction of visibility, the information display device 1 may impose the same restrictions on Zone B as on Zone A during the driving mode. Furthermore, the information display device 1 may impose restrictions on Zone B with different specified values ​​than those on Zone A during the driving mode. In this case, the display restriction on Zone A is set stricter than the display restriction on Zone B.

[0109] 15 is a diagram illustrating an example of the type of display image, whether it can be displayed in a display area, etc. As an example, the information display device 1 may store data in which the type of display image is associated with the display priority and whether it can be displayed in each area. In this example, an area with a high priority for use in driving operations is represented as Zone A, and other areas are represented as Zone B. Furthermore, a priority order is set based on the priority, with priority being given in the order of 1 to 5 (in other words, the smaller the number, the higher the priority).

[0110] In the figure, images of Examples 1 to 4 are permitted to be displayed in a high-priority area (Zone A) as information necessary for maintaining safety related to driving tasks. Images of Example 5, like images of Examples 1 to 4, are permitted to be displayed in a high-priority area. On the other hand, images of Examples 6 to 9 are restricted from being displayed in a high-priority area as information that may affect driving tasks, and are therefore not displayed in the high-priority area. However, these images are permitted to be displayed in a low-priority area (Zone B) as information that has little impact on driving tasks (i.e., information that will not distract the user from the surrounding environment if the user focuses on it).

[0111] Furthermore, as shown in Examples 10 to 12, message displays (text information), video displays, web displays (including text, images, and videos), etc. are considered to pose a high risk of distracting the driver while driving, and therefore their use is prohibited during driving mode.

[0112] These examples are examples of restrictions on the types of display that can be displayed. Also, images that reduce attention to driving (information that takes time to decipher, information that requires long viewing times due to changing images), etc., may be prohibited from display. Note that in situations where it is expected that attention to the surroundings may be reduced while driving, such as when the vehicle is stopped or parked, the information display device 1 may temporarily permit and display information that is prohibited from display. Also, the driving mode setting may be temporarily canceled.

[0113] Also, with reference to the same figure, an example of limiting the number of images that can be displayed will be described. Compared to normal mode, which displays images normally, driving mode involves driving tasks, so the number of objects that must be viewed (such as pedestrians, vehicles, and signs) increases, and the driver is required to recognize, acknowledge, and judge each object in a short time. Therefore, it is desirable to limit and reduce the number of images (information) displayed in driving mode compared to normal mode. This, for example, prevents the user's field of view from being obstructed by images, preventing the user from missing objects that need attention, such as pedestrians, and reduces the user's processing burden on tasks other than driving, thereby achieving safe driving. Therefore, as an example, as shown in the same figure, a high priority is set for displays related to driving. As a result, when the number of images that can be displayed is reduced, the information display device 1 displays images with higher priority first, according to the priority order.

[0114] The number of images that can be displayed in Zone A and Zone B may also be set. Here, by setting Zone A to have a smaller number of images that can be displayed or a smaller number of images that can be displayed per unit area than Zone B, it is possible to prevent the images from obscuring objects in the field of view. In this case, if the area in Zone A cannot be displayed due to the limit on the number of images that can be displayed, the images may be displayed in the area in Zone B.

[0115] Fig. 16 is a diagram showing an example of the process for display restriction in S22. As described above, restrictions may be imposed on the spatial density of the displayed image, and Fig. 16 shows an example of restrictions on the spatial density of the displayed image.

[0116] In driving mode, the vehicle's own (own vehicle) moving speed is faster than in normal mode. Furthermore, because other objects are also moving, the relative speed may become even faster. Therefore, it is necessary to instantly recognize, recognize, and judge objects in the field of view. In this case, as shown in the left diagram of (a) in the same figure, when there is no image in the field of view, it is possible to sufficiently recognize objects that should be viewed, such as surrounding vehicles. However, as shown in the right diagram of (a) in the same figure, when the information display device 1 (head-mounted display terminal) displays an image (in this example, a graphic indicating the vehicle's traveling direction), the object to be viewed (in this example, the vehicle) overlaps with the image, making it difficult to recognize the object, which may result in delayed recognition and judgment.

[0117] Therefore, unlike normal mode, in driving mode, care must be taken to prevent obstruction of the field of view due to the display of images within the field of view. While limiting the number of images displayed as mentioned above is one way to do this, here we will show how to improve the field of view obstruction caused by images alone.

[0118] The method shown in (b) of the same figure is a method for defining the size of an image, and is a method used, for example, in head-up displays. With this method, the information display device 1 defines the size of the image by tracing an envelope of a circle of a specified size (defined circle 22) on the display image 21. It is known that the spatial resolution in the high-frequency region of the MTF (Modulation Transfer Function: spatial frequency), which is a characteristic of the human eye, is low. Therefore, object recognition ability through sharp-angled parts of the image decreases, so it is desirable to define the size of the image by using an envelope to remove sharp-angled spatial regions. Here, for example, the circle used may be the same specified circle as that used in a head-up display.

[0119] The degree of visual field obstruction caused by an image in space is determined by the spatial density, which is the extent to which the image occupies a space. Therefore, the information display device 1 evaluates the calculated image size using a reference figure 23. The method shown in (c) of the same figure is a method for specifying the spatial density of an image, and is a method used in head-up displays. In this example, a vertically elongated rectangle with a predetermined height and width is specified as the reference figure 23, and the information display device 1 determines whether the image can be displayed by determining whether the occupancy rate of the image within this area is within a predetermined value. Note that, although a vertically elongated reference figure 23 is set assuming a pedestrian, a figure of another shape may also be used as the reference figure 23.

[0120] That is, the information display device 1 evaluates the display image 21 calculated in (b) based on the area covered by the reference figure 23, and evaluates the coverage ratio (i.e., the ratio of the display image 21 to the reference figure 23) as shown in (c). If the information display device 1 determines that the coverage ratio exceeds a specified value, it does not display the image, and if the information display device 1 determines that the coverage ratio does not exceed the specified value, it limits the display of the image, thereby improving visual field obstruction. Here, as an example, the information display device 1 may set a lower specified value for areas with a higher priority for driving tasks than for other areas when limiting the display of the image. Another example is the "coverage ratio = area covered by the reference figure / area of ​​the entire reference figure."

[0121] The reference figure 23 may be the same as the figure used in the evaluation of spatial density in the head-up display, and may have, for example, the same shape and size as the figure. Furthermore, the specified value of the coverage ratio in the information display device 1 may be set to the same as the specified value of the coverage ratio used in the evaluation of spatial density by the head-up display.

[0122] Furthermore, the information display device 1 may set a spatial density for the entire display area separately from the spatial density evaluated using the single graphic described above, and determine whether display is possible at both spatial densities. That is, when multiple images are displayed, even if the spatial density of each individual image is below a specified value, the entire image formed by these images may overlap with the target object. Therefore, the information display device 1 may use a reference graphic for evaluating the entire image to calculate the spatial density of the entire image and determine whether or not the entire image can be displayed. In this case, the specified value for the spatial density of the entire image may be set higher than the specified value for the spatial density of each individual image.

[0123] Furthermore, since the spatial resolution of the user's eyes is low in the area surrounding the center of the visual field, the specified value for the coverage ratio in the outer periphery of the visual field may be set smaller than the specified value for the coverage ratio in the center of the visual field to reduce the spatial density in the outer periphery of the visual field to make it easier to see surrounding objects.

[0124] For example, the specified value of the coverage ratio may be set to decrease depending on the angle from the center of the field of view. Also, the setting of the coverage ratio may be different for Zone A and Zone B. In this case, the specified value for Zone A may be set to be smaller than the specified value for Zone B.

[0125] A specific example of a restriction on changing the display image for driving mode will be described with reference to Figure 17. As an example, the information display device 1 may store data in which the image type is linked to image data for normal mode (normal display mode) and image data for driving mode. Note that the image type is not limited to the types shown in the figure, and image data for normal mode and image data for driving mode are set according to the image type.

[0126] In contrast to the normal mode, in the driving mode, it is necessary to recognize the display on the information display device 1 (head-mounted display terminal) while driving, so a display that takes time to interpret the image display can lead to inattentive driving, which poses a risk of hindering safe driving. Therefore, it is preferable that the image display be an image whose content can be understood instantly. Here, an example is shown of switching between the display used in the normal mode and the driving mode that is activated while driving.

[0127] In normal mode, the information display device 1 displays, as an example of a navigation display, a map of the surrounding area, information on landmarks, and a display that allows the user to know the relative positional relationship between the current location and the destination. In contrast, in driving mode, the information display device 1 displays only the minimum information necessary for the immediate driving operation. Furthermore, the information display device 1 reduces the number of characters displayed and displays them in a larger font size. Furthermore, the information display device 1 displays only words and does not display sentences. The displayed image is displayed in white, making it easy to distinguish even when the image overlaps with surrounding objects.

[0128] As an example of displaying received email information, the information display device 1 displays the number of received emails, a portion of the content of the received emails, etc. in normal mode. In the illustrated example, the sender, subject, and a portion of the text of the received email are displayed. If the user wants to know more detailed information, the user can access it by launching an email app. In contrast, in driving mode, the information display device 1 displays only information that an email has been received. In driving mode, launching the email app is restricted because it requires the user to pay close attention to decipher the text information. However, even in driving mode, when the vehicle is parked, stopped, or otherwise stationary and not being driven, the information display device 1 may exceptionally temporarily disable displays that are not permitted during driving and allow displays until the vehicle starts moving.

[0129] As an example of displaying incoming call information, in normal mode, the information display device 1 displays an image and text indicating that a call is being received. In normal mode, the information display device 1 notifies the user of an incoming call, and also displays information about the caller, such as the caller's phone number and whether the call is a registered phone number. In contrast, in driving mode, the information display device 1 only displays that there is an incoming call. In addition to this example, in driving mode, the information display device 1 may convey information that was previously conveyed to the user by image or image and audio, by combining audio with simplified image as in the above example. Furthermore, the information display device 1 may convey information by audio only, without displaying an image.

[0130] As an example of a music playback display, in normal mode, the information display device 1 displays which of the various operation buttons is selected during music playback, as well as images such as music artwork, lyrics, etc. In contrast, in driving mode, the information display device 1 displays only one icon indicating whether music is being played or is stopped. In this way, by the information display device 1 replacing the display with a single icon, there is no need to search for which icon is selected from multiple icons, and interference with driving is reduced.

[0131] In this way, in driving mode, unlike normal mode, only displays that have been previously permitted in driving mode are displayed, and only images that have been previously prepared for driving mode are displayed, thereby enabling safe driving without interfering with driving operations.

[0132] Next, referring to FIG. 18 , an example of image processing performed to assist the user's recognition when using the driving mode will be described. This image processing is part of the display restriction processing in the driving mode and can be executed in the display area control (S2). Unlike the normal mode, the driving mode requires the user to recognize, acknowledge, and judge the displayed image while continuing driving. Therefore, in order to judge and understand the meaning of the image in a short time compared to the normal mode, it is desirable to change the display state from the normal mode. The information display device 1 may perform at least one of the image processing examples shown below.

[0133] The information display device 1 may perform a process of reducing the number of colors as a response to image processing in driving mode (6A). Reducing the number of colors reduces the time required to distinguish colors, making the image easier to understand. For example, the information display device 1 may reduce the number of colors and display using primary colors. Furthermore, from the viewpoint of displaying clear, unadulterated hues, the information display device 1 may reduce the number of colors and display them so that each color is easily distinguishable.

[0134] The information display device 1 may reduce the number of gradations as part of image processing in driving mode (6B). Since intermediate gradations suppress the color output level to a low level, resulting in reduced brightness and making it difficult to distinguish differences between colors, intermediate gradations are considered unnecessary for clear display. Therefore, the information display device 1 may reduce the number of gradations and perform display.

[0135] The information display device 1 may change the display color in response to image processing in driving mode (6C). From the viewpoint of providing a clear display, the information display device 1 may change the display color and display based on a color combination that provides high color contrast, such as only black and white, only the three primary colors, or only complementary colors. The information display device 1 may also perform control so that red is used only for warning displays.

[0136] The information display device 1 may change the number of characters displayed as part of image processing in driving mode (6D). If the number of characters displayed is large, it is difficult to grasp the entire text from a bird's-eye view and understand its content, and the viewer must follow each character with their eyes in order to understand its meaning. Therefore, by reducing the number of characters displayed, it becomes easier to quickly grasp the meaning of the image from a bird's-eye view.

[0137] The information display device 1 may change the icon size as a measure for image processing in driving mode (6E). Small images have low spatial resolution for the eye from the viewpoint of MTF, making them difficult to distinguish and requiring time for recognition. Therefore, the information display device 1 may change the icon size to an easily viewable size and display the image.

[0138] The information display device 1 may change the limit on the number of icons to be displayed as a response to image processing in driving mode (6F). That is, the information display device 1 reduces the number of icons that can be displayed in driving mode compared to normal mode. Since the display is difficult to see in the peripheral vision area, the information display device 1 may reduce the number of icons to be displayed in the peripheral vision area to make them less noticeable. Furthermore, by responding to the peripheral vision area in this way, it is possible to prevent, for example, distracted driving.

[0139] The information display device 1 may stop occlusion processing as a response to image processing in driving mode (6G). Performing occlusion processing and displaying an image behind surrounding objects results in information being lost from the image and increases the time required for recognition. The information display device 1 may stop occlusion processing of the image and continue displaying it.

[0140] The information display device 1 may perform a change to reduce the resolution of the displayed icons in the peripheral vision region as part of image processing in driving mode (6H). The information display device 1 may also perform a change to change the size of the displayed icons in the peripheral vision region as part of image processing in driving mode (6I). The information display device 1 may also perform a change to reduce the number of displayed gradations in the peripheral vision region as part of image processing in driving mode (6K). It is known that the spatial resolution of the human eye is lower in the peripheral vision region than in the central vision region. Therefore, displaying icons with lower resolution in the peripheral vision region can reduce the processing load of the information display device 1 and achieve power savings. Furthermore, increasing the size of the displayed icons in the peripheral vision region can compensate for the reduced spatial resolution and aid in information recognition.

[0141] As a response to image processing in the driving mode, the information display device 1 may change the transmittance of the image when light reception information of the headlights of an oncoming vehicle is acquired (6L). Here, the information display device 1 may be equipped with a light reception sensor to acquire the light reception information. Furthermore, the information display device 1 may acquire the light reception information from the vehicle itself via communication.

[0142] The information display device 1 may change the display range according to the speed as a response to image processing in the driving mode (6M). For example, the information display device 1 may limit the display area to the center as the driving speed increases. Here, as described above, the vehicle's driving speed can be obtained by an appropriate method.

[0143] The information display device 1 may change the display color of the image to correspond to the color of the surrounding illumination as part of image processing in the driving mode (6N).

[0144] Furthermore, the information display device 1 may change text information into icons as a way of processing images in the driving mode.

[0145] 19 to 23 show specific examples of the processing of S23. In S23, the information display device 1 sets a display coordinate system for each set area. It is desirable that the coordinate system displayed for each image displayed on the information display device 1 (head-mounted display terminal) be different depending on the content of the image. For example, images used for driving assistance, particularly images superimposed on real space (navigation information, collision risk warnings with other vehicles, etc.), can be displayed on the road on which the user is traveling, allowing the user to immediately understand the intent of the displayed information while driving. In contrast, there is no reason to display images of information unrelated to driving (email reception information, incoming phone call information, etc.) in relation to the surrounding real-space conditions. Therefore, the display coordinate system may be different for each display.

[0146] 19 shows an example of processing by the information display device. In Example 2-3-1, the information display device 1 sets areas with different display coordinate systems for each divided area, and proceeds to the next step (2G). In other words, this example is an example in which a different display coordinate system is set for each divided area.

[0147] As described above, the coordinate system for displaying driving-related displays and other displays may be different. The display coordinate system here may be, for example, a coordinate system fixed to the user's face orientation or a coordinate system fixed to the vehicle. In this case, if the display coordinate system were changed for each display, images that move differently for each display would be mixed together, which is expected to confuse the user. Therefore, in this example, the information display device 1 divides the displayable area. Each divided area contains a display, and the information display device 1 changes the display coordinate system for each divided area. This allows for a display that changes the display coordinate system while effectively separating driving-related displays from other displays.

[0148] In Example 2-3-2, the information display device 1 sets different display coordinate systems for the area including the vehicle's axis of travel and the driver's horizontal viewing position and the other areas within the divided areas, and then proceeds to the next step (2H). That is, this example shows how different display coordinate systems are set for each divided area, and further how different display coordinate systems are set for the area of ​​the user's horizontal viewing position and the other areas. Here, the vehicle's axis of travel is a direction including a velocity vector in a coordinate system based on the vehicle. When the vehicle is traveling forward, it is the forward direction of the vehicle, and when the vehicle is traveling backward, it is the rearward direction of the vehicle. Furthermore, when the vehicle is turning, it is a direction including a tangent to the radius of curvature of the curve. Here, the horizontal viewing position is set taking into account the inclination of the vehicle. However, if the road into which the vehicle is entering has a slope or other road surface inclination, the vertical position of the horizontal viewing position may be adjusted. The road surface inclination may be detected using a sensor such as the gyro sensor 155 of the information display device 1, or the road surface inclination may be detected from an image captured by the outer camera 133. Furthermore, the above adjustment may be performed when the slope of the road surface exceeds a certain threshold. In another example, a function may be provided to adjust the variation in the eye gaze position for each user while driving. Some users may normally use a position that is several degrees lower than the water-level eye gaze position while driving. Such differences between users may be adjusted by providing a function to adjust the horizontal eye gaze position.

[0149] It is preferable that driving-related displays be displayed in the user's field of view while driving. In particular, among driving-related displays, displays that are linked to the surrounding environment should always be placed on the vehicle's driving path, allowing the user to immediately understand warnings regarding pedestrians, vehicles, etc. on the road. Furthermore, information not related to driving or displays related to driving that are not linked to the surrounding environment do not necessarily need to be placed on the vehicle's driving path. Therefore, it is preferable to display driving-related displays in an area that includes the horizontal viewing position, and display non-driving displays or driving-related displays that are not linked to the surrounding environment in other areas.

[0150] In Example 2-3-3, the information display device 1 sets different display coordinate systems for the area including the vehicle traveling axis and the driver's horizontal viewing position and the other areas within the divided areas, sets the area including the vehicle traveling axis and the driver's horizontal viewing position to a display coordinate system fixed to the vehicle, and proceeds to the next step (2J). That is, in this example, a different display coordinate system is set for each divided area, and further, a different display coordinate system is set for the user's horizontal viewing position and the other areas, and further, the area including the horizontal viewing position and the vehicle traveling axis is set to a coordinate system fixed to the vehicle.

[0151] It is preferable that driving-related displays be displayed in a coordinate system linked to the vehicle. This allows displays displayed in that area to always be positioned on the vehicle's driving path, allowing the user to immediately understand warnings about pedestrians and vehicles on the road. Here, driving-related displays can be set in a coordinate system fixed to the vehicle. On the other hand, displays other than driving-related displays can be set in a coordinate system linked to the user's head, allowing them to be constantly displayed within the user's field of view, allowing the user to constantly obtain information.

[0152] The display coordinate system will be described with reference to Figures 20 and 21. Figure 20 shows a coordinate system centered on the head of a user wearing an information display device (head-mounted display terminal), with the Y axis representing the forward direction of the user's face, the Z axis representing the upward direction of the user's head, and the X axis representing the left-right direction of the user's head. Note that although the figure shows the center of the head as the origin of coordinates, another location within the head (for example, the position of the eyeballs) may also be the origin of coordinates.

[0153] As shown in (a), the image display area of ​​the information display device 1 (head-mounted display terminal) is the area in front of the direction in which the user 13 is facing. The image display 31 is provided to the user 13 by the display 131 of the information display device 1, and therefore the location where the image is displayed changes in space in conjunction with the head of the user 13. If the user 13 turns to the right, an image is displayed in front of the user 13 facing right in conjunction with the direction of their face. If the user 13 turns to the left, an image is displayed in front of the user 13 facing left in conjunction with the direction of their face. The width of the image display 31 is defined by the FOV (Field Of View), and generally has an angular field of view of 40 to 120 degrees. Furthermore, in this example, this area is divided into multiple areas and images are displayed.

[0154] (b) shows an example of each divided area. The left diagram shows the positional relationship between the user's facial orientation and Zone A and Zone B divided within the field of view on the Y-Z plane. The right diagram shows the positional relationship between the user's facial orientation and Zone A and Zone B divided within the field of view on the X-Y plane. Note that the "user local coordinate system" refers to a coordinate system based on the position and orientation of the information display device 1 (in other words, the position and orientation of the user 13).

[0155] The left diagram shows a vertical cross section in a coordinate system based on a user wearing an information display device 1 (head-mounted display terminal), and the information display device 1 can set a display area that is divided into an area including a central visual field area and other areas. The right diagram shows a horizontal cross section in a coordinate system based on a user wearing an information display device 1 (head-mounted display terminal), and in this example, the information display device 1 divides the display area only vertically, so the divided areas are set to the same width in the horizontal direction. However, as in another example, the horizontal area may be divided and areas may be set.

[0156] Zone A is an important area for visually recognizing the surrounding environment in driving mode, and therefore is an area in which display restrictions are different from those for other areas. Different display coordinate systems may be used in each area. For example, a display coordinate system fixed to the vehicle may be used in Zone A, and a user local coordinate system may be used in Zone B.

[0157] The relationship with the coordinate system fixed to the vehicle will be described with reference to Fig. 21. Fig. 21 shows a user local coordinate system related to a coordinate system based on the user's head, and a vehicle local coordinate system related to a coordinate system based on the vehicle. More specifically, the "vehicle local coordinate system" is a coordinate system fixed to the vehicle and based on the position and orientation of the vehicle.

[0158] Here, the user-based coordinate system and the vehicle-based coordinate system are not linked to each other. When the user 13 looks directly ahead while driving, the angular deviation between the axes of the user-based coordinate system with respect to the vehicle-based coordinate system is small. However, when the user 13 turns his / her head while driving, the inter-axis angle deviates significantly. For example, when the user 13 turns his / her head to look directly to the side of the vehicle or behind the vehicle, the dot product of the vector representing the traveling direction of the vehicle 11 and the vector representing the direction of the user's face may be 0 or negative.

[0159] When the information display device 1 (head-mounted display terminal) is used while driving, it is expected that there will be two cases: (1) when it displays information related to driving assistance, and (2) when it displays information that is not directly related to driving.

[0160] Among the displays related to (1), there are displays that are particularly superimposed on real space to enhance the display effect. For example, route guidance informs the user 13 of the route the vehicle 11 should take, so by displaying it on the road while the vehicle 11 is traveling, awareness of the direction to take and where to turn right or left is enhanced. For example, warnings about the distance to the vehicle ahead and pedestrians ahead can be displayed together with the target position in real space ahead of the vehicle, thereby enhancing awareness of the warning content. The same applies to road signs and road markings placed on the road while the vehicle is traveling.

[0161] Therefore, for example, when a driving-related image is displayed and the image is adapted to the surrounding environment in the direction of travel (such as a navigation display or a warning about a vehicle or pedestrian ahead), it is desirable that the image be always displayed on the axis of movement of the vehicle 11 (the direction of travel of the vehicle) regardless of the orientation of the head of the user 13. For example, if a navigation display, warning display, or the like is displayed in a location other than the windshield, the correct intention may not be conveyed, or it may take time to understand the intention.

[0162] Displays related to (2) (such as received emails, incoming calls, weather information, information about music currently being played, and operation panels used for operations such as playing / stopping music) do not need to be limited to positions in real space. Furthermore, by continuously displaying them in the direction of the user's face, rather than being limited to positions in real space, they can be easily accessed when needed.

[0163] Therefore, the location where a specific image should be displayed differs depending on the display. Therefore, as an example, by determining which of the divided display areas a specific image should be displayed in, and then changing for each area whether the area should be fixed to a user-based coordinate system or a vehicle-based coordinate system, it is possible to realize a display that is suitable for the user 13.

[0164] For example, the display coordinate system of an area (e.g., Zone A) where driving-related images are displayed may be set to the vehicle local coordinate system, and the image may be locked to the vehicle's direction of travel rather than following facial movements. In this case, it is possible to continue displaying driving-related images in the location where they should be displayed. Even when the vehicle 11 obtains and displays information, the coordinate systems of the information display device 1 and the vehicle 11 are close to each other, making it easy to display the information. On the other hand, the display coordinate system of an area (e.g., Zone B) where images other than driving-related images are displayed may be set to the user local coordinate system, and the image may be displayed following the user's head.

[0165] The information display device 1 may automatically set the display coordinate system of a display-restricted area (e.g., Zone A) where driving-related images are displayed to the vehicle local coordinate system, and set the display coordinate system of a surrounding area (e.g., Zone B) to the user local coordinate system. However, the operation for setting the display coordinate system may be performed by the user via the input I / F 120, and the information display device 1 may set the display coordinate system based on the user's operation.

[0166] Fig. 22 is a diagram illustrating an example of display in different display coordinate systems. In Fig. 22, the information display device 1 displays an area including the user's horizontal viewing position and the vehicle's traveling axis direction as Zone A, and the other area as Zone B. Zone A displays an image in a display coordinate system fixed to the vehicle, while Zone B displays an image in a display coordinate system linked to the movement of the user's head. Zone A displays the direction in which the vehicle should travel, and Zone B displays other information.

[0167] 1A shows the display state of the information display device (head-mounted display terminal) when the user faces the front of the vehicle. In contrast, 1B shows the display state when the user turns their head to the left. Here, since Zone A is fixed to the vehicle's coordinate system, the display in Zone A does not move and continues to be displayed superimposed on the vehicle's roadway. In other words, the display area in Zone A is locked to the vehicle's traveling direction, and the image does not follow even if the user turns their head. On the other hand, since Zone B is fixed to a coordinate system based on the user's head, the display moves to the left in conjunction with the user's head movement.

[0168] If the user further shakes their head to the left, the image disappears from the field of view, but when they return to their original position, the image reappears. Similarly, if the user shakes their head to the right from the display state of (a), the display of Zone A is locked, but the display of Zone B moves to the right in conjunction with the user's head movement. If the user further shakes their head to the right, the image of Zone A disappears from the field of view, but when they return to their original position, the image reappears.

[0169] (c) in the same figure shows the display state of the information display device (head-mounted display terminal) when the user shakes his / her head downward. As with (b), the area in Zone A is fixed to the vehicle's coordinate system, so the display does not move. The area in Zone B is fixed to a coordinate system based on the user's head, so the display moves downward in conjunction with the user's head movement.

[0170] This allows the display, especially the driving-related display linked to the surrounding environment, to be displayed in a state that is always aligned with the vehicle's direction of travel, regardless of the user's head movement, while non-driving-related information and driving-related display not linked to the surrounding environment can remain accessible to the user.

[0171] The information display device 1 may correct the image movement amount relative to the head movement amount in the user reference coordinate system of the display area. Fig. 23 is a diagram for explaining the correction of the image movement amount relative to the head movement amount.

[0172] The human effective field of view is generally said to be ±100 degrees, but the effective field of view in which objects can be recognized and recognized is thought to be ±30 degrees, and the central field of view in which objects can be recognized and recognized with even higher resolution is thought to be in the range of ±1 to ±2 degrees. Therefore, the further away from the center of the field of view one is, the less adequate object recognition and recognition function will be, leading to delays in detecting vehicles, pedestrians, etc. while driving. Therefore, when driving, it is desirable to minimize the time that the user's line of sight is away from the direction of travel of the vehicle.

[0173] As described above, even if the display of Zone A is fixed in a vehicle-based coordinate system, if the display of Zone B is fixed in a user-based coordinate system, there is a risk that the user's field of view will remain significantly distant from the front of the vehicle. In other words, the user may gaze at the display of Zone B with their field of view significantly distant from the front of the vehicle, and the user's head may remain facing the same direction. This may result in the user being unable to see the displayed information in Zone A for a long period of time. Therefore, correcting the amount of image movement in response to the user's head movement can prevent the user's head from significantly deviating from the vehicle's traveling direction. This makes it less likely that the information in Zone A will deviate from their field of view, reducing the amount of time the user takes their eyes off the screen and ensuring safety. It is also desirable to perform a correction so that the amount of image movement is small relative to the amount of head movement.

[0174] 1A shows an example in which the information display device 1 performs a correction by multiplying the angle of the face direction (amount of user head movement θ) by a coefficient to move the image display. That is, an example is shown in which the amount of image movement kθ is suppressed by multiplying the amount of user head movement θ by a correction value k. Here, as shown in FIG. 1B, for example, the information display device 1 suppresses image movement by multiplying by a correction value 0 for face directions up to 15 degrees, thereby displaying the image in the direction of vehicle travel. Then, the information display device 1 may multiply by the correction value k for angles of 15 degrees or more to linearly reduce the amount of image movement.

[0175] For example, in the region of 15 degrees or more, a correction value k is set so that the image can be seen in the peripheral vision region without being removed from the field of view. For example, in the region of ±30 degrees, the correction value k is set to 0.8 so that the image is not cut off at the edge of the region, and in the region of ±60 degrees, the correction value k is set to 0.4 so that the image is not cut off at the edge of the region.

[0176] In this example, linear correction is performed using the correction value k, but it is not limited to linear correction. For example, correction may be performed by changing k with respect to the angle (amount of user head movement), or correction may be performed using a polynomial.

[0177] The information display device 1 may store data on a correction value k corresponding to the amount of user head movement θ and correct the amount of image movement. Here, for example, data may be stored in which the correction value k is 0 from 0 degrees to 15 degrees and the correction value k is 0.8 or 0.4 for angles 15 degrees or greater, and the information display device 1 may perform processing based on this data. Alternatively, data may be stored for stepwise correction values ​​k, such as 0.8 from 15 degrees to 30 degrees and 0.4 from 30 degrees to 60 degrees, and the information display device 1 may perform processing based on this data.

[0178] Furthermore, for example, data on the correction value k that gradually decreases as the user's head movement amount θ increases from 15 degrees may be stored, and the information display device 1 may perform processing based on this data. Furthermore, data on the correction values ​​(k0, k1, k2, ... kn) in a polynomial may be stored, and the information display device 1 may perform processing to determine the image movement amount based on this data using the formula: image movement amount = k1θ + k2θ^2 + ... + knθ^n + k0.

[0179] As with Zone B, Zone A may also have a fixed angle range and a range of movement in a corrected state to prevent the object from moving out of the field of view. Even in this case, Zone A and Zone B may be moved in different ways. Furthermore, data on the correction value k used for correction in Zone A and Zone B may be registered in advance in the information display device 1.

[0180] Next, an example of the display adjustment control (S3) will be described in detail with reference to Figures 24 to 26. In the display adjustment control, when the driving mode is selected, processing is performed on the image display that was displayed in the normal mode, and when the setting was made to display an image in Zone A, processing is performed to check whether an image display such as a head-up display or information projection headlight is being displayed in the same area, and to adjust the display position.

[0181] FIG. 24 is a flowchart showing an example of processing in the display adjustment control (S3). When processing transitions from the previous step (S300) to the display adjustment control (S3), for example, as shown in (a) of the same figure, the information display device 1 adjusts the existing display (S31). Then, processing transitions to the next step (S301). Note that, as shown in (b) of the same figure, the information display device 1 may also adjust with other display devices (S32). Furthermore, either the processing of S31 or S32 may be performed first.

[0182] If the information display device 1 was used in the normal mode before switching to the driving mode, it is assumed that some display has already been made on the display screen. Here, as described above, in the driving mode, the display area and display restrictions are set. Specifically, (1) restrictions on the types of displays that can be displayed, (2) restrictions on the number of displays that can be displayed, (3) restrictions on the spatial density of the displayed image, and (4) restrictions on the displayed image for the driving mode are set. For this reason, it is necessary to temporarily change the display to something different from the display displayed in the normal mode. For example, the information display device 1 may forcibly stop all displays used in the normal mode and replace them with the display in the driving mode. Another example is to rearrange the image displayed in the normal mode in the divided area of ​​the driving mode, taking into consideration the area and content. The information display device 1 performs such adjustment of the existing display in S31.

[0183] Furthermore, in S32, the information display device 1 adjusts with other display devices. Vehicles include devices that display images through the windshield. For example, there are devices related to technology that displays information such as driving assistance using virtual image displays, such as head-up displays. Another example is a device related to technology called driving assistance projection, which displays images (information) on the road surface through the vehicle's lights and presents them to the driver through the windshield. The area where these displays are performed overlaps with the area where the real-world environment is viewed through the information display device 1 (head-mounted display terminal). Therefore, when the above technology is installed in a vehicle and displays are being performed, there is a possibility that the displays will overlap, making it impossible to correctly understand either information. Therefore, if there is a concern about image overlap, it is necessary to adjust the image in driving mode (such as canceling the display or correcting the display position).

[0184] Fig. 25 shows an example of processing in S31. That is, Fig. 25 is a diagram showing, in a table, display states in normal mode when transitioning from normal mode to driving mode, and examples of responses (examples of processing) when transitioning to driving mode based on each condition.

[0185] More specifically, it is assumed that in normal mode, video is displayed without any particular area division, and the example shows whether to continue displaying the video or change the display area depending on whether the area in which the video is displayed corresponds to Zone A and / or Zone B after the driving mode is activated. However, this is not limited to a state in which no area division is performed in normal mode, and the area may be divided even in normal mode, or may be divided according to the display content, etc., even in normal mode, and this also includes cases in which the division is reset to a range suited to the driving mode.

[0186] If a display image is already present in the area corresponding to Zone A before transition to driving mode, and the display of this display image in Zone A and Zone B is disabled after transition to driving mode (condition 2), the information display device 1 may stop displaying the image in all areas. Here, the information display device 1 may take measures for some or all of the displayed images that were in use in normal mode, such as stopping the app displaying the image, temporarily suspending the image display, ending the image display, or hiding the image display.

[0187] If there is already a display image in the area corresponding to Zone A before the transition to driving mode, and after the transition to driving mode, the display of this display image in Zone A and Zone B is permitted (condition 3), the information display device 1 may continue to display the image in Zone A.

[0188] If there is already a display image in the area corresponding to Zone A before switching to driving mode, and after switching to driving mode, the display of this display image in Zone A is set to be disabled, and the display of this display image in Zone B is set to be permitted (condition 4), the information display device 1 may move the image to Zone B and display it.

[0189] If an image is already displayed in the area corresponding to Zone B before the transition to driving mode, and the display of this displayed image in Zone A and Zone B is disabled after the transition to driving mode (Condition 6), the information display device 1 may stop displaying the image in all areas. Here, the information display device 1 may take measures such as stopping the application for the image display, temporarily stopping the image display, ending the image display, or hiding the image display, for some or all of the image display.

[0190] If there is already an image displayed in the area corresponding to Zone B before the transition to driving mode, and after the transition to driving mode, the display of this displayed image in Zone A and Zone B is permitted (condition 7), the information display device 1 may continue to display the image in Zone B.

[0191] If there is already a display image (e.g., information about driving operations) in the area corresponding to Zone B before switching to driving mode, and after switching to driving mode, the display of this display image in Zone B is set to be disabled, and the display of this display image in Zone A is set to be permitted (condition 9), the information display device 1 may move the image to Zone A and display the image.

[0192] Furthermore, if a video image is already displayed in the area corresponding to Zone B before the transition to driving mode, and display of this video image in Zone A and Zone B is permitted after the transition to driving mode (condition 8), the information display device 1 determines whether the number of images displayed in Zone A will exceed the set number when the video image is moved to Zone A, and whether the display priority of the video image satisfies the specified range for display in Zone A. If these requirements are met (YES), the information display device 1 may move the video image to Zone A and continue displaying the video image. On the other hand, if one or both requirements are not met (NO), the information display device 1 may continue displaying the video image in Zone B.

[0193] If, before transition to the driving mode, there is no display image in the area corresponding to Zone B (condition 5), the information display device 1 may, upon transition to the driving mode, display a new image that is permitted to be displayed in Zone B. If, before transition to the driving mode, there is no display image in the area corresponding to Zone A (condition 1), the information display device 1 may, upon transition to the driving mode, omit displaying a new image in Zone A and proceed with the processing.

[0194] When the display mode is switched to the driving mode, the area is divided, and the determination condition for this process is which area after the driving mode the image displayed before is displayed in. The information display device 1 then checks, in each divided area, whether the content (display image) displayed before the driving mode switch is still permitted to be displayed after the driving mode switch.

[0195] As an example, the information display device 1 can determine whether there is a display in each of Zone A and Zone B, select conditions 1 to 9, and perform the process related to S31. The information display device 1 can apply one of conditions 2 to 4 and conditions 6 to 9 to a given display and proceed with the process.

[0196] The information display device 1 may also select multiple conditions and perform processing. For example, the information display device 1 may select condition 1 or 5, and apply one of conditions 2 to 4 or 6 to 9 to a certain display, and proceed with processing. The information display device 1 may also apply one of conditions 2 to 4 or 6 to 9 to a certain display, and apply a different condition to another display, and proceed with processing.

[0197] Then, after processing is performed for each video display, the process proceeds to the next step. Note that data related to the conditions may be stored in advance in the storage device 110 of the information display device 1. The information display device 1 can perform processing using this data related to the conditions. Furthermore, these settings may be processed based on conditions selected by the user.

[0198] FIG. 26 shows a specific example of S32. That is, FIG. 26 shows an example of an operation for suppressing interference of image display when transitioning to driving mode. A vehicle has a device (information projection device) that presents images to the driver. Examples of information projection devices include a head-up display that projects images related to driving assistance through the windshield, and a driving assistance projection device that projects images from vehicle lights (e.g., information projection headlights). These devices provide driving assistance by displaying images from a device installed inside or outside the vehicle.

[0199] When a video display related to driving assistance from a vehicle is superimposed on real space, the video display from the information display device 1 (head-mounted display terminal), which also displays a video superimposed on real space, and the video display from the vehicle may overlap, raising concerns that the information from each device may be unreadable or may be misinterpreted. Therefore, to prevent overlapping, video display in the relevant area is prohibited and the video position is moved from the relevant area.

[0200] Furthermore, when the same category is displayed on multiple devices, the displayed information may differ. For example, it is conceivable that the speed information displayed by the video display from the vehicle and the speed information displayed by the video device of the information display device 1 may indicate different speed values. When the vehicle and the information display device 1 are not connected by wire or wirelessly, the information displayed by each device may differ. Therefore, to avoid confusion for the user, when the same category is displayed on another device, the category is not displayed.

[0201] Example 3-1 is an example in which, when there is a risk that an image of an information projection device that displays an image on the windshield and an image of an information display device will overlap, the display position is moved to a position or area where the images do not overlap. In this example, when the information display device 1 detects an image displayed by another device on the windshield (Condition 3A), the information display device 1 changes the image display position to a position or area where the image does not overlap with the image displayed by the other device on the windshield, and the process proceeds to the next step. In other words, the display of the other device is given priority, and the information display device 1 changes the image display position of the information display device 1.

[0202] Example 3-2 is an example in which a display of the same category as the display content of an information projection device that displays an image on the windshield is prevented. In this example, when the information display device 1 detects an image displayed on the windshield by another device (Condition 3A) and determines that the image displayed by the other device and the display content of the information display device 1 (head-mounted display terminal) are similar (same category) (Condition 3B), the information display device 1 stops displaying the display content from the information display device 1, and the process proceeds to the next step. In other words, the display of the other device is given priority, and the information display device 1 stops displaying the display by the information display device 1. Furthermore, by the information display device 1 not displaying the corresponding image, it is possible to prevent multiple similar displays and confusion caused by different displays.

[0203] Example 3-3 is an example in which, when there is a risk that an image of an information projection device that displays an image on the road surface and an image of an information display device will overlap, the display position is moved to a position or area where the images do not overlap. In this example, when the information display device 1 detects an image displayed on the road surface from the vehicle (Condition 3C), the information display device 1 changes the image display position to a position or area where the image does not overlap with the image displayed on the road surface, and the process proceeds to the next step. In other words, the display of the other device is given priority, and the information display device 1 changes the image display position of the information display device 1.

[0204] Example 3-4 is an example in which a display of the same category as the display content of an information projection device that displays an image on the road surface is not displayed. In this example, when the information display device 1 detects an image displayed on the road surface from the vehicle (Condition 3C), and when the information display device 1 determines that the image displayed on the road surface from the vehicle and the display content of the information display device 1 (head-mounted display terminal) are similar (same category) (Condition 3D), the information display device 1 stops displaying the display content from the information display device 1, and the process proceeds to the next step. In other words, the display of the other device is given priority, and the information display device 1 stops displaying the image on the information display device. Furthermore, by the information display device 1 not displaying the image, it is possible to prevent multiple similar displays and confusion caused by different displays.

[0205] In addition, as in Example 3-5, if it is determined that the information display device 1 does not meet any of conditions 3A to 3D, it is determined that there is no interference between the image of the information display device 1 and the image of another display device, and processing proceeds to the next step.

[0206] For example, the information display device 1 may detect a display image on the windshield, detect a display image on the road surface, obtain the position of the display image, detect overlapping of the display image, etc., using images acquired by the outer camera 133. Furthermore, the information display device 1 may obtain display coordinates of other devices from the vehicle side via communication and detect overlapping of the display image.

[0207] As another example, the information display device 1 may prohibit the display of images in the display range of an information projection device (a device that displays images on the windshield and / or a device that displays images on the road surface). For example, the information display device 1 may prohibit the display of images in Zone A (i.e., the area where displays related to driving are performed) that includes the display range of an information projection device (a device that displays images on the windshield and / or a device that displays images on the road surface). This eliminates the need to perform sequential image overlap determination that takes into account, for example, head movement. The information display device 1 may calculate the display range of other devices from images acquired by the outer camera 133. The information display device 1 may also acquire information on the display range of other devices from the vehicle.

[0208] Next, an example of the operation mode end determination (S5) will be described in detail with reference to FIGS.

[0209] FIG. 27 is a flowchart showing an example of processing in the driving mode termination determination (S5). As shown in FIG. 27, the information display device 1 can perform, for example, the processing shown in (a) or (b) of the same figure. For example, as shown in (a) of the same figure, when the processing transitions from the previous step (S500) to the driving mode termination determination (S5), the information display device 1 performs the processing of S51 and then transitions to the next step (S501). For example, as shown in (b) of the same figure, when the processing transitions from the previous step (S500) to the driving mode termination determination (S5), the information display device 1 performs the processing of S51 and S52 and then transitions to the next step (S501).

[0210] In the driving mode termination determination (S51), the information display device 1 determines the conditions for transitioning from the driving mode to the normal mode. For example, if the user stops using the information display device 1 (head-mounted display terminal) or does not request driving assistance from the information display device 1 (head-mounted display terminal), the information display device 1 may terminate the driving mode and return to the normal mode. Furthermore, the information display device 1 may terminate the driving mode and return to the normal mode, for example, when the user switches modes, gets off the vehicle, or switches to an autonomous driving mode. When returning to the normal mode, the information display device 1 may continue to display content other than driving content. Furthermore, the information display device 1 may return to the display state temporarily saved when transitioning to the driving mode and resume displaying content that was prohibited. Alternatively, the information display device 1 may be started up in a new state with no new content displayed.

[0211] In the driving mode suspend determination (S52), the information display device 1 relaxes the display restrictions, for example, when the load related to driving is temporarily reduced or released during driving, such as when the vehicle is parked or stopped. Furthermore, for example, when the user continues to be present in the vehicle due to a mode change by the user or switching to an autonomous driving mode, and there is a possibility that the user will return to driving, the information display device 1 may suspend the driving mode without terminating the driving mode and relax the display restrictions. An example of relaxation of the display restrictions is allowing the display of content that has been prohibited from being displayed. While the examples in which video display, web display, and text display are prohibited have been described, these displays may be displayed during the suspension period because they do not pose a risk to the user. When the vehicle is confirmed to have started moving or the vehicle is being driven, the information display device 1 releases the suspension, restores the display restriction state, and monitors whether the determination conditions of S51 and S52 are met. During suspension, the image displayed during driving may be moved to the edge of the screen or reduced in size, but may continue to be displayed on the screen.

[0212] For example, the information display device 1 may confirm driving operations such as starting the vehicle and holding the steering wheel based on an image acquired by the outer camera 133. Furthermore, the information display device 1 may confirm starting the vehicle based on detection results of the acceleration sensor 154, the positioning sensor 151, etc. Furthermore, the information display device 1 may confirm starting the vehicle, driving operations, etc. through communication with the vehicle.

[0213] 28 shows a specific example of the process of S51. In Example 51-1, the information display device 1 checks whether the mode has been changed to normal mode by a user operation, and if it is confirmed that the mode has been changed to normal mode by the user, the mode is changed to normal mode and the process proceeds to the next step (51A). Example 51-1 is an example in which the driving mode is ended by the user's own operation (operation on the setting screen, button operation, voice operation, gesture operation, etc.). The user may change to normal mode when switching driving duties with another person or when ending driving, etc.

[0214] In Example 51-2, the information display device 1 confirms a change to normal mode through communication with the vehicle or in-vehicle equipment, and when a command to change to normal mode is received, the information display device 1 changes to normal mode and proceeds to the next step (51B). Example 51-2 is an example in which the information display device 1 (head-mounted display terminal) ends the driving mode when it receives a signal notifying the end of driving through communication with the vehicle or in-vehicle equipment. The information display device 1 may obtain information from the vehicle when connected to the in-vehicle network. The information display device 1 may obtain information from, for example, OBD (On Board Diagnosis). Alternatively, when it detects that the connection with the vehicle has been lost, the information display device 1 may end the driving mode and switch to normal mode.

[0215] In Example 51-3, when the information display device 1 detects the end of the driving operation, it switches to normal mode and proceeds to the next step (51C). Example 51-3 is an example in which, without user input or communication with the vehicle, the end of vehicle driving is determined based on input from the camera and / or sensor of the information display device 1 (head-mounted display terminal), and the driving mode is terminated. Here, the information display device 1 may use the outer camera 133 to detect that the user is no longer holding the steering wheel or that the vehicle has stopped moving based on the scenery in real space. The information display device 1 may also detect that movement has stopped using the acceleration sensor 154, the positioning sensor 151, etc. The information display device 1 may also use the outer camera 133 to detect the user's movement from the driver's seat, the user's movement from inside the vehicle, etc.

[0216] In Example 51-4, when the information display device 1 detects that the information display device 1 has been removed, the information display device 1 ends the operation mode and proceeds to the next step (51D). Example 51-4 is an example in which the end of the operation mode is determined when the user detects an action of removing the information display device 1 from the user. In this case, not only the operation mode but also the display may be ended. Alternatively, the information display device 1 may be switched to a sleep mode or the power may be turned off.

[0217] In Example 51-5, when the information display device 1 detects a malfunction of the information display device 1, the information display device 1 terminates the driving mode and proceeds to the next step (51E). Example 51-5 is an example in which the information display device 1 determines to terminate the driving mode when it detects that a malfunction prevents normal display. Note that, for example, if only one display on one eye side malfunctions, the information display device 1 may continue displaying on the other display. For example, the information display device 1 has an eyeglass-type structure, and when a malfunction of either the left or right display is detected, the information display device 1 may continue displaying on the other display that is not malfunctioning. Furthermore, the information display device 1 may terminate the driving mode only when both displays malfunction. Also, in Example 51-5, the information display device 1 may terminate the display in addition to terminating the driving mode. Furthermore, a message informing of a display abnormality may be displayed on the screen of the information display device 1 or an audio message may be emitted to urge the user to stop using the information display device 1.

[0218] In Example 51-6, the information display device 1 continues the driving mode if the set driving mode termination determination conditions are not met. That is, if the set driving mode termination determination conditions (51A to 51E) as in the above examples are not met, the information display device 1 does not terminate the driving mode and continues displaying in the driving mode. In this case, the information display device 1 returns to step S4 and loops the determination in step S5.

[0219] For example, the information display device 1 may terminate the driving mode when at least one of the above-mentioned Examples 51-1 to 51-5 is satisfied, and may continue displaying in the driving mode without terminating the driving mode when all of these are not satisfied. One or more (including all) of the conditions in Examples 51-1 to 51-5 may be set, and when any one of them is satisfied, the information display device 1 terminates the driving mode.

[0220] Figure 29 shows a specific example of the process of S52. Here, a determination process for temporarily canceling the driving mode and returning to the driving mode will be described. If the conditions are not satisfied, the information display device 1 continues the driving mode. After starting display in the driving mode, the information display device 1 constantly performs the process of S52 and monitors the situation.

[0221] In Example 52-1, when the information display device 1 detects an instruction to suspend the driving mode by a user operation, it releases some of the display restrictions imposed by the driving mode (52A). Example 52-1 is an example in which the driving mode is temporarily suspended by the user's own operation (operation on the setting screen, button operation, voice operation, gesture operation, etc.). When a condition is met, the information display device 1 continues the driving mode, but temporarily releases some or all of the display restrictions set in the driving mode. The information display device 1 releases, for example, the types of restrictions mentioned above (restrictions on the information displayed in the area), restrictions on the number of displays, restrictions on spatial density, and restrictions on driving mode images, allowing the user to check email content, display web pages, etc. even in the driving mode. Furthermore, when the suspended state is released by a user operation, the information display device 1 returns to the driving mode. Alternatively, the information display device 1 may be configured to return to the driving mode, for example, when it detects vehicle movement or driving operations.

[0222] In Example 52-2, when the information display device 1 detects that the vehicle is stopped or parked through communication with the vehicle or an in-vehicle device, it releases some or all of the display restrictions imposed by the driving mode (52B). When the vehicle is stopped or parked, the load related to driving operations is reduced, so the display restrictions may be released. The information display device 1 may also determine whether the vehicle has been stopped for a specified time or more. By setting the specified value, it is possible to determine whether the stop is a very short time, such as when waiting at a traffic light. Furthermore, multiple specified values ​​may be set and the display restrictions may be released in stages. Furthermore, the information display device 1 may release the display restrictions not only based on information from the brake and vehicle speed sensors, but also by taking into account information from the shift lever, information on whether the parking brake is on, information on the driver's steering position, and so on. Then, when the information display device 1 detects the vehicle moving, it returns to the driving mode.

[0223] In Example 52-3, when the information display device 1 detects that the vehicle has stopped or parked, it cancels some or all of the display restrictions imposed by the driving mode (52C). Example 52-3 is an example in which the information display device 1 detects that the vehicle has stopped or parked based on input from a camera or sensor provided in the information display device 1, without any user input or communication with the vehicle. The information display device 1 may use the outer camera 133 to detect that the user is no longer holding the steering wheel or that the vehicle has stopped moving based on the scenery in real space. The information display device 1 may also detect that movement has stopped based on the acceleration sensor 154, the positioning sensor 151, or the like. The information display device 1 may also take into account information such as the user not holding the steering wheel. For example, the information display device 1 may detect that the steering wheel is being held based on an image acquired by the outer camera 133. Then, when the information display device 1 detects that the vehicle is moving, it returns to the driving mode.

[0224] In Example 52-4, when the information display device 1 detects a transition from manual driving to autonomous driving, it releases some of the display restrictions imposed by the driving mode (52D). Example 52-4 is an example in which the display restrictions imposed by the driving mode are released in part or in whole when it detects that the vehicle is traveling in autonomous driving mode through communication with the vehicle or input from the outer camera 133 and / or sensors provided on the information display device 1. Then, when a transition to manual driving is detected through communication with the vehicle or through sensors and / or cameras provided on the information display device 1, the information display device 1 returns to the driving mode.

[0225] In Example 52-5, the information display device 1 changes some of the display restrictions imposed by the driving mode when it detects that the image is superimposed on the vehicle's instruments and / or rearview mirror (52E). During driving, it is conceivable that the image displayed by the information display device 1 may obscure the vehicle's instruments, rearview mirror, etc. Therefore, when the camera of the information display device 1 detects that the displayed image overlaps the vehicle's instruments, rearview mirror, etc., the information display device 1 changes the display restrictions for the overlapping image portion, for example, pausing the image display in the area including the overlapping image portion. The information display device 1 may also pause the image display in other areas. Meanwhile, the information display device 1 may continue displaying the image in other areas. Then, when the information display device 1 detects that the overlapping image has been resolved, the information display device 1 returns to the driving mode.

[0226] In Example 52-6, when the set driving mode suspend determination condition is not satisfied, the information display device 1 performs a driving mode termination determination in step S51.

[0227] For example, the information display device 1 suspends the driving mode when at least one of the above examples 52-1 to 52-5 is satisfied, and proceeds to a driving mode termination determination (S51) when all of these are not satisfied.

[0228] Next, an example of division of the display area and video display will be described with reference to FIGS.

[0229] As shown in FIG. 30 , the information display device 1 may combine vertical and horizontal divisions in dividing the display area. As shown in (a) of the same figure, the information display device 1 may divide the display area so that a single Zone B extending horizontally (left and right on the display screen) is set, and Zone C is set horizontally on both sides of Zone A, which is the area used for driving. For example, Zone A is restricted from displaying images. Here, Zone C is located on both sides of Zone A in the horizontal direction, so Zone C may be set with display restrictions equivalent to those of Zone B, or more lenient than Zone A but stricter than Zone B, taking into account the user's line of sight while driving. Note that the dashed lines in the left and right directions in the same figure indicate the horizontal viewing position.

[0230] 1B shows an example in which the display area is divided so that Zone B is set on both vertical sides of Zone A and Zone C is set on both horizontal sides of Zone A. In this example, from the same perspective as above, Zone C may be set with display restrictions that are the same as those of Zone A, or that are looser than Zone A but stricter than Zone B. The dashed lines in the left and right directions in the figure indicate the horizontal viewing position.

[0231] Fig. 31 shows an example of a display in the divided areas in Fig. 30. A display restriction is set in the area (Zone A) where driving-related displays are performed. As shown in Fig. 31(c), Zone A displays, for example, an image showing the vehicle's axis of movement (vehicle travel direction). Zone B displays other types of images. Similarly, as shown in Fig. 31(d), Zone A displays an image showing the vehicle's axis of movement (vehicle travel direction), and Zone B displays other types of images.

[0232] As shown in FIG. 32 , the information display device 1 may divide the display area only vertically (up and down on the display screen). This sets an area that extends vertically on the display screen. As shown in (e) of the same figure, for example, Zone A, which is an area related to driving, is set in the central area of ​​the display area (i.e., the central area of ​​the field of view), and a display restriction is set for this Zone A. Then, in the driving mode, as shown in (f) of the same figure, due to the setting of the display restriction, the information display device 1 displays only permitted images (in this example, the direction of vehicle travel) in Zone A. Due to the setting of the display restriction, the information display device 1 does not display unauthorized images in Zone A.

[0233] As shown in Figure 33, the information display device 1 may divide the field of view into, for example, concentric circles, and set a central display area and a peripheral display area. Here, as shown in (g) of the figure, the information display device 1 may prohibit the display of an image in area OA that overlaps with an image from an in-vehicle device such as a head-up display. Furthermore, as shown in (h) of the figure, the information display device 1 may not display an image in the central display area of ​​the field of view, but may instead display an image in the peripheral area.

[0234] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, for example, other configurations may be added, deleted, or replaced with part of the configuration of the embodiment.

[0235] As shown in Fig. 34, the information display device 1 may perform processing in driving mode that omits display adjustment (S3). When processing starts (S1000), the information display device 1 executes a driving mode activation determination (S1). Then, when the driving mode is activated, the information display device 1 performs display area control (S2) and starts display based on the display control settings. The information display device 1 executes a driving mode termination condition determination (S5), and when it is determined that the driving mode should be terminated, the information display device 1 terminates the driving mode (S6). Then, processing ends (S1001).

[0236] The driving mode can be performed during manual driving, or the driving mode can be performed during automatic driving.

[0237] 1 Information display device (head-mounted display terminal) 101 Processor 131 Display

Claims

1. An information display device to be worn on the head, comprising: a processor; and a display for displaying images, wherein the processor displays images in a normal display mode for normal display or in a driving mode used when driving a vehicle, and in the driving mode, the display of images is restricted compared to the normal display mode.

2. An information display device according to claim 1, wherein the processor divides the image display area in the driving mode.

3. An information display device according to claim 2, wherein the processor divides the display area into at least two or more areas and sets two or more different display restrictions for each area.

4. An information display device according to claim 2, characterized in that the processor divides the display area into at least two or more areas so that at least one of the areas includes a central area that is an area toward the center of the display area.

5. An information display device according to claim 4, wherein the processor prohibits display in the central region when restricting the display of the image.

6. An information display device according to claim 4, wherein the processor sets the image display restriction so that the occupancy rate per unit area of ​​the displayed image in the central region is lower than that in other regions.

7. An information display device according to claim 4, wherein the processor, in limiting the display of images, sets the number of images displayed in the central region lower than in other regions.

8. An information display device according to claim 2, wherein the processor displays images using different image data in the driving mode and the normal display mode for the same type of image display.

9. An information display device as described in claim 2, characterized in that, when the processor determines that the display content of an image from an equipment mounted on the vehicle is similar to the display content of an image in the display area during the driving mode, it stops displaying the image in the display area that is determined to be similar.

10. An information display device as claimed in claim 2, characterized in that, when the processor confirms an instrument and / or a mirror provided on the vehicle within the display area during the driving mode, it stops displaying within the divided area including the confirmed instrument and / or mirror.

11. An information display device according to claim 1, wherein the processor divides the image display area in the driving mode and sets a different display coordinate system for each divided display area.

12. An information display device according to claim 11, wherein the processor divides the display area into at least two or more areas so that at least one area includes an image in the direction of travel of the vehicle.

13. An information display device according to claim 11, wherein the processor divides the display area into at least two or more areas, sets a vehicle-based display coordinate system in at least one area, and sets the information display device-based display coordinate system in at least one area.

14. An information display device to be worn on the head, comprising: a processor; and a display for displaying images, wherein the processor is capable of executing the following processes: determining the driving state; restricting the display when the driving state is determined; displaying the image; and determining when the display restriction while driving has ended.

15. An information display device according to claim 14, wherein the processor is capable of executing image processing on the image that was being displayed when the driving state was determined.

Images