Head-mounted display device and automatic driving system

Abstract

Provided is technology suitable when an occupant is using a head-mounted display device in a vehicle. A head-mounted display device that can be used by a user who is an occupant in a vehicle acquires driving information related to automatic driving from an in-vehicle system of the vehicle. The driving information includes information on automatic driving level or a system notification related to automatic driving. The system notification includes the case of a take-over request (TOR) from automatic driving to manual driving. The head-mounted display device limits the function of the head-mounted display device according to the acquired driving information.

Description

Head-mounted display device and autonomous driving system

[0001] The present disclosure relates to technology for head-mounted display devices such as head-mounted displays (HMDs) and smart glasses, and technology for autonomous driving of automobiles / vehicles.

[0002] Autonomous driving technology for automobiles is advancing. For example, as described in the Ministry of Land, Infrastructure, Transport and Tourism's document "Names of Autonomous Driving Vehicles," there are five levels of autonomous driving: Level 1, in which either the accelerator / brake operation or the steering wheel operation is partially automated, with the driver remaining in charge of driving, and the corresponding vehicle is called a driver-assisted vehicle. Level 2, in which both the accelerator / brake operation and the steering wheel operation are partially automated, with the driver remaining in charge of driving, and the corresponding vehicle is called a driver-assisted vehicle. Level 3, in which the automated driving system takes over all driving operations in limited areas where specific driving environment conditions are met. However, if there is a risk that the automated driving system may not function properly while in operation, an alarm is sounded urging the driver to take over driving operations, and the driver must respond appropriately. The automated driving system remains in charge of driving operations, and if the automated driving system is unable to operate properly, the driver remains in charge. The corresponding vehicle is called a conditionally automated vehicle. Level 4, in which the automated driving system takes over all driving operations in limited areas where specific driving environment conditions are met. The subject of driving operation is the automated driving device, and the corresponding vehicle is called an autonomous vehicle. Level 5 is a state in which the automated driving device takes over all driving operations, the subject of driving operation is the automated driving device, and the corresponding vehicle is called a fully automated vehicle.

[0003] For example, an override function is necessary for Level 3 autonomous driving, where autonomous and manual driving coexist. Under certain circumstances (in other words, in an environment that satisfies a specified operational design domain), such as when the vehicle is congested on a highway (e.g., traveling at a speed of less than X km / h), autonomous driving is possible. However, when the autonomous driving system (in other words, a car, an in-vehicle system, etc.) determines that it is difficult to continue autonomous driving for some reason in a situation that does not satisfy the operational design domain, or even if the operational design domain is satisfied, it issues a takeover request (TOR) to the driver, requesting a switch from autonomous driving to manual driving (in other words, a handover, override, etc.).

[0004] An example of the prior art is Japanese Patent Laid-Open Publication No. 2022-137590 (Patent Document 1). Patent Document 1 describes an "in-vehicle head-mounted display device equipped with a takeover request notification function for an autonomously driven vehicle." Patent Document 1 describes that "when the in-vehicle HMD device 10 is worn by a driver or passenger P while the vehicle is in autonomous driving mode and a content image is displayed on the image display, upon receiving TOR, an image of the area ahead in the direction of travel of the vehicle is displayed on the image display by gradually increasing the brightness and superimposing it on the content image."

[0005] Japanese Patent Application Laid-Open No. 2022-137590

[0006] A vehicle may change its autonomous driving level / autonomous driving control state depending on the situation. In particular, as described above, when switching from an autonomous driving state to a state where the driver is in manual driving mode, in other words, when lowering the autonomous driving level, the vehicle may issue a notification such as the TOR (sometimes referred to as a system notification) to the driver or other occupants. The driver must respond to such a notification promptly and take appropriate action. For example, in response to a TOR, the driver must shift control of the driving from the autonomous driving system to manual driving operation by the driver. If the driver does not respond to a TOR, the autonomous driving system will implement emergency measures. Such action must be taken appropriately while ensuring traffic safety. Notifications such as a TOR are important and must not be overlooked. At autonomous driving level 3, the autonomous driving system requires a driver monitoring function that monitors hands-off, line of sight, seating, etc., and a system status recording function that records the driver's operations and the system status.

[0007] On the other hand, in an autonomous vehicle, users such as the driver who is an occupant can wear and use a display device such as an HMD or smart glasses (in other words, a head-mounted display device). In particular, in an autonomous driving mode with a high level of autonomous driving, the driver and other occupants do not need to keep looking at the outside world. Therefore, in a situation where traffic safety is ensured in the autonomous driving mode, the occupants can wear an HMD or the like and use or view applications, content, etc.

[0008] However, when a user wears and uses a head-mounted display device such as an HMD inside a vehicle, there is a risk that the user will overlook important system notifications / driving information such as TOR, abnormality notifications, and alerts that occur during automated driving. If an important system notification such as TOR occurs while the occupant is immersed in, for example, VR content using an HMD, the occupant must not miss the system notification, and must check the external situation around the vehicle and take appropriate action, such as switching to manual driving operation.

[0009] When a user who is a passenger in a vehicle wears and uses a head-mounted display device such as an HMD to use or view applications / content related to display or audio, it is desirable to be able to prioritize and ensure traffic safety while also achieving convenience, such as allowing the user to use the desired applications / content.

[0010] An object of the present disclosure is to provide, with regard to the above-mentioned head-mounted display device and autonomous driving technology, technology that is suitable for when a passenger uses a head-mounted display device inside a vehicle.

[0011] The present disclosure provides a technology for enabling a user who is an occupant of an autonomous vehicle to use a head-mounted display device such as an HMD to receive a notification such as TOR from the vehicle without overlooking the notification and to reliably accept the notification and take appropriate action in response. The vehicle transmits a notification such as TOR to the head-mounted display device, and the head-mounted display device receives the notification. The head-mounted display device may output the notification (or information associated with the notification) to the user by display or audio. The head-mounted display device restricts the functions of the head-mounted display device in accordance with the notification / information. For example, in the case of TOR, it is necessary to switch to manual driving, so the function restrictions include not wearing or not using the head-mounted display device, or restricting the use of at least some functions / applications / content, etc.

[0012] A representative embodiment of the present disclosure has the following configuration: The embodiment is a head-mounted display device that can be used by a user who is an occupant in a vehicle, and acquires driving information related to autonomous driving from an in-vehicle system of the vehicle, the driving information including information related to an autonomous driving level or a system notification related to autonomous driving, and the system notification includes a takeover request (TOR) from autonomous driving to manual driving, and restricts the functions of the head-mounted display device according to the acquired driving information.

[0013] According to a representative embodiment of the present disclosure, with regard to the head-mounted display device and autonomous driving technology, it is possible to provide a technology suitable for when a passenger uses a head-mounted display device in a vehicle. According to this embodiment, the passenger can use the head-mounted display device in a vehicle while prioritizing and ensuring traffic safety. Problems, configurations, effects, etc. other than those described above are described in the description of the embodiment of the invention.

[0014] 1 shows an example of the configuration of a vehicle in this embodiment. 2 shows examples of vehicle information of a controller and HMD information of an HMD in this embodiment. 3 shows an example of the configuration of seats and the like in a vehicle in this embodiment. 4 shows examples of HMD modes and the like in this embodiment. 5 shows an example of the configuration of the interior of an HMD in this embodiment. 6 shows an example of the configuration of an HMD exterior in this embodiment. 7 shows an example of the configuration of an in-vehicle system in this embodiment. 8 shows whether or not HMD restrictions exist for a manually driven vehicle and an automatically driven vehicle in this embodiment. 9 shows a processing flow of an HMD in this embodiment. 10 shows an example of the processing of step S1 in this embodiment. 11 shows an example of the processing of step S2 in this embodiment. 12 shows an example of determining a seating position in this embodiment. 13 shows an example of an HMD function restriction stage in this embodiment. 14 shows an example of the importance of a notification in this embodiment. 15 shows an example of the processing of step S3 in this embodiment. 16 shows an example of the processing of step S4 in this embodiment. 17 shows an example of the processing of step S5 in this embodiment. 18 shows a method in which an HMD recognizes the output of an in-vehicle system in this embodiment. 19 shows an example of the processing of step S6 in this embodiment. 1 shows an example of processing in step S7 in the present embodiment. 2 shows an example of notification output from an HMD in the present embodiment. 3 shows an example of notification output from an HMD in the present embodiment. 4 shows an example of notification output from an HMD in the present embodiment. 5 shows an example of notification output from an HMD in the present embodiment. 6 shows an example of output of a menu or the like on an HMD in the present embodiment. 7 shows an example of processing in step S2 in the present embodiment when there are multiple HMDs. 8 shows an example of processing in step S3 in the present embodiment when there are multiple HMDs. 9 shows an example of processing in step S5 in the present embodiment when there are multiple HMDs. 10 shows an example of processing in step S6 in the present embodiment when there are multiple HMDs. 11 shows an example of processing in step S7 in the present embodiment when there are multiple HMDs. 12 shows an example of processing in cooperation between an in-vehicle system and an HMD in the present embodiment. 13 shows an example of association between HMD function restriction stages and autonomous driving levels in the present embodiment. 14 shows an example of a case where a user's decision / input is required in response to a system notification in the present embodiment.1 shows an example of notification output according to whether an HMD is worn or not worn during automatic driving in this embodiment. 2 shows an example of notification output according to whether an HMD is worn or not worn during manual driving in this embodiment. 3 shows an example of control on the time axis in this embodiment.

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, identical parts are generally designated by the same reference numerals, and repeated explanations will be omitted. In the drawings, the representation of components may not represent their actual positions, sizes, shapes, ranges, etc., in order to facilitate understanding of the invention.

[0016] For the purpose of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main subject, but the main hardware subject in these cases is the processor, or a controller, device, computer, system, etc. configured with the processor, etc. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A computer executes processing according to a program read into memory using resources such as memory and communication interfaces as appropriate through the processor. This realizes predetermined functions, processing units, etc. A processor is configured, for example, with semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing and can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs, ASICs, and CPLDs can be used.

[0017] The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer. The program source may be a program distribution server on a communication network, or a non-transitory computer-readable storage medium, such as a memory card or disk. The program may be composed of multiple modules. The computer system may be composed of multiple devices. The computer system may be composed of a client-server system, a cloud computing system, an IoT system, etc. Various data and information may be composed of structures such as tables and lists, for example, but are not limited to these. Expressions such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0018] [Solution, etc.] In the present embodiment, a head-mounted display device such as an HMD that can adapt to an environment in which a vehicle is autonomously driven, and an autonomous driving system (in-vehicle system) that can adapt to the presence of the head-mounted display device such as an HMD are provided. During autonomous driving of the vehicle, a user such as a driver or a passenger uses a head-mounted display device such as an HMD as appropriate. The vehicle notifies the user / wearer of vehicle information / driving information as appropriate. In this case, the vehicle may output the notification using a display, audio, or other means provided by the in-vehicle system, or the notification may be transmitted from the in-vehicle system to the head-mounted display device such as an HMD via communication, causing the head-mounted display device to output the notification.

[0019] For example, during autonomous driving, an in-vehicle system (autonomous driving system) of a vehicle generates a notification such as TOR, which requests a user to make a decision or take an action, depending on an event or situation (e.g., an abnormality / danger related to driving, or a state in which it is impossible to make a decision). The in-vehicle system transmits the notification such as TOR from a controller to an HMD of the user (e.g., the driver). The HMD receives the notification and outputs the notification to the user by display, sound, or vibration. The user (e.g., the driver) recognizes the notification (TOR), makes a decision, and switches to manual driving operation, for example.

[0020] In the system of this embodiment, the in-vehicle system and head-mounted display device grasp the conditions inside and outside the vehicle and notify / output vehicle information / driving information (including notifications such as TOR) to appropriate occupants. Furthermore, this system (in-vehicle system and head-mounted display device) controls the operation restrictions of the head-mounted display device according to the autonomous driving level, etc., and changes (strengthens or relaxes) the level of restrictions as appropriate. For example, this system strengthens the level of restrictions when the autonomous driving level is low or decreasing, and relaxes the level of restrictions when the autonomous driving level is high or increasing.

[0021] 1. Functions of the head-mounted display device.

[0022] 1-1: HMD function restrictions. A head-mounted display device such as an HMD acquires driving information / vehicle information such as the autonomous driving level from a vehicle (in-vehicle system) and restricts the HMD's functions / applications / contents (sometimes collectively referred to as functions) according to the autonomous driving level. In addition to the autonomous driving level, levels of function restriction are established. For the sake of explanation and distinction, these levels of function restriction are referred to as function restriction stages (stages). For example, if there are stages from 0 to M, the higher the stage number, the stronger the restriction.

[0023] For example, in the case of manual driving (level 0 of automated driving) or level 1 or level 2 partial automation, the HMD's function restriction level is set to unusable / unwearable (e.g., level 4). In the case of level 3 conditional automated driving, level 3 allows some functions to be available, and some functions are unavailable (e.g., level 3 or level 2). In the case of level 4 or level 5 automated driving, or when not in the vehicle, all or many functions are available (e.g., level 0 or level 1).

[0024] 1-2: Control when receiving autonomous driving control information notification: A head-mounted display device such as an HMD receives driving information / vehicle information, including system notifications such as TOR, from a vehicle (in-vehicle system), and determines a control operation based on the importance of the driving information / vehicle information. For example, if the notification is of high importance, the processing and output of the notification is controlled to take priority over content use, and the HMD's function may be restricted to a greater level.

[0025] 1-3: Control based on the response of the notified occupant: A head-mounted display device such as an HMD outputs and transmits notifications, etc. to the user (the occupant) based on driving information / vehicle information such as notifications received from the in-vehicle system, and determines the subsequent control action depending on the user's response (reaction) to the notifications, etc. For example, if the user immediately takes over manual driving as a response to TOR, it is determined that there is no problem. If the user does not take a response action, it is determined that there is a problem, and control to deal with the problem is executed, such as notifying again, strengthening the notification, notifying other occupants, or automatic safety stop control.

[0026] 1-4: Understanding the position and status of the HMD and control based thereon: This system (head-mounted display device and autonomous driving system) grasps the status and state of the head-mounted display device such as an HMD and the user wearing the HMD as information. This may be grasped using HMD sensors, vehicle sensors (in-vehicle cameras, seat sensors, etc.), a mobile device paired with the HMD, registration information / settings information for the in-vehicle system / HMD, etc. For example, it may grasp whether the user wearing the HMD is a driver (a person with a driver's license) or a non-driver passenger. It may grasp whether the user holds a driver's license, whether they are an adult, etc. It may also grasp the user's seating position and status in the vehicle. It may also grasp whether the user is wearing an HMD or the like. It may also grasp the power on / off status of the HMD or the like, the functions / applications / contents being used, and the usage status (whether playing / watching, etc.). It may also grasp the user's line of sight, wakefulness state (drowsiness), etc. This system controls notification destinations and HMD function restriction levels based on the grasped situation and state.

[0027] 1-5: Control based on content usage status: This system determines the timing and notification means / method of system notifications, etc., depending on the function / application / content being used (during playback, etc.) on a head-mounted display device such as an HMD. For example, when a user is watching VR content, this system interrupts the content with a notification between segments. For example, this system controls whether to output a notification while continuing playback of the content, or to pause playback of the content and output the notification. For example, this system controls whether to display only the notification, superimpose the notification, or display it in parallel with the image / video of the content. For example, this system may change the notification means / method depending on whether the sense of vision (display) or hearing (audio) is primary.

[0028] The present system may enable / disable, depending on the situation, a transparent mode in which an optical image of the outside world (surroundings of the vehicle) is visible and a non-transparent mode in which an optical image of the outside world is not visible, as examples of HMD function restriction stages. For example, when the vehicle is in autonomous driving mode, the transparent mode (AR content, etc.) may be permitted and the non-transparent mode (VR content, etc.) may not be permitted as a function restriction stage for the HMD of the driver in the driver's seat. Alternatively, when the vehicle is in autonomous driving mode, both the transparent mode and the non-transparent mode may be permitted as a function restriction stage for the HMD of the driver in the driver's seat. In either case, important notifications such as TOR are given priority over the use of content, etc. Furthermore, when the driver is manually driving, the function restriction stage for the driver's HMD basically disables both the transparent mode and the non-transparent mode (not worn). As a variant, the transparent mode may be permitted. The non-transparent mode is basically disabled because it blocks the driver's vision and hearing from optical images of the outside world.

[0029] 1-6: Control using the relationship between multiple HMDs in a vehicle: This system considers the case where multiple HMDs are used by multiple occupants (users) in a vehicle. In this case, the users' HMDs may operate in cooperation with each other. For example, a master-slave relationship may be established between the users' HMDs. For example, the driver's HMD may be the master (first HMD) as a single HMD, and one or more HMDs of other occupants may be slaves (second HMDs). The master first HMD always receives important system notifications and the like from the in-vehicle system. Then, the master first HMD may transfer (for example, transmit all at once) the system notifications and the like to the slave second HMD.

[0030] For example, suppose a vehicle contains an HMD (first HMD) for a driver who is qualified to drive and can operate the vehicle, and an HMD (second HMD) for a passenger who is not qualified to drive. During autonomous driving, the driver is viewing content on the first HMD. In this case, the system may transmit a system notification such as TOR to the driver's first HMD and a notification to the passenger's second HMD urging the driver to stop using the HMD (viewing content) and switch to manual driving. Alternatively, if the driver's first HMD receives the notification but the driver does not take a corresponding action (switch to manual driving), the first HMD may transmit a notification to the second HMD urging the driver to stop using the HMD (viewing content) and switch to manual driving. Upon receiving the notification, the passenger can urge the driver to stop using the HMD (viewing content) and switch to manual driving.

[0031] Furthermore, for example, when there are two or more occupants capable of driving a vehicle, the HMDs of the occupants can be linked together. For example, a first HMD for a first user capable of driving and a second HMD for a second user capable of driving can be assigned. Priority may be set between users and HMDs. For example, the first user seated in the driver's seat is assigned first priority, and the second user seated in the passenger seat next to him is assigned second priority. The system first transmits a system notification, such as TOR, to the first HMD of the first user with first priority. The system then checks whether the first user who received the notification has taken appropriate action (e.g., switching to manual driving). If the first user does not take appropriate action, the system transmits a similar notification to the second HMD of the second user with second priority. Alternatively, the system transfers a similar notification from the first HMD to the second HMD. This allows the second user to take appropriate action (e.g., switching to manual driving). Alternatively, the system may send notifications simultaneously to both a first HMD of a first user and a second HMD of a second user.

[0032] 1-7: Resume playback function: This system has a resume playback function that records and saves information at the time of interruption so that the content can be resumed from the interrupted point after the user completes the corresponding action when the content playback on the HMD is interrupted due to a system notification such as TOR and the user is required to perform the corresponding action. After the user completes the corresponding action, when the content can be resumed, this system reads the information at the time of interruption and resumes it from the interrupted point.

[0033] 2. Functions of the in-vehicle system.

[0034] 2-1. Notification to HMD: The vehicle's in-vehicle system (in other words, the autonomous driving system) communicates driving information / vehicle information, including system notifications, to a head-mounted display device such as an HMD in the vehicle. The in-vehicle system changes the notification means / method depending on the type, content, importance, etc. of the target information. The in-vehicle system monitors the status of the user and HMD in the vehicle and determines the user / HMD to notify, priority, etc. When issuing a notification, the in-vehicle system determines whether to output the notification from the in-vehicle system's equipment (display or audio), from the HMD, or both, depending on the situation.

[0035] 2-2. Understanding Vehicle Status: The in-vehicle system understands the number of users who are occupants in the vehicle, their personal information, and their status, as well as the number and status of HMDs associated with each user. Based on this understanding, the in-vehicle system performs registration and configuration necessary for notifications. Note that the correspondence between users and HMDs is not limited to one-to-one. A single user may be able to use multiple HMDs (the user selects and wears the HMD they want to use), or multiple users may use a single HMD (the user who wants to use that HMD wears that HMD). The means used for understanding the status are not limited to, for example, cameras, sensors, communications, etc. The system understands the number of users and HMDs in the vehicle, which users are wearing and using which HMDs, which users are sitting in which seats, whether the users are licensed drivers, etc. The destination to which the in-vehicle system notifies / sends system notifications, etc., is not limited to HMDs, but may also be a mobile device paired with the HMD. In the case of a passenger who is not wearing an HMD, notification to the passenger's mobile device is effective.

[0036] 2-3. Selection of Notification Destination and Priority: The in-vehicle system determines the notification destination, priority / priority, notification method, etc., depending on the status of each user and HMD, the content and importance of the notification, etc., as described in 2-2 above. For example, in the case of a request for manual driving operation, such as TOR, the in-vehicle system assigns the highest priority (first priority) to the occupant sitting in the driver's seat or a licensed driver as the notification destination. Other occupants may also be notified as an additional occupant, and such other occupants may be assigned a lower priority. Furthermore, in the case of a notification requiring a decision, such as a request for a route change, the in-vehicle system may not only notify the occupant in the driver's seat but also occupants in seats other than the driver's seat. The notification destination for a certain system notification may be single or multiple. Furthermore, for occupants who are not using or wearing an HMD, the notification destination may be the occupant's mobile device. Furthermore, for occupants who are using or wearing an HMD but are not viewing or using immersive content, the notification destination may be the HMD itself. Furthermore, the in-vehicle system sets a function restriction level for the HMD in response to the notification, and determines a notification method in response to the level.

[0037] 2-4. Driving Operation While Wearing an HMD: When manually driving a vehicle, the in-vehicle system basically restricts the HMD function by disabling (prohibiting) the driver from wearing and using the HMD. When a user performs a driving operation while wearing and / or using an HMD, the in-vehicle system may invalidate the driving operation and prevent it from being reflected in driving control. For example, during autonomous driving, the in-vehicle system generates a TOR based on a condition judgment and transmits the TOR to the driver's HMD. Upon receiving the TOR, the HMD suspends the content and outputs the TOR to the user. That is, the HMD prompts the user to remove the HMD, stop viewing the content, and perform manual driving operations. In the normal flow, the user recognizes the TOR, stops viewing the content, removes the HMD (puts it off), and performs manual driving operations. In another case, the user manually operates the TOR while wearing the HMD, without stopping viewing the content.

[0038] The in-vehicle system (or HMD) detects and determines the user's actions and status as described above. Based on this, the in-vehicle system determines whether to enable or disable the manual driving operation of the user. In the former case, the in-vehicle system enables the manual driving operation because the user is not wearing the HMD (in other words, not using it). In the latter case, the in-vehicle system disables the manual driving operation because the user is wearing the HMD (in other words, using it). In such a case of disabling the manual driving operation, the in-vehicle system may output a notification such as a warning to the user / HMD that the manual driving operation is disabled because the user is wearing the HMD.

[0039] 3. Other Solutions.

[0040] 3-1. Recording of information: This system records various data and information such as the status of the user and HMD, the status of the vehicle and in-vehicle system, driving information / vehicle information, system notification information, and user operation history, and makes it possible to refer to and check this information later.

[0041] 3-2. Evaluation of User Response to Notification: When a notification is received from the in-vehicle system requesting a user's decision or operation, the system checks and evaluates the user's response to the notification (i.e., response), and stores and outputs the evaluation. For example, the system may check and understand whether or not the user responded to a high-priority notification, such as TOR, and whether the response was fast or slow, and assign an evaluation value to the user. The system may determine subsequent control actions for the user based on the evaluation value. In particular, if the user does not respond to a high-priority notification or responds slowly, the system may warn the user, lower the evaluation, and set an HMD function restriction level corresponding to the low evaluation. For users who do not respond or who repeatedly respond slowly, the system may temporarily set a stricter HMD function restriction level. Furthermore, if the user's response improves, the system may subsequently return the HMD function restriction level to its original level.

[0042] <Embodiment 1> A head-mounted display device (HMD) and an automatic driving system according to an example (Embodiment 1) will be described using Figure 1 and subsequent figures. Embodiment 1 assumes a case in which there is one driver as an occupant in a vehicle and the driver uses an HMD. The following description will be based on an example in which there is one HMD in the vehicle. Embodiment 2, which will be described later, assumes a case in which there are multiple occupants in the vehicle, including a driver and non-drivers, and multiple HMDs.

[0043] Currently, there are no legal restrictions on the use of head-mounted displays such as HMDs and smart glasses by vehicle occupants, and it is left up to the user's morals, etc. Although there are risks in using HMDs while manually driving a vehicle, as partial and fully automated driving advances in the future, it is expected that there will be an increasing number of decisions that there is little risk if the user uses an HMD under appropriate conditions during automated driving.

[0044] In this embodiment, even if a passenger uses a head-mounted display device such as an HMD in an autonomous vehicle, the system controls the operation of the head-mounted display device to prevent accidents. The system of this embodiment determines functional limitations on the operation of the HMD, taking into account the vehicle's driving control level (e.g., manual driving and autonomous driving levels 1 to 5), the user's seating position in the vehicle, and the state of the HMD. The system of this embodiment notifies the HMD of driving information such as TOR from the in-vehicle system. At that time, the system controls the notification, taking into account the HMD's functional limitations.

[0045] [Vehicle / In-Vehicle System / Autonomous Driving System] FIG. 1A shows a vehicle (or an autonomous vehicle) 2 equipped with an in-vehicle system (or an autonomous driving system) 100, in which a user U1, who is a passenger (e.g., a driver), is wearing an HMD 1 inside the vehicle 2. The in-vehicle system 100 includes an autonomous driving system and a controller 10 that controls the autonomous driving, etc. The controller 10 is, for example, a control device / control unit also called an ECU (Electronic Control Unit), and controls the entire in-vehicle system 100. The autonomous driving system, which is the in-vehicle system 100, is an arbitrary function or system that can appropriately switch between at least an autonomous driving mode and a manual driving mode and control the operation of each mode. The controller 10 may be connected to other devices, such as a car navigation system.

[0046] There are no limitations on the type or detailed configuration of the HMD 1. The HMD 1 may be a transmissive (optical see-through or video see-through) HMD, a non-transmissive (VR) HMD, or an HMD that has both a transmissive mode and a non-transmissive mode.

[0047] The HMD 1 and the controller 10 may be connected by any technical means. As an example, the HMD 1 and the controller 10 may be communicatively connected via a Car Area Network (CAN). There may also be cases where the HMD 1 and the controller 10 are not connected. In that case, the HMD 1 may indirectly acquire information output by the controller 10 (in-vehicle system 100) by a predetermined technical means (image recognition or voice recognition, which will be described later).

[0048] The controller 10 generates or acquires vehicle information 4 (in-vehicle system information) as shown in Fig. 1B. The controller 10 can output the vehicle information 4 to the user by display or sound using the equipment of the in-vehicle system 100, and can also transmit the vehicle information 4 to the HMD 1 via communication. On the other hand, the HMD 1 generates or acquires HMD information 7 as shown in Fig. 1B. The HMD 1 can transmit the HMD information 7 to the controller 10 via communication. In addition, any control information for cooperative operation as a system may be exchanged between the controller 10 and the HMD 1 via communication.

[0049] The controller 10 and the HMD 1 may each be communicatively connected to an external device, such as a server 8, via a communication network 9 (including a mobile network, the Internet, etc.). The controller 10 and the HMD 1 may each receive and acquire data from the external server 8, etc., or may transmit and register data to the external server 8, etc.

[0050] The images, videos, and sounds played back by the HMD 1 may be images, videos, and sounds of any function / application / content, etc. The various functions, etc. also include a graphical user interface (GUI). Examples of applications include video games, movies, music, etc. Another example of an application is an AR-type application that superimposes an assistance image, etc. on an optical image of the outside world (image of the vehicle's surroundings). Data such as content played back by the HMD 1 may be stored in advance in memory resources within the HMD 1, or may be obtained from outside the HMD 1 (for example, the in-vehicle system 100, the server 8, etc.).

[0051] Applications and the like provided in the HMD 1 execute predetermined processes according to programs. For example, a video game application controls the operation input, screen display, and audio output of the game. For example, a movie application controls the operation input, screen display, and audio output of the movie. For example, a music application controls the operation input, screen display, and audio output of music. Note that the programs and data of the applications and the like of the HMD 1 may be stored in an external device (such as the server 8), and processing / services may be realized in the form of a client-server system or the like. For example, data for games / movies / music may be distributed to the HMD 1 from the external server 8 (which may be located in the vehicle 2).

[0052] The HMD 1 may be equipped with a controller 1B (e.g., a remote control or a sensor). For example, when a user U1 plays a video game using the HMD 1, the user U1 operates the controller 1B, which is a game controller. Input / detection / operation signals from the controller 1B are transmitted to the HMD 1 via communication. The processor of the HMD 1 processes the game and controls the video and audio of the game based on the operation signals. In addition to operations using the controller 1B, the HMD 1 may also accept operations based on the user U1's gaze or gestures. For example, the camera 110 and the gaze sensor, acceleration sensor, and gyro sensor included in the sensor group 115 may detect the user U1's gaze, move a selection cursor based on the gaze movement, and select / determine the location where the cursor is pointing when actions such as closing the eyes or blinking are detected. Alternatively, the camera 110, acceleration sensor, gyro sensor, etc. may detect the movement of the user U1's hand, move the selection cursor based on the hand movement, and select / determine the location where the cursor is pointing when actions such as touching a virtual icon are detected. Cursor movement by eye movement and selection / determination by gestures may be combined.

[0053] The HMD 1 may be connected to a mobile terminal 15 such as a smartphone carried by the user U1 via communication. Instructions, video data, and the like may be transmitted from the mobile terminal 15 to the HMD 1.

[0054] [Vehicle Information and HMD Information] FIG. 1B shows an example of the contents of vehicle information (in-vehicle system information) 4 of the vehicle 2 and HMD information 7 of the HMD 1.

[0055] The controller 10 acquires / generates vehicle information 4 from various sensors ( FIG. 4 ) installed in the vehicle 2. The various sensors detect various events that occur in the vehicle 2 and detect various parameter values ​​related to the driving conditions of the vehicle 2. The controller 10 uses the vehicle information 4 to control the driving of the vehicle 2.

[0056] The controller 10 can transmit the vehicle information 4 to the HMD 1 based on its own judgment or based on a request from the HMD 1. The vehicle information 4 to be transmitted may be information such as numerical values ​​or character strings, or may be video data or the like that has been processed to be suitable for display on the HMD 1. The HMD 1 may process the vehicle information 4 to generate video data or the like.

[0057] The vehicle information 4 may include, for example, the following information: speed information (also referred to as vehicle speed) of the vehicle 2, gear information, steering angle information, lamp illumination information, external light information, distance information, seat occupancy information, engine ON / OFF information, camera image information, acceleration gyro information, GPS (Global Positioning System) information, navigation information, vehicle-to-vehicle communication information, road-to-vehicle communication information, and information input by the driver.

[0058] The vehicle information 4 includes driving information (automated driving related information) 4A. The driving information 4B includes automated driving level information, driving control information, system notifications, other user interface information, etc. The system notifications are notification information from the in-vehicle system 100. The system notifications include a request for handover from automated driving to manual driving (TOR), a notification of a transition from manual driving to automated driving, a notification of automatic safety stop control, an abnormality / hazard alert, navigation information, etc.

[0059] The seat occupancy information is information on the seating state, such as which occupant is seated in which seat 6, obtained based on the seating sensor 510. The camera image information is data on images captured by on-board cameras (cameras 531 and 532 in FIG. 4) or information on the analysis results of the images. The image of the vehicle surroundings may be acquired by the on-board camera or by the camera 110 (FIG. 2) of the HMD 1.

[0060] The input information from the driver is any information input by the driver through an operation input device (such as the vehicle operation switch 511 in FIG. 4 or a touch operation on a console panel) provided in the vehicle 2 (in-vehicle system 100). An example of the operation input device is a button (such as the vehicle operation switch 511 in FIG. 4) provided on the steering wheel (steering) 5.

[0061] The HMD information 7 includes information about the HMD 1, such as device information, ID, attributes, and status. The HMD information 7 includes information about the user, such as person information, ID, attributes, and status. The HMD information 7 includes, as wearing information, information about whether the user is wearing or not wearing the HMD 1. The HMD information 7 includes, as usage information, information about functions / applications / contents used by the user and their usage status (playing, paused, etc.). The HMD information 7 may also include seat occupancy information (information indicating the seat in which the user of the HMD 1 is seated, etc.). The HMD information 7 may also include corresponding action information (such as a confirmation result of whether the user of the HMD 1 has performed a corresponding action in response to a system notification). The HMD information 7 may also include input information, operation information, sensor detection information, etc., of the user of the HMD 1.

[0062] The in-vehicle system (autonomous driving system) 100 may transmit a notification to the occupant's HMD 1 when it is necessary to ask for the occupant's decision, not limited to TOR. For example, the in-vehicle system 100 may set a route to a destination based on a navigation system and traffic conditions and perform autonomous driving according to the route. The in-vehicle system 100 determines whether to change the route to the destination depending on changes in traffic conditions. For example, when a traffic jam occurs, the in-vehicle system 100 may determine whether to detour around the traffic jam. The in-vehicle system 100 may notify the user to ask for their decision on whether to change the route to the destination. The in-vehicle system 100 may also notify the user of the status, such as when the vehicle approaches or arrives at the destination, or the distance to the destination. The in-vehicle system 100 may also notify the user of the status of the energy (gasoline, charging power, etc.) of the autonomous vehicle.

[0063] [Seats, Multiple Occupants, Multiple HMDs] Fig. 1C shows an example of the configuration of seats 6 and the like in a vehicle 2, in which there are multiple occupants (users) and multiple HMDs 1. The seats 6 include seats 61, 62, and 63. Seat 61 is a driver's seat equipped with a steering wheel 5 (an example of a right-hand drive vehicle) and is occupied by user U1, who is the driver. Seat 62 is a passenger seat next to the driver's seat, in which user U2, who is a passenger (non-driver), is seated. Seat 63 is a rear seat, in which users U3, U4, and U5, who are passengers (non-drivers), are seated.

[0064] User U1, who is the driver, is qualified to drive (in other words, holds a driver's license) and wears HMD #1 as HMD 1. When vehicle 2 is in automatic driving mode, user U1 is not performing any driving operations, but wears HMD #1 and can use functions, etc. When vehicle 2 is in manual driving mode, user U1 is performing driving operations and is not wearing HMD #1. User U2 wears and uses HMD #2. User U3 wears and uses HMD #3. User U4 is not wearing HMD 1. User U5 wears and uses HMD #5.

[0065] The in-vehicle system 100 generates a system notification in response to the operating status of the vehicle 2, for example, in response to detection of an abnormality / danger, occurrence of an inability to make a judgment, etc. The in-vehicle system 100 may transmit the system notification to the HMD 1 to communicate it to the user wearing the HMD 1.

[0066] [HMD Modes] FIG. 1D is an explanatory diagram of the modes of the HMD 1. (A) shows an optical image of the outside world (such as the roads around the vehicle 2 and other vehicles) seen through the windshield 3 by a user U1 (in the case of a left-hand drive vehicle) who is the driver of the vehicle 2. (B) shows a case in which the user U1 in the vehicle 2 wears the HMD 1 and, in the transmission mode of the HMD 1, displays an image 1D1 corresponding to the optical image of the outside world on the display surface 1D0 of the display device (display 112 in FIG. 2). In the transmission mode (in other words, see-through mode, AR mode, etc.), the HMD 1 displays an optical image of the outside world (such as an image of the vehicle's surroundings) or any image to be superimposed on the optical image. (C) shows a case in which the user U1 wears the HMD 1 and, in the non-transmission mode of the HMD 1, displays an image 1D2 of VR content on the display surface 1D0 of the display device. In a non-transparent mode (in other words, a VR mode, a closed mode, etc.), the HMD 1 displays an image (for example, an image of a game or a movie) that is independent of an optical image of the outside world (such as an image of the surroundings of the vehicle).

[0067] When the user is not wearing the HMD 1 in the vehicle 2, the user places the HMD 1 (and the controller 1B) in any location (for example, the center console between the driver's seat and the passenger seat). The vehicle 2 may be provided with an HMD storage area (which may also serve as a charging device) in which the HMD 1 is placed when not being worn. This HMD storage area may be detected by distinguishing between a state in which the HMD 1 is placed (i.e., a non-wearing state) and a state in which the HMD 1 is not placed. The present system may grasp the state of such an HMD storage area and use it for control.

[0068] 2 shows an example configuration of the HMD 1. The HMD 1 includes a camera 110, a distance measurement sensor 111, a display 112, a speaker 113, a microphone 114, a group of sensors 115, an operation input unit 116, an internal bus 117, a control unit 130, etc. The control unit 130 includes a communication unit (communication interface) 120, a processor 121, a memory 122, a storage 123, etc. The storage 123 includes a basic operation program 124, an application program 125, a riding control program 126, etc. Various programs are loaded into the memory 122 and executed by the processor 121. The storage 123 also stores data for various functions, applications, content, etc.

[0069] The processor 121 is configured using a CPU / MPU / GPU, etc. The memory 122 is configured from a RAM, etc. The storage 123 is configured from a non-volatile storage medium, etc., such as a flash ROM. The processor 121 executes processing in accordance with a program read into the memory 122. This realizes various functions.

[0070] The basic operation program 124 is a program such as an OS for controlling the basic operation of the HMD 1. The application program 125 realizes the processing of applications / content such as games / movies / music, etc. The application program 125 includes, for example, an application (VR application) that provides virtual reality (VR) content to match the screen in non-transparent mode, and an application (AR application) that provides augmented reality (AR) content to match the screen in transparent mode.

[0071] The on-board control program 126 is a program that realizes control processing when a user wearing and using the HMD 1 is on board the vehicle 2 (in other words, when the HMD 1 is inside the vehicle 2). In this embodiment, when on board, the HMD 1 has the functions of communicating with the controller 10, receiving driving information (vehicle information 4 in FIG. 1B ) from the controller 10, outputting the driving information to the user by display or voice, and transmitting HMD information 7 to the controller 10. The on-board control program 126 is a program for controlling such functions.

[0072] The user turns on the power of the HMD 1 and wears it in the vehicle 2. The HMD 1 starts control during riding based on communication with the controller 10 by the riding control program 126. The user views content, for example, by an application program 125, on the HMD 1. The processor 121 acquires content data based on processing of the application program 125, for example, a VR application, displays content video on the display 112, and outputs audio from the speaker 113. The processor 121 processes the application program 125 based on operation input from the operation input unit 126, audio input from the microphone 114, detection signals from the sensor group 115, etc. The processor 121 may acquire content data from an external server 8 or the like via communication with the communication unit 120 using the VR application, for example, or may register data in the external server 8 or the like.

[0073] The camera 110 captures images of real space such as the area in front of the HMD 1. The camera 110 may include multiple cameras and may capture images of the sides or rear of the HMD 1. The camera 110 may include a 360-degree camera and may capture images of the 360-degree surroundings of the HMD 1. The camera 110 may include an infrared camera or the like. Depending on the application, video data captured by the camera 110 may be used. For example, a certain application may display an image of the front of the vehicle captured by the camera 110 on a display surface either as is or after processing.

[0074] The ranging sensor 111 measures the distance to real objects around the HMD 1. Alternatively, the distance to real objects can be measured using multiple cameras 110 (stereo cameras). The ranging sensor 111 may be a sensor that performs three-dimensional measurement using laser light, such as LiDAR. The processor 121 can acquire position information of detected objects using distance measurement data from the ranging sensor 111. The processor 121 can also generate a distance image using the ranging sensor 111 to spatially grasp the distance to surrounding objects. The position information may be world coordinate system information based on real space, or local coordinate system information based on the position of the HMD 1. The processor 121 may detect real objects (e.g., people, vehicles, etc.) inside and outside the vehicle 2 based on the images from the cameras 110 and the information from the ranging sensor 111.

[0075] The display 112 is a display device that displays images and videos in the user's field of view. The display 112 displays images and videos on the display surface based on control by the processor 121 and video data. The display 112 displays images and videos of the user interface of the HMD 1, applications / contents, etc. The display method of the display device (display 112) in the HMD 1 applicable to this embodiment may be a transmissive display device, a non-transmissive display device, or a display device that has both a transmissive mode and a non-transmissive mode.

[0076] The image processing unit realized by the processor 121 performs control processing for displaying an image on the display surface of the display 112 based on data such as functions, applications, and content. Note that an image processing unit may be provided separately from the processor 121. The processor 121 may use the communication unit 120 to acquire application or content data from the outside and store the data in the storage 123. The processor 121 may also use the communication unit 120 to acquire data such as images from the controller 10. The processor 121 may also use the camera 110 to acquire an optical image of the outside world (such as the surroundings of the vehicle 2) seen from the HMD 1, and display the image on the display 112 based on the acquired image data.

[0077] The speaker 113 outputs audio of the user interface, applications, content, etc. of the HMD 1 based on control and audio data from the processor 121. The microphone 114 inputs the voice of the user, etc., as needed, to obtain audio information.

[0078] The sensor group 115 includes sensors for performing various types of sensing, such as sensing the up / down, forward / backward, left / right movements of the wearer, and rotational movements accompanied by changes in the direction of the line of sight. The sensor group 15 includes, for example, a gaze sensor, a GNSS sensor, an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like. The gaze sensor detects the movement and direction of the eyes to capture the viewpoint at the line of sight. The GNSS sensor receives signals from GNSS satellites to detect the current position. The sensor group 15 may also include sensors that use wireless LAN signals, which can be used to detect distance and position. The acceleration sensor can detect acceleration in each axial direction. The gyro sensor can detect angular velocity in each rotational direction. The geomagnetic sensor can detect orientation. These sensors can capture the movement, posture, and the like of the HMD 1 worn on the user's head.

[0079] The operation input unit 116 includes an operation input device such as a button, a light receiving unit (which may be part of the communication unit 120) that receives a signal from the controller 1B, and the like.

[0080] The communication unit 120 is a device or circuit in which a protocol implementation unit corresponding to various communication interfaces (e.g., LAN, Internet, short-range communication, etc.) is implemented, and can be used depending on the purpose. In this embodiment, the HMD 1 communicates with the controller 10 of the in-vehicle system 100 of the vehicle 2. The communication interface of the communication unit 120 includes a communication interface (e.g., CAN) between the HMD 1 and the controller 10. The communication unit 120 may be paired with a user's mobile terminal 15 to communicate. Furthermore, in a second embodiment described below, the HMD 1 may communicate with another HMD 1 in the vehicle 2 through the communication unit 120 (e.g., a short-range communication interface).

[0081] Furthermore, when using the voice recognition and image recognition functions (FIG. 15) described below, the HMD 1 also includes a voice recognition unit and an image recognition unit (not shown) that realize the voice recognition and image recognition functions. These functions may be implemented by hardware or software processing.

[0082] With regard to the processing of a certain application, the processor 121 of the HMD 1 may execute all of the processing, or the processor 121 of the HMD 1 may execute some of the processing, and an external server 8 or the like may execute other parts of the processing. For example, processing with a high computational load, such as processing of a generation AI / LLM, may be performed by a server of a cloud computing system.

[0083] [Appearance of HMD] Fig. 3 shows an example of the appearance of the HMD 1. State 1A is a state seen from the front, and state 1B is a state seen from the back. The example in Fig. 3 shows a goggle-type HMD, but this is not limiting, and glass-type (smart glasses) HMDs are also possible.

[0084] 3, the HMD 1 is a fully immersive HMD and includes a non-transparent display device (display 112). In this example, the display 112 includes a plurality of display elements for the left and right eyes, and displays an image for the left eye and an image for the right eye, thereby allowing the user to recognize a three-dimensional image. The display 112 is capable of three-dimensional display, but may be configured to use other methods.

[0085] The speakers 113 include a speaker 113a for the left ear and a speaker 113b for the right ear. The HMD 1 includes housings 118a and 118b as housings (wearable housings) 118. The housing 118a is worn by placing it against the side or top of the head. The housing 118b is located on the front side and has the display 112, camera 110, distance measurement sensor 111, etc. mounted thereon.

[0086] Note that a system in which the HMD 1 as shown in the figure and the controller 1B or the mobile terminal 15 (FIG. 1) are paired and connected may be considered as an HMD.

[0087] 4 shows an example of the configuration of the in-vehicle system 100. The in-vehicle system 100 has the components shown in the figure that are connected to the controller 10. The in-vehicle system 100 includes a vehicle speed sensor 501, a shift position sensor 502, a steering wheel steering angle sensor 503, a headlight sensor 504, an illuminance sensor 505, a chromaticity sensor 506, a distance measurement sensor 507, an infrared sensor 508, an engine start sensor 509, a seating sensor 510, a vehicle operation switch 511, an acceleration sensor 512, a gyro sensor 513, a temperature sensor 514, a wireless transceiver for road-to-vehicle communication 515, a wireless transceiver for vehicle-to-vehicle communication 516, a wired wireless communication unit for mobile terminal-to-vehicle communication 517, a GPS receiver 518, a VICS (Vehicle Information and Communication System, registered trademark) receiver 519, a communication unit 520, an image generation unit 521, an image analysis unit 522, a camera (in-vehicle camera) 531, a camera (out-vehicle camera) 532, an audio input device 541 (including a microphone, etc.), an audio output device 542 (including a speaker, etc.), an automatic driving control unit 550, etc. The sensors and devices are not limited to these, and can be added, deleted, replaced, etc.

[0088] FIG. 4 illustrates a case where the controller 10 and the automatic driving control unit 550 are provided separately, but this is not limiting, and the automatic driving control unit 550 may be implemented as an integral part of the controller 10.

[0089] The vehicle speed sensor 501 detects the speed of the vehicle 2 (also referred to as vehicle speed) and generates speed information as the detection result. The shift position sensor 502 detects the current gear and generates gear information as the detection result. The steering wheel steering angle sensor 503 detects the current steering angle of the steering wheel 5 and generates steering wheel steering angle information as the detection result. The headlight sensor 504 detects whether the headlights are on or off and generates lamp illumination information as the detection result. The illuminance sensor 505 and chromaticity sensor 506 detect external light and generate external light information as the detection result.

[0090] The distance measurement sensor 507 detects the distance between the vehicle 2 and an external object and generates distance information as the detection result. The infrared sensor 508 detects the presence or absence of an object and the distance to the object in the vicinity of the vehicle 2 and generates infrared information as the detection result. The engine start sensor 509 detects whether the engine is on or off and generates on or off information as the detection result.

[0091] An occupancy sensor 510 is provided in each seat 6 (FIGS. 1A and 1C) in the vehicle 2. The occupancy sensor 510 detects whether an occupant is seated in the seat 6, whether the occupant is fastening a seat belt, and other conditions. The occupancy status and the fastening status may be detected individually. Even if the occupancy sensor 510 is not provided, the occupancy status of each seat 6 may be determined and detected using a sensor such as a camera 531 in the vehicle 2.

[0092] The vehicle operation switch 511 is, for example, a steering switch provided on the steering wheel 5, and receives operation by the driver to generate input information from the driver. The acceleration sensor 512 and gyro sensor 513 detect the acceleration and angular velocity of the vehicle 2 and generate, as the detection results, acceleration gyro information that represents the attitude and behavior of the vehicle 2. The temperature sensor 514 detects the temperature inside and outside the vehicle 2 and generates temperature information that is the detection result.

[0093] The road-to-vehicle communication wireless transceiver 515 generates road-to-vehicle communication information through road-to-vehicle communication between the vehicle 2 and roads, signs, traffic lights, etc. The vehicle-to-vehicle communication wireless transceiver 516 generates vehicle-to-vehicle communication information through vehicle-to-vehicle communication between the vehicle 2 and other nearby vehicles. The mobile terminal-to-vehicle communication wired wireless communication unit 517 generates mobile terminal-to-mobile terminal communication information through communication with the mobile terminal 15. The GPS receiver 518 generates GPS / GNSS information by receiving GPS / GNSS signals from GPS / GNSS satellites. The GPS / GNSS information includes information such as the current time, latitude, and longitude. The VICS receiver 519 generates VICS information by receiving VICS signals. The GPS receiver 518 and the VICS receiver 519 may be provided as part of a navigation system.

[0094] The communication unit 520 is a device or circuit that implements a communication interface with the external communication network 9 and a communication interface (e.g., CAN) with the HMD 1. The image generation unit 521 is a unit that generates image information when an image is displayed on a console panel (to be described later) or the HMD 1. The image analysis unit 522 is a unit that performs image analysis based on images captured by the cameras 531 and 532 and generates analysis result information.

[0095] The in-vehicle camera 531 captures images of the interior of the vehicle 2 to generate in-vehicle camera video information. The exterior camera 532 captures images of the exterior of the vehicle 2 to generate exterior camera video information. In a specific example, the in-vehicle camera 531 captures images of the occupants' positions, postures, eye positions, and movements, constituting a DMS (Driver Monitoring System). The exterior camera 532 captures images of the surrounding conditions, such as the front, rear, and sides of the vehicle 2. Analysis of the exterior camera video information makes it possible to determine the presence or absence of other vehicles or people around the vehicle 2, buildings, terrain, road conditions such as rain, snow, ice, and unevenness, and road signs. The exterior camera 532 also includes a drive recorder that records video of the driving situation.

[0096] The audio input device 541 includes a microphone and the like, and inputs audio. The audio output device 542 includes a speaker and the like, and outputs audio.

[0097] The autonomous driving control unit 550 controls the autonomous driving of the vehicle 2 in accordance with the autonomous driving level. The autonomous driving control unit 550 is capable of generating and outputting driving information 4A (FIG. 1B).

[0098] 5 shows the basic concept of this embodiment (embodiment 1), and illustrates scenes in which a user U1, who is a passenger (such as a driver) in a vehicle 2, uses the HMD 1, where (A) shows a state during manual driving and (B) shows a state during automatic driving. In this embodiment, the HMD function restriction level is controlled and set according to the automatic driving level, etc.

[0099] In (A), the vehicle 2 is being manually driven by the user U1, who is the driver (level 0). The HMD 1 of the user U1 acquires vehicle information 4 (including seating information and driving information 4A) from the controller 10 of the in-vehicle system 100, and determines whether to restrict the use of the HMD 1 (in other words, restrict its functions) based on the vehicle information 4. Alternatively, the controller 10 of the in-vehicle system 100 determines whether to restrict the use of the HMD 1. The determination is made based on the autonomous driving level and the seating state, and the HMD function restriction level is determined. In the case of the (A) state, the determination results in the HMD being unusable / unmountable. This unusable / unmountable HMD corresponds to, for example, level 4 (see FIGS. 10 and 26 described below) as an HMD function restriction level. Note that there are exceptions, such as when the user is a passenger (non-driver), the HMD is allowed to be used.

[0100] In (B), the vehicle 2 is being driven autonomously by the in-vehicle system 100 (e.g., at level 3). The HMD 1 of the user U1 acquires vehicle information 4 (including driving information 4A and seating information) from the controller 10 of the in-vehicle system 100 and makes a determination regarding usage restrictions based on the information. Alternatively, the controller 10 of the in-vehicle system 100 makes a determination regarding usage restrictions for the HMD 1. In the state of (B), the determination results in conditional HMD usability (level 3). This conditional HMD usability corresponds to, for example, level 2 (see FIGS. 10 and 26 described below) as an HMD function restriction level. Note that even during autonomous driving, all functions of the HMD 1 are not permitted (allowed) to be used, but are permitted under certain conditions, such as with priority given to notification output.

[0101] An operation unit 1B (e.g., a device held in the hand for sensing and inputting operations) is attached to the HMD 1. For example, when the occupant plays a video game as an example of an application using the HMD 1, the occupant operates the operation unit 1B.

[0102] In this embodiment, for example, in operation at autonomous driving level 3 or the like, the autonomous driving system (in-vehicle system 100) monitors, as a driver monitoring function, the wearing and use state of the HMD 1, etc. by the occupant (driver). In addition, as a system status recording function, the autonomous driving system records the wearing and use state of the HMD 1, etc. by the occupant in association with the driving operation by the driver and the driving control state of the autonomous driving system.

[0103] [Basic Flow] FIG. 6 shows the basic flow of the HMD 1 in the system of this embodiment. User registration and other procedures are completed in advance for the HMD 1. The HMD 1 also initially includes a guardian setting as one of its settings. The guardian setting is a setting for the range of movement of the user U1 of the HMD 1. The guardian setting is performed so that the user U1 can prevent or guard against hitting the vehicle body when, for example, they reach out and move their hands inside the vehicle 2. Note that the guardian setting may be performed at any timing. For example, when the HMD 1 is put on or before content use, a provisional setting may be performed to a rough range, such as a predetermined range, regardless of the content. Then, when content use begins, the range may be fine-tuned and detailed conditions may be set depending on the content, and a guardian with an appropriate range may be set. At this time, the provisional setting is reviewed and the guardian setting is updated.

[0104] In step S1 , the HMD 1 performs processing to establish a communication connection with the controller 10 of the in-vehicle system 100 .

[0105] In step S2, the HMD 1 performs a process of checking the HMD usage environment. Examples of the usage environment include the level of automatic driving (including whether the driving is manual or automatic), the attributes of the user U1 (for example, whether the user U1 is a qualified driver or a non-driver (passenger)), the applications / contents to be used, etc. In the first embodiment, in step S2, control is performed to check whether the driver is seated in the driver's seat, but this control can be omitted.

[0106] In step S3, the HMD 1 performs a process of setting the functions of the HMD 1, in other words, a process of restricting the functions of the HMD 1. As shown in Fig. 10 and Fig. 26 described later, the HMD function restriction level is set for the HMD 1 depending on the situation.

[0107] In step S4, the HMD 1 performs processing to start operation based on the function setting in step S3.

[0108] In step S5 , the HMD 1 performs processing when it receives driving information such as a system notification from the controller 10 of the in-vehicle system 100 .

[0109] In step S6, the HMD 1 performs a process of checking a response action by the user in response to the received system notification, etc. In step S6, the HMD 1 or the in-vehicle system 100 detects and determines whether the user has started a manual driving operation as a response action to the system notification (e.g., TOR). This detection and determination can be performed using various sensors and cameras of the in-vehicle system 100, based on the state of gripping the steering wheel 5, depressing the accelerator / brake, etc.

[0110] In step S7, the HMD 1 performs post-reception processing such as system notification in accordance with the corresponding operation in step S6. The above processing steps are repeated as appropriate.

[0111] In the following FIG. 7 and the like, a case where the HMD 1 is a single unit in the vehicle 2 is shown.

[0112] [Step S1: Communication Establishment Processing] Fig. 7 is a table summarizing a detailed configuration example regarding the communication establishment processing of step S1. In the table, each column shows a basic operation example, variations (other processing examples, etc.), and remarks. Rows indicated with symbols such as "1-1" indicate processes, units, steps, etc. in a series of processes where divisions, parts, branches, case distinctions, main routines, etc. occur. Symbols such as (1) and (2) in cells indicate units such as subroutines that are processed in chronological order within the processing.

[0113] Process 1-1: (1) The power of the HMD 1 is turned on. (2) The HMD 1 analyzes the image of the camera 110 and determines whether or not the HMD 1 is inside the vehicle 2. (3) If the HMD 1 is inside the vehicle 2, the process proceeds to process 1-2 (in other words, the HMD 1 is in the on-board control mode). If the HMD 1 is not inside the vehicle 2, i.e., if the HMD 1 is outside the vehicle 2, the HMD 1 performs normal operation.

[0114] Variation of process 1-1: (2') The HMD 1 attempts to send a connection request to the in-vehicle system 100 (vehicle 2) (using a predefined protocol, as in 1-2 described below). Then, the HMD 1 determines whether or not the HMD 1 is inside the vehicle 2 based on whether or not there is a response from the in-vehicle system 100 (vehicle 2).

[0115] Process 1-2: (1) The HMD 1 sends a connection request to the in-vehicle system 100. (2) If there is a response to the connection request of (1) from the in-vehicle system 100 and the HMD 1 and the in-vehicle system 100 are newly connected, proceed to (3). (3) Information is exchanged between the HMD 1 and the in-vehicle system 100, and in-vehicle system information is newly registered in the HMD 1, and HMD information is newly registered in the in-vehicle system 100. (4) When the above (3) is completed normally, communication is established between the HMD 1 and the in-vehicle system 100.

[0116] Modification of Process 1-2: (2') If there is a connection history between the HMD 1 and the in-vehicle system 100, proceed to Process 1-3. If there is no response from the in-vehicle system 100, proceed to Process 1-4.

[0117] Process 1-3: (1) The HMD 1 establishes communication between the HMD 1 and the in-vehicle system 100 based on the registered information.

[0118] Modification of Process 1-3: (1') When there is a change in the registered information, the HMD 1 and the in-vehicle system 100 each update the registered information and re-register it.

[0119] Process 1-4: (1) The HMD 1 stops the communication establishment process if there is no response from the in-vehicle system 100. (2) The HMD 1 restricts the functions of the HMD 1.

[0120] Modification of Process 1-4: (1') The HMD 1 may also abort the communication establishment process when a predetermined amount of data cannot be exchanged within a predetermined time due to a delay.

[0121] Process 1-5: (1) When the HMD 1 stops using the HMD 1 or when the in-vehicle system 100 stops, the HMD 1 stops communication between the HMD 1 and the in-vehicle system 100. (2) To facilitate the next connection, the HMD 1 holds the in-vehicle system information, and the in-vehicle system 100 holds the HMD information.

[0122] Modification of Process 1-5: (1') After communication is established, if a communication failure occurs during use of the HMD, the HMD 1 restricts the functions of the HMD 1.

[0123] [Step S2: Checking the HMD Usage Environment] FIG. 8 is a table summarizing an example of a detailed configuration regarding the HMD usage environment checking process in step S2.

[0124] Process 2-1: (1) The HMD 1 checks whether the user is wearing or not wearing the HMD 1. (2) If the HMD 1 is being worn, proceed to (3), and if the HMD 1 is not being worn, proceed to process 2-2. (3) The HMD 1 checks the seating position of the user who is wearing the HMD 1. Based on the check result, the HMD 1 determines the type of seat (for example, driver's seat, passenger seat, rear seat, etc.) in which the user is sitting.

[0125] The method / means for confirming (3) above can be the group of sensors 115 and camera 110 (FIG. 2) provided in the HMD 1, the seating sensor 510 (FIG. 4) and camera 531 of the vehicle 2, or a combination thereof.

[0126] Modification of Process 2-1: (3') includes (3'-1) and (3'-2). (3'-1) If the HMD 1 cannot confirm information about the seating position, it determines that the seating position of the user who is wearing the HMD 1 is unknown, and restricts the functions of the HMD 1. (3'-2) The HMD 1 may also confirm personal information of the user who is wearing the HMD 1 (for example, attributes such as age) and driving qualifications. Based on the confirmation results, the HMD 1 determines whether the user who is wearing the HMD 1 is able to drive.

[0127] The method / means for checking (3'-2) above may utilize user registration information in the HMD 1 or the in-vehicle system 100, the user's mobile terminal 15, or the like.

[0128] Process 2-2: (1) If the HMD 1 is not being worn, it waits until it is worn or the power is turned off.

[0129] Modification of Process 2-2: (1') When the HMD 1 is in a power-on but non-wearable state for a predetermined period of time or more, the HMD 1 transitions to a sleep state or a power-off state.

[0130] In the above-described modified example, it is confirmed whether the occupant sitting in the driver's seat is a qualified driver. Depending on the situation, the occupant sitting in the driver's seat may not be a qualified driver. In this system, when sending important notifications such as TOR to the HMD 1, priority is given to sending the notifications to the HMD 1 of the qualified driver. For this reason, in the above-described modified example, personal information, driving qualifications, etc. are also confirmed to confirm whether the HMD wearer is an appropriate driver.

[0131] [Determining Position Within Vehicle] FIG. 9 shows an example of determining the wearer's position within the vehicle (confirming the seating position) by the HMD 1 in step S2. (A) shows a case where the seating position is determined using information from the camera 110, position sensors, etc. (sensor group 115) of the HMD 1. The HMD 1 estimates and grasps its own position within the vehicle 2. The HMD 1 recognizes the space within the vehicle 2 based on the image from the camera 110, etc., and determines the seating position of the wearer of the HMD 1. The HMD 1 may also make such determination using configuration information (e.g., information representing the configuration shown in FIG. 1C ) of the seats 6, etc., within the vehicle 2. The HMD 1 may also acquire such seat information / configuration information from the in-vehicle system 100. The HMD 1 notifies the in-vehicle system 100 of the seating information (information representing which user is seated in which seat) resulting from the determination.

[0132] (B) is a case where the seating position is determined by acquiring image analysis result information (in-vehicle camera information) of the in-vehicle camera 531 of the vehicle 2 (in-vehicle system 100) from the controller 10. The in-vehicle system 100 notifies the HMD 1 of the image analysis result information (in-vehicle camera information) of the in-vehicle camera 531, seating information (information indicating which user is seated in which seat) of the seating sensor 510, etc. The HMD 1 determines the seating position of the wearer of the HMD 1 based on the information acquired from the controller 10.

[0133] Alternatively, a method that combines both of the above (A) and (B) may be applied. Furthermore, as will be described later, when there are multiple occupants in the vehicle 2, for example, the first HMD of a first user may determine the seating position of another second user.

[0134] [Determining HMD Wearing Status] Any technology can be applied to detect and determine whether the user U1 is wearing the HMD 1 on their head, and the details are not limited. The following examples can be applied: The HMD 1 may detect whether the HMD 1 is worn on their head (worn / unworn) using a sensor of the HMD 1 (for example, a contact sensor provided inside the housing 118 in FIG. 3 ). The HMD 1 may use the sensor group 115 to determine whether the HMD 1 is worn or unworn based on the movement, position, etc. of the HMD 1. The HMD 1 may determine whether the HMD 1 is worn or unworn based on the video image captured by the camera 110. Furthermore, the in-vehicle system 100 may detect and determine whether the user is wearing the HMD 1 based on video analysis result information from the in-vehicle camera 531. Furthermore, if there are multiple occupants in the vehicle 2, it is possible, for example, for the first HMD of a first user to determine the wearing state of the second HMD of another second user.

[0135] [HMD Function Restriction Stages] Fig. 10 is an explanatory diagram regarding the HMD function restriction stages. The table in Fig. 10 lists the functions and notes of the HMD 1 for each HMD function restriction stage. Stage 0 means that all functions / applications / contents, etc. of the HMD 1 are available and there are no restrictions. As a note, in Stage 0, even when a notification is received from the in-vehicle system 100, the applications / contents, etc. are not stopped, and the display of the contents, etc. remains full-screen.

[0136] In stage 1, all functions / applications / contents, etc. of the HMD 1 are available, but there are restrictions when receiving driving information such as system notifications. In stage 1, when the HMD 1 receives a system notification from the in-vehicle system 100, the display of the content, etc. is restricted, for example, by switching from full-screen display to partial display (referred to as notification method 1). Note that in stage 1, after the user performs a corresponding action in response to the system notification from the in-vehicle system 100, the HMD 1 returns the display of the content, etc. to its original state (in other words, before the notification). An example of notification method 1 is example B in FIG. 18A described below.

[0137] In stage 2, all functions / applications / contents, etc. of the HMD 1 are available, but there are restrictions when receiving driving information such as system notifications. In stage 2, when the HMD 1 receives a system notification from the in-vehicle system 100, the content, etc. is temporarily stopped (interrupted) and the display switches to a see-through display of an optical image of the outside world (image of the vehicle's surroundings) in a transparent mode. Note that in stage 2, after the user responds to the system notification from the in-vehicle system 100, the HMD 1 resumes the content, etc., and returns the display of the content, etc., and various play-related settings (including guardian settings, etc.) to their original states. An example of notification method 2 is Example A in Figure 18A, which will be described later.

[0138] The see-through display in stage 2 (notification method 2) is intended to display system notifications such as TOR prominently with as little delay as possible. In this case, the HMD 1 displays the system notification image as is on the screen after pausing, rather than performing video processing such as combining the system notification image with video data such as content.

[0139] Stage 3 is a state in which some functions / applications / content, etc. of the HMD 1, such as immersive content (such as VR games), are unavailable. Other functions / applications / content, etc. are available. Also, in stage 3, similar to notification method 2 in stage 2, when a system notification is received, content, etc. is paused and the display switches to see-through. Note that in stage 3, use as a non-immersive work display, etc. is permitted. For example, a function that superimposes a work / assistance GUI, etc., on an optical image of the outside world in a transparent mode is available. Also, in stage 3, similar to stage 2, after responding to the system notification, content, etc. is resumed and play-related settings are restored to their original state.

[0140] In stage 4, all functions / applications / contents, etc. of the HMD 1 are unavailable. In stage 4, the user must not wear the HMD 1. Note that in stage 4, system notifications from the in-vehicle system 100 are not sent to the HMD 1 set in stage 4 (or even if a system notification is sent, it is not received by the HMD 1). In stage 4, the system notification from the in-vehicle system 100 is switched from a notification to the HMD 1 to a display or audio notification on the in-vehicle system 100 equipment in the vehicle 2, or to a mobile terminal 15 or the like owned by the user. In stage 4, if the HMD 1 detects that the user is wearing the HMD 1, it may output an alert or the like to the effect that wearing the HMD 1 is prohibited.

[0141] The lower part of Fig. 10 shows a state transition diagram relating to stages 0 to 4, and transitions from one stage to the next are possible. The HMD 1 grasps and manages its own function restriction stage (in other words, the stage setting state). The in-vehicle system 100 also grasps and manages the function restriction stage (in other words, the stage setting state) of the HMD 1. The controller 10 of the in-vehicle system 100 may transmit information on the HMD function restriction stage to the HMD 1 as vehicle information 4 (Fig. 1B). The HMD 1 may transmit information on the HMD function restriction stage to the controller 10 of the in-vehicle system 100 as HMD information 7.

[0142] As shown in FIG. 10 , in this embodiment, in addition to known autonomous driving levels, HMD function restriction levels are provided as control levels in this embodiment. The levels are numerically represented as 0 to 4, for example, with larger numerical values ​​indicating stricter restrictions. The number and content of levels may be different, and are not limited to the example of FIG. 10 . Multiple levels may be provided for whether or not functions / applications / content, etc. of the HMD 1 are available, or multiple levels may be provided for notification methods such as system notifications (how to control the output of content, etc. and notifications when notifications are received). For example, in level A, up to application group A may be unavailable, and in level B, up to application groups A and B may be unavailable.

[0143] [Autonomous driving levels and stages] FIG. 26 shows a table showing an example of the correspondence between autonomous driving levels and HMD function restriction stages. Stage 0 corresponds to when the driver is not in the vehicle or to autonomous driving level 5 or 4. Stage 1 corresponds to autonomous driving level 4. Stage 2 corresponds to autonomous driving level 3. Stage 3 corresponds to autonomous driving level 3. Stage 4 corresponds to manual driving (level 0) or autonomous driving level 1 or 2. Details of the correspondence depend on the details of the control implementation of this embodiment.

[0144] As a modified example, an HMD function restriction stage may be associated one-to-one with each autonomous driving level.

[0145] 11 shows an example of setting the importance of a system notification from the in-vehicle system 100. This example is an example of defining the importance according to the content of the system notification. The importance levels are "high," "medium," and "low," with high > medium > low.

[0146] Notification contents with a "high" importance level include TOR, abnormality / danger / fault occurrence notification (alerts, etc.), emergency action execution notification (e.g., automatic safety stop control), etc. Note that such notifications are notifications requesting user operation (e.g., manual driving operation) and notifications regarding abnormalities. Such notifications include important requests / instructions, cautions / warnings, and notifications requesting immediate decisions or response actions from occupants.

[0147] Notifications with a "medium" level of importance include notifications for changes to the autonomous driving level, especially notifications for lowering the level, and requests for route changes. Note that these notifications are notifications that alert the user or require a user decision. These notifications are low-urgency notifications that prompt the occupant to confirm or make a decision.

[0148] Notification contents with a "low" importance level include notifications of changes to the autonomous driving level, particularly notifications when the level is increased, notifications regarding route progress, notifications regarding vehicle control, and notifications regarding the recognition of obstacles, etc. Note that these notifications are notifications that do not require user operation or judgment. Examples of notifications regarding route progress include notifications regarding arrival at the destination, estimated arrival time, and current location. Examples of notifications regarding vehicle control include notifications regarding starting, stopping, turning right or left, etc. These notifications are notifications that do not necessarily require the occupant to recognize, judge, or take action in response.

[0149] The more important the system notification, such as the "high" level of importance, the higher the HMD function restriction level (i.e., the stronger the restriction).

[0150] The importance of the above-mentioned notification may be defined and set in advance by the in-vehicle system 100 or may be defined and set by the HMD 1. Importance information may be added to the vehicle information 4 transmitted from the in-vehicle system 100 to the HMD 1. For example, when a TOR notification is transmitted, the TOR information is assigned a level of importance of "high" and transmitted.

[0151] [Step S3: Function Setting Process] FIG. 12 is a table summarizing an example of a detailed configuration regarding the function setting process in step S3.

[0152] Process 3-1: (1) The HMD 1 confirms the seating position of the user who is wearing the HMD 1. (2) The HMD 1 confirms the driving control status of the in-vehicle system 100. (3) If the wearer is sitting in the driver's seat and the autonomous driving level is 4 or 5, the HMD 1 proceeds to (4). If the wearer is sitting in the driver's seat and the autonomous driving level is 3, the HMD 1 proceeds to process 3-2. If the wearer is sitting in the driver's seat and the autonomous driving level is 1 or 2, the HMD 1 proceeds to process 3-3. If the wearer's seating position is unknown, the HMD 1 proceeds to process 3-4.

[0153] (4) The HMD 1 sets the HMD function restriction level of the user seated in the driver's seat to level 0. (5) When a change to an autonomous driving level other than 4 or 5 occurs, the HMD 1 outputs a notification to the user via the HMD 1 or other devices.

[0154] Modifications of process 3-1: (4'-1) When the autonomous driving level is 4, the HMD function restriction stage may be set to a stage other than 0. (4'-2) The above (4'-1) may be applied only to users who are seated in the driver's seat and have driving qualifications. (5') When an abnormality occurs in communication with the in-vehicle system 100, the HMD function restriction stage is set to stage 3 or higher.

[0155] Process 3-2: (1) The HMD 1 sets the HMD function restriction level of the user seated in the driver's seat to level 3 or higher. (2) When a change to an autonomous driving level other than 3 occurs, the HMD 1 outputs a notification to the user via the HMD 1 or other devices and changes the HMD function restriction level as appropriate.

[0156] Variations on process 3-2: (1'-1) Before setting the level to level 3, the HMD 1 confirms that there are no problems with communication with the in-vehicle system 100 and notifies the user that a corresponding action will be required if a notification of "high" importance is received. (1'-2) If there is an abnormality in communication with the in-vehicle system 100 in the above (1'-1), the user sitting in the driver's seat is set to level 4. (2') When changing the autonomous driving level, the notification method is changed depending on whether the level is changed from level 3 to level 1 or 2 (in the case of a level downgrade) or from level 3 to level 4 or 5 (in the case of a level upgrade). When the level is downgraded, the driving becomes closer to manual driving, so the notification method prioritizes safety, and system notifications are made more noticeable than content, etc.

[0157] Process 3-3: (1) The HMD 1 sets the HMD function restriction level of the user seated in the driver's seat to level 4. (2) When the autonomous driving level is changed to a level other than 1 or 2, the HMD 1 notifies the user by means other than the HMD 1. Following the change, the HMD 1 appropriately sets the HMD function restriction level when the user seated in the driver's seat uses the HMD 1.

[0158] Modifications of process 3-3: (1'-1) When the HMD 1 is set to stage 4, the HMD 1 issues a warning notice to the HMD 1. (1'-2) If the user does not stop using the HMD 1 even after a predetermined time has passed since the warning notice in (1'-1) above, the HMD 1 notifies the in-vehicle system 100 to change the autonomous driving level or perform an automatic safety stop.

[0159] Process 3-4: (1) When the seating position of the wearer is unknown, the HMD 1 sets the function restriction of the HMD 1 to level 4.

[0160] Modification of Process 3-4: (1') When the seated position of the wearer can be identified, the HMD 1 can change the HMD function restriction level at that point.

[0161] As described above, in this system, the HMD function restriction level is determined, set, and controlled according to the occupant's state of wearing an HMD, seating, etc., and driving information such as the autonomous driving level.

[0162] In addition, in process 3-4, when the seating position is unknown, it is not limited to setting the level to level 4, but in a modified example, it may be set to a level set for a driver who should be most restricted (for example, the driver is set to level 3). Generally speaking, drivers need to be able to handle manual driving as well, so HMD function restrictions are the strictest compared to non-drivers.

[0163] [Step S4: Operation Start Processing] FIG. 13 is a table summarizing an example of a detailed configuration regarding the operation start processing in step S4.

[0164] Process 4-1: (1) The HMD 1 presents to the user selectable and available functions / applications / contents etc. at the HMD function limitation stage according to the setting of the stage. For example, a list of selectable and available items may be presented on the screen. (2) The HMD 1 starts playing the application / content selected by the user.

[0165] Modifications of Process 4-1: (1') The HMD 1 outputs a notification to the user that functions / applications / contents, etc. that cannot be selected and used due to the setting of the HMD function restriction level cannot be selected and used. In this case, different output modes may be used for selectable and unselectable content. Note that even if selectable and usable content, etc. are not presented, if content, etc. that the user has instructed to play is unavailable due to restrictions, an output may be made to indicate that the content is unavailable due to restrictions. (2') The same process as Process 3-1 is performed when stopping content, etc. that is being played and reselecting other content, etc.

[0166] [Step S5: Processing Upon Receipt of System Notification] FIG. 14 is a table summarizing an example of a detailed configuration regarding the processing upon receipt of a system notification in step S5.

[0167] Process 5-1: (1) The HMD 1 checks whether a system notification has been received from the in-vehicle system 100. (2) When the HMD 1 receives a system notification, it checks the importance of the notification (FIG. 11). If the importance is "high", it proceeds to (3). If the importance is "medium", it proceeds to process 5-2. If the importance is "low", it proceeds to process 5-3. (3) If the importance is "high", the HMD 1 changes the HMD function restriction level of the user seated in the driver's seat to level 4 if it is other than level 4.

[0168] Variations of Process 5-1: (2') If the HMD 1 has not received a system notification, it returns to (1). (3'-1) The HMD 1 changes the HMD function limitation level and outputs a notification prompting the user to take action in response to the system notification. (3'-2) The above (3'-1) is controlled to be completed within the grace period until a user operation (response) is required. (3'-3) The notification of the above (3'-1) is not limited to one time. The system may repeatedly issue the notification when it is determined that the user's response is slow or when the remaining time in the grace period falls below a threshold. (3'-4) A notification of "high" importance may be output to the user not only via the HDM 1 but also via a display, audio, or other means on equipment in the vehicle 2. (3'-5) The HMD 1 may perform image or audio recognition on the display and audio in the vehicle 2 in (3'-4) to determine whether the system notification has been received and its content (see FIG. 15).

[0169] Process 5-2: (1) When the importance is "medium", if the HMD function restriction level of the user sitting in the driver's seat is level 0, the HMD 1 changes it to level 1 or higher.

[0170] Modifications of Process 5-2: (1') When the HMD function limitation stage is one of stages 1 to 4, the HMD 1 does not need to change the stage.

[0171] Process 5-3: (1) If the importance level is "low", the HMD 1 notifies the user of the HMD 1.

[0172] Modifications of process 5-3: (1'-1) The HMD 1 may not need to stop functions / applications / contents, etc. (1'-2) Depending on the HMD function restriction stage or the type and content of functions / applications / contents, etc., the in-vehicle system 100 may not need to send a notification. (1'-3) In the case of (1'-2) above, the HMD 1 may store notifications received during playback of functions / applications / contents, etc., and output the stored notifications together when the functions / applications / contents, etc. are stopped.

[0173] 15 is an explanatory diagram of a method by which the HMD 1 recognizes and determines a specific notification by display or audio in the in-vehicle system 100 in the vehicle 2. This method can omit implementation related to the communication, as compared with a method realized by communication between the in-vehicle system 100 and the HMD 1.

[0174] In FIG. 15 , a user U1 wearing an HMD 1 is seated in a driver's seat 6 in a vehicle 2 (in-vehicle system 100) such as that shown in FIG. 1 . The vehicle 2 (in-vehicle system 100) includes, as display means, a console display 1501 on a dashboard 1510, a windshield projection display 1502, and a virtual image display 1503. The vehicle 2 (in-vehicle system 100) also includes, as audio means, a speaker 1504. The in-vehicle system 100 can use these display means and audio means when notifying the user. The in-vehicle system 100 can issue the above-mentioned system notification (e.g., TOR) using these display means and audio means. The in-vehicle system 100 can also vibrate when issuing a notification using a vibration mechanism provided in the steering wheel 5, the seat 6, a seat belt, etc.

[0175] 15 , the in-vehicle system 100 uses the display means to display output 1511 such as "Driving operation is required." Also, using the audio means, it outputs voice output 1512 such as "Driving operation is required." Also, it vibrates 1513 in response to the system notification.

[0176] The HMD 1 detects the display, audio, vibration, etc. associated with the system notification from the in-vehicle system 100 in the vehicle 2 as described above. Specifically, for example, for the display output 1511, the HMD 1 performs image recognition processing based on the image / video from the camera 110, grasps the notification content (e.g., "Driving operation required"), and determines and detects whether a driving operation request is present. If the HMD 1 determines that the driving operation request (e.g., TOR) is present, the HMD 1 changes the HMD function limitation level, for example, to level 4, and uses the display and audio functions of the HMD 1 to output a notification to the user U1 prompting the user U1 to perform a manual driving operation corresponding to the driving operation request. This notification output may be similar to the notification output of the in-vehicle system 100, for example, a display or audio such as "Driving operation required." Alternatively, this notification output may be a notification output regarding the content of the HMD function limitation level, for example, a display or audio such as "Level 4: HMD unavailable."

[0177] [Step S6: System Notification Compatible Operation Check Process] FIG. 16 is a table summarizing an example of a detailed configuration regarding the system notification compatible operation check process in step S6.

[0178] Process 6-1: (1) The HMD 1 checks the status of the user's response to the system notification received from the in-vehicle system 100. (2) If the importance of the notification is "high", the HMD 1 proceeds to (3). If the importance is "medium", the HMD 1 proceeds to process 6-2. (3) If the importance is "high", the HMD 1 checks whether the user seated in the driver's seat has stopped using the HMD 1 (HMD use stopped state), has removed the HMD 1 (non-wearing state), and has performed an action in response to the system notification (for example, manual driving operation).

[0179] Modifications of Process 6-1: (3') The status of HMD use suspension can be grasped from the internal status of the HMD 1 or the status of communication with the server 8, etc. Whether the HMD is not worn or whether an action has been taken in response to the notification can be determined from the images of the camera 110 of the HMD 1 or the in-vehicle camera 531, the content of the notification from the in-vehicle system 100, etc.

[0180] Process 6-2: (1) If the importance of the notification is "medium," the HMD 1 confirms that the user sitting in the driver's seat has taken action in response to the notification.

[0181] Variations on Process 6-2: (1') Whether or not an action has been taken in response to the notification can be determined similarly from the images of the camera 510 of the HMD 1 or the in-vehicle camera 531, the content of the notification from the in-vehicle system 100, and the like.

[0182] [Step S7: System Notification Post-Reception Processing] FIG. 17 is a table summarizing an example of a detailed configuration regarding the system notification post-reception processing in step S7.

[0183] Process 7-1: (1) If the HMD 1 determines from the confirmation result of step S6 that the action to respond to the system notification has been completed, it transmits a notification response completion result (information informing that the action to respond to the system notification has been completed) to the in-vehicle system 100 and proceeds to (2). If it determines that the action to respond to the notification has not been completed, it proceeds to process 7-2. (2) If the importance of the notification for which the action has been taken is "high", the HMD 1 waits for the next system notification from the in-vehicle system 100. If the importance of the notification for which the action has been taken is other than "high", that is, "medium" or "low", the HMD 1 ends the flow. (3) When the HMD 1 receives the next notification from the in-vehicle system 100 (a notification that the restriction may be lifted), it returns the temporarily changed HMD function restriction level to the original level.

[0184] Modifications of Process 7-1: (3'-1) When the operation of the HMD 1 has been stopped or changed, the HMD 1 resumes or returns to the operation before the change. (3'-2) After receiving the next notification from the in-vehicle system 100, the HMD 1 may return to the operation immediately or after a predetermined time has elapsed.

[0185] Process 7-2: (1) When it is determined that the user has not completed the corresponding action in response to the notification, the HMD 1 repeatedly outputs a notification to prompt the user to perform the corresponding action.

[0186] Modifications of process 7-2: (1'-1) Repeated notification output may be performed at predetermined intervals from the previous notification output. (1'-2) When the grace period for taking action in response to a system notification falls below a threshold, the frequency of notification output may be increased, the intensity of notification output may be increased, or the notification output method may be changed.

[0187] [Major Usage Flow] The system of this embodiment, with respect to the basic flow ( FIG. 6 ), mainly assumes the following usage flow. In steps S1 to S4, autonomous driving is the basis whenever possible, and functional limitations of the HMD 1 are minimized. In step S5, while the vehicle 2 is autonomously driving, the HMD 1 receives a system notification (e.g., TOR) from the in-vehicle system 100 requesting the user to take action, for example, in an emergency. In step S6, the HMD 1 confirms that the user has successfully performed an action in response to the system notification (e.g., switching to manual driving). In step S7, the system behaves according to the system notification response action result. As a result, if the action in response to the system notification is completed, the HMD 1 notifies the in-vehicle system 100 of the notification response completion result. This corresponds to sending the "notification response completion result" in process 7-1 of FIG. 17 and the "send system notification response result" in step S210 of FIG. 25 (described later). The notification response completion result corresponds to the response action information in the HMD information 7 in FIG. 1B.

[0188] If the HMD 1 has received a system notification (e.g., TOR) with a "high" priority in step S5, after transmitting the notification response completion result, the in-vehicle system 100 (vehicle 2) temporarily stops autonomous driving as a response to an emergency or the like and drives manually by the user. The in-vehicle system 100 determines whether this temporary response (manual driving) can be canceled, i.e., whether autonomous driving can be resumed, depending on the driving situation, etc. If the in-vehicle system 100 determines that the situation has become one in which it is acceptable to cancel this response, it transmits a notification of the cancellation to the HMD 1 ("next notification" in FIG. 17). This notification indicates that an emergency situation or the like has been avoided by the temporary manual driving response, but that the emergency situation has been canceled and autonomous driving can be resumed. In other words, this notification is a notification regarding the transition from manual driving to autonomous driving. When the HMD 1 receives this system notification, it returns to the original HMD function restriction stage setting. The original setting is the setting before the emergency response, i.e., the setting at the autonomous driving stage.

[0189] As a result of such control, in some cases, the system may be turned off. That is, the in-vehicle system 100 continues to respond by manual driving until the vehicle 2 arrives at a destination (e.g., an emergency pit, a dealer, etc.), at which point the system is turned off. When the system is turned off, the system may forget the settings, such as the original stage, that were stored to be restored when the response was released, or may store the information on the original settings in, for example, a non-volatile memory to restore the original settings the next time the system is turned on.

[0190] [Examples of Notification Output] Figures 18A, 18B, 18C, and 18D show examples of notification output (in other words, notification methods) from the in-vehicle system 100 to the HMD 1 and the user wearing it in this system. Example A in Figure 18A shows an example in which, when a user U1 receives a system notification (e.g., TOR) from the in-vehicle system 100 while viewing content (e.g., VR content) on the HMD 1, the HMD 1 pauses the content being played, changes the display to a transparent mode (see-through display), and displays a notification message. In Example A, initially, the user U1 wears and uses the HMD 1, and a video 1801 of, for example, VR content is displayed across the entire display surface of the display device (display 112). At this time, when a system notification is generated from the in-vehicle system 100, the HMD 1 displays an image (message) 1802 such as "System Notification Received (TOR)" above the video 1801.

[0191] After that, the HMD 1 temporarily suspends the use of the VR content, i.e., the display of the image 1801 and the playback of audio, and displays an optical image 1803 of the outside world (image of the vehicle surroundings) in a transparent mode across the entire display surface. The HMD 1 then displays, on a portion of the image 1803, an image (message) such as "Content has been stopped," indicating HMD function limitations, or an image (message) 1805 corresponding to the system notification content (e.g., TOR), such as "Please remove the HMD before driving." Not only display, but audio output can also be controlled in a similar manner. Upon receiving the notification output, the user U1 recognizes TOR and the function limitations, removes the HMD 1, and drives manually.

[0192] Example B in Figure 18A is an example in which, when a system notification occurs while VR content is being played on HMD1, HMD1 pauses the VR content, displays the image of the VR content in a content window 1811 (in other words, a partial display) on the display surface 1810 of the display device, moves the content window 1811 to the background, and displays a notification image (message) 1812 such as "Please remove the HMD before driving" so that it is superimposed on top of the content window 1811.

[0193] 18B shows an example in which, when a system notification occurs during playback of VR content on the HMD 1, the HMD 1 displays a pop-up message 1821 corresponding to the content of the system notification ("Please drive") in a portion above the VR content image 1801. This pop-up message 1821 is a non-transparent area, and the image 1801 behind it is not visible. When displaying this pop-up message 1821, the display content, such as the image size, blinking cycle, and color, may be changed according to the urgency / importance of the notification, the notification content, the notification recipient (user), etc.

[0194] In example D, when a system notification occurs while VR content is being played back on the HMD 1, the HMD 1 displays a pop-up message 1822 corresponding to the content of the system notification in a portion above the VR content image 1801, and at that time controls and changes the transparency of the displayed pop-up message 1822 according to the urgency / importance of the notification. This pop-up message 1822 is a transparent area according to the transparency, and the image 1801 behind it can be seen through.

[0195] Example E is an example in which, when a system notification occurs during playback of VR content on the HMD 1, the HMD 1 displays a pop-up message 1821 (or 1822) on the VR content image 1801 and also superimposes an image 1823 of an optical image of the outside world (e.g., an image of the vehicle's surroundings) on a portion of the image 1801. The transparency of the optical image of the outside world image 1823 may be controlled to be transparent or opaque depending on the importance of the notification, etc. The pop-up message 1821 and the image 1823 are displayed side by side. This allows the user U1 to check the situation around the vehicle. Furthermore, during this display, the ratio between the display of the content image 1801 and the display of the optical image of the outside world image 1823 may be changed depending on the importance of the notification, etc. In Example E, the VR content image 1801 remains displayed over the entire display screen, and the image 1823 is superimposed as a transparent area of ​​a predetermined size in the upper left corner. For example, the size of the video 1823 may be increased as the importance increases, and the size of the content video 1801 may be decreased as the importance increases.

[0196] Example F is an example in which, when a system notification occurs while VR content is being played back on the HMD 1, the HMD 1 forms a message display area 1824 (e.g., near the top or top edge) on the display surface and displays a message corresponding to the content of the system notification ("Please drive") in the message display area 1824. On the display surface, such a message display area 1824 may be provided as a fixed area that is always displayed, or may be generated as a dynamic area when necessary. The message display area 1824 may be a transparent area or a non-transparent area.

[0197] Example G in FIG. 18C is an example of a case where the user does not take action in response to the system notification. Assume that the HMD 1 receives a system notification (e.g., TOR) while user U1 is viewing VR content on the HMD 1. As shown in the upper part of Example G in FIG. 18C , during a predetermined transition time during the stage transition, the HMD 1 pauses the VR content image 1801 and displays an image (message) 1831 on top of the VR content image 1801, prompting the user to take action in response to the content of the system notification, such as "Please remove the HMD and start driving." The normal response action is to remove the HMD 1 and start manual driving. Assume that user U1, in response, keeps the HMD 1 on and does not start manual driving.

[0198] During a predetermined transition time during the stage transition, the HMD 1 or the in-vehicle system 100 detects and determines that the user has not taken any corresponding action. Then, as shown in the middle part of Example G in FIG. 18C , the HMD 1 outputs a warning to the user U1. For example, the HMD 1 displays an optical image 1832 of the outside world in a transmissive mode on the display surface, and displays a warning image (message) 1833 on top of it. Additionally, a warning icon or the like may be displayed, or a warning sound may be output. The icon may be an icon representing a manual driving operation. Furthermore, the HMD 1 may output a warning repeatedly as appropriate during the predetermined transition time. Furthermore, the intensity of the warning may be increased depending on the time elapsed since the system notification was issued.

[0199] If the user U1 does not take any action during the transition time even after receiving the warning, the HMD 1 notifies the in-vehicle system 100 that no action was taken, and causes the in-vehicle system 100 to take appropriate action, such as changing the autonomous driving level or implementing automatic safety stop control, and also changes the HMD function restriction level. As shown in the lower part of example G in FIG. 18C , for example, when the in-vehicle system 100 executes automatic safety stop control, the HMD 1 displays an image (message) 1834 on the display surface, such as "Performing an automatic safety stop," indicating that automatic safety stop control will be executed. Automatic safety stop control is control in which the in-vehicle system (autonomous driving system) 100 automatically stops the vehicle 2 in a safe location. At this time, the level is set to, for example, a level in which content, etc., is unavailable.

[0200] Example H in FIG. 18D shows a case where user U1 is using an AR application as a non-immersive application on the HMD1. The HMD1 displays an optical image 1841 of the outside world on the entire display surface in a transparent mode. The HMD1 appropriately superimposes an AR image 1842 for work / assistance on the image 1841. The AR application generates and displays an image 1842 of information about an object based on recognition of the object, such as a road, traffic sign, or building in the outside world. For example, an image 1842 of information about a detected building such as a store or facility is superimposed as a balloon image. In addition, an arrow image 1843 for navigation based on a set route is superimposed.

[0201] In such a situation, when a system notification such as TOR occurs, the HMD 1 displays an image (message) 1844, such as "Please remove the HMD before driving," corresponding to the content of the system notification, on top of the video 1841 on the display surface and the AR images 1842 and 1843. The image (message) 1844 is displayed with priority over the AR images 1842 and 1843.

[0202] As shown in Figure 18A, even when user U1 is immersed in VR content or the like on HMD1 during autonomous driving, when a system notification such as TOR occurs, an image corresponding to the notification is displayed according to the HMD function restriction level, the VR content or the like is paused, and an optical image of the outside world (video of the vehicle's surroundings, etc.) is displayed. This allows user U1 to recognize notifications such as TOR without missing them and to smoothly take corresponding actions such as switching to manual driving operation. The HMD function restriction level is set according to the system notification (driving information, etc.), and the use of functions / applications / content is restricted according to the level.

[0203] 18A and other notification output control, particularly display control, may be applied in a modified example by combining a technique for controlling transparency as in Patent Document 1 with a configuration such as an HMD function restriction stage. That is, in the modified example, the HMD 1 may control the transparency of the display of content being played and the transparency of the display of system notifications and optical images of the outside world, so that, for example, the image of content gradually becomes transparent and disappears while the system notifications and optical images of the outside world gradually appear.

[0204] [Examples of User Settings] Fig. 19 shows examples of user settings, menus, etc. provided by the HMD 1. The HMD 1 displays images of menus / GUIs, etc. on the display surface of the display device. Example A is an example of content selection, in which the use of some content is restricted in HMD function restriction stage 3. Contents numbered 1 and 2 are non-immersive content, and therefore are usable in HMD function restriction stage 3, and the selection button is operable. Content numbered 3 is immersive content, and therefore is not usable in HMD function restriction stage 3, and is displayed in gray, and the selection button is inoperable.

[0205] Example B is an example of settings when a system notification occurs (settings when receiving a notification of each importance level). For "high" importance, the setting cannot be changed because the user should not ignore important notifications, but for "medium" and "low" importance, the setting can be changed, and the change button is operable. If the user decides and operates to change the setting for a notification of "medium" importance, for example, on the initial screen, the screen below is displayed, displaying a detailed setting screen for "medium" importance. On this screen, detailed settings can be made for each setting item, such as the notification method (e.g., "audio"), content operation (e.g., "pause"), and repeat notification (e.g., "none").

[0206] [Cases where user decision / input is required for system notification] There may be cases where the user of the HMD 1 needs to make a decision and provide an operational input according to the decision result in response to the system notification from the in-vehicle system 100. In step S5 of Fig. 6, the HMD 1 receives the system notification from the in-vehicle system 100. This system notification requests the user to make a decision / selection and to provide an operational input according to the decision result (selection).

[0207] FIG. 27 is an explanatory diagram showing an example of this case. In this example, the user U1 is wearing the HMD 1 during automatic driving. The HMD 1 outputs the contents of the system notification received from the controller 10 by display or sound. The notification contents may be, for example, a request to select between option A and option B. The notification output is, for example, an output such as "Please select: option A / option B." The options may be YES / NO. As a specific example, the options may be "do / do not change the route" in a route change request.

[0208] The user U1 makes a decision and selection based on the notification content. The user U1 selects, for example, option A and performs an operation input according to the decision and selection. The user U1 may perform this operation input to the HMD 1, or may perform this operation input to the in-vehicle system 100. The user U1 may input option A to the HMD 1, for example, by operating the controller 1B or by voice input. The user U1 may input option A to the in-vehicle system 100, for example, by operating a button on the vehicle operation switch 511 provided on the steering wheel 5 or by voice input.

[0209] The HMD 1 and the in-vehicle system 100 receive operation input from the user U1. When an operation input is made to the HMD 1, the HMD 1 transmits information about the operation input from the user U1 (in other words, a response to a system notification request) to the in-vehicle system 100 as a type of HMD information 7. The in-vehicle system 100 performs a control operation according to the information about the operation input from the user U1.

[0210] The above-described operation input may be considered as one type of the aforementioned corresponding action.

[0211] [System notification according to HMD wearing state] As a supplement, using Fig. 28 and Fig. 29, the output of a system notification according to whether the user is wearing the HMD 1 during automatic driving or manual driving will be described. This system may change the system notification output method according to the HMD wearing state, etc. Fig. 28 shows an example of a system notification output method according to whether the user U1 is wearing the HMD 1 during automatic driving of the vehicle 2. Fig. 29 shows an example of a system notification output method according to whether the user U1 is wearing the HMD 1 during manual driving of the vehicle 2.

[0212] If a system notification occurs when the user U1 is not wearing the HMD 1, a notification from the in-vehicle system 100 to the HMD 1 is ineffective because the user U1 would have difficulty recognizing the notification. In this case, the notification may be output using any means provided in the in-vehicle system 100, such as a display (visual), sound (auditory), or vibration (tactile). The in-vehicle system 100 may determine whether the user U1 is wearing the HMD 1 based on communication with the HMD 1. The in-vehicle system 100 may determine a system notification output method depending on whether the user U1 is wearing the HMD 1. That is, the in-vehicle system 100 may send a notification to the HMD 1 (i.e., output a notification from the HMD 1) when the user U1 is wearing the HMD 1, and output a notification from the in-vehicle system 100 when the user U1 is not wearing the HMD 1.

[0213] 28 , State A indicates a case where a system notification such as TOR is generated when the user U1 is not wearing the HMD 1 during automatic driving of the vehicle 2. The HMD function restriction at this time is low, for example, at stage 2, but the user U1 is not using the HMD 1. If the in-vehicle system 100 recognizes that the user U1 is not wearing the HMD 1 when the system notification is generated, the in-vehicle system 100 outputs the notification (e.g., "Please drive") using a display / audio / vibration mechanism provided in the in-vehicle system 100. For example, a console display 1501, a windshield projection display 1502, a virtual image display 1503, a speaker 1504, or vibration of the seat 6, seat belt, or steering wheel 5 can be used.

[0214] State B is a case where a system notification such as TOR occurs while the user U1 is wearing the HMD 1 during automatic driving of the vehicle 2. If the in-vehicle system 100 recognizes that the user U1 is wearing the HMD 1 when the system notification occurs, it transmits the system notification to the HMD 1. The HMD 1 outputs the notification by display or sound. Details of this case are as described above.

[0215] As a modified example, when the user U1 is wearing the HMD 1, the in-vehicle system 100 may output a system notification from the in-vehicle system 100 and also transmit the system notification to the HMD 1. When notification outputs from both the in-vehicle system 100 and the HMD 1 are used in this manner, the user U1 only needs to recognize the system notification from at least one of the notification outputs.

[0216] The system notification may also be sent to the aforementioned mobile terminal 15 or the operation device 1B. As shown in state A, the system notification may be sent from the controller 10 to the mobile terminal 15. As shown in state B, the controller 10 may vibrate the operation device 1B via the HMD 1.

[0217] In FIG. 29 , state A indicates a case where a system notification occurs while the user U1 is not wearing the HMD 1 during manual driving of the vehicle 2. The HMD function restrictions at this time are severe, such as at stage 4, and the user U1 is not using the HMD 1. The controller 10 outputs a system notification (e.g., "Switching to automatic driving") using a display / audio / vibration mechanism provided in the in-vehicle system 100. The vehicle (in-vehicle system 100) may switch from manual driving to automatic driving without prior notification to the user U1 or a switching operation by the user U1. However, in most cases, the system switches to automatic driving upon receiving a request to switch to automatic driving from the user U1. In other words, the system notification in FIG. 29 can be considered to notify the user U1 that the switch to automatic driving has been completed in response to a request from the user U1. Here, the HMD 1 is not being worn, but if the power is on, it can receive the system notification, and upon receiving the system notification and recognizing the content of the notification, it can determine that the completion of switching to automatic operation means that the user U1 may wear and start using the HMD 1, and the HMD 1 can be immediately transitioned to a usable state, allowing the user U1 to use the HMD 1 promptly after switching to automatic operation.

[0218] State B is a case where a system notification occurs while the user U1 is wearing the HMD 1 while manually driving the vehicle 2. During manual driving, as described above, the basic principle is to restrict HMD functions to level 4, making all HMD functions unavailable and not wearing the HMD. However, in future operations, manual driving with the HMD 1 worn may be permitted with the HMD 1's functions restricted. Therefore, as a variant, in this case, the HMD 1 is set to a predetermined functional restriction level, making many of its functions / applications / content unavailable (e.g., VR content in non-transparent mode is prohibited), and the display surface is set to transparent mode, displaying an optical image of the outside world (such as a video of the vehicle's surroundings). AR applications such as those shown in FIG. 18D may also be permitted. The HMD 1 accepts and outputs system notifications from the in-vehicle system 100 as a partially enabled function.

[0219] During manual driving, when a system notification is generated, if the in-vehicle system 100 recognizes that the user U1 is wearing the HMD 1 and in a specific state, such as non-transparent mode, the system notification is sent to the HMD 1. The HMD 1 displays the system notification received from the in-vehicle system 100 (e.g., "Transitioning to automatic driving"), for example, as a voice output, superimposed on an optical image of the outside world displayed on the screen in transparent mode. Furthermore, after transitioning to automatic driving, the function restriction level of the HMD 1 may be changed automatically or by user selection. When transitioning from manual driving to automatic driving, the function restriction level of the HMD 1 may be lowered, allowing the HMD 1 to use more functions, applications, content, etc. Furthermore, a list of applications and content that have changed from unavailable to available due to the change in the function restriction level may be displayed, allowing the user U1 to select an application or content from the list and begin using the selected application or content.

[0220] [Example of control on the time axis] As a supplement, regarding the change of the HMD function limitation stage, a time (transition time) during the transition from one stage to another may be set in control. During this transition time, the user's decision or operation may be accepted. Alternatively, in another form, there may be no transition time, and the user's decision or operation may be accepted within the time for each stage.

[0221] FIG. 30 shows an example of HMD function restriction stage control on a time axis as a supplement. The upper part (A) shows the changes when transitioning from autonomous driving to manual driving. Along the time axis, the autonomous driving level of the in-vehicle system 100, the actions of the HMD 1 and user U1, and the HMD function restriction stage are shown. Initially, the vehicle is in autonomous driving mode at autonomous driving level 3. User U1 is wearing the HMD 1 and viewing content. For example, at level 3, the HMD function restriction stage is set to level 2. The in-vehicle system 100 generates a system notification such as TOR in response to an event (time t1). Examples of events include determining that manual driving should be initiated due to an abnormality / danger situation outside the vehicle 2, or prompting the user to make a decision because it is impossible to determine whether to transition to manual driving. The in-vehicle system 100 transmits the TOR system notification to the HMD 1 of the user (the driver). The HMD 1 receives the notification and outputs it as a display or sound.

[0222] In FIG. 30A , the period from time t1 when the system notification is generated to time t2 when the user has completed responding to the system notification is represented by a stage transition time (transition time) 3001. User U1 recognizes the system notification (TOR), makes a decision, and responds by removing the HMD 1 and switching to manual driving operation. The HMD 1 stops content playback. The HMD 1 and the in-vehicle system 100 detect and confirm the user U1's response and status. If the HMD 1 detects and confirms the response and status, it transmits a notification response confirmation result to the in-vehicle system 100 (time t2). Alternatively, the in-vehicle system 100 may detect the response and status based on an equipment sensor or the like, in which case, obtaining the notification response confirmation result from the HMD 1 may be omitted. Following user U1's normal response, the vehicle enters manual driving (level 0), the transition time ends, and the HMD 1's function limitation level is changed to, for example, level 4.

[0223] If the user U1 does not take a normal action during the transition time 3001, the in-vehicle system 100 performs control depending on the notification response confirmation result and the situation, and also controls the stage of the HMD 1 as appropriate. For example, depending on the situation, automatic driving may continue, or the aforementioned automatic safety stop control may be executed. It is desirable to set an upper limit on the transition time 3001 from automatic driving to manual driving. If a user action is taken within the upper limit of the transition time, t2 is determined at that point (upper limit of the transition time > t2). On the other hand, if no user action is taken even after the upper limit of the transition time is exceeded, t2 is not determined, i.e., t2 > upper limit of the transition time. In this case, for example, the transition to manual driving may be abandoned, and the in-vehicle system may stop the vehicle in a safe place. Alternatively, if no user action is taken even after upper limit 1 of the transition time is exceeded, the user may be notified again, and then if no user action is taken even after upper limit 2 of the transition time is exceeded, the vehicle may be stopped.

[0224] The lower part (B) shows the changes that occur when transitioning from manual driving to automatic driving. Initially, manual driving is in progress (level 0). User U1 is not wearing the HMD1. Stage 4 is set for the HMD1 when level 0 is reached. User U1 operates the vehicle operation switch 511 or the like to request the in-vehicle system 100 to switch to automatic driving, and this operation generates a system notification to transition to automatic driving (time t3). Upon receiving this notification, the in-vehicle system 100 recognizes the notification, checks whether the operating conditions for transitioning to automatic driving are met, and if the conditions are met, it accepts the request to switch to automatic driving and notifies user U1, for example, via the vehicle's audio or display device, that the switching process has begun.

[0225] The in-vehicle system 100 performs a transition process to autonomous driving, and when autonomous driving starts, notifies the user U1 that autonomous driving will begin via the vehicle's audio or display device, and autonomous driving begins (e.g., autonomous driving level 3). Upon receiving the autonomous driving start notification, the user U1 may wear the HMD 1, and the HMD 1 may play content once the system has switched to autonomous driving. At this time, the function restriction level of the HMD 1 is determined according to the autonomous driving level set by the in-vehicle system (time t4). In the case of autonomous driving level 3, the HMD 1 operates at, for example, function restriction level 2.

[0226] Although (B) of Figure 30 clearly illustrates a transition time 3002 during the HMD function restriction phase, in reality, this time may be very short. (B) of Figure 30 differs from (A) of Figure 30 in that (B) shows a transition from a state in which user U1 was performing driving operations (manual driving) to a state in which user U1 does not need to perform driving operations (automated driving). Therefore, the in-vehicle system 100 may quickly switch to automatic driving upon receiving a request to switch to automatic driving. Even if such a switch is performed, the likelihood of a dangerous situation that could lead to a vehicle accident occurring is low. In other words, the transition time 3002 can be relatively shorter than the transition time 3001. On the other hand, because the transition time 3001 also represents an emergency situation in which a TOR notification has been generated, it is desirable to control the transition time as short as possible while reliably understanding the user's actions, for example by setting an upper limit on the transition time as described above.

[0227] As described above, according to this example (first embodiment), when a driver in an automatically driven vehicle 2 uses the HMD 1, suitable assistance can be provided.

[0228] <Embodiment 2> Next, embodiment 2 will be described. As a modification of embodiment 1, embodiment 2 describes a configuration example that takes into consideration a case where there are multiple HMDs 1 in the vehicle 2, a case where an occupant other than the user in the driver's seat is wearing an HMD 1, etc. As a situation, an example is assumed in which there are multiple occupants and multiple HMDs 1 in the vehicle 2, as shown in FIG. 1C above.

[0229] The basic flow in the second embodiment is the same as that in Fig. 6. The details of step S1 are the same as those in Fig. 7.

[0230] [Multiple HMDs: Step S2] Fig. 20 shows a modification of the HMD usage environment confirmation process in step S2 in Fig. 8, assuming multiple HMDs 1 and a user other than the driver. Here, for example, a modification of process 2-1 in Fig. 8 (when there is a single HMD) is shown as process 2-1b.

[0231] Process 2-1b: (1) This system (HMD 1 and in-vehicle system 100) checks the wearing / non-wearing status of the HMD 1, personal information, seating position, whether or not the person is qualified to drive, etc., for all occupants in the vehicle 2. (2) If there are users wearing HMDs in seats other than the driver's seat, proceed to (3); if not, proceed to process 2-2b. (3) This system checks the seating position of each wearer. Based on the check result, this system determines whether the wearer is sitting in the driver's seat, passenger seat, rear seat, or other. (4) Establishes communication between multiple HMDs 1 in use in the vehicle 2.

[0232] Modifications of Process 2-1b: (1'-1) In a situation where the only HMD wearer / possessor is the user in the driver's seat (driver), the process is the same as Process 2-1 (Figure 8) when there is only one HMD. (3'-1) If the seating position of the HMD wearer cannot be confirmed, the system determines that the seating position of the HMD wearer is unknown and restricts the functionality of the HMD 1 as necessary. (4'-1) An HMD 1 worn by a user whose seating position cannot be confirmed is excluded from communication establishment. (4'-2) An HMD 1 with which communication cannot be established is determined to be an HMD with which communication is not possible. (4'-3) If the user in (4'-1) or (4'-2) above possesses another communication-enabled device such as a mobile terminal 15 (Figure 1), communication may be established between the communication-enabled device and the HMD 1. In the event that the user does not wear the HMD 1 or the HMD 1 cannot communicate, a notification is sent to the mobile terminal 15 or the like in an emergency.

[0233] Process 2-2b: (1) The system waits until the HMD 1 is worn or the power is turned off.

[0234] Modifications of Process 2-2b: (1') If the HMD 1 is powered on but not worn for a predetermined period of time or longer, the HMD 1 is transitioned to a sleep state or a power-off state.

[0235] [Multiple HMDs: Step S3] FIG. 21 shows a modification of the function setting process in step S3 of FIG. 12, which is intended for a user other than the driver.

[0236] Process 3-1b: (1) The system checks the seating positions of all HMD wearers in the vehicle 2. (2) The system checks the driving control status of the in-vehicle system 100. (3) If the autonomous driving level is level 4 or 5, proceed to (4). If it is level 3, proceed to process 3-2b. If it is level 1 or 2, proceed to process 3-3b. If the seating positions of the HMD wearers are unknown, proceed to process 3-4b. (4) Set the function restrictions of all HMDs to level 0. (5) If a change to a level other than 4 or 5 occurs, a notification is output to the user via HMD 1 or other devices.

[0237] Modifications of process 3-1b: (4'-1) When autonomous driving level 4 is reached, the HMD function restriction may be set to a level other than 0. (4'-1) The HMD 1 of a user seated in a seat other than the driver's seat is not subject to (4'-1). (5') When an abnormality occurs in communication with the in-vehicle system 100, the level is set to 3 or higher.

[0238] Process 3-2b: (1) The HMD 1 of users other than those seated in the driver's seat is set to level 0 or higher. (2) When a change to a level other than 3 occurs, notify all users via the HMD 1 and other devices, and change the level as appropriate. (3) Not all users in the vehicle 2, including the user seated in the driver's seat, are permitted to use the same functions / applications / contents, etc.

[0239] Modified examples of process 3-2b: (1'-1) When setting a user seated in a seat other than the driver's seat to level 3, it is confirmed in advance that there are no problems with communication with the in-vehicle system 100, and if a notification of "high" importance is received, a notification is given that the user seated in the driver's seat may need support. (1'-2) When a user seated in a seat other than the driver's seat is qualified to drive, at least one HMD 1 is set to level 3 or lower. Particular priority is given to users close to the driver's seat. (1'-3) When an abnormality occurs in communication with the in-vehicle system 100, at least one HMD 1 of a user seated in a seat other than the driver's seat is set to level 4. (2'-1) When changing the level, the notification method or notification destination (such as which user's HMD) is changed when changing from level 3 to 1 / 2 and when changing from level 3 to 4 / 5.

[0240] Process 3-3b: (1) The HMD 1 of users seated other than the driver's seat is set to level 3 or lower. (2) When changing to a level other than 1 / 2, the HMD 1 or other devices notify the users seated other than the driver's seat and change the level appropriately. (3) All users in the vehicle, including the user seated in the driver's seat, are not permitted to use the same functions / applications / contents, etc.

[0241] Variations on process 3-3b: (1'-1) When a user seated in the driver's seat is wearing an HMD 1, a notification is sent to the HMD 1 or mobile terminal 15 of users seated in other seats to prompt the user seated in the driver's seat to stop using the HMD 1.

[0242] Process 3-4b: (1) If the seating position of the HMD user is unknown, the HMD 1 is assigned to stage 4.

[0243] Modifications of process 3-4b: (1'-1) When the seating position can be identified, the stage can be changed at that point. (1'-2) When the seating position cannot be identified by the HMD 1 alone and communication with another HMD 1 is possible, the seating position may be estimated from the relative position with respect to the other HMD 1.

[0244] Note that the reason why the use of the same function / application / content, etc. is not permitted in (3) of process 3-2b and (3) of process 3-3b is intended to be as follows: For example, when everyone is playing the same game, etc. (e.g., competitive or cooperative play), there is a possibility that everyone will ignore system notifications, etc., and this possibility is reduced. In this case, if everyone takes a break in the game, etc. at the same time (e.g., when the competition ends), there is a possibility that everyone will ignore notifications until that break occurs. If each occupant uses a different function / application / content, etc., such as a different game, the timing of the break will vary, increasing the possibility that someone will notice the notification.

[0245] The process of step S4 is the same as that shown in FIG. 13, with no particular difference between a single HMD 1 and a plurality of HMDs 1.

[0246] [Multiple HMDs: Step S5] FIG. 22 shows a modification of the system notification reception process in step S5 of FIG. 14, which assumes a user other than the driver.

[0247] Process 5-1b: (1) The HMD 1 checks whether a system notification has been received. (2) If a notification has been received, the HMD 1 checks the importance of the notification. If the importance is "high", proceed to (3). If the importance is "medium", proceed to process 5-2. If the importance is "low", proceed to process 5-3. (3) This system sends a notification from the in-vehicle system 100 to the HMD 1 of a user seated in a seat other than the driver's seat. The notification is sent via the HMD 1 of the user seated in the driver's seat that received the notification from the in-vehicle system 100. In other words, the HMD 1 of the user seated in the driver's seat sends a system notification with "high" importance (similar information) to the HMD 1 of a user seated in a seat other than the driver's seat.

[0248] Modifications of process 5-1b: (3'-1) Notification to the HMD 1 of a user seated in a seat other than the driver's seat may be sent directly from the in-vehicle system 100 to the HMD 1. (3'-2) The HMD 1 of the user seated in a seat other than the driver's seat who has received the notification may change the level (e.g., from level 0 to 1) or stop applications / contents, etc., depending on the HMD function restriction level at that time. (3'-3) If the user seated in the driver's seat does not take action in response to the received system notification even after a predetermined time has passed, the system may repeatedly send notifications to users seated in seats other than the driver's seat.

[0249] Process 5-2b: (1) If the importance of the notification is "medium," the notification is also sent to the HMDs 1 of users seated in seats other than the driver's seat.

[0250] Modifications of process 5-2b: (1'-1) The notification may be sent only to the HMD 1 of the user in a specific seat in the vehicle 2 (for example, the passenger seat, which can be set by the user). (1'-2) The notification may be sent only to a specific HMD 1 of a specific user who wishes to receive the notification (the specific user and the specific HMD 1 can be set by the user). (1'-3) Depending on the HMD function restriction level of the user receiving the notification, the level may be changed (for example, from level 0 to 1), or applications / content may be stopped. (1'-4) If the user in the driver's seat does not take action in response to the received system notification even after a predetermined time has passed, the notification may be sent repeatedly to users other than the driver's seat. (1'-5) Another user may take action in response to the system notification on behalf of the user in the driver's seat (for example, granting permission to a route change request).

[0251] Process 5-3b: (1) If the importance of the notification is "low," the notification may be sent to the HMD 1 of a user seated in a seat other than the driver's seat.

[0252] Modifications of process 5-3b: (1'-1) Notification may not be sent to the HMD 1 of users other than the user sitting in the driver's seat. (1'-2) Notification may be sent only to specific HMD 1s of specific users who wish to receive the notification. (1'-3) It may be possible to set to receive only notifications related to specific content (such as notifications of a specific type or a specific level of importance).

[0253] [Multiple HMDs: Step S6] Fig. 23 shows a modification of the system notification response operation check process in step S6 in Fig. 16, which is intended for a user other than the driver. In particular, a modification of process 6-2 is shown.

[0254] Process 6-2b: (1) If the importance of the notification is "medium," the system confirms that the user in the driver's seat or a user other than the driver's seat has taken action in response to the notification.

[0255] Modifications of process 6-2b: (1'-1) The system determines whether the user has taken action in response to the notification from the images of the camera 110 of the HMD 1 or the in-vehicle camera 531, the content of the notification from the in-vehicle system 100, etc. (1'-2) When multiple HMDs 1 have received the same notification, it is sufficient to confirm that one of the users has taken action in response.

[0256] [Multiple HMDs: Step S7] FIG. 24 shows a modification of the system notification post-reception process in step S7 of FIG. 17, which is intended for a user other than the driver.

[0257] Process 7-1b: (1) If the system determines from the confirmation result of step S6 that the action to respond to the notification has been completed, it transmits a notification response completion result from the HMD 1 to the in-vehicle system 100 and proceeds to (2). If it determines that the action to respond to the notification has not been completed, it proceeds to process 7-2b. (2) If the importance of the notification for which the action has been taken is "high", the HMD 1 waits for the next notification from the in-vehicle system 100. If it is other than "high" (medium, low), the flow ends. (3) When the next notification is received from the in-vehicle system 100, all HMDs 1 whose HMD function restriction level has been temporarily changed are returned to their original level.

[0258] Modifications of Process 7-1b: (2'-1) After determining that the response action is complete, the HMD 1 may be returned to the original state immediately or after a predetermined time has elapsed. (2'-2) The conditions for returning to the original state may be changed depending on the seat. For example, when the HMD 1 is returned to the original state after a predetermined time has elapsed since the response action to the notification is completed, the HMD 1 of the user in the back seat may be returned to the original state in a shorter time than the HMD 1 of the user in the driver's seat. (2'-3) When the operation of the HMD 1 has been stopped or changed, the HMD 1 is resumed or returned to the operation before the change. (2'-4) The conditions for resuming or returning to the operation before the change may differ depending on the seat. For example, when the HMD 1 is resumed or returned to the operation before the change after a predetermined time has elapsed since the response action to the notification is completed, the HMD 1 of the user in the back seat may be resumed or returned to the operation before the change in a shorter time than the HMD 1 of the user in the driver's seat.

[0259] Process 7-2b: (1) If the system determines that the user has not completed the corresponding action in response to the notification, it will repeatedly notify the user to prompt them to take the corresponding action.

[0260] Modifications of process 7-2b: (1'-1) If the notification requires an operation such as driving, and the user in the driver's seat does not take a corresponding action even after a predetermined time has passed, a notification may be sent to the HMD 1 of a user other than the driver's seat to prompt the user in the driver's seat to take an operation. (1'-2) In the case of the above (1'-1), the notification destination is not limited to the HMD 1 of a user other than the driver's seat, and may be a mobile terminal 15 owned by the user. (1'-3) If the notification does not require an operation such as driving, and the notification destination user (e.g., the user in the driver's seat or the user in the passenger seat) does not take a corresponding action even after a predetermined time has passed, the notification may be sent to the HMD 1 of another user (e.g., the user in the back seat).

[0261] [Cooperative Operation Between In-Vehicle System and HMD] FIG. 25 shows an example of a flow / sequence of cooperative operation through communication between the in-vehicle system 100 and the HMD 1. In FIG.

[0262] In step S101, the in-vehicle system 100 starts control. In step S201, the HMD 1 is powered on. In step S202, the HMD 1 transmits a communication establishment request to the in-vehicle system 100. In step S102, the in-vehicle system 100 receives the communication establishment request from the HMD 1. In step S103, the in-vehicle system 100 transmits a system information notification destination registration request to the HMD 1. In step S203, the HMD 1 receives the system information notification destination registration request from the in-vehicle system 100.

[0263] In step S204, the HMD 1 transmits registration information corresponding to the system information notification destination registration request to the in-vehicle system 100. In step S104, the in-vehicle system 100 receives the registration information from the HMD 1 and completes the system information notification destination registration.

[0264] In step S205, the HMD 1 transmits a system information transmission request to the in-vehicle system 100. In step S105, the in-vehicle system 100 receives the system information transmission request from the HMD 1. In step S106, the in-vehicle system 100 transmits system information corresponding to the system information transmission request to the HMD 1. In step S206, the HMD 1 receives the system information from the in-vehicle system 100.

[0265] In step S207, the HMD 1 performs function setting of the HMD 1. This function setting includes setting of the HMD function restriction level. Note that a default level setting may be applied initially.

[0266] In step S208, the HMD 1 starts operating. Subsequent operations are appropriately restricted based on the function settings.

[0267] In step S107, when an event occurs, the in-vehicle system 100 transmits a system notification corresponding to the event to the HMD 1. The event is a predetermined event in driving control, including abnormality detection, etc. The system notification is the above-mentioned TOR, etc. In step S209, the HMD 1 receives the system notification from the in-vehicle system 100. The HMD 1 outputs the system notification to the user who is wearing the HMD 1. The user takes a corresponding action in response to the system notification (for example, switching to manual driving). Furthermore, the HMD 1 and the in-vehicle system 100 appropriately change the function restriction level of the HMD 1 in accordance with the system notification.

[0268] In step S210, the HMD 1 checks whether the user has performed a corresponding action, and transmits the notification response check result to the in-vehicle system 100. In step S108, the in-vehicle system 100 receives the notification response check result from the HMD 1, and determines the next control action according to the notification response check result. In step S109, the in-vehicle system 100 determines whether to continue control, and continues or ends the control. In step S211, the HMD 1 determines whether to resume operation, etc., and resumes or ends the operation.

[0269] Steps S209 to S210 may be performed as follows. The HMD 1 counts time upon receiving the system notification and checks whether the user has performed the corresponding action requested in the notification within a predetermined time. A time threshold or the like may be set for this predetermined time according to the system notification. If the corresponding action is performed within the predetermined time, the normal flow is achieved, and the function restriction level is changed appropriately. If the corresponding action is not performed within the predetermined time, the exceptional flow is achieved, and the in-vehicle system 100 performs predetermined control or countermeasures.

[0270] Furthermore, whether or not to pause the playback of content, etc. on the HMD 1 during such notification response operation check (the notification method described above) can be controlled in various ways, depending on the specifications of the function restriction stage and the control specifications. For example, if the response operation is not performed within a predetermined time, the content, etc. may be paused or forcibly terminated, or the HMD 1 itself may be forcibly terminated, and the HMD 1 may notify the in-vehicle system 100 that the response operation will not be performed.

[0271] The following control example may be used in steps S107 and S108. The in-vehicle system 100 first transmits a system notification to a determined first notification destination, for example, the HMD 1 of the driver. If the in-vehicle system 100 determines, based on the result of step S210, that the user of the first notification destination has not taken any action in response to the notification, the in-vehicle system 100 may determine a next second notification destination and transmit the system notification to the second notification destination, for example, the HMD 1 of a passenger. The notification in this case may be a notification that prompts the user of the second notification destination to take a corresponding action. In another control example, notifications may be transmitted to multiple notification destinations from the beginning.

[0272] As described above, according to the second embodiment, suitable assistance is possible even when there are multiple occupants, including those other than the driver, in the vehicle 2 and multiple HMDs 1 .

[0273] In the above embodiment, a case has been described in which at least the HMD 1 grasps and manages its own HMD function limitation level and appropriately changes or controls the HMD function limitation level. However, it is also possible for the in-vehicle system 100 to grasp and manage the HMD function limitation level of one or more HMDs 1 in the vehicle 2 and appropriately change or control the HMD function limitation level. The in-vehicle system 100 may automatically determine the function limitation level of the HMD 1 based on the driving status, system notifications, the status of the HMD 1 and the user, etc. Control information regarding the HMD function limitation level may be exchanged through communication between the in-vehicle system 100 and the HMD 1. For example, the in-vehicle system 100 appropriately transmits a function limitation level instruction to the HMD 1 as a type of driving information 4A based on its judgment. The HMD 1 sets its own device to the level specified in the received instruction and transmits a response to the in-vehicle system 100 indicating that the specified level has been set. In response to the response, the in-vehicle system 100 grasps the setting status of the function limitation level of each HMD 1 and retains the information.

[0274] Although the embodiments of the present disclosure have been specifically described above, they are not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. Except for essential components, components can be added, deleted, or replaced in each embodiment. Unless otherwise specified, each component may be singular or plural. A combination of each embodiment and its variations is also possible.

[0275] 1...HMD (head-mounted display device), 2...vehicle (autonomous driving vehicle), 4...vehicle information, 6...seat, 10...controller, 100...in-vehicle system, U1...user

Claims

1. A head-mounted display device that can be used by a user who is a passenger in a vehicle, which acquires driving information related to autonomous driving from the vehicle's on-board system, the driving information including information related to the level of autonomous driving or a system notification related to autonomous driving, and the system notification including a takeover request (TOR) from autonomous driving to manual driving, and which restricts the functions of the head-mounted display device according to the acquired driving information.

2. A head-mounted display device according to claim 1, wherein the function restrictions include setting a function restriction stage according to the autonomous driving level or the system notification, and the setting of the function restriction stage comprises: changing the function restriction stage to a more restrictive stage when the autonomous driving level is lowered to a lower level or when manual driving is specified; and changing the function restriction stage to a less restrictive stage when the autonomous driving level is raised to a higher level or when autonomous driving is specified.

3. A head-mounted display device according to claim 1, wherein the function restriction includes setting a function restriction stage according to the autonomous driving level or the system notification, and the function restriction stage has a plurality of stages including a stage in which all functions, applications or content are available for the head-mounted display device, a stage in which all functions, applications or content are unavailable, and a stage in which some functions, applications or content are unavailable.

4. A head-mounted display device according to claim 1, wherein the head-mounted display device is a device worn by a user who is the driver of the vehicle, the driving information includes information intended for the driver, and the acquired driving information is output to the driver by display or audio.

5. A head-mounted display device according to claim 1, wherein the head-mounted display device receives and acquires the driving information transmitted to the head-mounted display device by communication from the in-vehicle system of the vehicle.

6. A head-mounted display device according to claim 1, wherein the driving information output by display from the in-vehicle system of the vehicle is obtained by image recognition, or the driving information output by voice from the in-vehicle system of the vehicle is obtained by voice recognition.

7. A head-mounted display device according to claim 1, wherein the seating state of an occupant in a seat in the vehicle is acquired, and the head-mounted display device has functional limitations according to the seating state.

8. A head-mounted display device according to claim 1, which acquires information about the driving qualifications of a user who is a passenger in the vehicle, and which has functional limitations on the head-mounted display device according to the driving qualifications.

9. A head-mounted display device according to claim 1, wherein the driving information from the vehicle's in-vehicle system has a level of importance depending on the content, and the head-mounted display device has functional limitations depending on the level of importance.

10. A head-mounted display device according to claim 1, wherein the function restriction includes setting a function restriction stage according to the autonomous driving level or the system notification, and the function restriction stage includes provisions regarding a method of notifying the driving information when the driving information is received from the vehicle while using a function, application, or content of the head-mounted display device, and the notification method includes provisions regarding at least one of whether to pause playback of the function, application, or content being used, whether the playback is to be displayed in full screen or partial screen, whether an optical image of the outside world is to be displayed transparently, the position and size of the display of the driving information, and the timing of outputting the driving information.

11. A head-mounted display device according to claim 1, wherein the user may be requested to take a corresponding action in response to the system notification, and after receiving the system notification, the head-mounted display device outputs a visual or audio output to the user at least once to prompt the user to take the corresponding action.

12. A head-mounted display device according to claim 1, wherein the user may be requested to take action in response to the system notification, and the head-mounted display device checks whether the user has taken the action in response to the system notification, and transmits the check result to the in-vehicle system of the vehicle.

13. A head-mounted display device as claimed in claim 1, wherein the user's response to the system notification may be to stop using the head-mounted display device or to not wear the head-mounted display device, and the head-mounted display device detects whether the user's response to the system notification is to stop using the head-mounted display device or to not wear the head-mounted display device.

14. A head-mounted display device as claimed in claim 1, wherein the user may be requested to take a corresponding action in response to the system notification, and if the user does not take the corresponding action in response to the system notification within a predetermined time, a warning is output to the user, or a notification is sent to the vehicle's on-board system to change the autonomous driving level or to perform automatic safety stop control of the vehicle.

15. A head-mounted display device according to claim 1, wherein, when there are multiple users as occupants in the vehicle, the head-mounted display device is a head-mounted display device of a plurality of head-mounted display devices that can be worn and used by each of the multiple users, and a distinction is made between drivers and non-drivers among the multiple users, and the functional restrictions include functional restrictions on the head-mounted display device worn by the driver and functional restrictions on the head-mounted display device worn by the non-drivers.

16. A head-mounted display device according to claim 15, wherein communication is established between the head-mounted display device of the driver and the head-mounted display device of the non-driver, and the head-mounted display device of the driver receives the driving information from the vehicle and transmits information corresponding to the driving information to the head-mounted display device of the non-driver.

17. A head-mounted display device according to claim 15, wherein the head-mounted display devices of the plurality of users are controlled so as not to simultaneously use the same function, application, or content.

18. A head-mounted display device as claimed in claim 15, wherein, when the driver is wearing and using the head-mounted display device and is requested to stop using or not wear the head-mounted display device as a response action to the system notification, the head-mounted display device of the non-driver issues a notification urging the driver to stop using or not wear the head-mounted display device.

19. An autonomous driving system comprising a controller mounted on a vehicle, wherein the controller communicates with a head-mounted display device available to a user who is a passenger in the vehicle, and transmits driving information related to autonomous driving to the head-mounted display device, wherein the driving information includes information related to the level of autonomous driving or a system notification related to autonomous driving, and wherein the system notification includes a takeover request (TOR) from autonomous driving to manual driving, and the controller restricts the functions of the head-mounted display device according to the driving information.

Images