State information presentation program and state information presentation method

Abstract

This state information presentation program, which is used for causing a head-mounted-display-type computer including a first communication unit, a storage unit, and a display to present state information pertaining to a vehicle including a second communication unit, causes the computer to execute: a process (S10) for acquiring, via the first communication unit, user information related to the vehicle or an occupant of the vehicle and status information pertaining to the vehicle; a process (S20) for determining whether a computer identifier stored in the storage unit and the user information satisfy a prescribed condition; a process (S30) for determining the state of the vehicle on the basis of the status information pertaining to the vehicle when the prescribed condition is satisfied; and a process (S40) for presenting the state information pertaining to the vehicle on the display in a display format in which a facial expression is added to an icon that evokes the vehicle.

Description

Status information presentation program and status information presentation method

[0001] The present invention relates to a status information presentation program and a status information presentation method for causing a head-mounted display type computer to present vehicle status information.

[0002] One known example of this type of technology is a technology for displaying virtual content in a 3D real-world spatial environment (also called a mixed reality (MR) environment). For example, Patent Literature 1 discloses that the content to be displayed may include balloons carrying everyday messages.

[0003] JP 2023-36711 A

[0004] While conventional technology allows multiple users of a mixed reality environment system to share content, there is no means, other than video calls, to constantly connect with a specific user and know their status (in other words, to watch over them). Generally, watching over a specific user through a constantly connected video call constitutes surveillance, which can be psychologically burdensome for both the watching user and the user being watched. Therefore, there is a need for a system that minimizes the psychological burden, allowing a watcher (e.g., a person working away from home or telecommuting) to grasp the status of a close friend (the person being watched) who is not in the same space, to the extent that they can sense their presence, using a mixed reality environment system.

[0005] A status information presentation program according to one aspect of the present invention causes a head-mounted display computer including a first communication unit, a storage unit, and a display to present status information of a vehicle including a second communication unit. The status information presentation program causes the computer to execute the following processes: acquire user information related to the vehicle or a vehicle occupant and vehicle status information via the first communication unit; determine whether a computer identifier stored in the storage unit and the user information satisfy a predetermined condition; determine the vehicle status based on the vehicle status information if the predetermined condition is satisfied; and present the vehicle status information on the display in a display format in which an expression is added to an icon reminiscent of the vehicle. Another aspect of the present invention is a status information presentation method causes a head-mounted display computer including the first communication unit, a storage unit, and a display to present status information of a vehicle including the second communication unit. The status information presentation method includes the steps of acquiring user information relating to the vehicle or the vehicle occupants and vehicle status information via a first communication unit, determining whether a computer identifier stored in a memory unit and the user information satisfy predetermined conditions, determining the vehicle status based on the vehicle status information if the predetermined conditions are satisfied, and presenting the vehicle status information on a display in a display format in which an icon reminiscent of the vehicle is given an expression.

[0006] According to the present invention, monitoring can be performed with less psychological burden on the vehicle being monitored.

[0007] 1A. A schematic diagram showing an example of an information providing system. A schematic diagram explaining a virtual space that can be viewed by a user on a display. A block diagram illustrating an example of the configuration of the main parts of the computer of FIG. 1A. A block diagram showing an example of the configuration of the IVI system of FIG. 1A. A diagram illustrating an example of an icon during steady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A diagram illustrating an example of an icon during unsteady state. A flowchart explaining status information presentation processing by a program. A flowchart explaining addition processing 1. A flowchart explaining addition processing 2. A flowchart explaining addition processing 3. A flowchart explaining addition processing 4.

[0008] Hereinafter, an embodiment of the invention will be described with reference to the drawings. <Outline of Status Information Presentation Program> A status information presentation program according to an embodiment is executed by a head-up display computer (hereinafter simply referred to as a computer) as an example of a mixed reality device. A computer executing the status information presentation program performs a monitoring function for monitoring a moving vehicle such as an automobile. For example, user A of the computer monitors, via the computer, a vehicle in which user B, who is a family member of user A, is riding. In other words, the status information presentation program provides user A with a monitoring service for the vehicle (and therefore user B).

[0009] A computer running a status information presentation program acquires information about the vehicle in which User B is riding via a public communication network. The computer then displays the vehicle information on the computer display in a 3D real-world space (which may also be called a virtual space). Furthermore, if the computer detects an abnormality in the acquired vehicle information, it switches the display mode of the vehicle information so that User A notices it.

[0010] In this embodiment, a vehicle-side communication device is configured as one of the functions of an IVI (In-Vehicle Infotainment) system installed in a vehicle. The vehicle-side communication device transmits information detected by sensors and the like installed in the vehicle to a computer via a public communication network as vehicle status information. An information provision system including such a head-up display type computer, a public communication network, and the vehicle-side communication device will be described in further detail below.

[0011] 1A is a schematic diagram showing an example of an information provision system 500. In the embodiment, a system including a head-up display type computer 200, a public communication network 300, an IVI system 110 as a communication device on the vehicle 100 side, and a server device 400 is exemplified as the information provision system 500.

[0012] FIG. 1B is a schematic diagram illustrating a virtual space R3 that user A can view on the display of computer 200 used by user A in FIG. 1A. Computer 200 displays worksheets 72, 73, and 74 of an office work application program (hereinafter simply referred to as an app) running on computer 200, overlaid with a stereoscopic image of the scenery in front of user A captured by the computer's built-in camera. User A uses worksheets 72, 73, and 74, which appear to float in the 3D virtual space R3 recreated by the stereoscopic images, to perform office work, for example, by moving his / her eyes and right hand or speaking. The stereoscopic images of the backgrounds of worksheets 72, 73, and 74 may be stereo images captured in advance at a different location instead of stereo images captured by the built-in camera. Alternatively, instead of worksheets 72, 73, and 74 of the office work app, the screen of a video or movie viewing app may be displayed floating in the 3D virtual space R3. In general, the ability of humans to perceive depth based on binocular disparity is called binocular stereopsis. In this embodiment, the brain reconstructs a three-dimensional structure from a pair of left and right two-dimensional images displayed on a display 207 (described later) using binocular stereopsis, and the reconstructed space is called virtual space R3.

[0013] Furthermore, when the computer 200 launches an app for presenting status information (a status information presentation program according to the embodiment), it displays an icon 71 on the display that evokes the vehicle 100, and changes the appearance of the icon 71 based on information about the vehicle 100.

[0014] In FIG. 1B , region R1 indicated by a dashed line in virtual space R3 corresponds to a person's central visual field, which has a visual field angle of approximately 1 to 2 degrees. Region R2 indicated by a dashed line corresponds to a person's effective visual field, which has a visual field angle of approximately 4 to 20 degrees. The area outside the effective visual field (in other words, region R2) corresponds to a person's peripheral visual field, which has a visual field angle of approximately 200 degrees combined. Computer 200 displays icon 71 in the peripheral visual field. In this embodiment, icon 71 is further displayed in an area that does not interfere with worksheets 72, 73, and 74.

[0015] Returning to FIG. 1A , vehicle 100 is equipped with an IVI system 110. IVI system 110, which also functions as a communication device, is configured to communicate with server device 400 via communication network 300. Communication network 300 includes not only public wireless communication networks such as the Internet and mobile phone networks, but also closed communication networks established for each designated management area, such as wireless local area networks (LANs) and Wi-Fi (registered trademark). While FIG. 1A illustrates only vehicle 100 in which user B rides, there may be many other vehicles in addition to vehicle 100. In this case, multiple pieces of information obtained when multiple vehicles are traveling are collected in server device 400. Furthermore, while FIG. 1A illustrates only computer 200 used by user A, there may be many other computers in addition to computer 200. In this case, a monitoring service can be provided to many other users in addition to user A.

[0016] The server device 400 is managed, for example, by a business entity that provides maintenance services for the vehicle 100. The server device 400 may be configured using a virtual server function on the cloud, or may be configured as a plurality of distributed devices. For example, the server device 400 may be distributed to each predetermined management area. The server device 400 acquires status information of the vehicle 100 along with time information at predetermined time intervals via the communication network 300 from the IVI system 110, which serves as a communication device for the traveling vehicle 100. The status information of the vehicle 100 includes, for example, information indicating the vehicle ID, direction of travel (azimuth), speed, traveling position, and the operating status of the turn signals (direction indicators). The vehicle ID is, for example, a vehicle identification number (VIN). The server device 400 also acquires, via the communication network 300, information on an abnormality detected by the control system of the vehicle 100. The abnormality information includes, for example, a trouble code (called a DTC (Diagnostic Trouble Code)), the date and time of occurrence (detection), and information indicating the location (latitude and longitude) of the vehicle 100 at the time of occurrence (detection).

[0017] Computer 200, which has launched the app for presenting status information, acquires status information of vehicle 100 that has been registered in advance (in other words, that has been approved for pairing with computer 200) from server device 400 via communication network 300. For example, computer 200 acquires, in real time, the status information of vehicle 100 that server device 400 has acquired from IVI system 110 of vehicle 100 from server device 400. As described above, computer 200 changes the appearance of icon 71 illustrated in FIG. 1B based on the status information of vehicle 100.

[0018] The configuration of such information providing system 500 will be further described with reference to a block diagram. <Computer> Fig. 2A is a block diagram illustrating an example of the configuration of the main parts of computer 200 in Fig. 1A. Computer 200 includes a group of cameras 201, a group of microphones 202, a speaker 203, a group of sensors 204, a calculation unit 205, a storage unit 206, a display 207, and a communication unit 208.

[0019] (Camera Group) The camera group 201 is configured by, for example, a plurality of cameras for capturing images of the space around the user A wearing the computer 200 and the face (particularly the eyes) of the user A.

[0020] (Microphone Group) The microphone group 202 is made up of a plurality of microphones for collecting sounds around the user A wearing the computer 200 and the voice emitted by the user A.

[0021] (Speaker) The speaker 203 is configured by a small speaker system for stereophonically reproducing the sound of the content for the user A wearing the computer 200 .

[0022] (Sensor Group) The sensor group 204 includes an inertial measurement unit, a magnetic sensor, a GPS (Global Positioning System) unit, an eye tracking unit, a hand tracking unit, etc. The IMU is a unit that detects inertial motion (translational motion and rotational motion) in three orthogonal axial directions, and outputs a detection signal that indicates the motion state (movement, posture, etc.) of the head of user A wearing computer 200. The eye tracking unit measures the line of sight of user A wearing computer 200. For example, it measures where the line of sight is directed in space within 3D virtual space R3. The hand tracking unit analyzes the movement of user A's fingers based on camera images.

[0023] The eye tracking unit acquires, as internal information, correction value information for measuring the line of sight of user A wearing computer 200, based on field of view information of user A acquired by computer 200 when computer 200 is worn by user A. Then, based on the correction value information, it determines region R1 corresponding to the central visual field, region R2 corresponding to the effective visual field, and the range of peripheral visual field outside region R2.

[0024] (Calculation Unit) The calculation unit 205 includes a calculation processing unit such as an MPU (not shown), and reads and executes a predetermined program (application) stored in the storage unit 206. When the calculation unit 205 executes the status information presentation application, the calculation unit 205 causes the computer 200 to perform a monitoring function. While the status information presentation application is executing, the calculation unit 205 causes the display 207 to display an icon 71 reminiscent of the vehicle 100 in the peripheral vision, as illustrated in FIG. 1B .

[0025] 3A is a diagram illustrating an example of icon 71 in a steady state. In the embodiment, icon 71 has a shape that reminds user A of vehicle 100. Computing unit 205 presents icon 71 on display 207 in a display format that includes an expression corresponding to status information of vehicle 100.

[0026] 3B and 3C, 4A to 4C, and 5A to 5C are diagrams illustrating icons 71A to 71H in an unsteady state, which change their expressions depending on the state of the vehicle 100. Fig. 3B shows an icon 71A whose expression changes due to the energy state of the vehicle 100, indicating that the battery SOC or fuel is below a predetermined value. Fig. 3C shows an icon 71B whose expression changes due to the sudden braking state of the vehicle 100, indicating that the acceleration during braking is above a predetermined value.

[0027] Fig. 4A shows an icon 71C whose expression changes due to the condition of the tires of the vehicle 100, indicating that the tire pressure is below a predetermined value or that a predetermined period of time has passed since the last inspection. Fig. 4B shows an icon 71D whose expression changes due to the driving condition of the vehicle 100, indicating that the value on the inclinometer is an uphill slope of a predetermined angle or more. Fig. 4C shows an icon 71E whose expression changes due to the operation of the wipers of the vehicle 100, indicating that the operation duration of the wipers has exceeded a predetermined time. Icon 71E may be an animated icon showing the movement of the wipers.

[0028] 5A shows an icon 71F whose expression changes due to the lighting status of the vehicle 100, indicating that the lighting duration of a lighting device such as a headlight has exceeded a predetermined time. FIG. 5B shows an icon 71G whose expression changes due to the content playback status of the vehicle 100, indicating that music content is being played by the IVI system 110. FIG. 5C shows an icon 71H whose expression changes due to the driving status of the vehicle 100, indicating that an airbag or the like has been deployed. As described above, the calculation unit 205 displays the icons 71 (71A to 71H) whose expressions change depending on the status of the vehicle 100 in the virtual space R3 corresponding to the peripheral vision of the user A, which is less noticeable than the central vision. In other words, the status of the vehicle 100 is indirectly communicated to the user A by the expressions of the icons 71, etc. Therefore, this is sometimes referred to as the presentation of icons 71 (71A to 71H) by display 207, in the sense that it is different from the method of directly conveying numerical and textual information indicating the status of vehicle 100 by displaying it in virtual space R3 corresponding to user A's central visual field.

[0029] When user A performs a free operation with his / her fingers on icons 71 (71A to 71H) floating in virtual space R3 as illustrated in FIG. 1B , the calculation unit 205 moves the icons 71 (71A to 71H) within virtual space R3 or changes their size (enlarges or reduces them). Specifically, the calculation unit 205 rotates the icons 71 (71A to 71H) in accordance with the movement of user A's fingers, moves them within virtual space R3, or changes their presentation size (enlarges or reduces them). When user A performs a free operation, the calculation unit 205 may move the icons 71 (71A to 71H) to region R1 or R2 corresponding to the central visual field or the effective visual field.

[0030] Furthermore, when sound information transmitted from the IVI system 110 is received by the communication unit 208 via the communication network 300, the calculation unit 205 may share the sound information between the vehicle 100 (IVI system 110) and the computer 200. This sharing allows sound based on the sound information to be reproduced from the speaker 203. That is, user A can enjoy the same sound content as user B. Furthermore, when image information transmitted from the IVI system 110 is received by the communication unit 208 via the communication network 300, the calculation unit 205 may share the image information between the vehicle 100 (IVI system 110) and the computer 200. This sharing allows video based on the image information to be displayed on the display 207. That is, user A can enjoy the same video content as user B.

[0031] Furthermore, when the voice information sent from the IVI system 110 is received by the communication unit 208 via the communication network 300, the calculation unit 205 determines whether the voice information is a predetermined word. If the calculation unit 205 determines that the voice information is a predetermined word (e.g., "XX airport"), it displays a predetermined message corresponding to the word (e.g., "Now, XX airport") near the icon 71 displayed on the display 207.

[0032] (Storage Unit) The storage unit 206 has a volatile or non-volatile memory (not shown). The storage unit 206 stores various applications executed by the calculation unit 205, various data, etc. The calculation unit 205 records and reads data etc. in the storage unit 206.

[0033] (Display) The display 207 is capable of providing stereoscopic images using high-definition displays provided for the left and right eyes, respectively.

[0034] (Communication Unit) The communication unit 208 is a wireless communication module that performs wireless communication with the communication network 300 .

[0035] <IVI System> Fig. 2B is a block diagram showing an example configuration of the IVI system 110 in Fig. 1A. The IVI system 110 includes a vehicle communication device 160, a group of vehicle sensors 112 of a vehicle 100 as a moving body, an output device 114 provided in the vehicle 100, an input device 111 provided in the vehicle 100, and an interior camera 113 provided in the vehicle 100.

[0036] The vehicle communication device 160, the input device 111, the vehicle sensor group 112, the in-vehicle camera 113, and the output device 114 are configured to be able to communicate with each other via wired communication using a CAN (Controller Area Network) or the like.

[0037] <Vehicle communication device> The vehicle communication device 160 includes, as its functional configuration, an occupant detection unit 161, a vehicle state detection unit 162, an output unit 163, an input unit 164, a control unit 165, and a communication unit 166, and controls the IVI system 110 to function as the vehicle communication device 160.

[0038] In the embodiment, as an example, when user B gets into vehicle 100, the vehicular communication device 160 calls out to user B via a voice reproduction unit (not shown) constituting the output device 114, and the voice of user B responding (e.g., stating his / her name) is collected by a microphone (not shown) constituting the input device 111, and an image of user B is captured by the in-vehicle camera 113. The vehicular communication device 160 associates (or may associate) the face of user B, the name of user B, and the frequency components of the voice emitted by user B.

[0039] (Occupant Detection Unit) The occupant detection unit 161 detects the user B who is riding in the vehicle 100. For example, the occupant detection unit 161 detects the user B based on a camera image acquired by the interior camera 113.

[0040] (Vehicle State Detection Unit) The vehicle state detection unit 162 acquires vehicle speed information, position information, and other information as status information of the vehicle 100 from the vehicle sensor group 112 via the communication unit 166. This enables the vehicular communication device 160 to detect the state of the vehicle 100.

[0041] (Output Unit) When an audio signal is included in a communication signal acquired from an external device (e.g., server device 400) communicating with communication unit 166, output unit 163 causes an audio playback unit (speaker, etc.) constituting output device 114 to play back audio based on the audio signal output from output unit 163. When a video signal is included in a communication signal acquired from an external device (e.g., server device 400) via communication network 300, output unit 163 causes a display unit constituting output device 114 to display video based on the video signal output from output unit 163.

[0042] (Input Unit) The input unit 164 inputs the voice uttered by the user B. More specifically, the input unit 164 inputs a voice signal collected by a microphone (not shown).

[0043] (Control Unit) The control unit 165 (essentially, the MPU (Micro Processing Unit) of the IVI system 110) reads and executes an application for the vehicle communication device 160 stored in a memory unit (not shown) to perform various information processing, control processing, etc. required for the vehicle communication device 160.

[0044] The control unit 165 may control the input unit 164 to start inputting a voice signal (in other words, start collecting sound using a microphone not shown) when the occupant detection unit 161 detects that the user B has gotten into the vehicle 100 and, for example, the vehicle sensor group 112 detects an ignition-on operation. Furthermore, the control unit 165 may control the input unit 164 to start inputting a voice signal (in other words, start collecting sound using a microphone not shown) when the occupant detection unit 161 detects that the user B has gotten into the vehicle 100 and the vehicle sensor group 112 detects the opening and closing of a door as an opening / closing body of the vehicle 100.

[0045] The control unit 165 may further control the communication unit 166 so that, when the communication unit 166 receives a first request signal sent from the computer 200 via the communication network 300, the communication unit 166 transmits a voice signal of the user B collected in the vehicle 100 to the computer 200 via the communication network 300. The first request signal is a signal from the computer 200 to the IVI system 110 requesting that the IVI system 110 transmit a voice signal input to the input unit 164 (e.g., a microphone not shown) to the computer 200.

[0046] Furthermore, when the second request signal sent from computer 200 is received by communication unit 166 via communication network 300, control unit 165 causes a sound playback unit (such as a speaker) constituting output device 114 to play predetermined sound information (e.g., an electronic sound) that has been stored in advance in control unit 165. The second request signal is a signal from computer 200 to request IVI system 110 to play predetermined sound information in vehicle 100.

[0047] (Communication Unit) The communication unit 166 includes a wireless communication module (not shown) that performs wireless communication with the communication network 300, and a wired communication module (not shown) that performs wired communication using a CAN or the like.

[0048] <Input Device> The input device 111 includes an operation detection unit and a microphone (not shown). (Operation Detection Unit) The operation detection unit is operated by user B, who is an occupant of the vehicle 100, and outputs an operation signal to the vehicle communication device 160. The operation detection unit may be provided on a display surface of a display device constituting the output device 114 (described later).

[0049] (Microphone) The microphone collects the voice uttered by the user B and outputs a voice signal to the vehicle communication device 160 .

[0050] <Vehicle Sensor Group> The vehicle sensor group 112 includes a vehicle speed sensor, a positioning sensor, a magnetic sensor (direction sensor), a brake operation sensor, a steering sensor, an accelerator operation sensor, an opening / closing body (door, rear gate, etc.) sensor, an energy (battery SOC or fuel) sensor, a turn signal operation sensor, a wiper operation sensor, an air conditioner operation sensor, an ignition switch sensor, etc. The vehicle sensor group may also include a drive recorder (not shown).

[0051] Information detected by each sensor of the vehicle sensor group 112 is output to the vehicle communication device 160. The vehicle communication device 160 may acquire driving history information of the vehicle 100 and information about roads traveled by the vehicle 100 based on the information detected by the vehicle sensor group 112. The vehicle communication device 160 may also acquire operation information of a navigation device as one of the functions of the IVI system 110 of the vehicle 100, operation information of other functions of the IVI system 110 (e.g., content playback information), and information about recorded video from a drive recorder.

[0052] <In-Vehicle Camera> The in-vehicle camera 113 captures an image of, for example, the upper body of a user B who is an occupant of the vehicle 100, and outputs data of the subject image to the vehicle communication device 160 as image information.

[0053] <Output Device> The output device 114 includes a display unit and an audio playback unit (not shown). The output device 114 is used, for example, to display video content or play content. (Display Unit) The display unit has a screen such as a liquid crystal display, and displays image information based on a display signal output from the vehicle communication device 160. The image information includes images of the video content, etc.

[0054] (Audio Reproducing Unit) The audio reproducing unit is configured as a speaker that reproduces and outputs audio and the like, and reproduces sound content and the like based on a reproduction signal output from the vehicle communication device 160 .

[0055] 6 is a flowchart illustrating an example of a status information presentation process performed by an app executed by the calculation unit 205 of the computer 200. When a launch icon of a status information presentation app (not shown) is selected by the user A (by line of sight and finger operation) in the virtual space R3 as illustrated in FIG. 1B, the calculation unit 205 executes the app that causes the computer 200 to function as a status information presentation device.

[0056] 6, the calculation unit 205 acquires information about the vehicle 100 via the communication network 300 using the communication unit 208, and proceeds to step S20. The calculation unit 205 acquires, for example, user information about the vehicle 100 or an occupant (user B) of the vehicle 100, and status information about the vehicle 100.

[0057] In step S20, the calculation unit 205 makes a determination on the user information based on the acquired information, and proceeds to step S30. The calculation unit 205 determines, for example, whether or not a combination of the ID information of the computer 200 and the above user information has been stored (may also be called registered) in advance in the storage unit 206.

[0058] In step S30, the calculation unit 205 determines the state of the vehicle 100, and the process proceeds to step S40. The calculation unit 205 determines the state of the vehicle 100, for example, based on the status information of the vehicle 100 acquired in step S10. More specifically, the calculation unit 205 determines the state of the vehicle 100 for each item, such as brake operation information and wiper operation information. Note that the calculation unit 205 can use at least one of the following as the status information of the vehicle 100: driving history information of the vehicle 100, driving road information, direction information, brake operation information, steering operation information, accelerator operation information, opening / closing device operation information, energy information, turn signal operation information, wiper operation information, air conditioner operation information, navigation device operation information, information from the IVI system 110, and information from the drive recorder.

[0059] In step S40, the calculation unit 205 visualizes the vehicle 100 as illustrated in Fig. 1B, and presents an icon 71 indicating the status information of the vehicle 100 in the peripheral vision on the display 207, and then the process proceeds to step S50. The icon 71 is initially presented in the manner illustrated in Fig. 3A (steady display).

[0060] In step S50, the calculation unit 205 quantifies the state of the vehicle 100, and the process proceeds to step S60. The calculation unit 205 quantifies the state of the vehicle 100 for each of the above items into, for example, five levels between the highest and lowest values.

[0061] In step S60, the calculation unit 205 determines whether the quantified numerical value is equal to or greater than a predetermined threshold. If the numerical value is equal to or greater than the threshold (or equal to or less than the threshold) (for example, equal to or greater than the fourth level when the worst value is level 5 and the best value is level 0, or equal to or less than the first level when the best value is level 5 and the worst value is level 0), the calculation unit 205 makes a positive judgment in step S60 and proceeds to step S70. If the numerical value is not equal to or greater than the threshold (or equal to or less than the threshold), the calculation unit 205 makes a negative judgment in step S60 and proceeds to step S100.

[0062] In step S70, the calculation unit 205 switches the mode of the icon 71 (steady display) to one of the modes (non-steady display) of the icons 71A to 71H, and proceeds to step S80. By switching the steady display icon 71 to the non-steady display icon 71A to 71H, it is possible to gently inform the user A that some kind of state change has occurred in the vehicle 100 in which the user B is riding.

[0063] In step S80, the calculation unit 205 determines whether the digitized numerical value is less than a predetermined threshold. If the numerical value is less than the threshold (for example, less than or equal to the third level when the worst value is level 5 and the best value is level 0, or more than or equal to the second level when the best value is level 5 and the worst value is level 0), the calculation unit 205 makes a positive judgment in step S80 and proceeds to step S90. If the numerical value is not less than the threshold, the calculation unit 205 makes a negative judgment in step S80 and proceeds to step S100.

[0064] In step S90, the calculation unit 205 returns the state of any one of the non-steady-state display icons 71A to 71H to the state of the steady-state display icon 71, and proceeds to step S100. Note that the processing of steps S50 to S90 is performed based on the respective thresholds for each item of the state of the vehicle 100. As a result, if there are multiple items that are above (or below) the threshold, the calculation unit 205 causes the display 207 to present the corresponding icons 71A to 71H in the peripheral vision in sequence at predetermined time intervals (for example, 2 seconds).

[0065] In step S100, the calculation unit 205 determines whether or not an operation to terminate the app has been performed by user A. If a predetermined termination operation has been performed, the calculation unit 205 makes a positive judgment in step S100 and proceeds to step S110, and if a termination operation has not been performed, the calculation unit 205 makes a negative judgment in step S100 and proceeds to step S120.

[0066] In step S110, the calculation unit 205 performs a termination process to terminate the process shown in FIG. 6 . The termination process includes, for example, the termination of presentation of icons 71, 71A to 71H, and the termination of acquisition of information from vehicle 100 by communication unit 208. Furthermore, if a sharing setting is made in additional process 1 described later, the termination process includes the cancellation of the sharing setting. Furthermore, if a first request signal is sent to vehicle 100 in additional process 3 described later, and if a second request signal is sent to vehicle 100 in additional process 4 described later, the termination process includes the cancellation of these request signals. Note that wireless communication with communication network 300 by communication unit 208 does not need to be terminated if it is required for an app other than the status information presentation app.

[0067] In step S120, the calculation unit 205 reacquires the status information of the vehicle 100 via the communication network 300 using the communication unit 208, and returns to step S40. When returning to step S40, the calculation unit 205 repeats the above-described processing.

[0068] <Explanation of Additional Process 1> A process according to the flowchart illustrated in Fig. 7A may be added between step S90 and step S100 in Fig. 6 described above. In step S91 in Fig. 7A, the calculation unit 205 determines whether or not the quantified values ​​representing the state of the vehicle 100 have not changed for a predetermined period of time. If there has been no change in all items representing the state of the vehicle 100 for, for example, 10 minutes, the calculation unit 205 makes a positive determination in step S91 and proceeds to step S92. If a change is recognized in the quantified values ​​representing the state of the vehicle 100, the calculation unit 205 makes a negative determination in step S91, terminates the process in Fig. 7A, and proceeds to step S100 in Fig. 6.

[0069] In step S92, the calculation unit 205 starts to acquire content information from the vehicle 100 via the communication network 300 by the communication unit 208, and the process proceeds to step S93.

[0070] In step S93, the calculation unit 205 performs sharing settings on the acquired content information, terminates the processing of Fig. 7A, and proceeds to step S100 of Fig. 6. By performing sharing settings, sound can be played from the speaker 203 of the computer 200, and video content can be played on the display 207. This allows user A to enjoy the same audio content and / or video content as user B.

[0071] <Explanation of Additional Process 2> Processing according to the flowchart illustrated in Fig. 7B may be added between step S90 and step S100 in Fig. 6 described above. In step S95 in Fig. 7B, the calculation unit 205 determines whether or not movement of the computer 200 (in other words, a change in position or posture) has been detected. When the calculation unit 205 detects movement of the computer 200, i.e., movement of the head of user A, based on a signal from the IMU constituting the sensor group 204, the calculation unit 205 makes a positive determination in step S95 and proceeds to step S96. When the calculation unit 205 does not detect movement of the computer 200, the calculation unit 205 makes a negative determination in step S95, terminates the processing according to Fig. 7B, and proceeds to step S100 in Fig. 6.

[0072] In step S96, the calculation unit 205 suspends the presentation of the icon 71 (or 71A to 71H) on the display 207, and the process proceeds to step S97. The presentation may be suspended, for example, by causing the icon 71, etc., representing the status information of the vehicle 100, to exit the virtual space R3 on the display 207.

[0073] In step S97, the calculation unit 205 determines whether the acceleration based on the signal from the IMU constituting the sensor group 204 is equal to or less than a predetermined threshold and has remained at or below the predetermined threshold for a predetermined period of time. If the calculation unit 205 detects that the acceleration is equal to or less than the predetermined threshold and has remained at or below the predetermined threshold for a predetermined period of time, the calculation unit 205 makes a positive determination in step S97 and proceeds to step S98. If the calculation unit 205 does not detect that the acceleration is equal to or less than the predetermined threshold and has remained at or below the predetermined threshold for a predetermined period of time, the calculation unit 205 repeats this determination process and waits for the computer 200 to finish moving.

[0074] In step S98, the calculation unit 205 resumes presenting the icon 71 in the peripheral vision on the display 207, ends the processing of Fig. 7B, and proceeds to step S100 of Fig. 6. The presentation of the icon 71 is resumed by making the icon 71 appear in the peripheral vision on the display 207 from the same direction as the direction of acceleration based on the signal from the IMU that constitutes the sensor group 204.

[0075] <Explanation of Additional Process 3> A process according to the flowchart illustrated in FIG. 7C may be added between step S30 and step S40 in FIG. 6 described above. The process in FIG. 7C may be added as a set with the process in FIG. 7D described below. In step S31 in FIG. 7C, the calculation unit 205 causes the communication unit 208 to send a first request signal to the vehicle 100 via the communication network 300. The first request signal is a signal that requests the vehicle 100 to turn on the microphone (sound collection state) and send information about the collected sound. After sending the first request signal, the calculation unit 205 ends the process in FIG. 7C and proceeds to step S40 in FIG. 6.

[0076] <Explanation of Additional Process 4> A process according to the flowchart illustrated in Fig. 7D may be added between step S90 and step S100 in Fig. 6 described above. The process of Fig. 7D may be added as a set with the process of Fig. 7C described above. In step S910 in Fig. 7D, the calculation unit 205 starts acquiring voice information from the vehicle 100 via the communication network 300 using the communication unit 208, and proceeds to step S920. The voice information is voice information collected by the microphone of the vehicle 100, in other words, voice uttered by the user B and ambient sounds reaching the cabin of the vehicle 100.

[0077] In step S920, the calculation unit 205 determines whether the voice information includes a predetermined word. If the voice information includes the predetermined word, the calculation unit 205 makes a positive determination in step S920 and proceeds to step S930. If the voice information does not include the predetermined word, the calculation unit 205 makes a negative determination in step S920 and proceeds to step S940. The predetermined word is not limited to one word, and may be multiple words. For example, it may be "station," "airport," "interchange," "traffic jam," etc.

[0078] In step S930, the calculation unit 205 displays a predetermined message corresponding to a predetermined word near the currently presented icon 71 (or 71A to 71H) on the display 207, and then proceeds to step S940. The message may be predetermined for each predetermined word and stored in the storage unit 206, such as "You are now at XX Station," "You are now at △△ Airport," "You are now at XX Interchange," or "There is a traffic jam."

[0079] In step S940, the calculation unit 205 determines, using the hand tracking unit constituting the sensor group 204, whether or not an operation of pinching the icon 71 (or 71A to 71H) being presented in the virtual space R3 on the display 207 with the fingers of the user A has been performed. If a pinching operation has been performed, the calculation unit 205 makes a positive judgment in step S940 and proceeds to step S950. If a pinching operation has not been performed, the calculation unit 205 makes a negative judgment in step S940, terminates the processing in FIG. 7D, and proceeds to step S100 in FIG. 6.

[0080] In step S950, calculation unit 205 causes communication unit 208 to send a second request signal to vehicle 100 via communication network 300. The second request signal is a signal that requests vehicle 100 to reproduce predetermined audio information from output device 114. The predetermined audio information is, for example, a beep, a door knock, or other sound that attracts user B's attention. After sending the second request signal, calculation unit 205 ends the processing in FIG. 7D and proceeds to step S100 in FIG. 6.

[0081] One or more of the additional processes 1, 2, and 3 described above may be arbitrarily combined with the process shown in Fig. 6, or may not be combined with the process shown in Fig. 6. However, as mentioned above, additional processes 3 and 4 are used as a set.

[0082] The above-described embodiment provides the following advantageous effects. (1-1) A status information presentation program that causes head-mounted display computer 200, including communication unit 208 as a first communication unit, storage unit 206, and display 207, to present status information of vehicle 100, including communication unit 166 as a second communication unit, causes computer 200 to execute the following steps: a process of acquiring user information about vehicle 100 or user B as an occupant of vehicle 100 and status information of vehicle 100 via communication unit 208 (S10); a process of determining whether ID information as an identifier of computer 200 stored in storage unit 206 and the user information satisfy predetermined conditions (S20); a process of determining the status of vehicle 100 based on the status information of vehicle 100 if the predetermined conditions are satisfied (S30); and a process of presenting the status information of vehicle 100 on display 207 in a display format in which facial expressions are added to icons 71 or the like that evoke vehicle 100 (S40). With this configuration, it becomes possible for user A to watch over user B in vehicle 100 without causing both user A and user B trouble, such as by having user A using computer 200 keep a phone call to user B in vehicle 100 (the pairing partner). To explain in more detail, user B in vehicle 100 experiences less psychological stress from being watched over than if he or she were constantly monitored by a camera or the like. Furthermore, user A watching over user B does not simply watch the behavior of user B in vehicle 100, but watches over the state of vehicle 100 in which user B is riding, so the psychological burden of watching over user B is reduced.

[0083] (1-2) The process (S30) for determining the state of the vehicle 100 determines the state of the vehicle 100 by using at least one of the following as status information of the vehicle 100: driving history information of the vehicle 100, road information, direction information, brake operation information, steering operation information, accelerator operation information, open / close body operation information, energy information, blinker operation information, wiper operation information, air conditioner operation information, navigation device operation information, information from the IVI system 110 as in-vehicle infotainment information, and drive recorder information. This configuration enables user A to appropriately keep an eye on user B through the state of the vehicle 100 in which user B is riding.

[0084] (1-3) The status information presentation program further causes the computer 200 to execute a process of quantifying the determined status of the vehicle 100 into a predetermined number of stages (S50), and a process of switching the display format of the vehicle status information from a steady display to a non-steady display when the quantified value is equal to or greater than a predetermined threshold (S60, S70). With this configuration, the user A can appropriately watch over the user B through the expressions of the icon 71, etc., that evoke the vehicle 100.

[0085] (1-4) The status information presentation program further causes the computer to execute the following processes (S91, S92) when the status information of the vehicle 100 does not change for a predetermined time: acquiring, via the communication unit 208, at least one of sound information being played by the IVI system 110 in the vehicle 100 and image information acquired by the drive recorder (vehicle sensor group 112) of the vehicle 100 as content information; and sharing the acquired sound information and image information with the computer 200 (S93). If the status information of the vehicle 100 does not change for a long time, the icon 71 presented to user A does not change for a long time. At this time, user A may become anxious about whether the status information of the vehicle 100 in which user B is riding is being correctly received by the computer 200. In such a situation, by sharing the sound information being played in the vehicle 100 or the image information acquired by the drive recorder of the vehicle 100 between the vehicle 100 and the computer 200 used by user A, it is possible to alleviate user A's anxiety.

[0086] (2-1) A status information presentation program that causes a head-mounted display type computer 200 including a communication unit 208 as a first communication unit, a storage unit 206, and a display 207 to present status information of the vehicle 100 including the communication unit 166 as a second communication unit includes a process (S10) of acquiring first user information about the vehicle 100 or user B as an occupant of the vehicle 100 and status information of the vehicle 100 via the communication unit 208, ID information as an identifier of the computer 200 stored in the storage unit 206, and the first user information. The computer 200 executes the following processes: a process of determining whether the vehicle status information satisfies a predetermined condition (S20); a process of determining the state of the vehicle 100 based on the status information of the vehicle 100 if the predetermined condition is satisfied (S30); and a process of presenting the state information of the vehicle 100 in a virtual space R3 as a first predetermined range of the display 207 corresponding to the visible range of the second user A wearing the computer 200, and in an area other than the area R1 as a second predetermined range of the display 207 corresponding to the central visual field of the user A. With this configuration, the user A can watch over the user B through the state of the vehicle 100 (pairing partner) in which the user B is riding, without interfering with the main work of the user A using the computer 200 (e.g., work using the worksheet 72) and without requiring the user A to take time to do something (e.g., have the user A specify the display position of the state information in the virtual space R3).

[0087] (2-2) The process of presenting the status information of the vehicle 100 on the display 207 involves presenting the information in a display format in which an expression is added to an icon 71 or the like that evokes the vehicle 100 in a virtual space R3 as a first predetermined range of the display 207, in an area other than the area R1 as a second predetermined range of the display 207, and in an area other than the area R2 as a third predetermined range of the display 207 that corresponds to the effective visual field of the user A. With this configuration, it becomes possible to present the status information of the vehicle 100 in an appropriate position within the virtual space R3 of the display 207 that does not interfere with the main work of the user A using the computer 200 (for example, work using the worksheet 72).

[0088] (2-3) The status information presentation program further causes the computer to execute the following processes: a process (S10) for acquiring acceleration information of the computer 200; a process (S96) for suspending the presentation of the status information of the vehicle 100 when a change in the position of the computer 200 is detected based on the acceleration information; and a process (S97, S98) for resuming the presentation of the status information of the vehicle 100 when the acceleration information falls below a predetermined threshold and a predetermined time has elapsed. With this configuration, when the head of user A using the computer 200 moves, the presentation position of the status information of the vehicle 100 temporarily disappears outside the field of view (in other words, exits the virtual space R3 of the display 207). After a predetermined time has elapsed since the movement of the user A's head has stopped, the presentation of the status information of the vehicle 100 is restored (in other words, returns to the virtual space R3 of the display 207). Therefore, the icon 71 or the like representing the status information of the vehicle 100 does not move around in the virtual space R3, thereby reducing the annoyance to the user A.

[0089] (2-4) The process (S98) of resuming presentation of the status information of the vehicle 100 causes the icon 71 or the like, which is status information of the vehicle 100, to appear in the same direction as the direction of acceleration indicated by the acceleration information within the virtual space R3, which is the first predetermined range of the display 207. With this configuration, for example, when the user A looks to the side or behind while moving, it is possible to make the icon 71 or the like appear within the virtual space R3 as if the icon 71 or the like is following the user A as he or she moves.

[0090] (2-5) In the process (S40) of presenting status information of the vehicle 100 on the display 207, a first predetermined range (virtual space R3) is a viewing angle of approximately 200 degrees, which is the combined right and left visual fields of the user A, a second predetermined range (region R1) is a viewing angle of approximately 1 to 2 degrees as the central visual field of the user A, and a third predetermined range (region R2) is a viewing angle of approximately 4 to 20 degrees as the effective visual field of the user A. With this configuration, it is possible to present to the user A an icon 71 or the like as status information of the vehicle 100 at an appropriate position in the virtual space R3.

[0091] (2-6) In the process (S40) of presenting the status information of the vehicle 100 on the display 207, the first predetermined range (virtual space R3), the second predetermined range (region R1), and the third predetermined range (region R2) of the display 207 are determined based on the visibility information of the user A acquired by the computer 200 when the computer 200 is worn by the user A. With this configuration, it is possible to present the user A with an icon 71 or the like as the status information of the vehicle 100 at an appropriate position in the virtual space R3.

[0092] (3-1) An information transmission program that causes a head-mounted display type computer 200 including a communication unit 208 as a first communication unit, a microphone group 202 and a speaker 203 as a first audio input / output unit, a storage unit 206, and a display 207 to transmit information to a vehicle communication device 160 (IVI system 110) as a vehicle-side device including a communication unit 166 as a second communication unit, an input device 111 and an output device 114 as a second audio input / output unit, transmits, when the information transmission program is started on the computer 200, first user information regarding the vehicle 100 or user A as an occupant of the vehicle 100, The computer 200 executes the following processes: a process of acquiring the first user information and status information of the vehicle 100 via the communication unit 208 (S10); a process of determining whether the first user information and ID information as identifier information of the computer 200 stored in the storage unit 206 satisfy a predetermined first condition and whether the status information of the vehicle 100 satisfies a predetermined second condition (S20, S30); and a process of sending, if the first condition and the second condition are satisfied, a first request signal via the communication unit 208 to request the vehicular communication device 160 to transmit, via the communication unit 166, voice information input to the input device 111. With this configuration, it is possible to request the vehicular communication device 160 of the vehicle 100 (pairing partner) in which the user B is riding to transmit, for example, information about voice uttered by the user B and ambient sounds reaching the cabin of the vehicle 100 via the communication unit 166.

[0093] (3-2) The process (S30) of determining whether the status information of the vehicle 100 satisfies a predetermined second condition is performed by using at least one of information indicating whether the doors or the like are open or closed as an opening / closing body of the vehicle 100 and information indicating that the ignition is on as the status information of the vehicle 100. With this configuration, it is possible to appropriately determine whether the user B has boarded the vehicle 100 and is about to move the vehicle 100.

[0094] (3-3) The information transmission program further causes the computer 200 to execute the following processes: a process (S910) of acquiring, via the communication unit 208, voice information that has been input to the input device 111 and transmitted via the communication unit 166; a process (S920) of determining whether the acquired voice information is a predetermined word; and a process (S930) of displaying a predetermined message corresponding to the word on the display 207 if the predetermined word is determined. This configuration makes it possible to deliver a predetermined message corresponding to a word uttered by the user B on the vehicle 100 to the user A using the computer 200. For example, a text message that is pre-associated with a word such as a place name, a facility name, or weather may be presented.

[0095] (3-4) When the first condition is satisfied, the information transmission program further causes the computer to execute the following processes: determining the status of the vehicle 100 based on the situation information of the vehicle 100 (S30); presenting an icon 71 or the like indicating the status information of the vehicle 100 on the display 207 (S40); and, when detecting an operation of pinching the icon 71 or the like by a user A wearing the head-mounted display-type computer 200 (S940), transmitting a second request signal via the communication unit 208 to request the vehicle communication device 160 to play predetermined audio information from the output device 114 (S950). With this configuration, when user A using the computer 200 performs an operation such as pinching the icon 71 or the like in the virtual space R3, it becomes possible to request, for example, the output device 114 to play a telegraph sound in the vehicle 100 paired with the computer 200. In other words, user A can use his or her fingers to send a signal or the like to user B in the vehicle 100.

[0096] The above-described embodiment can be modified in various ways. Modifications are described below. (Modification 1) In the above-described embodiment, an example has been described in which the vehicle 100 accesses the communication network 300 using the communication unit 166 of the vehicle communication device 160 constituting the IVI system 110. Alternatively, the communication network 300 may be accessed by a carry-on terminal such as a smartphone used by the user B in the vehicle 100. In this case, among the conditions of the vehicle 100 detected by the vehicle sensor group 112, information that can be obtained directly by sensors provided on the smartphone or indirectly based on information obtained by the smartphone, such as driving history information, road information, and direction information, may be substituted for the information detected by the vehicle sensor group 112. Furthermore, when accessing the communication network 300 using a carry-on terminal, a unique ID specific to the carry-on terminal may be used instead of or in addition to the vehicle ID.

[0097] (Variation 2) In the embodiment described above, an example was given of computer 200 of the type that displays on a display worksheets 72, 73, and 74 of an office work application running on computer 200, with the scenery ahead of user A superimposed on a stereo image captured by a built-in camera of computer 200. Alternatively, a computer may be used that displays worksheets 72, 73, and 74 of the office work application on a holographic display, and uses a prism or a half mirror to synthesize the image from the holographic display with a real image of the scenery ahead of user A. In this case, icons 71 and the like indicating the status of vehicle 100 are displayed on the holographic display.

[0098] (Variation 3) In the above-described embodiment, an integrated computer 200 has been described as an example in which the calculation unit 205 and the display 207 are integrally formed together with other components. Alternatively, a separate computer in which the main calculation unit 205 and the display 207 are formed separately may be used to execute the program according to the embodiment. The display 207 worn by the user A on his / her head and the calculation unit 205 that controls the display of this display 207 are connected, for example, by short-range wireless communication.

[0099] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications, as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other.

[0100] 71, 71A to 71H icons, 72 to 74 worksheets, 100 vehicle, 110 IVI system, 111 input device, 112 vehicle sensor group, 113 in-vehicle camera, 114 output device, 160 vehicle communication device, 200 computer, 201 camera group, 202 microphone group, 203 speaker, 204 sensor group, 205 calculation unit, 206 memory unit, 207 display, 208 communication unit, 300 communication network, 400 server device, 500 information provision system, A, B user, R1, R2 area, R3 virtual space

Claims

1. A status information presentation program that causes a head-mounted display type computer including a first communication unit, a memory unit, and a display to present status information of a vehicle including a second communication unit, characterized in that the computer executes the following processes: a process of acquiring user information regarding the vehicle or the occupants of the vehicle and status information of the vehicle via the first communication unit; a process of determining whether the identifier of the computer stored in the memory unit and the user information satisfy predetermined conditions; a process of determining the status of the vehicle based on the status information of the vehicle if the predetermined conditions are satisfied; and a process of presenting the status information of the vehicle on the display in a display format in which an expression is added to an icon that evokes the vehicle.

2. A status information presentation program as claimed in claim 1, characterized in that the process of determining the vehicle's status determines the vehicle's status using at least one of the vehicle's driving history information, driving road information, direction information, brake operation information, steering operation information, accelerator operation information, opening and closing device operation information, energy information, turn signal operation information, wiper operation information, air conditioner operation information, navigation device operation information, in-vehicle infotainment information, and drive recorder information as the vehicle's situation information.

3. A status information presentation program as claimed in claim 1 or 2, further comprising causing the computer to execute the following processes: a process of quantifying the determined vehicle status into a predetermined number of stages; and a process of switching the display format of the vehicle status information from a steady state display to a non-steady state display when the quantified value is equal to or greater than a predetermined threshold value.

4. A status information presentation program as described in claim 3, characterized in that the program further causes the computer to execute the following processes: when the status information of the vehicle does not change for a predetermined period of time, acquiring at least one of information about the sound being played inside the vehicle and image information acquired by the vehicle's drive recorder via the first communication unit; and sharing the acquired sound information and image information with the computer.

5. A status information presentation method for causing a head-mounted display type computer including a first communication unit, a memory unit, and a display to present status information of a vehicle including a second communication unit, comprising the steps of: acquiring user information relating to the vehicle or an occupant of the vehicle and status information of the vehicle via the first communication unit; determining whether the computer identifier stored in the memory unit and the user information satisfy predetermined conditions; determining the status of the vehicle based on the status information of the vehicle if the predetermined conditions are satisfied; and presenting the status information of the vehicle on the display in a display format in which an expression is added to an icon reminiscent of the vehicle.

Images