Travel information reproduction program and travel information reproduction method

Abstract

Provided is a travel information reproduction program for causing a head-mounted-display-type computer including a first communication unit, a storage unit, and a display to display travel information pertaining to a vehicle including a second communication unit, the travel information reproduction program causing the computer to execute: a process for acquiring, via the first communication unit, vehicle information including position information and heading information pertaining to the vehicle; a process for acquiring map information from the storage unit or via the first communication unit; a process for displaying, on the display, a map for a prescribed range including the position of the vehicle, on the basis of the position information and the map information; a process for displaying, on the display, an icon image representing the vehicle in an orientation based on the heading information, in accordance with the map; a process for detecting a first operation for instructing a first viewpoint mode and a second operation for instructing a second viewpoint mode from a user wearing the computer; and a display control process for switching the map to a bird's-eye view mode when the first operation is detected, and switching the map to a mode of viewing from behind the vehicle when the second operation is detected.

Description

Driving information reproduction program and driving information reproduction method

[0001] The present invention relates to a driving information reproduction program and a driving information reproduction method for displaying vehicle status information and a map on a head-mounted display type computer.

[0002] A related technique is known in which image data of a virtual space is displayed in accordance with a user's current position and input operations, as if the user were moving within the virtual space (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2006-292616

[0004] The conventional technology is intended to meet the needs of users, such as for preparing a route to travel, and when a user actively performs input operations, an image corresponding to the scenery seen from the user's perspective is displayed on the image display unit, as if the user were moving within the virtual space. Therefore, it is not suited to usage patterns such as a user who uses a head-mounted display computer in daily life freely viewing another user's driving a vehicle from their own perspective.

[0005] A driving information reproduction program according to one aspect of the present invention causes a head-mounted display computer including a first communication unit, a storage unit, and a display to display driving information of a vehicle including a second communication unit. The driving information reproduction program causes the computer to execute the following processes: acquiring vehicle information including vehicle position information and orientation information via the first communication unit; acquiring map information from the storage unit or via the first communication unit; displaying a map of a predetermined range including the vehicle's position on the display based on the position information and the map information; displaying an icon image on the display indicating the vehicle oriented based on the orientation information in accordance with the map; detecting a first operation from a user wearing the computer to instruct a first viewpoint mode and a second operation to instruct a second viewpoint mode; and display control processing to switch the map to a bird's-eye view mode when the first operation is detected and to switch the map to a viewable from behind the vehicle when the second operation is detected. Another aspect of the present invention is a driving information reproduction method causing a head-mounted display computer including the first communication unit, a storage unit, and a display to display driving information of a vehicle including the second communication unit. The driving information reproduction method includes the steps of: acquiring vehicle information including vehicle position information and orientation information via a first communication unit; acquiring map information from a memory unit or via the first communication unit; displaying a map of a predetermined range including the vehicle's position on a display based on the position information and the map information; displaying an icon image on the display indicating the vehicle oriented based on the orientation information in accordance with the map; detecting a first operation from a user wearing the computer that indicates a first viewpoint mode and a second operation that indicates a second viewpoint mode; and performing a display control step of switching the map to a bird's-eye view mode when the first operation is detected, and switching the map to a viewable mode from behind the vehicle when the second operation is detected.

[0006] According to the present invention, a user using a head-mounted display type computer can freely view another user driving a vehicle from his or her own viewpoint.

[0007] 1A 。 FIG. 1B is a schematic diagram showing an example of an information provision system. FIG. 1C is a schematic diagram illustrating a virtual space that can be viewed by a user on a display. FIG. 1D is a block diagram illustrating an example of the configuration of the main parts of the computer of FIG. 1A. FIG. 1E is a block diagram illustrating an example of the configuration of the IVI system of FIG. 1A. FIG. 1F is a diagram illustrating an icon for a steady state. FIG. 1G is a diagram illustrating an icon for a non-steady state. FIG. 1H is a diagram illustrating an icon for a non-steady state. FIG. 1I is a diagram illustrating an icon for a non-steady state. FIG. 1I is a diagram illustrating an icon for a non-steady state. FIG. 1J is a diagram illustrating an icon for a non-steady state. FIG. 1J is a diagram illustrating an icon for a non-steady state. FIG. 1J is a diagram illustrating an icon for a non-steady state.

[0008] Hereinafter, an embodiment of the invention will be described with reference to the drawings. <Outline of Status Information Presentation Program> A status information presentation program according to an embodiment is executed by a head-up display computer (hereinafter simply referred to as a computer) as an example of a mixed reality device. A computer executing the status information presentation program performs a monitoring function for monitoring a moving vehicle such as an automobile. For example, user A of the computer monitors, via the computer, a vehicle in which user B, who is a family member of user A, is riding. In other words, the status information presentation program provides user A with a monitoring service for the vehicle (and therefore user B).

[0009] A computer running a status information presentation program acquires information about the vehicle in which User B is riding via a public communication network. The computer then displays the vehicle information on the computer display in a 3D real-world space (which may also be called a virtual space). Furthermore, if the computer detects an abnormality in the acquired vehicle information, it switches the display mode of the vehicle information so that User A notices it.

[0010] In addition, in parallel with the execution of the status information presentation program, the computer executes a driving information reproduction program according to the embodiment for providing a three-dimensional view display based on map information. The view display can be selected from a bird's-eye view of the cityscape from above the vehicle in which user B is riding (first viewpoint view) and a bird's-eye view of the cityscape from behind the vehicle in which user B is riding (second viewpoint view).

[0011] In this embodiment, a vehicle-side communication device is configured as one of the functions of an IVI (In-Vehicle Infotainment) system installed in a vehicle. The vehicle-side communication device transmits information detected by sensors and the like installed in the vehicle to a computer via a public communication network as vehicle status information. An information provision system including such a head-up display type computer, a public communication network, and the vehicle-side communication device will be described in further detail below.

[0012] 1A is a schematic diagram showing an example of an information provision system 500. In the embodiment, a system including a head-up display type computer 200, a public communication network 300, an IVI system 110 as a communication device on the vehicle 100 side, and a server device 400 is exemplified as the information provision system 500.

[0013] FIG. 1B is a schematic diagram illustrating a virtual space R3 that user A can view on the display of computer 200 used by user A in FIG. 1A. Computer 200 displays worksheets 72, 73, and 74 of an office work application program (hereinafter simply referred to as an app) running on computer 200, overlaid with a stereoscopic image of the scenery in front of user A captured by the computer's built-in camera. User A uses worksheets 72, 73, and 74, which appear to float in the 3D virtual space R3 recreated by the stereoscopic images, to perform office work, for example, by moving his / her eyes and right hand or speaking. The stereoscopic images of the backgrounds of worksheets 72, 73, and 74 may be stereo images captured in advance at a different location instead of stereo images captured by the built-in camera. Alternatively, instead of worksheets 72, 73, and 74 of the office work app, the screen of a video or movie viewing app may be displayed floating in the 3D virtual space R3. In general, the ability of humans to perceive depth based on binocular disparity is called binocular stereopsis. In this embodiment, the brain reconstructs a three-dimensional structure from a pair of left and right two-dimensional images displayed on a display 207 (described later) using binocular stereopsis, and the reconstructed space is called virtual space R3.

[0014] Furthermore, when the computer 200 launches an app for presenting status information (a status information presentation program according to the embodiment), it displays an icon 71 on the display that evokes the vehicle 100, and changes the appearance of the icon 71 based on information about the vehicle 100.

[0015] In FIG. 1B , region R1 indicated by a dashed line in virtual space R3 corresponds to a person's central visual field, which has a visual field angle of approximately 1 to 2 degrees. Region R2 indicated by a dashed line corresponds to a person's effective visual field, which has a visual field angle of approximately 4 to 20 degrees. The area outside the effective visual field (in other words, region R2) corresponds to a person's peripheral visual field, which has a visual field angle of approximately 200 degrees combined. Computer 200 displays icon 71 in the peripheral visual field. In this embodiment, icon 71 is further displayed in an area that does not interfere with worksheets 72, 73, and 74.

[0016] Returning to FIG. 1A , vehicle 100 is equipped with an IVI system 110. IVI system 110, which also functions as a communication device, is configured to communicate with server device 400 via communication network 300. Communication network 300 includes not only public wireless communication networks such as the Internet and mobile phone networks, but also closed communication networks established for each designated management area, such as wireless local area networks (LANs) and Wi-Fi (registered trademark). While FIG. 1A illustrates only vehicle 100 in which user B rides, there may be many other vehicles in addition to vehicle 100. In this case, multiple pieces of information obtained when multiple vehicles are traveling are collected in server device 400. Furthermore, while FIG. 1A illustrates only computer 200 used by user A, there may be many other computers in addition to computer 200. In this case, a monitoring service can be provided to many other users in addition to user A.

[0017] The server device 400 is managed, for example, by a business entity that provides maintenance services for the vehicle 100. The server device 400 may be configured using a virtual server function on the cloud, or may be configured as a plurality of distributed devices. For example, the server device 400 may be distributed to each predetermined management area. The server device 400 acquires status information of the vehicle 100 along with time information at predetermined time intervals via the communication network 300 from the IVI system 110, which serves as a communication device for the traveling vehicle 100. The status information of the vehicle 100 includes, for example, information indicating the vehicle ID, direction of travel (azimuth), speed, traveling position, and the operating status of the turn signals (direction indicators). The vehicle ID is, for example, a vehicle identification number (VIN). The server device 400 also acquires, via the communication network 300, information on an abnormality detected by the control system of the vehicle 100. The abnormality information includes, for example, a trouble code (called a DTC (Diagnostic Trouble Code)), the date and time of occurrence (detection), and information indicating the location (latitude and longitude) of the vehicle 100 at the time of occurrence (detection).

[0018] Computer 200, which has launched the app for presenting status information, acquires status information of vehicle 100 that has been registered in advance (in other words, that has been approved for pairing with computer 200) from server device 400 via communication network 300. For example, computer 200 acquires, in real time, the status information of vehicle 100 that server device 400 has acquired from IVI system 110 of vehicle 100 from server device 400. As described above, computer 200 changes the appearance of icon 71 illustrated in FIG. 1B based on the status information of vehicle 100.

[0019] While the computer 200 is running the status information presentation app, the computer 200 may also run a driving information reproduction app that displays a three-dimensional view based on map information in parallel with the status information presentation app. The computer 200 that has started the driving information reproduction app indicates the traveling direction of the vehicle 100 using an icon 71 displayed on the display, and also displays on the display a map of a streetscape as seen from above the vehicle 100 in which the user B is riding (a first viewpoint view) or a map of a streetscape as seen from behind the vehicle 100 in which the user B is riding (a second viewpoint view). The user A can observe the map floating like an island in the virtual space R3 illustrated in FIG. 1B . The position where the map floats in the virtual space R3 can be moved closer to the user A or further away from the user A in the depth direction within the virtual space R3.

[0020] The configuration of such information providing system 500 will be further described with reference to a block diagram. <Computer> Fig. 2A is a block diagram illustrating an example of the configuration of the main parts of computer 200 in Fig. 1A. Computer 200 includes a group of cameras 201, a group of microphones 202, a speaker 203, a group of sensors 204, a calculation unit 205, a storage unit 206, a display 207, and a communication unit 208.

[0021] (Camera Group) The camera group 201 is configured by, for example, a plurality of cameras for capturing images of the space around the user A wearing the computer 200 and the face (particularly the eyes) of the user A.

[0022] (Microphone Group) The microphone group 202 is made up of a plurality of microphones for collecting sounds around the user A wearing the computer 200 and the voice emitted by the user A.

[0023] (Speaker) The speaker 203 is configured by a small speaker system for stereophonically reproducing the sound of the content for the user A wearing the computer 200 .

[0024] (Sensor Group) The sensor group 204 includes an inertial measurement unit, a magnetic sensor, a GPS (Global Positioning System) unit, an eye tracking unit, a hand tracking unit, etc. The IMU is a unit that detects inertial motion (translational motion and rotational motion) in three orthogonal axial directions, and outputs a detection signal that indicates the motion state (movement, posture, etc.) of the head of user A wearing computer 200. The eye tracking unit measures the line of sight of user A wearing computer 200. For example, it measures where the line of sight is directed in space within 3D virtual space R3. The hand tracking unit analyzes the movement of user A's fingers based on camera images.

[0025] The eye tracking unit acquires, as internal information, correction value information for measuring the line of sight of user A wearing computer 200, based on field of view information of user A acquired by computer 200 when computer 200 is worn by user A. Then, based on the correction value information, it determines region R1 corresponding to the central visual field, region R2 corresponding to the effective visual field, and the range of peripheral visual field outside region R2.

[0026] (Calculation Unit) The calculation unit 205 includes a calculation processing unit such as an MPU (not shown), and reads and executes a predetermined program (application) stored in the storage unit 206. When the calculation unit 205 executes the status information presentation application, the calculation unit 205 causes the computer 200 to perform a monitoring function. While the status information presentation application is executing, the calculation unit 205 causes the display 207 to display an icon 71 reminiscent of the vehicle 100 in the peripheral vision, as illustrated in FIG. 1B .

[0027] 3A is a diagram illustrating an example of icon 71 in a steady state. In the embodiment, icon 71 has a shape that reminds user A of vehicle 100. Computing unit 205 presents icon 71 on display 207 in a display format that includes an expression corresponding to status information of vehicle 100.

[0028] 3B and 3C, 4A to 4C, and 5A to 5C are diagrams illustrating icons 71A to 71H in an unsteady state, which change their expressions depending on the state of the vehicle 100. Fig. 3B shows an icon 71A whose expression changes due to the energy state of the vehicle 100, indicating that the battery SOC or fuel is below a predetermined value. Fig. 3C shows an icon 71B whose expression changes due to the sudden braking state of the vehicle 100, indicating that the acceleration during braking is above a predetermined value.

[0029] Fig. 4A shows an icon 71C whose expression changes due to the condition of the tires of the vehicle 100, indicating that the tire pressure is below a predetermined value or that a predetermined period of time has passed since the last inspection. Fig. 4B shows an icon 71D whose expression changes due to the driving condition of the vehicle 100, indicating that the value on the inclinometer is an uphill slope of a predetermined angle or more. Fig. 4C shows an icon 71E whose expression changes due to the operation of the wipers of the vehicle 100, indicating that the operation duration of the wipers has exceeded a predetermined time. Icon 71E may be an animated icon showing the movement of the wipers.

[0030] 5A shows an icon 71F whose expression changes due to the lighting status of the vehicle 100, indicating that the lighting duration of a lighting device such as a headlight has exceeded a predetermined time. FIG. 5B shows an icon 71G whose expression changes due to the content playback status of the vehicle 100, indicating that music content is being played by the IVI system 110. FIG. 5C shows an icon 71H whose expression changes due to the driving status of the vehicle 100, indicating that an airbag or the like has been deployed. As described above, the calculation unit 205 displays the icons 71 (71A to 71H) whose expressions change depending on the status of the vehicle 100 in the virtual space R3 corresponding to the peripheral vision of the user A, which is less noticeable than the central vision. In other words, the status of the vehicle 100 is indirectly communicated to the user A by the expressions of the icons 71, etc. Therefore, this is sometimes referred to as the presentation of icons 71 (71A to 71H) by display 207, in the sense that it is different from the method of directly conveying numerical and textual information indicating the status of vehicle 100 by displaying it in virtual space R3 corresponding to user A's central visual field.

[0031] When user A performs a free operation with his / her fingers on icons 71 (71A to 71H) floating in virtual space R3 as illustrated in FIG. 1B , the calculation unit 205 moves the icons 71 (71A to 71H) within virtual space R3 or changes their size (enlarges or reduces them). Specifically, the calculation unit 205 rotates the icons 71 (71A to 71H) in accordance with the movement of user A's fingers, moves them within virtual space R3, or changes their presentation size (enlarges or reduces them). When user A performs a free operation, the calculation unit 205 may move the icons 71 (71A to 71H) to region R1 or R2 corresponding to the central visual field or the effective visual field.

[0032] (Storage Unit) The storage unit 206 has a volatile or non-volatile memory (not shown). The storage unit 206 stores various applications executed by the calculation unit 205, various data, etc. The calculation unit 205 records and reads data etc. in the storage unit 206.

[0033] (Display) The display 207 is capable of providing stereoscopic images using high-definition displays provided for the left and right eyes, respectively. Simply put, when the relative distance between objects (e.g., maps) included in common on both the left and right sides of a pair of two-dimensional images displayed on the left and right high-definition displays becomes closer, the depth distance to the image of the map in virtual space R3 perceived by user A becomes longer, and when the relative distance between the maps included in common on both the left and right sides becomes wider, the depth distance to the image of the map in virtual space R3 perceived by user A becomes shorter.

[0034] (Communication Unit) The communication unit 208 is a wireless communication module that performs wireless communication with the communication network 300 .

[0035] <IVI System> Fig. 2B is a block diagram showing an example configuration of the IVI system 110 in Fig. 1A. The IVI system 110 includes a vehicle communication device 160, a group of vehicle sensors 112 of a vehicle 100 as a moving body, an output device 114 provided in the vehicle 100, an input device 111 provided in the vehicle 100, and an interior camera 113 provided in the vehicle 100.

[0036] The vehicle communication device 160, the input device 111, the vehicle sensor group 112, the in-vehicle camera 113, and the output device 114 are configured to be able to communicate with each other via wired communication using a CAN (Controller Area Network) or the like.

[0037] <Vehicle communication device> The vehicle communication device 160 includes, as its functional configuration, an occupant detection unit 161, a vehicle state detection unit 162, an output unit 163, an input unit 164, a control unit 165, and a communication unit 166, and controls the IVI system 110 to function as the vehicle communication device 160.

[0038] In the embodiment, as an example, when user B gets into vehicle 100, the vehicular communication device 160 calls out to user B via a voice reproduction unit (not shown) constituting the output device 114, and the voice of user B responding (e.g., stating his / her name) is collected by a microphone (not shown) constituting the input device 111, and an image of user B is captured by the in-vehicle camera 113. The vehicular communication device 160 associates (or may associate) the face of user B, the name of user B, and the frequency components of the voice emitted by user B.

[0039] (Occupant Detection Unit) The occupant detection unit 161 detects the user B who is riding in the vehicle 100. For example, the occupant detection unit 161 detects the user B based on a camera image acquired by the interior camera 113.

[0040] (Vehicle State Detection Unit) The vehicle state detection unit 162 acquires vehicle speed information, position information, and other information as status information of the vehicle 100 from the vehicle sensor group 112 via the communication unit 166. This enables the vehicular communication device 160 to detect the state of the vehicle 100.

[0041] (Output Unit) When an audio signal is included in a communication signal acquired from an external device (e.g., server device 400) communicating with communication unit 166, output unit 163 causes an audio playback unit (speaker, etc.) constituting output device 114 to play back audio based on the audio signal output from output unit 163. When a video signal is included in a communication signal acquired from an external device (e.g., server device 400) via communication network 300, output unit 163 causes a display unit constituting output device 114 to display video based on the video signal output from output unit 163.

[0042] (Input Unit) The input unit 164 inputs the voice uttered by the user B. More specifically, the input unit 164 inputs a voice signal collected by a microphone (not shown).

[0043] (Control Unit) The control unit 165 (essentially, the MPU (Micro Processing Unit) of the IVI system 110) reads and executes an application for the vehicle communication device 160 stored in a memory unit (not shown) to perform various information processing, control processing, etc. required for the vehicle communication device 160.

[0044] (Communication Unit) The communication unit 166 includes a wireless communication module (not shown) that performs wireless communication with the communication network 300, and a wired communication module (not shown) that performs wired communication using a CAN or the like.

[0045] <Input Device> The input device 111 includes an operation detection unit and a microphone (not shown). (Operation Detection Unit) The operation detection unit is operated by user B, who is an occupant of the vehicle 100, and outputs an operation signal to the vehicle communication device 160. The operation detection unit may be provided on a display surface of a display device constituting the output device 114 (described later).

[0046] (Microphone) The microphone collects the voice uttered by the user B and outputs a voice signal to the vehicle communication device 160 .

[0047] <Vehicle Sensor Group> The vehicle sensor group 112 includes a vehicle speed sensor, a positioning sensor, a magnetic sensor (direction sensor), a brake operation sensor, a steering sensor, an accelerator operation sensor, an opening / closing body (door, rear gate, etc.) sensor, an energy (battery SOC or fuel) sensor, a turn signal operation sensor, a wiper operation sensor, an air conditioner operation sensor, an ignition switch sensor, etc. The vehicle sensor group may also include a drive recorder (not shown).

[0048] Information detected by each sensor of the vehicle sensor group 112 is output to the vehicle communication device 160. The vehicle communication device 160 may acquire driving history information of the vehicle 100 and information about roads traveled by the vehicle 100 based on the information detected by the vehicle sensor group 112. The vehicle communication device 160 may also acquire operation information of a navigation device as one of the functions of the IVI system 110 of the vehicle 100, operation information of other functions of the IVI system 110 (e.g., content playback information), and information about recorded video from a drive recorder.

[0049] <In-Vehicle Camera> The in-vehicle camera 113 captures an image of, for example, the upper body of a user B who is an occupant of the vehicle 100, and outputs data of the subject image to the vehicle communication device 160 as image information.

[0050] <Output Device> The output device 114 includes a display unit and an audio playback unit (not shown). The output device 114 is used, for example, to display video content or play content. (Display Unit) The display unit has a screen such as a liquid crystal display, and displays image information based on a display signal output from the vehicle communication device 160. The image information includes images of the video content, etc.

[0051] (Audio Reproducing Unit) The audio reproducing unit is configured as a speaker that reproduces and outputs audio and the like, and reproduces sound content and the like based on a reproduction signal output from the vehicle communication device 160 .

[0052] <Explanation of Flowchart> First, the flow of the status information presentation process by the status information presentation app will be explained, and then the flow of the driving information reproduction process by the driving information reproduction app will be explained. Figure 6 is a flowchart illustrating an example of the status information presentation process by an app executed by the calculation unit 205 of the computer 200. When the launch icon of the status information presentation app (not shown) is selected by the user A in the virtual space R3 as shown in Figure 1B (for example, by directing the gaze toward the icon and clicking with the finger), the calculation unit 205 executes the app that causes the computer 200 to function as a status information presentation device.

[0053] 6, the calculation unit 205 acquires information about the vehicle 100 via the communication network 300 using the communication unit 208, and proceeds to step S20. The calculation unit 205 acquires, for example, user information about the vehicle 100 or an occupant (user B) of the vehicle 100, and status information about the vehicle 100.

[0054] In step S20, the calculation unit 205 makes a determination on the user information based on the acquired information, and proceeds to step S30. The calculation unit 205 determines, for example, whether or not a combination of the ID information of the computer 200 and the above user information has been stored (may also be called registered) in advance in the storage unit 206.

[0055] In step S30, the calculation unit 205 determines the state of the vehicle 100, and the process proceeds to step S40. The calculation unit 205 determines the state of the vehicle 100, for example, based on the status information of the vehicle 100 acquired in step S10. More specifically, the calculation unit 205 determines the state of the vehicle 100 for each item, such as brake operation information and wiper operation information. Note that the calculation unit 205 can use at least one of the following as the status information of the vehicle 100: driving history information of the vehicle 100, driving road information, direction information, brake operation information, steering operation information, accelerator operation information, opening / closing device operation information, energy information, turn signal operation information, wiper operation information, air conditioner operation information, navigation device operation information, information from the IVI system 110, and information from the drive recorder.

[0056] In step S40, the calculation unit 205 visualizes the vehicle 100 as illustrated in Fig. 1B, and presents an icon 71 indicating the status information of the vehicle 100 in the peripheral vision on the display 207, and then the process proceeds to step S50. The icon 71 is initially presented in the manner illustrated in Fig. 3A (steady display).

[0057] In step S50, the calculation unit 205 quantifies the state of the vehicle 100, and the process proceeds to step S60. The calculation unit 205 quantifies the state of the vehicle 100 for each of the above items into, for example, five levels between the highest and lowest values.

[0058] In step S60, the calculation unit 205 determines whether the quantified numerical value is equal to or greater than a predetermined threshold. If the numerical value is equal to or greater than the threshold (or equal to or less than the threshold) (for example, equal to or greater than the fourth level when the worst value is level 5 and the best value is level 0, or equal to or less than the first level when the best value is level 5 and the worst value is level 0), the calculation unit 205 makes a positive judgment in step S60 and proceeds to step S70. If the numerical value is not equal to or greater than the threshold (or equal to or less than the threshold), the calculation unit 205 makes a negative judgment in step S60 and proceeds to step S100.

[0059] In step S70, the calculation unit 205 switches the mode of the icon 71 (steady display) to one of the modes (non-steady display) of the icons 71A to 71H, and proceeds to step S80. By switching the steady display icon 71 to the non-steady display icon 71A to 71H, it is possible to gently inform the user A that some kind of state change has occurred in the vehicle 100 in which the user B is riding.

[0060] In step S80, the calculation unit 205 determines whether the digitized numerical value is less than a predetermined threshold. If the numerical value is less than the threshold (for example, less than or equal to the third level when the worst value is level 5 and the best value is level 0, or more than or equal to the second level when the best value is level 5 and the worst value is level 0), the calculation unit 205 makes a positive judgment in step S80 and proceeds to step S90. If the numerical value is not less than the threshold, the calculation unit 205 makes a negative judgment in step S80 and proceeds to step S100.

[0061] In step S90, the calculation unit 205 returns the state of any one of the non-steady-state display icons 71A to 71H to the state of the steady-state display icon 71, and proceeds to step S100. Note that the processing of steps S50 to S90 is performed based on the respective thresholds for each item of the state of the vehicle 100. As a result, if there are multiple items that are above (or below) the threshold, the calculation unit 205 causes the display 207 to present the corresponding icons 71A to 71H in the peripheral vision in sequence at predetermined time intervals (for example, 2 seconds).

[0062] In step S100, the calculation unit 205 determines whether or not an operation to terminate the app has been performed by user A. If a predetermined termination operation has been performed, the calculation unit 205 makes a positive judgment in step S100 and proceeds to step S110, and if a termination operation has not been performed, the calculation unit 205 makes a negative judgment in step S100 and proceeds to step S120.

[0063] In step S110, the calculation unit 205 performs a termination process to terminate the process shown in FIG. 6. The termination process includes, for example, the termination of presentation of icons 71, 71A to 71H, and the termination of acquisition of information from vehicle 100 by communication unit 208. Furthermore, if a map is displayed in the driving information reproduction process (FIG. 7A) described below, the termination of the driving information reproduction process (map display) is also included. Note that wireless communication with communication network 300 by communication unit 208 does not need to be terminated if it is required for an app other than the status information presentation app.

[0064] In step S120, the calculation unit 205 reacquires the status information of the vehicle 100 via the communication network 300 using the communication unit 208, and returns to step S40. When returning to step S40, the calculation unit 205 repeats the above-described processing.

[0065] <Explanation of Traveling Information Reproduction> Computer 200 may process an app for traveling information reproduction in parallel with the processing of the app for presenting status information in Fig. 6. Fig. 7A is a flowchart illustrating an example of traveling information reproduction processing by an app executed by calculation unit 205 of computer 200. When user A selects a launch icon (e.g., an icon reminiscent of a map) of the traveling information reproduction app (not shown) in virtual space R3 as exemplified in Fig. 1B, calculation unit 205 executes the app that causes computer 200 to function as a traveling information reproduction device.

[0066] 7A, the calculation unit 205 acquires information about the vehicle 100 via the communication network 300 using the communication unit 208, and proceeds to step S220. The calculation unit 205 acquires at least direction information indicating the direction in which the vehicle 100 is traveling and location information indicating the current location of the vehicle 100. Note that the calculation unit 205 may use the direction information and location information included in the situation information about the vehicle 100 acquired in step S10 (FIG. 6).

[0067] In step S220, the calculation unit 205 acquires map information of a predetermined range (for example, 200 to 300 m square) including the current location of the vehicle 100 from the storage unit 206 or from an external server (not shown) via the communication unit 208, and proceeds to step S230. The predetermined range may be changed as appropriate.

[0068] In step S230, the calculation unit 205 displays an icon 71 such as that shown in Fig. 8 in the virtual space R3 on the display 207, and then the process proceeds to step S240. More specifically, the calculation unit 205 displays the icon 71 in a manner that indicates the direction in which the vehicle 100 is traveling, based on the direction information.

[0069] In step S240, the calculation unit 205 displays a map 80 of a predetermined range including the current location of the vehicle 100, as illustrated in FIG. 8, in the virtual space R3 on the display 207, and then proceeds to step S250. In the embodiment, a display of the map 80 with the north direction at the top is called a north-up display. FIG. 9A is a diagram illustrating the map 80 and icons 71 in the north-up display. In contrast, a display of the map 80 with the direction in which the vehicle 100 is traveling at the top is called a head-up display. FIG. 9B is a diagram illustrating the map 80 and icons 71 in the head-up display. Switching between the displays of FIG. 9A and FIG. 9B is possible by operating the display change button 81. Furthermore, a mode in which a map 80 of a streetscape (first viewpoint view) of the current location of the vehicle 100 in which user B is riding is displayed as if it were viewed from above is called a first viewpoint mode. Both FIG. 9A and FIG. 9B are display examples in the first viewpoint mode. In contrast, a mode in which a map 80 showing the current location of the vehicle 100 in which user B is riding, as seen from a bird's-eye view of the cityscape (second viewpoint view) from behind the vehicle 100, is called the second viewpoint mode. Fig. 10 shows an example of a display in the second viewpoint mode. The display of Fig. 9B (first viewpoint mode) and Fig. 10 (second viewpoint mode) can be switched by a viewpoint change operation in additional process 2, which will be described later. As an example, the calculation unit 205 sets the first viewpoint mode and north-up display as the default settings in step S240.

[0070] When the map 80 is displayed in a north-up orientation, the calculation unit 205 indicates the direction in which the vehicle 100 is traveling by the state of the icon 71. That is, the calculation unit 205 changes the state of the icon 71 to follow changes in the traveling direction of the vehicle 100. When the traveling direction is indicated by text or the like, the icon 71 includes text or the like indicating the traveling direction. When the traveling direction is indicated by the orientation of the icon 71, the calculation unit 205 changes the orientation of the displayed icon 71 according to the traveling direction of the vehicle 100. On the other hand, when the map 80 is displayed in a head-up orientation, the calculation unit 205 fixes the state of the icon 71 to a state showing a rear view of the vehicle 100 (the traveling direction is facing backward). During the head-up display, the calculation unit 205 changes the orientation of the upper side of the map 80 to follow changes in the traveling direction of the vehicle 100 (in other words, the displayed map 80 is rotated around an axis indicating the vertical direction of the map 80). When the map 80 is displayed in a head-up view, the state of the icon 71 does not change even if the traveling direction of the vehicle 100 changes, and therefore the setting of the head-up display may be referred to as a setting that fixes the state of the icon 71 (for example, the orientation of the icon 71). Conversely, when the map 80 is displayed in a north-up view, the state of the icon 71 changes depending on the traveling direction of the vehicle 100, and therefore the setting of the north-up display may be referred to as a setting that does not fix the state of the icon 71.

[0071] In step S250, the calculation unit 205 determines whether or not an operation to terminate the travel information reproduction app has been performed by the user A. If a predetermined termination operation (for example, an operation on the end button) has been performed, the calculation unit 205 makes a positive judgment in step S250 and proceeds to step S260, but if an termination operation has not been performed, the calculation unit 205 makes a negative judgment in step S250 and proceeds to step S270.

[0072] In step S260, calculation unit 205 performs a predetermined termination process to terminate the process shown in Fig. 7A. The termination process includes terminating the display of map 80 and icon 71. Note that wireless communication with communication network 300 by communication unit 208 does not need to be terminated if it is required for an app other than the app for reproducing driving information.

[0073] In step S270, the calculation unit 205 causes the communication unit 208 to reacquire information about the vehicle 100 via the communication network 300, and the process proceeds to step S280.

[0074] In step S280, the calculation unit 205 reacquires map information of a predetermined range including the current location of the vehicle 100 from the storage unit 206 or from an external server (not shown) via the communication unit 208, and proceeds to step S290.

[0075] In step S290, the calculation unit 205 determines whether or not the map 80 is currently being displayed in a north-up orientation. If the map 80 is currently being displayed in a north-up orientation (FIGS. 8 and 9A), the calculation unit 205 makes an affirmative decision in step S290 and returns to step S230. After returning to step S230, the calculation unit 205 repeats the above-described processing. If the map 80 is not currently being displayed in a north-up orientation (FIGS. 9B and 10), the calculation unit 205 makes a negative decision in step S290 and proceeds to step S320 of additional processing 1 (FIG. 7B), which will be described later.

[0076] When user A performs a free operation with his / her fingers on map 80 floating in virtual space R3, calculation unit 205 moves map 80 within virtual space R3 or changes its size (enlarges or reduces it). Specifically, calculation unit 205 rotates map 80 in accordance with the movement of user A's fingers, moves map 80 within virtual space R3, or changes its display size (enlarges or reduces it).

[0077] <Explanation of Additional Process 1> Processing according to the flowchart illustrated in Fig. 7B may be added between step S240 and step S250 in Fig. 7A described above. In step S310 in Fig. 7B, the calculation unit 205 determines whether or not a head-up operation has been detected. When the calculation unit 205 detects an operation on the display change button 81 in the virtual space R3 illustrated in Fig. 8, the calculation unit 205 makes a positive judgment in step S310 and proceeds to step S320. When the calculation unit 205 does not detect an operation on the display change button 81, the calculation unit 205 makes a negative judgment in step S310, terminates the processing according to Fig. 7B, and proceeds to step S250 in Fig. 7A.

[0078] In step S320, the calculation unit 205 fixes the icon 71 displayed in virtual space R3 on the display 207 facing backward (e.g., as shown in FIG. 9B ), and proceeds to step S330. In step S330, the calculation unit 205 switches the map 80 displayed in virtual space R3 on the display 207 to head-up display (e.g., as shown in FIG. 9B ), and proceeds to step S340. As described above, during head-up display, the map 80, which displays the direction in which the vehicle 100 is traveling on the upper side, is rotated around an axis indicating the vertical direction of the map 80.

[0079] In step S340, the calculation unit 205 determines whether or not a north-up operation has been detected. If the calculation unit 205 detects an operation by user A on the display change button 81 in virtual space R3, the calculation unit 205 makes an affirmative judgment in step S340 and proceeds to step S350. If the calculation unit 205 does not detect an operation on the display change button 81, the calculation unit 205 makes a negative judgment in step S340, terminates the processing in FIG. 7B, and proceeds to step S250 in FIG. 7A.

[0080] In step S350, the calculation unit 205 does not fix the orientation of the icon 71 displayed in virtual space R3 on the display 207 (e.g., as shown in FIG. 9A ), and proceeds to step S360. In step S360, the calculation unit 205 switches the map 80 displayed in virtual space R3 on the display 207 to a north-up display (e.g., as shown in FIG. 9A ), ends the processing shown in FIG. 7B , and proceeds to step S250 in FIG. 7A . As described above, during the north-up display, the appearance of the icon 71 is changed to indicate the direction in which the vehicle 100 is traveling.

[0081] <Explanation of Additional Process 2> Processing according to the flowchart illustrated in Fig. 7C may be added between step S240 and step S250 in Fig. 7A described above. In step S410 in Fig. 7C, the calculation unit 205 determines whether or not a viewpoint change operation has been detected. When the calculation unit 205 detects a viewpoint change operation (e.g., movement of user A's head, or an operation by user A using his / her line of sight and fingers) in virtual space R3 as illustrated in Fig. 8, the calculation unit 205 makes a positive judgment in step S410 and proceeds to step S420. When the calculation unit 205 does not detect a viewpoint change operation, the calculation unit 205 makes a negative judgment in step S410, terminates the processing according to Fig. 7C, and proceeds to step S250 in Fig. 7A.

[0082] Here, the viewpoint change operation will be described in detail. (Example of an operation performed by user A moving his / her head) The calculation unit 205 switches between the first viewpoint mode and the second viewpoint mode using a detection signal from the IMU that constitutes the sensor group 204 of the computer 200. Specifically, based on the detection signal from the IMU, the calculation unit 205 detects a change in the position of user A's head relative to the map 80 displayed in virtual space R3 on the display 207 (in other words, the map 80 perceived by user A in virtual space R3), and switches between the first viewpoint mode and the second viewpoint mode in response to the change in head position. For example, if the head position of user A looking at the map 80 in virtual space R3 is fixed in a position substantially directly facing the map 80 (for example, the posture is maintained for approximately two seconds), this is detected as a viewpoint change operation to the first viewpoint mode (for example, Figures 9A and 9B). In contrast, when user A, viewing map 80 in virtual space R3, fixes the head position at an angle to map 80 (similarly, maintaining this position for approximately two seconds), this is detected as a viewpoint change operation to the second viewpoint mode (e.g., FIG. 10 ). At this time, the altitude of the bird's-eye view display (in other words, the height of the viewpoint in the second viewpoint mode) may be changed depending on the relative angle of user A's line of sight with respect to map 80 as perceived by user A. By fixing the position of user A's head relative to map 80, user A can display map 80 viewed from different viewpoints in virtual space R3.

[0083] (Example of an Operation Performed by User A Using Gaze and Fingers) The calculation unit 205 switches between the first viewpoint mode and the second viewpoint mode using detection signals from the eye tracking unit and the hand tracking unit that constitute the sensor group 204 of the computer 200. Specifically, based on the detection signals from the eye tracking unit and the hand tracking unit, the calculation unit 205 detects the movement of the user A's fingers relative to the map 80 perceived by the user A in the virtual space R3, and switches between the first viewpoint mode and the second viewpoint mode based on the movement of the fingers relative to the map 80. For example, when user A focuses (directs his / her gaze) on a building or the like in the map 80 and moves his / her fingers, the calculation unit 205 detects this as a viewpoint change operation. The calculation unit 205 gradually switches the display of the map 80 between the first viewpoint mode and the second viewpoint mode in response to the movement of the fingers in space. By moving the position of the user A's fingers in space, the map 80 can be displayed as viewed from a viewpoint moving through the virtual space R3.

[0084] Returning to the description of the flowchart, in step S420, the calculation unit 205 determines whether or not an overhead map (in other words, the first viewpoint mode) is being displayed as the map 80. If the first viewpoint mode (e.g., FIG. 9B ) is being displayed, the calculation unit 205 makes a positive determination in step S420 and proceeds to step S430, whereas if the second viewpoint mode (e.g., FIG. 10 ) is being displayed, the calculation unit 205 makes a negative determination in step S420 and proceeds to step S440.

[0085] In step S430, the calculation unit 205 switches the map 80 displayed in the virtual space R3 on the display 207 to a bird's-eye view (in other words, the second viewpoint mode), terminates the processing of FIG. 7C, and proceeds to step S250 of FIG. 7A.

[0086] In step S440, the calculation unit 205 switches the map 80 displayed in the virtual space R3 on the display 207 to a bird's-eye view display (in other words, the first viewpoint mode), terminates the processing of FIG. 7C, and proceeds to step S250 in FIG. 7A.

[0087] The display in the second viewpoint mode illustrated in FIG. 10 includes a weather display button 82. When the calculation unit 205 detects user A's operation on the weather display button 82 in the virtual space R3, the calculation unit 205 may control the display 207 to display a weather image showing the weather (which may also be referred to as "weather") of the area currently displayed as the map 80 vertically above the map 80. Weather information for the area indicated by the vehicle 100's location information may be acquired, for example, from an external server (not shown) via the communication unit 208. Based on the acquired weather information, the calculation unit 205 generates a weather image including at least one of the sun, moon, clouds, rain, and snow, and displays the weather image vertically above the map 80 in the virtual space R3. FIGS. 11A and 11B are diagrams illustrating an example of the map 80 displayed together with several weather images. User A can view the map 80 as driving information reproduced based on information acquired by the vehicle 100 driven by user B, including the weather information. When the calculation unit 205 detects an operation by the user A on the weather display end button 83 in the virtual space R3, it ends the display of the weather image and returns to the display shown in FIG.

[0088] One or more of the additional processing 1 and additional processing 2 described above may be arbitrarily combined with the processing shown in FIG. 7A.

[0089] The above-described embodiment provides the following advantageous effects. (1-1) A driving information reproduction program that causes head-mounted display computer 200, including communication unit 208 as a first communication unit, storage unit 206, and display 207, to display driving information of vehicle 100, including communication unit 166 as a second communication unit, causes computer 200 to execute the following processes: acquiring vehicle information, including position information and orientation information of vehicle 100, via communication unit 208 (S210); acquiring map information from storage unit 206 or via communication unit 208 (S220); displaying icon 71 that evokes vehicle 100 on display 207 in an orientation based on the orientation information (S230); and displaying map 80 of a predetermined range including the position of vehicle 100, based on the position information and map information, in the same direction as icon 71 and at the back in the depth direction, in virtual space R3 perceived by user A wearing computer 200 (S240). With this configuration, user A can view user B's driving of vehicle 100 while wearing display 207 of head-mounted display computer 200. For example, user A can passively view map 80 based on information acquired by vehicle 100 and the driving direction of vehicle 100, as if viewing the scenery of vehicle 100 driving.

[0090] (1-2) The traveling information reproduction program further causes the computer 200 to execute a process (S310) of detecting an operation (head-up operation) from the user A to fix the orientation of the icon 71 facing backward in the depth direction of the virtual space R3, and a process (S330) of changing the display orientation of the map 80 in the virtual space R3 in accordance with a change in the orientation information when the orientation of the icon 71 is fixed facing backward (S320). With this configuration, the user A can use the head-up display according to his / her preference when viewing the map 80 and the traveling orientation of the vehicle 100.

[0091] (1-3) The driving information reproduction program further causes computer 200 to execute the following processes via communication unit 208: acquiring weather information for the area indicated by the location information; generating a weather image including at least one of the sun, moon, clouds, rain, and snow based on the weather information; and displaying the weather image vertically above the map in virtual space (FIGS. 11A and 11B). With this configuration, user A can view map 80, including the weather information, based on information acquired by vehicle 100 driven by user B.

[0092] (2-1) A driving information reproduction program that causes a head-mounted display type computer 200 including a communication unit 208 as a first communication unit, a storage unit 206, and a display 207 to display driving information of the vehicle 100 including a communication unit 166 as a second communication unit includes a process of acquiring vehicle information including position information and direction information of the vehicle 100 via the communication unit 208 (S210), a process of acquiring map information from the storage unit 206 or via the communication unit 208 (S220), and a process of displaying a map 80 of a predetermined range including the position of the vehicle 100 on the display 207 based on the position information and the map information. The computer 200 executes the following display control processes: a process for displaying an icon 71 indicating the orientation of the vehicle 100 based on the azimuth information on the display 207 in accordance with the map 80 (S230); a process for detecting a first operation for instructing the first viewpoint mode and a second operation for instructing the second viewpoint mode from the user A wearing the computer 200 (S410); and a display control process for switching the map 80 to a bird's-eye view mode when the first operation is detected (S440), and switching the map 80 to a viewable mode from behind the vehicle 100 (bird's-eye view display) when the second operation is detected (S430). With this configuration, the user A can freely view the driving of the vehicle 100 by the user B from his / her own viewpoint while wearing the display 207 of the head-mounted display type computer 200. For example, the map 80 based on information acquired by the vehicle 100 can be viewed from a viewpoint that suits the user A's preference.

[0093] (2-2) The process (S240) of displaying the map 80 on the display 207 displays the map 80 at a predetermined position in the virtual space R3 perceived by the user A, the process (S410) of detecting the first operation and the second operation is performed based on a change in the position of the user A's head relative to the map 80 in the virtual space R3 detected by the sensor group 204 mounted on the computer 200, and the display control process (S430, S440) switches to the map 80 in a mode that can be viewed from a fixed viewpoint in the virtual space R3 in response to the change in the head position. With this configuration, the user A wearing the display 207 of the head-mounted display type computer 200 can view the map 80 presented in views from different viewpoints in the virtual space R3 by changing the position of his or her head relative to the map 80 perceived by the user A in the virtual space R3.

[0094] (2-3) The process (S240) of displaying the map 80 on the display 207 displays the map 80 at a predetermined position in the virtual space R3 perceived by the user A, the process (S410) of detecting the first operation and the second operation is performed based on the movement of the user A's finger detected by the sensor group 204 mounted on the computer 200, and the display control process (S430, S440) switches to the map 80 in a mode that can be viewed from a viewpoint that moves in the virtual space R3 in response to the finger movement. With this configuration, the user A wearing the display 207 of the head-mounted display type computer 200 can view the map 80 displayed in views from different viewpoints in the virtual space R3 by looking at the map 80 perceived by the user A in the virtual space R3 and moving the position of his or her fingers.

[0095] The above-described embodiment can be modified in various ways. Modifications are described below. (Modification 1) In the above-described embodiment, an example has been described in which the vehicle 100 accesses the communication network 300 using the communication unit 166 of the vehicle communication device 160 constituting the IVI system 110. Alternatively, the communication network 300 may be accessed by a carry-on terminal such as a smartphone used by the user B in the vehicle 100. In this case, among the conditions of the vehicle 100 detected by the vehicle sensor group 112, information that can be obtained directly by sensors provided on the smartphone or indirectly based on information obtained by the smartphone, such as driving history information, road information, and direction information, may be substituted for the information detected by the vehicle sensor group 112. Furthermore, when accessing the communication network 300 using a carry-on terminal, a unique ID specific to the carry-on terminal may be used instead of or in addition to the vehicle ID.

[0096] (Variation 2) In the embodiment described above, an example was given of computer 200 of the type that displays on a display worksheets 72, 73, and 74 of an office work application running on computer 200, with the scenery ahead of user A superimposed on a stereo image captured by a built-in camera of computer 200. Alternatively, a computer may be used that displays worksheets 72, 73, and 74 of the office work application on a holographic display, and uses a prism or a half mirror to synthesize the image from the holographic display with a real image of the scenery ahead of user A. In this case, icons 71 and the like indicating the status of vehicle 100 are displayed on the holographic display.

[0097] (Variation 3) In the above-described embodiment, an integrated computer 200 has been described as an example in which the calculation unit 205 and the display 207 are integrally formed together with other components. Alternatively, a separate computer in which the main calculation unit 205 and the display 207 are formed separately may be used to execute the program according to the embodiment. The display 207 worn by the user A on his / her head and the calculation unit 205 that controls the display of this display 207 are connected, for example, by short-range wireless communication.

[0098] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications, as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other.

[0099] 71, 71A to 71H icons, 72 to 74 worksheets, 80 map, 100 vehicle, 110 IVI system, 111 input device, 112 vehicle sensor group, 113 in-vehicle camera, 114 output device, 160 vehicle communication device, 200 computer, 201 camera group, 202 microphone group, 203 speaker, 204 sensor group, 205 calculation unit, 206 memory unit, 207 display, 208 communication unit, 300 communication network, 400 server device, 500 information provision system, A, B user, R1, R2 area, R3 virtual space

Claims

1. A driving information reproduction program that causes a head-mounted display type computer including a first communication unit, a memory unit, and a display to display driving information of a vehicle including a second communication unit, the driving information reproduction program causing the computer to execute the following processes: a process of acquiring vehicle information including position information and orientation information of the vehicle via the first communication unit; a process of acquiring map information from the memory unit or via the first communication unit; a process of displaying a map of a predetermined range including the position of the vehicle on the display based on the position information and the map information; a process of displaying an icon image of the vehicle oriented based on the orientation information on the display in accordance with the map; a process of detecting a first operation to instruct a first viewpoint mode and a second operation to instruct a second viewpoint mode from a user wearing the computer; and a display control process of switching the map to a bird's-eye view when the first operation is detected, and switching the map to a view visible from behind the vehicle when the second operation is detected.

2. A driving information reproduction program as described in claim 1, characterized in that the process of displaying the map on the display displays the map at a predetermined position in a virtual space perceived by the user, the process of detecting the first operation and the second operation is performed based on a change in the position of the user's head relative to the map in the virtual space detected by a sensor mounted on the computer, and the display control process switches to the map in a form that can be viewed from a fixed viewpoint in the virtual space in response to the change in the position of the head.

3. A driving information reproduction program as described in claim 1, characterized in that the process of displaying the map on the display displays the map at a predetermined position in a virtual space perceived by the user, the process of detecting the first operation and the second operation is performed based on finger movements of the user detected by a sensor mounted on the computer, and the display control process switches the map to a form that can be viewed from a viewpoint moving in the virtual space in response to the finger movements.

4. A driving information reproduction method for displaying driving information of a vehicle including a second communication unit on a head-mounted display type computer including a first communication unit, a memory unit, and a display, comprising the steps of: acquiring vehicle information including position information and orientation information of the vehicle via the first communication unit; acquiring map information from the memory unit or via the first communication unit; displaying a map of a predetermined range including the position of the vehicle on the display based on the position information and the map information; displaying an icon image on the display indicating the vehicle oriented based on the orientation information in accordance with the map; detecting a first operation to instruct a first viewpoint mode and a second operation to instruct a second viewpoint mode from a user wearing the computer; and a display control step of switching the map to a bird's-eye view when the first operation is detected, and switching the map to a view visible from behind the vehicle when the second operation is detected.

Images