Virtual reference display device, virtual reference display method, and program

Abstract

To realize a virtual reference display device, a virtual reference display method and a program which display a virtual reference for appropriately supporting stable travel.SOLUTION: This virtual reference display device displays at least two virtual reference objects that are virtual references for appropriately supporting stable travel in which coordinate positions of a three-dimensional space are different. Since the virtual objects are not actually present, the virtual reference display device can set positions of the virtual images in arbitrary places. The virtual reference display device can arrange the virtual objects in a place having a high effect of supporting stable travel. Therefore, the virtual reference display device can easily arrange the virtual objects in a place having a high effect of supporting stable travel, for example, according to a situation of a road. As a result, the virtual reference display device can achieve effective support of stable travel in any situation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for displaying a virtual reference to assist stable driving. [Background technology]

[0002] In recent years, various systems have been developed that provide driving assistance information to occupants of moving objects such as vehicles. For example, Patent Document 1 discloses a technology in which a head-up display (HUD) is installed in a vehicle (e.g., an automobile) and driving information (e.g., an arrow indicating the vehicle's traveling direction) is displayed as a virtual image in front of the vehicle's windshield as seen by the occupant of the vehicle, superimposed on the foreground in the forward field of view. With the technology of Patent Document 1, driving assistance is realized by having the occupant (driver) of the vehicle (e.g., an automobile) check the driving information (e.g., an arrow indicating the vehicle's traveling direction) displayed on the HUD.

[0003] Among driving assistance technologies, there is a demand for technology that supports stable driving. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-64760 A Summary of the Invention [Problem to be solved by the invention]

[0005] The above-mentioned technology (the technology disclosed in Patent Document 1) makes it easy for a vehicle (e.g., automobile) occupant (driver) to confirm the direction of travel by checking driving information (e.g., an arrow indicating the vehicle's direction of travel) displayed on the HUD, but it may be difficult to appropriately support stable driving. This is because it is difficult to identify factors that realize stable driving, and whether stable driving is realized or not depends largely on the driving skill of the vehicle's (e.g., automobile) occupant (driver). In other words, when moving (driving a vehicle), a lack of visual targets or insufficient attention can prevent the occupant (driver) from perceiving clues to grasp the space, resulting in a failure of movement control (the vehicle's travel wobbles), and this factor is greatly influenced by the occupant's (driver's) driving skill.

[0006] There is a need for technology that can solve these problems and appropriately support stable driving, but no technology has been established that can appropriately support stable driving.

[0007] In view of the above, an object of the present invention is to provide a virtual reference display device, a virtual reference display method, and a program that display a virtual reference for appropriately supporting stable driving. [Means for solving the problem]

[0008] In order to solve the above problem, a first invention is a virtual reference display device for displaying a group of virtual reference objects ahead in the direction of travel when a user moves, comprising a travel path shape acquisition unit, a relative position acquisition unit, an attitude information acquisition unit, a virtual object position calculation unit, a display parameter calculation unit, and a display unit.

[0009] The travel route shape acquisition unit acquires travel route shape information, which is information relating to the shape of the travel route along which the user is traveling.

[0010] The relative position acquisition unit acquires relative position information, which is information relating to the relative position between the user's position and the travel route.

[0011] The posture information acquisition unit acquires user posture information, which is information relating to the posture of the user.

[0012] The virtual object position calculation unit sets a three-dimensional coordinate space based on the position of the user, and calculates the positions of the virtual reference objects in the three-dimensional space.

[0013] The display parameter calculation unit calculates, based on the user's position and the position of the virtual object, display parameters for displaying the virtual reference object group at the position of the virtual reference object group in three-dimensional space when viewed from the user's viewpoint.

[0014] The display unit displays the virtual reference objects based on the display parameters.

[0015] The group of virtual reference objects includes at least two virtual reference objects that are substantially parallel to the bottom surface of the travel path along which the user is traveling and that are located ahead in the travel direction and have different coordinate positions in three-dimensional space.

[0016] This makes it possible for this virtual reference display device to display virtual references (at least two virtual reference objects having different coordinate positions in three-dimensional space) for appropriately supporting stable driving.

[0017] Furthermore, in this virtual reference display device, since the virtual object does not actually exist, its position (the position of its virtual image) can be set to any location. Therefore, in this virtual reference display device, the virtual object can be placed in a location where it is most effective in supporting stable driving. Therefore, in this virtual reference display device, the virtual object can be easily placed in the optimal location (a location where it is most effective in supporting stable driving) depending on, for example, the road conditions (road width, road curvature, etc.). As a result, in this virtual reference display device, effective support for stable driving can be achieved regardless of the situation.

[0018] The second invention is the first invention, further comprising a stable driving path prediction unit that predicts the direction in which the user should move to achieve stable driving based on the travel path shape information and the user's relative position information, and obtains a stable driving vector that defines the predicted direction.

[0019] The virtual reference object group includes two virtual reference objects, which are arranged at positions symmetrical with respect to a line extending from the stable driving vector.

[0020] As a result, this virtual reference display device can provide stable driving support to the user by using two virtual objects placed in line-symmetric positions with respect to a straight line extending from the stable driving vector.

[0021] A third aspect of the present invention is the second aspect of the present invention, further comprising a stable running degree acquisition unit and an optimal virtual reference position detection unit.

[0022] The stable driving level acquisition unit acquires a stable driving level indicating the degree of stable driving by comparing the stable driving vector with the route actually traveled by the user.

[0023] The optimum virtual reference position detection unit detects a virtual reference object position that is likely to realize stable driving.

[0024] The two virtual reference objects are placed at one pattern position among N (N: natural number, N≧2) pattern positions that are line-symmetric with respect to a straight line extending from the stable running vector.

[0025] The virtual object position calculation unit calculates the positions in three-dimensional space of the two virtual reference objects in a state where one selected from the N pattern positions is placed at the pattern position, as the positions of the virtual reference object group.

[0026] The display unit displays the virtual reference object group so that the virtual reference object group is displayed at the positions calculated by the virtual object position calculation unit.

[0027] The stable running degree acquisition unit acquires the stable running degree when a pattern position selected from the N pattern positions is displayed on the display unit, in association with the selected pattern position.

[0028] The optimum virtual reference position detection unit detects, as the optimum virtual reference object position, the pattern position when the running state closest to the stable running state is realized, based on the stable running degree associated with the pattern position.

[0029] As a result, the virtual reference display device displays virtual objects on the display unit based on an arrangement pattern that is likely to achieve more stable driving, thereby enabling the virtual reference display device to provide more appropriate support for stable driving.

[0030] A fourth aspect of the present invention is the vehicle of any one of the first to third aspects of the present invention, further comprising a vehicle that moves when a user rides on it and operates it.

[0031] The relative position acquisition unit acquires information relating to the relative position between the position of the moving body and the traveling route as relative position information.

[0032] As a result, this virtual reference display device can realize appropriate stable driving assistance even when using a mobile body that moves by being operated by a user while riding on it.

[0033] The fifth invention is the fourth invention, wherein the posture information acquisition unit acquires user posture information based on at least one of information on the user's head position when the user is riding on the moving body and operating the moving body, and information on the user's posture state.

[0034] As a result, this virtual reference display device can detect the position of the user's viewpoint and perform display processing of virtual objects based on the detected viewpoint position, allowing for more accurate display of virtual objects and, as a result, providing more appropriate assistance for stable driving.

[0035] The sixth invention is the fourth or fifth invention, wherein the display unit includes a plurality of light-emitting elements arranged in a row, is attached to the moving body so as to be positioned in front of the user when the user is riding on the moving body and operating it, and causes the light-emitting elements arranged at positions corresponding to the group of virtual reference objects to emit light based on the display parameters.

[0036] As a result, this virtual reference display device can realize assistance for stable driving using a display unit (for example, an LED array) including a plurality of light-emitting elements arranged in a line.

[0037] A seventh invention is a virtual reference display method for displaying a group of virtual reference objects ahead in the direction of travel when a user moves, comprising a travel path shape acquisition step, a relative position acquisition step, an attitude information acquisition step, a virtual object position calculation step, a display parameter calculation step, and a display step.

[0038] The travel route shape acquisition step acquires travel route shape information, which is information relating to the shape of the travel route along which the user is traveling.

[0039] The relative position acquisition step acquires relative position information, which is information relating to the relative position between the user's position and the travel route.

[0040] The posture information acquisition step acquires user posture information that is information relating to the posture of the user.

[0041] The virtual object position calculation step sets a three-dimensional coordinate space based on the position of the user, and calculates the positions of the virtual reference objects in the three-dimensional space.

[0042] The display parameter calculation step calculates, based on the user's position and the virtual object's position, display parameters for displaying the virtual reference object group at the position of the virtual reference object group in three-dimensional space when viewed from the user's viewpoint.

[0043] The display step displays the virtual reference objects based on the display parameters.

[0044] The group of virtual reference objects includes at least two virtual reference objects that are substantially parallel to the bottom surface of the travel path along which the user is traveling and that are located ahead in the travel direction and have different coordinate positions in three-dimensional space.

[0045] This makes it possible to realize a virtual reference display method that has the same effects as the first aspect of the invention.

[0046] An eighth aspect of the present invention is a program for causing a computer to execute the virtual reference display method of the seventh aspect of the present invention.

[0047] This makes it possible to realize a program for causing a computer to execute a virtual reference display method that has the same effects as the first aspect of the invention. [Effects of the Invention]

[0048] According to the present invention, it is possible to realize a virtual reference display device, a virtual reference display method, and a program for displaying a virtual reference for appropriately supporting stable driving. [Brief explanation of the drawings]

[0049] [Figure 1] FIG. 1 is a schematic configuration diagram of a virtual reference display device 100 according to a first embodiment. [Figure 2] 1 is a diagram schematically illustrating a schematic configuration of a virtual reference display system 1000 when the virtual reference display device 100 according to the first embodiment is mounted on an automobile (an example of a moving body). [Figure 3] 10 is a flowchart of a process executed by the virtual reference display system 1000. [Figure 4] FIG. 10 is a diagram showing the positional relationship between a car Vhcl1 and two virtual objects Mkr1L and Mkr1R. [Figure 5] FIG. 10 is a diagram showing an example of the foreground seen by the driver Drv1 when two virtual objects Mkr1L and Mkr1R are superimposed and displayed on the display screen of the display unit 2 by the above processing. [Figure 6] FIG. 10 is a schematic configuration diagram of a virtual reference display system 2000 according to a second embodiment. [Figure 7] FIG. 10 is a schematic configuration diagram of a virtual reference display device 200 according to a second embodiment. [Figure 8] 10 is a flowchart of a process executed by a virtual reference display system 2000 according to a second embodiment. [Figure 9] 10 is a flowchart of a process executed by a virtual reference display system 2000 according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a placement pattern of virtual objects when N=2. [Figure 11] FIG. 10 is a diagram for explaining a stable driving route prediction process. [Figure 12] FIG. 10 is a diagram showing the arrangement pattern of virtual objects when an automobile Vhcl1 is traveling in the center lane of a five-lane expressway. [Figure 13] 1 is a diagram showing the foreground seen by a driver Drv1 (the foreground seen from the front window) and an LED array LED_arry as a display unit 2. FIG. [Figure 14] A diagram showing the CPU bus configuration. DETAILED DESCRIPTION OF THE INVENTION

[0050] [First embodiment] The first embodiment will be described below with reference to the drawings.

[0051] In the following, an example will be described in which the virtual reference display system and the virtual reference display device are mounted on an automobile.

[0052] <1.1: Virtual Reference Display System Configuration> FIG. 1 is a schematic diagram of a virtual reference display device 100 according to the first embodiment.

[0053] Figure 2 is a diagram (a side view of the vehicle Vhcl1 in the direction of travel when the driver Drv1 is driving the vehicle Vhcl1) showing a schematic configuration of the virtual reference display system 1000 when the virtual reference display device 100 of the first embodiment is mounted on a vehicle (an example of a moving body).

[0054] As shown in FIG. 1, the virtual reference display device 100 includes a display control unit 1 and a display unit 2.

[0055] As shown in FIG. 2, the virtual reference display system 1000 includes a GPS receiving unit Rx1, a front camera F_cam, a sensor Sen1, a map data storage unit Dev1, an attitude detection unit Dev2, and a virtual reference display device 100 (display control unit 1, display unit 2).

[0056] The GPS receiver Rx1 receives GPS signals transmitted from GPS satellites and extracts predetermined parameters from the received GPS signals. The GPS receiver Rx1 then performs calculations using the extracted parameters to calculate the current position of the vehicle Vhcl1. The GPS receiver Rx1 then generates data D_GPS including information about the calculated current position of the vehicle Vhcl1 and outputs the generated data D_GPS to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0057] The front camera F_cam is installed, for example, at the front of the vehicle Vhcl1. The front camera F_cam is an imaging device configured by a camera using a solid-state imaging element such as a CMOS image sensor or a CCD image sensor, and captures the scenery ahead in the traveling direction of the vehicle. The front camera F_cam outputs the captured image data as data D_F_cam to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0058] The sensor Sen1 is, for example, a sensor that can measure physical quantities related to the situation outside the vehicle Vhcl1, and is installed at a position where it can measure the physical quantities of the measurement target. The sensor Sen1 is, for example, a sensor that detects obstacles outside the vehicle Vhcl1, a distance sensor that measures the distance to an obstacle outside the vehicle Vhcl1, or a sensor that detects the condition of the road on which the vehicle Vhcl1 is traveling (for example, a sensor that detects white lines on the road). Note that the sensor Sen1 may be composed of multiple sensors. The sensor Sen1 outputs data including the acquired physical quantities (measurement result data by the sensor) as data D_sensor to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0059] The map data storage unit Dev1 is a storage unit for storing map data (e.g., map road data). The map data is stored in the map data storage unit Dev1, for example, by downloading it from a map data distribution center or by obtaining it from a recording medium such as a DVD or a memory card. In accordance with a request (request command) from the display control unit 1, the map data storage unit Dev1 reads out predetermined map data specified by the request. Then, the map data storage unit Dev1 outputs the read map data as data D_map to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0060] The posture detection unit Dev2 is a device that detects the posture of the driver Drv1, and is, for example, a camera (image capture device) installed on the dashboard, steering wheel, or the like of the vehicle Vhcl1. When the posture detection unit Dev2 is a camera (image capture device), the camera is installed in a position that allows it to capture an image of the posture of the driver Drv1 while driving the vehicle Vhcl1, and captures an image of the driver Drv1 so that the posture of the driver Drv1 can be detected. Then, the posture detection unit Dev2 outputs the captured data (captured image data) as data D_pose to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0061] The display control unit 1 of the virtual reference display device 100 includes an input interface unit 11, a travel path shape acquisition unit 12, a relative position acquisition unit 13, an attitude information acquisition unit 14, a virtual object position calculation unit 15, and a display parameter calculation unit 16.

[0062] The input interface unit 11 is an interface that is connected to one or more functional units mounted on the automobile Vhcl1 so as to be able to communicate data with the functional units, and is capable of performing predetermined data communication with the functional units. The input interface unit 11 receives data D_GPS output from the GPS receiver Rx1, data D_F_cam output from the front camera F_cam, data D_sensor output from the sensor Sen1, data D_map output from the map data storage unit Dev1, and data D_pose output from the attitude detection unit Dev2.

[0063] The input interface unit 11 extracts, from the input data, data necessary for the traveling path shape acquisition unit 12 to acquire the traveling path shape, and outputs the extracted data as data D1a to the traveling path shape acquisition unit 12. The input interface unit 11 also extracts, from the input data, data necessary for the relative position acquisition unit 13 to acquire the relative position, and outputs the extracted data as data D1b to the relative position acquisition unit 13. The input interface unit 11 also extracts, from the input data, data necessary for the attitude information acquisition unit 14 to acquire attitude information, and outputs the extracted data as data D1c to the attitude information acquisition unit 14.

[0064] The travel route shape acquisition unit 12 receives data D1a output from the input interface unit 11. Based on the data D1a, the travel route shape acquisition unit 12 acquires the shape of the road along which the vehicle Vhcl1 is currently traveling. For example, the data D1a includes (1) information about the current position of the vehicle Vhcl1 acquired from data D_GPS, (2) map information about the road along which the vehicle Vhcl1 is currently traveling acquired from data D_map, (3) the shape of the road along which the vehicle Vhcl1 is currently traveling (e.g., the road surface condition, etc.) acquired from data D_sensor, and (4) a captured image (captured video) of the scenery ahead of the vehicle Vhcl1 in the direction of travel of the road along which the vehicle Vhcl1 is currently traveling acquired from data D_F_cam. Based on this information, the travel route shape acquisition unit 12 acquires the shape of the road along which the vehicle Vhcl1 is currently traveling. The travel route shape acquisition unit 12 then outputs data including the acquired information about the travel route shape as data D2a to the virtual object position calculation unit 15.

[0065] The relative position acquisition unit 13 receives data D1b output from the input interface unit 11. Based on the data D1b, the relative position acquisition unit 13 acquires the relative position of the vehicle Vhcl1 (for example, a relative position for identifying where the vehicle Vhcl1 is located on the road on which the vehicle Vhcl1 is traveling (for example, a predetermined position on the road on which the vehicle Vhcl1 is traveling (for example, a relative position from the center position in the width direction))).

[0066] For example, data D1b includes (1) information about the current position of automobile Vhcl1 obtained from data D_GPS, (2) map information of the road on which automobile Vhcl1 is currently traveling obtained from data D_map, (3) the relative position of automobile Vhcl1 from a white line on the road on which automobile Vhcl1 is currently traveling obtained from data D_sensor (position information specifying how far automobile Vhcl1 is located from a white line on the road), and (4) a captured image (captured video) of the scenery ahead of automobile Vhcl1 in the direction of travel on the road on which automobile Vhcl1 is currently traveling obtained from data D_F_cam. Based on this information, relative position acquisition unit 13 acquires the relative position of automobile Vhcl1. Then, relative position acquisition unit 13 outputs data including the acquired information on the relative position of automobile Vhcl1 to virtual object position calculation unit 15 as data D2b.

[0067] The posture information acquisition unit 14 receives the data D1c output from the input interface unit 11. The posture information acquisition unit 14 acquires posture information of the driver Drv1 of the vehicle Vhcl1 based on the data D1c.

[0068] For example, data D1c includes captured image data of driver Drv1 of vehicle Vhcl1 acquired from data D_pose, and posture information acquisition unit 14 analyzes the captured image to identify the posture of driver Drv1 and identify the viewpoint of driver Drv1. Then, posture information acquisition unit 14 outputs data including the acquired information about the driver's posture (information including information about the driver's viewpoint position) to virtual object position calculation unit 15 as data D2c.

[0069] The virtual object position calculation unit 15 receives data D2a output from the traveling path shape acquisition unit 12, data D2b output from the relative position acquisition unit 13, and data D2c output from the attitude information acquisition unit 14. The virtual object position calculation unit 15 calculates the position of the virtual object based on the data D2a, D2b, and D2c. Specifically, the virtual object position calculation unit 15 defines a three-dimensional space, for example, with a predetermined point (this point is designated as Po) within the vehicle Vhcl1 as the origin. The virtual object position calculation unit 15 sets, for example, an x-axis, a y-axis, and a z-axis that are orthogonal to each other in the three-dimensional space, and defines the three-dimensional space defined by these coordinate axes. Then, the virtual object position calculation unit 15 calculates the position at which the virtual object is to be placed within the defined three-dimensional space, based on the data D2a, D2b, and D2c. Then, data including the calculated position information of the virtual object is output to the display parameter calculation unit 16 as data D3.

[0070] The display parameter calculation unit 16 receives data D3 output from the virtual object position calculation unit 15. Then, based on the data D3, the display parameter calculation unit 16 calculates display parameters for generating a display image so that when the driver Drv1 looks at the display screen of the display unit 2, the virtual object is recognized as existing at the position calculated by the virtual object position calculation unit 15. Then, the display parameter calculation unit 16 outputs data including the calculated display parameters and image data to be displayed on the display unit 2 to the display unit 2 as data D4.

[0071] The display unit 2 of the virtual reference display device 100 receives the data D4 output from the display parameter calculation unit 16. Then, the display unit 2 displays an image generated from the image data included in the data D4 on the display screen of the display unit 2, based on the display parameters included in the data D4.

[0072] The display unit 2 is realized by, for example, a HUD. In the present embodiment, a case where the display unit 2 is realized by a HUD will be described below. In this case, the display screen of the display unit 2 can be realized by, for example, installing a combiner (a half mirror that displays an image of the HUD by superimposing it on the scenery ahead of the driver) in a predetermined area on the windshield. For example, in the cross section of the windshield seen from the side in FIG. 2, the combiner is installed in the area from point P1 to point P2, and this area is used as the display screen of the display unit 2. An image is then projected onto the display screen from the projector of the display unit 2, thereby displaying an image. Note that the vertical length of the display screen is equal to or greater than the distance from point P1 to point P2 in the cross section of the windshield seen from the side in FIG. 2, and the horizontal length of the display screen is a length corresponding to the range covering the field of view of the driver Drv1 when the driver Drv1 looks forward through the windshield. The display area of ​​the display screen is not limited to the above and may be of other sizes and shapes.

[0073] The display unit 2 receives data D4 including the display parameters calculated by the display parameter calculation unit 16 and image data to be displayed on the display unit 2. Then, based on the display parameters included in the data D4, the display unit 2 displays the image data included in the data D4 on the display screen (in FIG. 1, the area from point P1 to point P2).

[0074] <1.2: Operation of the virtual reference display system> The operation of the virtual reference display system 1000 and the virtual reference display device 100 configured as above will be described below.

[0075] FIG. 3 is a flowchart of the process executed by the virtual reference display system 1000.

[0076] Figure 4 shows a car Vhcl1 and two virtual objects Mkr1. L , Mkr1 R 1 is a diagram showing the positional relationship between the

[0077] The operation of the virtual reference display system 1000 will now be described with reference to the flowchart of FIG.

[0078] (Step S1): In step S1, a travel route shape acquisition process is executed.

[0079] The GPS receiving unit Rx1 generates data D_GPS including information on the calculated current position of the automobile Vhcl1, and outputs the generated data D_GPS to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0080] The display control unit 1 requests the map data storage unit Dev1 to read out map data for the area in which the vehicle Vhcl1 is currently traveling, and in response to this request, the map data storage unit Dev1 outputs the map data for the area in which the vehicle Vhcl1 is currently traveling to the input interface unit 11 of the display control unit 1.

[0081] The sensor Sen1 measures the shape of the road on which the automobile Vhcl1 is currently traveling (for example, the surface condition of the road), and outputs the measurement data as data D_sensor to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0082] The front camera F_cam captures an image of the scenery ahead in the traveling direction of the vehicle, and outputs the captured image data as data D_F_cam to the input interface unit 11 of the display control unit 1 of the virtual reference display device 100.

[0083] The input interface unit 11 of the display control unit 1 of the virtual reference display device 100 inputs data D_GPS output from the GPS receiver Rx1, data D_F_cam output from the front camera F_cam, data D_sensor output from the sensor Sen1, and data D_map output from the map data storage unit Dev1.

[0084] Then, the input interface unit 11 extracts data necessary for the travel route shape acquisition unit 12 to acquire the travel route shape from the above input data, and outputs the extracted data to the travel route shape acquisition unit 12 as data D1a.

[0085] The travel route shape acquisition unit 12 acquires the shape of the road along which the vehicle Vhcl1 is currently traveling, based on data D1a output from the input interface unit 11. The data D1a includes (1) information about the current position of the vehicle Vhcl1 acquired from data D_GPS, (2) map information about the road along which the vehicle Vhcl1 is currently traveling, acquired from data D_map, (3) the shape of the road along which the vehicle Vhcl1 is currently traveling (e.g., the road surface condition, etc.) acquired from data D_sensor, and (4) a captured image of the scenery ahead of the vehicle Vhcl1 in the direction of travel of the road along which the vehicle Vhcl1 is currently traveling, acquired from data D_F_cam. Based on this information, the travel route shape acquisition unit 12 acquires the shape of the road along which the vehicle Vhcl1 is currently traveling. The travel route shape acquisition unit 12 then outputs data including the acquired information about the travel route shape to the virtual object position calculation unit 15 as data D2a.

[0086] (Step S2): In step S2, a relative position acquisition process is executed.

[0087] Specifically, the relative position acquisition unit 13 acquires the relative position of the automobile Vhcl1 (for example, a relative position for identifying where the automobile Vhcl1 is located on the road on which the automobile Vhcl1 is traveling (for example, a predetermined position on the road on which the automobile Vhcl1 is traveling (for example, a relative position from the center position in the width direction))) based on the data D1b output from the input interface unit 11.

[0088] For example, data D1b includes (1) information about the current position of automobile Vhcl1 obtained from data D_GPS, (2) map information of the road on which automobile Vhcl1 is currently traveling obtained from data D_map, (3) the relative position of automobile Vhcl1 from a white line on the road on which automobile Vhcl1 is currently traveling obtained from data D_sensor (position information specifying how far automobile Vhcl1 is located from a white line on the road), and (4) a captured image (captured video) of the scenery ahead of automobile Vhcl1 in the direction of travel on the road on which automobile Vhcl1 is currently traveling obtained from data D_F_cam. Based on this information, relative position acquisition unit 13 acquires the relative position of automobile Vhcl1. Then, relative position acquisition unit 13 outputs data including the acquired information on the relative position of automobile Vhcl1 to virtual object position calculation unit 15 as data D2b.

[0089] (Step S3): In step S3, a posture information acquisition process is executed.

[0090] Specifically, the posture information acquisition unit 14 acquires posture information of the driver Drv1 of the vehicle Vhcl1 based on the data D1c output from the input interface unit 11.

[0091] For example, data D1c includes captured image data of driver Drv1 of automobile Vhcl1 acquired from data D_pose output from posture detection unit Dev2, and posture information acquisition unit 14 analyzes the captured image to identify the posture of driver Drv1 and identify the viewpoint of driver Drv1. Then, posture information acquisition unit 14 outputs data including the acquired information about the driver's posture (information including information about the driver's viewpoint position) to virtual object position calculation unit 15 as data D2c.

[0092] In the virtual reference display system 1000, the processes of steps S1 to S3 may be executed in parallel.

[0093] (Step S4): In step S4, a virtual object position calculation process is executed.

[0094] The virtual object position calculation unit 15 receives data D2a output from the traveling path shape acquisition unit 12, data D2b output from the relative position acquisition unit 13, and data D2c output from the attitude information acquisition unit 14. The virtual object position calculation unit 15 calculates the position of the virtual object based on the data D2a, D2b, and D2c. Specifically, the virtual object position calculation unit 15 defines a three-dimensional space, for example, with a predetermined point (this point is designated as Po) within the vehicle Vhcl1 as the origin. The virtual object position calculation unit 15 sets, for example, an x-axis, a y-axis, and a z-axis that are orthogonal to each other in the three-dimensional space, and defines the three-dimensional space defined by these coordinate axes. Then, the virtual object position calculation unit 15 calculates the position at which the virtual object is to be placed within the defined three-dimensional space, based on the data D2a, D2b, and D2c.

[0095] For the sake of convenience, two virtual objects (virtual object Mkr1) are used as shown in FIG. L , Mkr1 R ), and in a planar view from above, the origin of the three-dimensional space is the point where the straight line connecting the two front wheels of the automobile Vhcl1 intersects with the center line of the automobile Vhcl1 in the traveling direction (the center line in a planar view from above), and is a plane that includes the bottom edge of the windshield Win_s of the automobile Vhcl1 and is parallel to the horizontal plane (ground) when the automobile Vhcl1 is traveling on a horizontal surface. The vector that indicates the direction when the automobile Vhcl1 is traveling straight is called Vec1, and its base point is called the origin Po. Then, two virtual objects Mkr1 L , Mkr1 R As shown in FIG. 4, the virtual object Mkr1 is positioned diagonally forward in the traveling direction of the vehicle Vhcl1 and in a position symmetrical with respect to the vector Vec1 in a plan view from above. L , Mkr1 R are assumed to be spherical in shape and have the same radius (same size).

[0096] The virtual object position calculation unit 15 calculates the positions of two virtual objects Mkr1 and Mkr2 in a three-dimensional space with the point Po as the origin based on the data D2a, D2b, and D2c. L , Mkr1 R The position of the calculated virtual object Mkr1 L The position of the center point in 3D space is Position(Mkr1 L ) and calculate the virtual object Mkr1 R The position of the center point in 3D space is Position(Mkr1 R )

[0097] Then, the virtual object position calculation unit 15 outputs data including the calculated position information of the virtual object to the display parameter calculation unit 16 as data D3.

[0098] (Step S5): In step S5, a display parameter calculation process is executed.

[0099] The display parameter calculation unit 16 calculates display parameters for generating a display image based on the data D3 output from the virtual object position calculation unit 15 so that when the driver Drv1 looks at the display screen of the display unit 2, the virtual object is recognized as existing at the position calculated by the virtual object position calculation unit 15. The display parameter calculation unit 16 can acquire the following information. (1) The shape and position (shape and position in three-dimensional space) of the display screen of the display unit 2 (in FIG. 1, the area from point P1 to point P2) (2) Viewpoint position of driver Drv1 (The viewpoint position of driver Drv1 can be acquired from data D2c including information about the driver's posture (information including information about the driver's viewpoint position) acquired by posture information acquisition unit 14.) (3) Virtual object Mkr1 L , Mkr1 R Position and shape in 3D space Therefore, using the information (1) to (3) above, the display parameter calculation unit 16 can calculate display parameters for generating a display image so that when the driver Drv1 looks at the display screen of the display unit 2, the virtual object is recognized as existing at the position calculated by the virtual object position calculation unit 15.

[0100] Then, the display parameter calculation unit 16 outputs data including the calculated display parameters and image data to be displayed on the display unit 2 to the display unit 2 as data D4.

[0101] (Step S6): In step S5, a display parameter calculation process is executed.

[0102] The display unit 2 receives data D4 including the display parameters calculated by the display parameter calculation unit 16 and image data to be displayed on the display unit 2. Then, based on the display parameters included in the data D4, the display unit 2 displays the image data included in the data D4 on the display screen (in FIG. 1, the area from point P1 to point P2).

[0103] FIG. 5 shows two virtual objects Mkr1 and Mkr2 displayed on the display screen of the display unit 2 by the above process. L , Mkr1 R 10 is a diagram showing an example of the foreground seen by a driver Drv1 when the following are superimposed and displayed.

[0104] As can be seen in Figure 5, two virtual objects, Mkr1 L , Mkr1 R are displayed on the display screen of the display unit 2 (superimposed in the foreground) so that the driver Drv1 can recognize that they are located at positions spaced apart on a plane parallel to the driving surface of the vehicle Vhcl1 (the road surface (ground) on which the vehicle is traveling). The driver Drv1 recognizes the two virtual objects Mkr1 superimposed in the foreground on the display screen of the display unit 2. L , Mkr1 RBy being aware of this, the driver Drv1 can easily recognize the driving state of the vehicle Vhcl1 that he is driving (easily recognize whether the vehicle is driving stably or not). In other words, the driver Drv1 can easily recognize the driving state of the vehicle Vhcl1 that he is driving (easily recognize whether the vehicle is driving stably or not). L , Mkr1 R When it is recognized that the virtual objects are not displayed stably (for example, two virtual objects Mkr1 L , Mkr1 R In other words, the virtual reference display system 1000 processes the virtual reference object Mkr1 as described above, and the virtual reference object Mkr2 is displayed in a manner similar to the above. L , Mkr1 R on the display screen of the display unit 2, the driver Drv1 can be assisted in stable driving of the vehicle Vhcl1.

[0105] As described above, the virtual reference display system 1000 (virtual reference display device 100) displays virtual references (in the above example, two virtual objects Mkr1 and Mkr2) for appropriately supporting stable driving. L , Mkr1 R ) can be displayed.

[0106] Furthermore, in the virtual reference display system 1000, since the virtual object does not actually exist, its position (the position of its virtual image) can be set to any location, and therefore the virtual object can be placed in a location that is most effective in supporting stable driving. Therefore, the virtual object can be easily placed in the optimal location (a location that is most effective in supporting stable driving) depending on the road conditions (road width, road curvature, etc.). As a result, the virtual reference display system 1000 can provide effective support for stable driving regardless of the situation.

[0107] [Second embodiment] Next, a second embodiment will be described. Note that the same parts as those in the above embodiment are given the same reference numerals and detailed description will be omitted.

[0108] FIG. 6 is a schematic configuration diagram of a virtual reference display system 2000 according to the second embodiment.

[0109] FIG. 7 is a schematic configuration diagram of a virtual reference display device 200 according to the second embodiment.

[0110] 8 and 9 are flowcharts of the processing executed by the virtual reference display system 2000 according to the second embodiment.

[0111] FIG. 10 is a diagram showing an example of a layout pattern of virtual objects when N=2.

[0112] FIG. 11 is a diagram for explaining the stable driving path prediction process.

[0113] <2.1: Virtual Reference Display System Configuration> The virtual reference display system 2000 of the second embodiment has a configuration in which the virtual reference display device 1000 of the first embodiment (display control unit 1 and display unit 2) is replaced with a virtual reference display device 200 (display control unit 1A and display unit 2).

[0114] As shown in Figure 7, the virtual reference display device 200 of the second embodiment has a configuration in which, in the virtual reference display device 100 of the first embodiment, the input interface unit 11 is replaced with an input interface unit 11A, the virtual object position calculation unit 15 is replaced with a virtual object position calculation unit 15A, and further, a stable driving path prediction unit 17, a stable driving degree acquisition unit 18, an acquired data storage processing unit 19, and an optimal virtual reference detection unit 20 are added.

[0115] The input interface unit 11A has the same functions as the input interface unit 11. The input interface unit 11A extracts, from the input data, data necessary for the stable driving path prediction unit 17 to execute processing for predicting a stable driving path, and outputs the extracted data as data D1d to the stable driving path prediction unit 17. The input interface unit 11A also extracts, from the input data, data necessary for the stable driving degree acquisition unit 18 to acquire a stable driving degree, and outputs the extracted data as data D1e to the stable driving degree acquisition unit 18.

[0116] The stable driving path prediction unit 17 receives data D1d output from the input interface unit 11A. Based on the data D1d, the stable driving path prediction unit 17 acquires information about the path on which the vehicle Vhcl1 is currently traveling, and predicts a stable path (a future path on which stable driving will be achieved) for the vehicle Vhcl1 based on the information. For example, data D1d includes (1) information about the current position of automobile Vhcl1 obtained from data D_GPS, (2) map information of the road on which automobile Vhcl1 is currently traveling obtained from data D_map, (3) the shape of the road on which automobile Vhcl1 is currently traveling (e.g., the surface condition of the road, etc.) obtained from data D_sensor, and (4) captured images (captured video) of the scenery ahead of automobile Vhcl1 in the direction of travel on the road on which automobile Vhcl1 is currently traveling obtained from data D_F_cam. Based on this information, stable driving path prediction unit 17 obtains information about the past path of automobile Vhcl1 (the path of automobile Vhcl1 from a specified time in the past to the present time), and predicts a stable path of automobile Vhcl1 (a future path that will achieve stable driving) based on this information, the current state of automobile Vhcl1 (speed, acceleration, direction of travel, etc.), map information, the shape of the road on which automobile Vhcl1 is currently traveling, etc.

[0117] For example, as shown in the left diagram of Fig. 11, when the automobile Vhcl1 is present at the position shown in the left diagram of Fig. 11 at time t1, the stable driving path prediction unit 17 predicts the stable driving path to be the path Path_opt shown in the left diagram of Fig. 11 based on the above information (information such as the position of the automobile Vhcl1 at time t1, the traveling direction, and the shape of the road of the automobile Vhcl1). Note that in Fig. 11, at time tk (k: integer), the origin position in the three-dimensional space set by the virtual object position calculation unit 15A is Po(tk), the vector indicating the traveling direction of the automobile Vhcl1 (the vector indicating the direction when the automobile Vhcl1 travels straight) is Vec1(tk), and the positions (positions of the virtual images) of the two virtual objects displayed in the virtual reference display system 2000 are P L (tk), P R 11 is a diagram showing the state of the vehicle Vhcl1 when it travels stably (passing the predicted stable path Path_opt) (the diagram showing the state of the vehicle Vhcl1 at times t1, t2, and t3).

[0118] The stable driving path predicting unit 17 outputs data including the prediction result to the stable driving degree acquiring unit 18 and the acquired data storage processing unit 19 as data D2d.

[0119] The stable driving degree acquisition unit 18 receives data D1e output from the input interface unit 11A and data D2d output from the stable driving path prediction unit 17. The stable driving degree acquisition unit 18 acquires the stable driving degree of the vehicle Vhcl1 based on the data D1e and D2d. For example, the stable driving degree acquisition unit 18 acquires the amount of deviation between the stable driving path (predicted path) acquired from the data D2d and the path actually traveled by the vehicle Vhcl1, integrates the amount of deviation over a predetermined period, and regards the value acquired by this integration as the stable driving degree, thereby acquiring the stable driving degree.

[0120] Then, the stable running degree acquiring unit 18 outputs data including the acquired stable running degree to the acquired data storage processing unit 19 as data D2e.

[0121] Virtual object position calculation unit 15A has the same functions as virtual object position calculation unit 15, and can further set a plurality of arrangement patterns of virtual objects. For example, virtual object position calculation unit 15A has N arrangement patterns of virtual objects (N: natural number, N≧2), selects one arrangement pattern from the N arrangement patterns, and executes the same processing as in the first embodiment based on the selected arrangement pattern.

[0122] Fig. 10 shows an example of a virtual object arrangement pattern when N = 2. For ease of explanation, the following will describe a case where the virtual object arrangement pattern shown in Fig. 10 is used (when N = 2).

[0123] In the arrangement pattern 1, two virtual objects Mkr1 are placed at the positions shown in Fig. 10. L , Mkr1 R is a pattern in which

[0124] In the arrangement pattern 2, two virtual objects Mkr2 are placed at the positions shown in Fig. 10. L , Mkr2 R is a pattern in which

[0125] The virtual object position calculation unit 15A outputs data including information on the arrangement pattern of the selected virtual object to the acquired data storage processing unit 19 as data Dsel.

[0126] Furthermore, the virtual object position calculation unit 15A receives the data D_det output from the optimum virtual reference detection unit 20, and can set the arrangement pattern of the virtual object based on the data D_det.

[0127] The acquired data storage processing unit 19 receives data D2d output from the stable driving path prediction unit 17, data D2e output from the stable driving degree acquisition unit 18, and data Dsel output from the virtual object position calculation unit 15A. The acquired data storage processing unit 19 generates data D_st that associates the stable driving degree with the arrangement pattern of the virtual object that was selected when the stable driving degree was acquired, and stores the data D_st. In addition, the acquired data storage processing unit 19 outputs the data D_st to the optimal virtual reference detection unit 20 in accordance with a read request from the optimal virtual reference detection unit 20.

[0128] The optimum virtual reference detection unit 20 reads the data D_st by making a data read request to the acquired data storage processing unit 19 (the data D_st is input from the acquired data storage processing unit 19). Then, based on the data D_st, the optimum virtual reference detection unit 20 identifies the arrangement pattern of virtual objects that has achieved the most stable running in a predetermined period, and outputs data including information on the identified arrangement pattern of the virtual objects as data D_det to the virtual object position calculation unit 15A.

[0129] <2.2: Operation of the virtual reference display system> The operation of the virtual reference display system 2000 configured as above will be described with reference to a flowchart.

[0130] (Step S21): In step S21, initialization processing is executed to set the first virtual object arrangement pattern to be selected. Here, a variable i that specifies the virtual object arrangement pattern is set to "1."

[0131] (Step S22): In step S22, loop processing (processing of steps S22 to S28) is started. Note that since N=2 here, the loop processing is executed twice.

[0132] (Step S23): In step S23, the virtual object position calculation unit 15A sets the arrangement pattern of the virtual object to the first pattern (arrangement pattern 1) (selects arrangement pattern 1).

[0133] (Step S24): In step S24, a stable driving route prediction process is executed.

[0134] The stable driving route prediction unit 17 acquires information about the route on which the vehicle Vhcl1 is currently traveling based on the data D1d output from the input interface unit 11A, and predicts a stable route (a future route on which stable driving will be achieved) for the vehicle Vhcl1 based on the information. For example, data D1d includes (1) information about the current position of automobile Vhcl1 obtained from data D_GPS, (2) map information of the road on which automobile Vhcl1 is currently traveling obtained from data D_map, (3) the shape of the road on which automobile Vhcl1 is currently traveling (e.g., the surface condition of the road, etc.) obtained from data D_sensor, and (4) captured images (captured video) of the scenery ahead of automobile Vhcl1 in the direction of travel on the road on which automobile Vhcl1 is currently traveling obtained from data D_F_cam. Based on this information, stable driving path prediction unit 17 obtains information about the past path of automobile Vhcl1 (the path of automobile Vhcl1 from a specified time in the past to the present time), and predicts a stable path of automobile Vhcl1 (a future path that will achieve stable driving) based on this information, the current state of automobile Vhcl1 (speed, acceleration, direction of travel, etc.), map information, the shape of the road on which automobile Vhcl1 is currently traveling, etc.

[0135] Then, the stable driving path predicting unit 17 outputs data including the prediction result to the stable driving degree acquiring unit 18 and the acquired data storage processing unit 19 as data D2d.

[0136] (Step S25): In step S25, a virtual reference display process is executed.

[0137] In the first processing, the virtual reference display process is executed with the arrangement pattern of the virtual objects set as arrangement pattern 1, and in the second processing, the virtual reference display process is executed with the arrangement pattern of the virtual objects set as arrangement pattern 2.

[0138] The virtual reference display process in step S25 is the same as the process in the first embodiment (the process in the flowchart in FIG. 3). That is, steps S251 to S256 (FIG. 9) constituting the virtual reference display process in step S25 are the same as steps S1 to S6 executed in the process of the virtual reference display system 1000 in the first embodiment, respectively.

[0139] (Step S26): In step S26, a stable running degree acquisition process is executed.

[0140] The stable driving degree acquisition unit 18 acquires the stable driving degree of the vehicle Vhcl1 based on the data D1e output from the input interface unit 11A and the data D2d output from the stable driving path prediction unit 17. For example, the stable driving degree acquisition unit 18 acquires the amount of deviation between the stable driving path (predicted path) acquired from the data D2d and the path actually traveled by the vehicle Vhcl1, integrates the amount of deviation over a predetermined period, and regards the value acquired by the integration as the stable driving degree, thereby acquiring the stable driving degree.

[0141] Then, the stable running degree acquiring unit 18 outputs data including the acquired stable running degree to the acquired data storage processing unit 19 as data D2e.

[0142] The process of step S25 and the process of step S26 are executed in parallel.

[0143] (Step S27): In step S27, an acquired data storage process is executed.

[0144] The acquired data storage processing unit 19 receives as input the data D2d output from the stable driving path prediction unit 17, the data D2e output from the stable driving degree acquisition unit 18, and the data Dsel output from the virtual object position calculation unit 15A. The acquired data storage processing unit 19 generates, as data D_st, data that associates the stable driving degree with the arrangement pattern of the virtual object that was selected when the stable driving degree was acquired, and stores the data D_st.

[0145] (Step S28): In step S28, a process for determining whether the loop process is complete is executed. If it is determined that the loop process is all complete, the process proceeds to step S29. If it is not determined that the loop process is all complete, the process returns to step S22.

[0146] (Step S29): In step S29, an optimum arrangement pattern detection process is executed.

[0147] The optimal virtual reference detection unit 20 reads the data D_st by sending a data read request to the acquired data storage processing unit 19 (the data D_st is input from the acquired data storage processing unit 19). Then, the optimal virtual reference detection unit 20 identifies the arrangement pattern of virtual objects that has achieved the most stable running in a predetermined period, based on the data D_st. For example, if the value of the stability degree when the arrangement pattern of the virtual objects is arrangement pattern 1 in a predetermined period indicates that the stability degree is better than the value of the stability degree when the arrangement pattern of the virtual objects is arrangement pattern 2, it can be determined that more stable running can be achieved by setting the arrangement pattern of the virtual objects to arrangement pattern 1. Therefore, in this case, the optimal virtual reference detection unit 20 identifies the arrangement pattern of virtual objects that has achieved the most stable running in a predetermined period as arrangement pattern 1, based on the data D_st.

[0148] Then, the optimum virtual reference detection unit 20 outputs data including information on the arrangement pattern of the identified virtual object as data D_det to the virtual object position calculation unit 15A.

[0149] Then, the virtual object position calculation unit 15A sets the arrangement pattern of the virtual object to the arrangement pattern detected by the optimum virtual reference detection unit 20, and executes the display process of the virtual object.

[0150] As a result, in the virtual reference display system 2000, virtual objects are displayed on the display screen of the display unit 2 based on an arrangement pattern that is likely to achieve more stable driving. As a result, the virtual reference display system 2000 can realize more appropriate support for stable driving.

[0151] The reason why the virtual reference display system 2000 provides a plurality of virtual object arrangement patterns and detects the optimum arrangement pattern from among them through the above processing is as follows.

[0152] It has been found that the placement pattern of virtual objects that ensures stable driving varies depending on the width of the field of view of the driver Drv1. When a driver Drv1 with a wide field of view drives the vehicle Vhcl1, the likelihood of stable driving increases if virtual objects are placed further forward in the direction of travel, and when a driver Drv1 with a narrow field of view drives the vehicle Vhcl1, the likelihood of stable driving increases if virtual objects are placed closer in the direction of travel than the optimal placement position for a driver Drv1 with a wide field of view.

[0153] Based on this knowledge, in order to detect the optimal placement pattern of virtual objects that changes depending on the width of the driver Drv1's field of view, the virtual reference display system 2000 performs the above-described process to provide multiple placement patterns of virtual objects and then executes a process to detect the optimal placement pattern from among them. As a result, the virtual reference display system 2000 can identify a placement of virtual objects that is likely to enable driver Drv1 to achieve stable driving, regardless of the width of driver Drv1's field of view, and can display the virtual objects on the display screen of the display unit 2 based on that placement. As a result, the virtual reference display system 2000 can appropriately support stable driving, regardless of the width of driver Drv1's field of view.

[0154] The process of detecting the optimum pattern of the virtual object in the above process may be updated at a predetermined interval.

[0155] Furthermore, in the above, the number N of arrangement patterns of virtual objects is "2", but this is not limitative and the number N of arrangement patterns of virtual objects may be another number.

[0156] FIG. 12 shows, as an example, a layout pattern of virtual objects when an automobile Vhcl1 is traveling in the center lane of a five-lane expressway.

[0157] In FIG. 12, the virtual object placement patterns are the following nine patterns. (1) Placement pattern 1: Virtual Object Ptn1 LR (Center position of Lane 3, distance L1 from the front wheel of vehicle Vhcl1) (2) Placement pattern 2: Virtual Object Ptn2 L (Center position of Lane 2, distance L1 from the front wheel position of vehicle Vhcl1) Virtual Object Ptn2 R (Center position of Lane 4, distance L1 from the front wheel of vehicle Vhcl1) (3) Placement pattern 3: Virtual Object Ptn3 L(Center position of Lane 1, distance L1 from the front wheel position of vehicle Vhcl1) Virtual Object Ptn3 R (Center position of Lane 5, Lane 4, distance L1 from the front wheel of vehicle Vhcl1) (4) Placement pattern 4: Virtual Objects Ptn4 LR (Center position of Lane 3, distance L2 from the front wheel of vehicle Vhcl1) (5) Placement pattern 5: Virtual Objects Ptn5 L (Center position of Lane 2, distance L2 from the front wheel position of vehicle Vhcl1) Virtual Objects Ptn5 R (Center position of Lane 4, distance L2 from the front wheel of vehicle Vhcl1) (6) Placement pattern 6: Virtual Objects Ptn6 L (Center position of Lane 1, distance L2 from the front wheel position of vehicle Vhcl1) Virtual Objects Ptn6 R (Center position of Lane 5, Lane 4, distance L2 from the front wheel of vehicle Vhcl1) (7) Placement pattern 7: Virtual Objects Ptn7 LR (Center position of Lane 3, distance L3 from the front wheel of vehicle Vhcl1) (8) Placement pattern 8: Virtual Objects Ptn8 L (Center position of Lane 2, distance L3 from the front wheel position of vehicle Vhcl1) Virtual Objects Ptn5 R (Center position of Lane 4, distance L3 from the front wheel of vehicle Vhcl1) (9) Placement pattern 9: Virtual Objects Ptn6 L (Center position of Lane 1, Lane 2, distance L3 from the front wheel position of vehicle Vhcl1) Virtual Objects Ptn6 R (Center position of Lane 5, Lane 4, distance L3 from the front wheel of vehicle Vhcl1) When vehicle Vhcl1 is traveling in the center lane of a five-lane highway, the above processing in the virtual reference display system 2000 may be performed using the nine virtual object placement patterns described above. In this case, experimental data has shown that when a driver Drv1 with a wide field of view drives vehicle Vhcl1, the vehicle is likely to be driven stably when placement pattern 9 is selected, whereas when a driver Drv1 with a narrow field of view drives vehicle Vhcl1, the vehicle is likely to be driven stably when placement pattern 6 is selected. By performing the above processing, the virtual reference display system 2000 can identify the optimal virtual object placement pattern even in the case of FIG. 12 (and is likely to obtain detection results that match or are similar to the experimental data).

[0158] Furthermore, in the virtual reference display system 2000, since the virtual object does not actually exist, its position (the position of its virtual image) can be set to any location, and therefore the virtual object can be placed in a location that is most effective in supporting stable driving. Therefore, the virtual object can be easily placed in the optimal location (a location that is most effective in supporting stable driving) depending on the road conditions (road width, road curvature, etc.). As a result, the virtual reference display system 2000 can provide effective support for stable driving in any situation.

[0159] [Third embodiment] Next, a third embodiment will be described. Note that the same parts as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0160] FIG. 13 is a diagram showing the foreground seen by the driver Drv1 (the foreground seen from the front window) and the LED array LED_arry as the display unit 2. As shown in FIG.

[0161] In the above embodiment, the case where the display unit 2 is configured using a HUD has been described, but in the third embodiment, the display unit 2 is realized by an LED array.

[0162] In the virtual reference display system of the third embodiment, instead of the display screen of the HUD, an LED array in which multiple LEDs are arranged in a line is installed on the dashboard of the automobile Vhcl1 as the display unit 2. The multiple LEDs (multiple light-emitting elements) are arranged so as to be approximately parallel to the bottom edge of the windshield.

[0163] In the virtual reference display system of the third embodiment, as shown in Fig. 13, in a virtual plane (a plane normal to the line of sight of the driver Drv1) when the driver Drv1 looks ahead in the traveling direction of the vehicle Vhcl1, when the positions of two virtual objects are projected onto the virtual plane, LEDs (light-emitting elements) installed at positions where the lateral (horizontal) coordinates match are turned on. That is, in the case of Fig. 13, in the virtual reference display system of the third embodiment, the LED (light-emitting element) set at position Pos_LED_on_L (virtual object Mkr1) is turned on. L ) and the LED (light-emitting element) set at position Pos_LED_on_R (corresponding to virtual object Mkr1 R (corresponding to) and turn on.

[0164] By controlling in this manner, the virtual reference display system of the third embodiment can achieve stable driving assistance using a low-cost LED array instead of a costly HUD as in the first and second embodiments.

[0165] [Other embodiments] In the above embodiment, a case has been described in which a HUD or an LED array is used as the display unit 2, but this is not limiting, and a head-mounted display (HMD) may also be used as the display unit 2. When an HMD is used, a virtual object may be displayed as a 3D image. That is, taking into consideration the binocular parallax of the user, a display image for the right eye of the HMD and a display image for the left eye of the HMD may be generated so that the virtual object is projected as a 3D image at the position described above on the display screen for the right eye and the display screen for the left eye of the HMD, and the generated images may be displayed on the display screen for the right eye and the display screen for the left eye of the HMD, respectively.

[0166] In addition, although the above description is based on the case where the moving body is an automobile, the present invention is not limited to this case, and the moving body may be a bicycle, for example, and the user operating the bicycle may wear an HMD and execute the above-described processing. This may provide appropriate stable riding support to a user riding a bicycle, for example.

[0167] The present invention may also be applied to a pedestrian, rather than a moving object. In this case, the pedestrian wears an HMD and executes the above-described process, thereby providing appropriate support for the pedestrian's stable running (stable walking).

[0168] In the above embodiment, the virtual object has been described as having a spherical shape, but the shape is not limited to this and the virtual object may have any shape. The size of the virtual object may also be any size.

[0169] In the above embodiment, when the moving body is the automobile Vhcl1, the process of detecting the posture of the driver Drv1 is performed to detect the position of the driver Drv1's viewpoint. However, this is not limited to this, and for example, the posture information acquisition unit or the posture information acquisition process may be omitted. In this case, the approximate position of the viewpoint of the driver Drv1 is known in advance from the arrangement, shape, etc. of the driver's seat of the automobile Vhcl1, and the processing of the first and second embodiments may be performed using this position of the viewpoint.

[0170] Furthermore, in the above embodiment, the processing is performed using vector Vec1 as a vector indicating the direction in which the automobile Vhcl1 moves straight ahead, but this is not limited to this. For example, instead of vector Vec1, the processing of the above embodiment may be performed by predicting the direction in which the moving body (or user) should move to achieve stable driving, and using a vector (stable driving vector) that defines the predicted direction. Furthermore, in the virtual reference display system and virtual reference display device described in the above embodiments, each block may be individually integrated into a single chip using a semiconductor device such as an LSI, or may be integrated into a single chip to include some or all of the blocks.

[0171] Although we have referred to it as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0172] Furthermore, the method of integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. It is also possible to use FPGAs (Field Programmable Gate Arrays), which can be programmed after the LSI is manufactured, or reconfigurable processors, which allow the connections and settings of circuit cells inside the LSI to be reconfigured.

[0173] Furthermore, part or all of the processing of each functional block in each of the above embodiments may be realized by a program. And part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Furthermore, the programs for performing each processing are stored in a storage device such as a hard disk or ROM, and are executed in the ROM or by being read into the RAM.

[0174] Furthermore, each process in the above-described embodiments may be realized by hardware, or by software (including cases where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by a combination of software and hardware.

[0175] For example, when each functional unit of the above embodiment is realized by software, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 14 (for example, a hardware configuration in which a CPU, ROM, RAM, input unit, output unit, etc. are connected via a bus).

[0176] Furthermore, the execution order of the processing methods in the above embodiments is not necessarily limited to that described in the above embodiments, and the execution order can be changed within the scope of the gist of the invention.

[0177] The scope of the present invention includes a computer program for causing a computer to execute the above-described method and a computer-readable recording medium having the program recorded thereon, including, for example, a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a large-capacity DVD, a next-generation DVD, and a semiconductor memory.

[0178] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0179] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0180] 1000, 2000 Virtual Reference Display System 100, 200 Virtual reference display device 1, 1A Display control unit 12. Travel path shape acquisition unit 13 Posture information acquisition unit 14 Posture information acquisition section 15, 15A Virtual object position calculation unit 16 Display parameter calculation unit 17 Stable driving path prediction unit 18 Stable running degree acquisition unit 20 Optimal virtual reference detection unit Drv1 Driver (User) LED_arry LED array

Claims

1. A virtual reference display device that is installed on a moving body used by a user when moving, and that can input image data of the user, and that displays a group of virtual reference objects including a plurality of virtual reference objects on a display screen ahead in the direction of travel of the user when the user moves, a travel route shape acquisition unit that acquires travel route shape information, which is information about the shape of a road on which a user is traveling; a relative position acquisition unit that acquires relative position information for identifying a position on a road on which the moving object is moving; a posture information acquisition unit that analyzes image data of the user to identify the posture of the user, identify the user's viewpoint, and acquire user posture information that includes information related to the identified posture of the user and information related to the user's viewpoint position; a virtual object position calculation unit that defines a three-dimensional coordinate space with a predetermined point as the origin, calculates positions at which to place the virtual reference object group in the three-dimensional coordinate space based on the travel path shape information acquired by the travel path shape acquisition unit, the relative position information acquired by the relative position acquisition unit, and the user attitude information acquired by the attitude information acquisition unit, and acquires data including the calculated position information of the virtual reference object group as data for calculating display parameters; a display parameter calculation unit that calculates display parameters for displaying a display image based on the display parameter calculation data so that, when the user looks at the display screen, the virtual reference object group is recognized as being present at the positions calculated by the virtual object position calculation unit; and a display unit that displays the virtual reference object group on the display screen based on the display parameters; Equipped with the virtual reference object group includes a first virtual reference object and a second virtual reference object, which are two virtual reference objects having different coordinate positions in the three-dimensional coordinate space, the first virtual reference object and the second virtual reference object are disposed in a position forward in the traveling direction in the three-dimensional coordinate space such that a straight line connecting the first virtual reference object and the second virtual reference object is substantially parallel to a bottom surface of a traveling path along which the user is traveling; Virtual reference display.

2. A virtual reference display device that is installed on a moving body used when a user moves, and that displays a group of virtual reference objects including a plurality of virtual reference objects on a display screen in front of the user in the traveling direction when the user moves, comprising: a travel route shape acquisition unit that acquires travel route shape information, which is information about the shape of a road on which a user is traveling; a relative position acquisition unit that acquires relative position information for identifying a position on a road on which the moving object is moving; a virtual object position calculation unit that defines a three-dimensional coordinate space with a predetermined point as the origin, calculates positions at which to place the group of virtual reference objects in the three-dimensional coordinate space based on the traveling path shape information acquired by the traveling path shape acquisition unit, the relative position information acquired by the relative position acquisition unit, and the user's viewpoint position identified from the arrangement and shape of the driver's seat when the user operates the moving object, and acquires data including the calculated position information of the group of virtual reference objects as data for calculating display parameters; a display parameter calculation unit that calculates display parameters for displaying a display image based on the display parameter calculation data so that, when the user looks at the display screen, the virtual reference object group is recognized as being present at the positions calculated by the virtual object position calculation unit; and a display unit that displays the virtual reference object group on the display screen based on the display parameters; Equipped with the virtual reference object group includes a first virtual reference object and a second virtual reference object, which are two virtual reference objects having different coordinate positions in the three-dimensional coordinate space, the first virtual reference object and the second virtual reference object are disposed in a position forward in the traveling direction in the three-dimensional coordinate space such that a straight line connecting the first virtual reference object and the second virtual reference object is substantially parallel to a bottom surface of a traveling path along which the user is traveling; Virtual reference display.

3. a stable driving path prediction unit that predicts a direction in which the user should move to achieve stable driving based on the travel path shape information and the relative position information of the user, and obtains a stable driving vector that defines the predicted direction; the group of virtual reference objects is two of the virtual reference objects, the two virtual reference objects are arranged in line-symmetric positions with respect to a straight line extending from the stable driving vector in the three-dimensional coordinate space; 3. A virtual reference display device according to claim 1 or 2.

4. a stable driving degree acquiring unit that acquires a stable driving degree indicating a degree of stable driving by comparing the stable driving vector with a route actually traveled by the user; an optimal virtual reference position detection unit that detects a virtual reference object position that is likely to realize stable driving; Furthermore, the two virtual reference objects are arranged at one pattern position among N (N: natural number, N≧2) pattern positions that are line-symmetric with respect to a straight line extending from the stable running vector in the three-dimensional coordinate space; The virtual object position calculation unit calculating, as positions of the virtual reference object group, positions in the three-dimensional coordinate space of the two virtual reference objects in a state in which one selected from the N pattern positions is placed at the pattern position; the display unit displays the virtual reference object group so that the virtual reference object group is displayed at the position calculated by the virtual object position calculation unit; the stable running degree acquisition unit acquires the stable running degree when the virtual reference object group is displayed at a pattern position selected from the N pattern positions, in association with the selected pattern position; the optimal virtual reference position detection unit detects, as an optimal virtual reference object position, a pattern position when running that is closest to stable running is realized based on the stable running degree associated with the pattern position.

4. The virtual reference display device of claim 3.

5. Further provided is a moving body that moves when operated by a user riding thereon; the relative position acquisition unit acquires, as the relative position information, information relating to a relative position between the position of the moving object and the traveling path; 5. A virtual reference display device according to claim 1.

6. The posture information acquisition unit acquiring the user posture information based on at least one of information on a head position of the user when the user is riding on a moving body and operating the moving body and information on a posture state of the user; 2. The virtual reference display device of claim 1.

7. The display unit the light emitting device includes a plurality of light emitting elements arranged in a line, and is attached to the moving body so as to be positioned in front of the user when the user rides on the moving body and operates it; causing the light-emitting elements arranged at positions corresponding to the group of virtual reference objects to emit light based on the display parameters; 7. A virtual reference display device according to claim 5 or 6.

8. A virtual reference display method used in a virtual reference display device that is installed on a moving body used by a user when moving and that can input image data of the user, the virtual reference display method being for displaying a group of virtual reference objects including a plurality of virtual reference objects on a display screen ahead in the direction of travel of the user when the user moves, comprising: a travel route shape acquisition step of acquiring travel route shape information which is information about the shape of a road on which the user is traveling; a relative position acquisition step of acquiring relative position information for identifying a position on a road on which the moving object is moving; a posture information acquisition step of identifying the posture of the user by analyzing image data of the user, identifying the user's viewpoint, and acquiring user posture information that includes information related to the identified posture of the user and information related to the user's viewpoint position; a virtual object position calculation step of defining a three-dimensional coordinate space with a predetermined point as an origin, calculating positions at which to place the group of virtual reference objects in the three-dimensional coordinate space based on the travel path shape information acquired in the travel path shape acquisition step, the relative position information acquired in the relative position acquisition step, and the user attitude information acquired in the attitude information acquisition step, and acquiring data including the calculated position information of the group of virtual reference objects as data for calculating display parameters; a display parameter calculation step of calculating, based on the display parameter calculation data, display parameters for displaying a display image so that, when the user looks at the display screen, the virtual reference object group is recognized as being present at the positions calculated in the virtual object position calculation step; a display step of displaying the virtual reference objects on the display screen based on the display parameters; Equipped with the virtual reference object group includes a first virtual reference object and a second virtual reference object, which are two virtual reference objects having different coordinate positions in the three-dimensional coordinate space, the first virtual reference object and the second virtual reference object are disposed in a position forward in the traveling direction in the three-dimensional coordinate space such that a straight line connecting the first virtual reference object and the second virtual reference object is substantially parallel to a bottom surface of a traveling path along which the user is traveling; Virtual reference display method.

9. A virtual reference display method for displaying a group of virtual reference objects including a plurality of virtual reference objects on a display screen in front of a moving direction of the user when the user moves, the method comprising: a travel route shape acquisition step of acquiring travel route shape information which is information about the shape of a road on which the user is traveling; a relative position acquisition step of acquiring relative position information for identifying a position on a road on which the moving body is moving; a virtual object position calculation step of defining a three-dimensional coordinate space with a predetermined point as an origin, calculating positions for placing the group of virtual reference objects in the three-dimensional coordinate space based on the travel path shape information acquired in the travel path shape acquisition step, the relative position information acquired in the relative position acquisition step, and the user's viewpoint position identified from the arrangement and shape of the driver's seat when the user operates the moving object, and acquiring data including the calculated position information of the group of virtual reference objects as data for calculating display parameters; a display parameter calculation step of calculating, based on the display parameter calculation data, display parameters for displaying a display image so that, when the user looks at the display screen, the virtual reference object group is recognized as being present at the positions calculated in the virtual object position calculation step; a display step of displaying the virtual reference objects on the display screen based on the display parameters; Equipped with the virtual reference object group includes a first virtual reference object and a second virtual reference object, which are two virtual reference objects having different coordinate positions in the three-dimensional coordinate space, the first virtual reference object and the second virtual reference object are disposed in a position forward in the traveling direction in the three-dimensional coordinate space such that a straight line connecting the first virtual reference object and the second virtual reference object is substantially parallel to a bottom surface of a traveling path along which the user is traveling; Virtual reference display method.

10. 10. A program for causing a computer to execute the virtual reference display method according to claim 8 or 9.

Images