Mixed reality or augmented reality display method and content presentation device

Abstract

This enhances the sense of realism in mixed reality or augmented reality display methods that overlay virtual objects around a vehicle. [Solution] In a mixed reality or augmented reality display method that superimposes virtual objects onto video footage of the area around a vehicle or the actual scenery around the vehicle and displays them on a display device, environmental information of the environment around the vehicle is detected (S1), factors that may cause collision with real-world moving objects other than the vehicle are estimated based on the environmental information (S2), the movement of the moving objects is predicted according to the estimated factors (S3), an avoidance position in real space that will not collide with the moving objects is selected based on the predicted movement of the moving objects (S4, S5, S7), and a virtual object is superimposed and displayed at the avoidance position (S6).

Description

Technical Field

[0001] The present invention relates to a method for displaying mixed reality or augmented reality and a content presentation device.

Background Art

[0002] As a technology for expressing virtual objects that do not exist in the real world as if they exist in the real space, mixed reality (MR) technology and augmented reality (AR) technology are known (for example, Patent Document 1).

[0003] In such mixed reality technology and augmented reality technology, virtual objects are superimposed and displayed on a video or scenery of the real world. For this reason, the shape of an object in the real world is recognized by image recognition, and virtual objects are arranged at natural positions with respect to the objects in the real world (for example, virtual objects are arranged on a table). In addition, virtual objects are displayed so as to avoid the situation where when an object in the real world collides with a virtual object, the virtual object appears to penetrate through it.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] As mixed reality content or augmented reality content provided to a user riding in a vehicle, entertainment content that displays virtual objects around the vehicle for production can be considered. In this case, if virtual objects are not arranged in consideration of the real-time driving environment, for example, objects are displayed at unnatural positions where they are likely to collide with other moving objects, such as stopping in the middle of an intersection with a red signal, and the sense of reality (so-called "being there feeling") is lost.

[0006] Furthermore, avoiding interference with fast-moving real-world objects (such as other vehicles) requires highly accurate and high-speed image recognition processing and virtual object rendering processing. If the processing speed or calculation accuracy is insufficient, interference between real-world vehicles and virtual objects can occur, causing users to perceive an unnatural experience, which is another challenge. The present invention aims to improve the sense of realism in a mixed reality or augmented reality display method that overlays virtual objects around a vehicle. [Means for solving the problem]

[0007] According to one aspect of the present invention, a mixed reality or augmented reality display method is provided, which superimposes virtual objects onto video footage of the area around a vehicle or the actual scenery around the vehicle and displays them on a display device. In this display method, environmental information of the environment around the vehicle is detected, factors that may cause collision with real-world moving objects other than the vehicle are estimated based on the environmental information, the movement of the moving objects is predicted according to the estimated factors, an avoidance position in real space that will not collide with the moving objects is selected based on the predicted movement of the moving objects, and virtual objects are superimposed and displayed at the avoidance position. [Effects of the Invention]

[0008] According to the present invention, the sense of realism in a mixed reality or augmented reality display method that overlays virtual objects around a vehicle can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of an example of content presentation in an embodiment. [Figure 2] (a) and (b) are schematic diagrams illustrating the problems of the embodiment. [Figure 3] (a) and (b) are schematic diagrams illustrating examples of the placement of virtual objects at an intersection. [Figure 4] (a) and (b) are schematic diagrams of examples of the placement locations of virtual objects in the confluence section. [Figure 5] (a) and (b) are schematic diagrams illustrating examples of virtual object placement locations when a vehicle prioritizes the passage of other traffic users. [Figure 6] (a) and (b) are schematic diagrams illustrating an example of changing the display format of a virtual object. [Figure 7] This is a flowchart showing an example of how to display the embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. Note that the drawings are schematic and may differ from actual ones. Furthermore, the embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0011] (composition) Figure 1 is a schematic diagram of an example of content presentation according to the embodiment. Vehicle 1 is equipped with a content presentation device 10 that presents mixed reality content or augmented reality content to a user riding in vehicle 1. The content display device 10 presents the user with a mixed reality or augmented reality world in which the real world and the virtual world are merged by displaying virtual objects that do not exist in the real world superimposed on the video of the area around the vehicle 1 or the real scenery around the vehicle 1 on the display device.

[0012] For example, the content display device 10 may provide entertainment content that displays and portrays characters from subcultures such as animation, manga, and games as virtual objects around the vehicle 1. For the sake of simplicity, the following description will focus on an example where the content presentation device 10 presents mixed reality content, but the present invention can also be applied to a content presentation device that presents augmented reality content. When applying the present invention to augmented reality, replace the term "mixed reality" with "augmented reality" in the following description.

[0013] The content display device 10 includes an external sensor 11, a vehicle sensor 12, a map database (map DB) 13, a positioning device 14, a communication device 15, an in-vehicle camera 16, a human-machine interface (HMI 17), and a controller 18.

[0014] The external sensor 11 detects environmental information, which is information about the surrounding environment of vehicle 1, such as the relative position of objects around vehicle 1 to vehicle 1, the distance between vehicle 1 and objects, and the direction in which objects are located. For example, the external sensor may be equipped with a camera that photographs the surrounding environment of vehicle 1 (for example, in front of vehicle 1). In the following description, the camera equipped with the external sensor 11 may be referred to as the "surrounding environment camera". Furthermore, the external sensor 11 may include, for example, a rangefinder (LRF), radar, or a LiDAR (Light Detection and Ranging) laser radar.

[0015] The vehicle sensor 12 detects various information (vehicle information) obtained from the vehicle 1. The vehicle sensor 12 includes, for example, a vehicle speed sensor that detects the vehicle speed of the vehicle 1, a wheel speed sensor that detects the rotational speed of the vehicle 1's tires, a three-axis acceleration sensor (G sensor) that detects the acceleration (including deceleration) of the vehicle 1 in three axes, a steering angle sensor that detects the steering angle (including turning angle), a gyro sensor that detects the angular velocity occurring in the vehicle 1, a yaw rate sensor that detects the yaw rate, an accelerator sensor that detects the accelerator opening of the vehicle 1, and a brake sensor that detects the amount of brake operation by the driver.

[0016] The map DB 13 stores road map data. For example, the map DB 13 may store map data for navigation (hereinafter simply referred to as "navigation map"). The map DB 13 may also store high-precision map data (hereinafter simply referred to as "high-definition map") suitable as map information for autonomous driving. The positioning device 14 includes a global navigation satellite system (GNSS) receiver, receives radio waves from a plurality of navigation satellites, and measures the current position of the vehicle 1. The GNSS receiver may be, for example, a global positioning system (GPS) receiver or the like. The positioning device 14 may also be, for example, an inertial navigation device.

[0017] The communication device 15 provides a communication function between the vehicle 1 and an external device. The communication method by the communication device 15 may be, for example, wireless communication via a public mobile communication network, satellite communication, vehicle-road communication, vehicle-to-vehicle communication, etc. The controller 18 exchanges information with other vehicles and cloud devices via the communication device 15. The in-vehicle camera 16 is a camera that captures the interior of the vehicle 1. The controller 18 can acquire the face image of the user from the captured image by the in-vehicle camera 16.

[0018] The HMI 17 is an interface device that exchanges information between the user boarding the vehicle 1 and the content presentation device 10. For example, the HMI 17 may include a display device visible to the user as an interface for presenting visual information. The display device of the HMI 17 may be mounted on, for example, the instrument panel, navigation device, in-vehicle infotainment (IVI) device, etc. of the vehicle 1, and may include a display device capable of displaying the video of the surroundings of the vehicle 1 captured by the surrounding environment camera.

[0019] Also, for example, the display device of the HMI 17 may include a head-up display (HUD) that displays virtual objects superimposed on the real scene in front of the vehicle 1 on the combiner arranged on the dashboard of the vehicle 1 or on the front windshield.

[0020] Furthermore, the display device of HMI17 may include, for example, an optical see-through type head-mounted display (HMD) or a video see-through type HMD. The HMI17 may be equipped with a speaker or buzzer as an interface for presenting auditory information. The HMI17 may also be equipped with an interface (touch panel, buttons, switches, etc.) for receiving user input.

[0021] The controller 18 is an electronic control unit (ECU) that performs information processing to generate mixed reality content in which virtual objects are superimposed on images of the area around the vehicle 1 or the real-world scenery around the vehicle 1. The controller 18 includes a processor and peripheral components such as memory devices. The processor may be, for example, a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit).

[0022] The memory device may include semiconductor memory devices, magnetic memory devices, optical memory devices, etc. The memory device may include registers, cache memory, and memory such as ROM (Read Only Memory) and RAM (Random Access Memory) used as main memory. The functions of the controller 18 described below are realized, for example, by the processor executing a computer program stored in the memory device.

[0023] The controller 18 may be formed by dedicated hardware for performing the information processing described below. For example, the controller 18 may include functional logic circuits set in a general-purpose semiconductor integrated circuit. For example, the controller 18 may have a programmable logic device (PLD) such as a field-programmable gate array (FPGA).

[0024] The controller 18 presents the user with augmented reality content via the display device of the HMI 17, in which virtual objects are superimposed on images of the area around the vehicle 1 or on the real-world scenery around the vehicle 1. When generating mixed reality content, the controller 18 sets the real-world location where the virtual object will be placed. In the following description, the real-world location where the virtual object will be placed may be referred to as the "object placement location." For example, the controller 18 may set the object placement location to a location within a predetermined range around the vehicle 1 (for example, a location at a predetermined distance in front of the vehicle 1).

[0025] The controller 18 superimposes virtual objects onto the locations where objects are placed in the video footage taken around the vehicle 1 and displays them on the HMI 17's display device. Alternatively, the controller 18 displays virtual objects on the HMI 17's display device so that the virtual objects appear superimposed onto the locations where objects are placed in the real-world scene around the vehicle 1. For example, when displaying mixed reality content on a display device mounted on an instrument panel, navigation system, IVI equipment, or video see-through type HMD, the controller 18 may display images of the vehicle 1's surroundings, captured by an ambient camera or a camera built into the HMD, on these display devices.

[0026] The controller 18 may calculate the pixel position on the display screen that corresponds to the object placement position in real space, based on a known correspondence between the position on the camera coordinate system (a relative coordinate system with the camera's position as the origin) in real space and the pixel position on the display screen, and may draw the virtual object at the calculated position.

[0027] For example, when presenting mixed reality content by displaying virtual objects on the HUD, the controller 18 may detect the user's viewpoint position and gaze direction based on the user's face image captured by the in-vehicle camera 16. Based on the detected viewpoint position and gaze direction, the controller 18 may calculate a correspondence between the position in the real space in front of the vehicle 1 and the display position of the virtual image displayed on the HUD so as to overlap with this position in the real space. Based on the calculated correspondence, the controller 18 may calculate a display position where the virtual image is displayed overlapping the object placement position, and draw the virtual object at the calculated position.

[0028] For example, when presenting mixed reality content by displaying virtual objects on an optical see-through HMD, the controller 18 detects the position and orientation of the HMD. For example, the controller 18 may detect the position and orientation of the HMD based on the image of the HMD captured by the in-vehicle camera 16, or it may detect the position and orientation of the HMD using a positioning sensor built into the HMD. Based on the detected position and orientation, the controller 18 may calculate a correspondence between the position in the real space around the vehicle 1 and the display position of the virtual image displayed on the HMD so as to overlap with this position in the real space. Based on the calculated correspondence, the controller 18 may calculate a display position where the virtual image is displayed overlapping the object placement position, and draw the virtual object at the calculated position.

[0029] Next, the problems of the embodiment will be explained with reference to Figures 2(a) and 2(b). Now, let's take the example of when vehicle 1 is traveling towards intersection C, as shown in Figure 2(a). The content presentation device 10 presents mixed reality content such that virtual objects Ob1 and Ob2 appear superimposed on pre-set standard object placement positions around vehicle 1.

[0030] In the following explanation, a pre-set standard object placement position may be referred to as the "standard position." For example, the standard position may be a predetermined distance in front of vehicle 1. Furthermore, in the following inventions, presenting mixed reality content such that a virtual object appears superimposed on a specific location in real space may be simply referred to as "placing a virtual object."

[0031] If the traffic signal S changes to red and vehicle 1 stops before intersection C, simply stopping the movement of the content placement positions of virtual objects Ob1 and Ob2 to match vehicle 1 will result in virtual objects Ob1 and Ob2 being placed inside intersection C and pedestrian crossing Pc, respectively, as shown in Figure 2(b). If virtual objects Ob1 and Ob2 are placed in positions where they are likely to collide with other moving objects in the real world (for example, other vehicles V1 and V2 or pedestrians Pd), users may perceive the virtual objects Ob1 and Ob2 as unnatural, potentially losing their sense of reality ("the feeling of being there").

[0032] Therefore, the content presentation device 10 of the embodiment detects environmental information of the environment surrounding the vehicle 1 and estimates the factors that may cause a collision with other moving objects in the real world (e.g., other vehicles V1, V2 or pedestrians Pd) based on the environmental information. For example, if vehicle 1 is stopped before an intersection due to a red light, the lane that intersects with vehicle 1's lane at the intersection, as well as the pedestrian crossing at the intersection, are estimated as factors that increase the likelihood of a collision (hereinafter referred to as collision likelihood factors).

[0033] The content display device 10 of this embodiment predicts the movement of a moving object (e.g., other vehicles V1, V2 or pedestrian Pd) according to estimated collision probability factors, selects an avoidance position in real space that will not collide with the moving object based on the predicted movement of the moving object, and displays virtual objects Ob1 and Ob2 superimposed on the avoidance position.

[0034] This reduces the unnaturalness of virtual objects Ob1 and Ob2 being placed in locations where they are likely to collide or interfere with real-world moving objects. Furthermore, by pre-determining the probability of collisions or interference between real-world moving objects and virtual objects, it is possible to reduce the frequency of failures in the calculation process (image recognition of moving objects and rendering of virtual objects Ob1 and Ob2) that would otherwise cause users to feel uneasy, thereby preventing collisions or interference between virtual objects Ob1 and Ob2 and fast-moving objects (e.g., other vehicles V1).

[0035] Refer to Figure 1. The functional configuration of the controller 18 will be described. Refer to Figure 1. The controller 18 comprises a collision probability prediction unit 20, a display position determination unit 21, a display format changing unit 22, and a display control unit 23. The collision probability prediction unit 20 detects the driving environment around the vehicle 1 based on environmental information detected by the external sensor 11, the current position of the vehicle 1 measured by the positioning device 14, and map information from the map DB 13.

[0036] For example, the collision probability prediction unit 20 may perform image recognition processing on the video footage captured by the surrounding environment camera to detect the road structure around the vehicle 1 (intersections, merging sections, lanes, etc.), surrounding landmarks (traffic signals, road signs, pedestrian crossings, etc.), and the behavior of other traffic users (surrounding vehicles and pedestrians) as part of the driving environment.

[0037] For example, the collision probability prediction unit 20 may detect the road structure and surrounding features of the vehicle 1 as part of the driving environment based on the current position of the vehicle 1 measured by the positioning device 14 and the map information in the map DB 13. Furthermore, the collision probability prediction unit 20 detects the vehicle behavior of vehicle 1 based on the vehicle information detected by the vehicle sensor 12.

[0038] The collision probability prediction unit 20 estimates collision probability factors, which are factors that make it possible for the vehicle 1 to collide with a real-world moving object other than the vehicle 1, based on the driving environment around the vehicle 1 and the vehicle behavior of the vehicle 1. When a collision possibility factor is detected, the collision possibility prediction unit 20 determines that the current driving scene of vehicle 1 is a scene in which the object placement position of virtual objects should be corrected from the standard position.

[0039] For example, the collision probability prediction unit 20 may estimate, as a collision probability factor, an area in real space where a moving object from the real world is likely to pass near the vehicle 1. In the following description, the area in real space where a moving object from the real world is likely to pass near the vehicle 1 may be referred to as the "expected traffic area".

[0040] Refer to Figure 3(a). Now, let us illustrate a case where vehicle 1 is traveling towards intersection C, and the content display device 10 is displaying mixed reality content such that virtual objects Ob1 and Ob2 appear superimposed at a standard position a predetermined distance in front of vehicle 1. When the collision probability prediction unit 20 detects that the traffic signal S at intersection C for vehicle 1 has changed to a yellow light, it determines that the current driving scene of vehicle 1 is the target scene and may estimate the intersecting lanes that cross vehicle 1's driving lane at intersection C, the oncoming lanes, and the area within intersection C as expected traffic areas where other vehicles are likely to pass near vehicle 1. The area At1 enclosed by the dashed line in Figure 3(a) indicates the expected traffic area where other vehicles are likely to pass in the vicinity of vehicle 1.

[0041] Furthermore, when the vehicle 1 detects that the traffic signal S at intersection C has changed to a yellow light, the collision probability prediction unit 20 may determine that the current driving scene of vehicle 1 is the target scene and estimate the pedestrian crossing at intersection C as a predicted traffic area where pedestrians are likely to pass near vehicle 1. The area At2 enclosed by the dashed line in Figure 3(a) indicates the expected traffic area where pedestrians are likely to pass in the vicinity of vehicle 1.

[0042] The collision probability prediction unit 20 may set the predicted traffic area At1 when it actually detects other vehicles traveling in the intersecting lane or oncoming lane, or it may set the predicted traffic area At1 even if it does not actually detect any vehicles. Similarly, the predicted traffic area At2 may be set when it actually detects pedestrians attempting to cross the crosswalk at intersection C, or it may set the predicted traffic area At2 even if it does not actually detect any pedestrians.

[0043] Refer to Figure 4(a). Now, let us illustrate a case where vehicle 1 is traveling on the main lane towards merging section M, and the content display device 10 is displaying mixed reality content such that a virtual object Ob1 appears superimposed at a standard position a predetermined distance ahead of vehicle 1. When the collision prediction unit 20 detects that vehicle 1 is approaching the merging section M, it may determine that the current driving scene of vehicle 1 is the target scene and estimate the area on the main lane near the exit of the merging lane as the expected traffic area At1 where another vehicle V1 is likely to pass near vehicle 1.

[0044] For example, the collision probability prediction unit 20 may estimate an area on the main lane where there is a high probability that another vehicle V1 in the merging lane will cut in front of vehicle 1 as the expected traffic area At1 where the other vehicle V1 is likely to pass near vehicle 1. The collision probability prediction unit 20 may set the predicted traffic area At1 when it actually detects another vehicle V1 traveling in the merging lane, or it may set the predicted traffic area At1 even if it does not actually detect any other vehicle V1.

[0045] Figure 5(a) is a schematic diagram of an example where vehicle 1 stops to give way to other traffic users (other vehicles and pedestrians). When the collision probability prediction unit 20 detects that vehicle 1 has stopped before a crosswalk or intersection, it may determine that the current driving scene of vehicle 1 is the target scene and estimate the expected traffic area in the vicinity of vehicle 1 where other traffic users who have been given way by vehicle 1 are likely to pass.

[0046] In the example shown in Figure 5(a), since vehicle 1 is stopped before the pedestrian crossing Pc, the pedestrian crossing Pc may be estimated as the expected traffic area At2 where pedestrian Pd is likely to pass near vehicle 1. The collision probability prediction unit 20 may set the expected traffic area At2 when it actually detects a pedestrian Pd crossing the pedestrian crossing Pc, or it may set the expected traffic area At2 even if it does not actually detect a pedestrian.

[0047] Next, the collision probability prediction unit 20 predicts the movement of moving objects passing through the predicted traffic areas At1 and At2. For example, the collision probability prediction unit 20 may predict the range and direction of movement of the moving objects as the movement of the moving objects. For example, the collision probability prediction unit 20 may predict the expected traffic areas At1 and At2 as the range of movement of the moving object.

[0048] For example, in the example shown in Figure 3(a), the direction of movement of vehicles traveling in intersecting lanes or oncoming lanes that intersect with the lane in which vehicle 1 is traveling may be predicted as the direction of movement of moving objects (other vehicles) passing through the predicted traffic area At1. Similarly, the direction of movement of pedestrians crossing a crosswalk may be predicted as the direction of movement of other moving objects (pedestrians) passing through the predicted traffic area At2.

[0049] For example, in the example shown in Figure 4(a), the direction of movement of a vehicle changing lanes from a merging lane to a main lane, or a vehicle immediately after changing lanes, may be predicted as the direction of movement of other moving objects (other vehicles) passing through the predicted traffic area At1. For example, in the example shown in Figure 5(a), the direction of movement of a pedestrian crossing the crosswalk Pc may be predicted as the direction of movement of other moving objects (pedestrians) passing through the expected traffic area At2.

[0050] Refer to Figure 1. The display position determination unit 21 selects a position in real space that will not collide with the moving object as an avoidance position, based on the movement of the moving object (i.e., the range of movement and direction of movement) predicted by the collision possibility prediction unit 20. For example, the display position determination unit 21 may select a position as an avoidance position where there is a low probability that other moving objects will pass through. The display position determination unit 21 corrects the object placement position from the standard position where the virtual object is currently placed to the avoidance position.

[0051] Figure 3(b) shows the state after correcting the positions of the virtual objects Ob1 and Ob2 in Figure 3(a) to avoidance positions. For example, the display position determination unit 21 may select as the avoidance position the position closest to the standard position of the virtual object Ob1 and Ob2 before correction, among the positions outside the expected traffic areas At1 and At2. Alternatively, the display position determination unit 21 may select as the avoidance position the position closest to vehicle 1, among the positions outside the expected traffic areas At1 and At2.

[0052] Figure 4(b) shows the state after correcting the position of the virtual object Ob1 in Figure 4(a) to the avoidance position. For example, the display position determination unit 21 may select as the avoidance position the position closest to the standard position of the virtual object Ob1 before correction, among the positions outside the expected traffic area At1. For example, the display position determination unit 21 may select an avoidance position in front of vehicle 1 on the adjacent lane L2, which is adjacent to the opposite side of the merging lane across the vehicle 1's driving lane L1. The longitudinal distance between the avoidance position and vehicle 1 may be the same as the longitudinal distance between the standard position before correction and vehicle 1.

[0053] Figure 5(b) shows the state after correcting the position of the virtual object Ob1 in Figure 5(a) to the avoidance position. For example, the display position determination unit 21 may select the position closest to the standard position of the virtual object Ob1 before correction, among the positions outside the expected traffic area At2, as the avoidance position. For example, the display position determination unit 21 may select the avoidance position to be further back than the pedestrian crossing Pc.

[0054] As another example, the collision probability prediction unit 20 may detect a scene where, for example, a vehicle 1 turning left at an intersection is stopped and there are pedestrians crossing the crosswalk ahead of the left turn. In this case, the display position determination unit 21 may select an avoidance position on the other side of the crosswalk ahead of the left turn.

[0055] For example, the collision probability prediction unit 20 may detect a scene where, for instance, a vehicle 1 turning right at an intersection stops within the intersection. In this case, the display position determination unit 21 may select an avoidance position near vehicle 1. This is because moving away from vehicle 1 could interfere with the expected traffic area of ​​oncoming vehicles or vehicles turning left.

[0056] For example, the collision probability prediction unit 20 may detect a scene in which, for instance, vehicle 1 stops at a level crossing because the barrier is down, and estimate the area within the level crossing where the barrier is down as a predicted passage area where a railway vehicle is likely to pass. In this case, the display position determination unit 21 may select an avoidance position on the other side of the level crossing.

[0057] For example, the collision probability prediction unit 20 may detect a scene in which vehicle 1 passes through a highway toll booth as the target scene. In this case, the display position determination unit 21 may select an avoidance position on the other side of the toll booth's barrier. In this case, the display control unit 23, described later, may display an image of the virtual object passing under the barrier. The display form changing unit 22, described later, may reduce the display size of the virtual object when the virtual object passes under the barrier.

[0058] Alternatively, the collision probability prediction unit 20 may detect a scene in which vehicle 1 passes through a roundabout as the target scene. In this case, the display position determination unit 21 may select an avoidance position near vehicle 1, or it may select an avoidance position in the central part of the roundabout.

[0059] Refer to Figure 1. If the display position determination unit 21 cannot select an avoidance position, that is, if it cannot select a ground position where the virtual object is unlikely to collide with a moving object as an avoidance position, the display form changing unit 22 changes the display form of the virtual object. For example, if a virtual object has the attribute of being placed on the ground, and it is not possible to select a ground avoidance position where the virtual object is unlikely to collide with a moving object if the current display size is maintained, the display mode change unit 22 may reduce the display size of the virtual object.

[0060] Figures 6(a) and 6(b) are schematic diagrams of examples of changes in the display format of virtual objects. In the example in Figure 6(a), a vehicle 1 is traveling toward intersection C, and the content presentation device 10 presents mixed reality content such that the virtual object Ob1 appears superimposed on a standard position a predetermined distance in front of the vehicle 1. When the collision prediction unit 20 detects that the traffic signal S at intersection C for vehicle 1 has changed to a yellow light, it determines that the current driving scene of vehicle 1 is the target scene and estimates the predicted traffic areas At1 and At2.

[0061] The virtual object Ob1 in Figure 6(a) has the attribute of being placed on the ground, and if the current display size is maintained, the display position determination unit 21 cannot select a ground avoidance position where the virtual object Ob1 is unlikely to collide with a moving object. Therefore, the display format changing unit 22 reduces the display size of the virtual object Ob1, as shown in Figure 6(b). The display position determination unit 21 selects a ground position as the avoidance position where the reduced virtual object Ob1 is unlikely to collide with a moving object.

[0062] Alternatively, the display mode changing unit 22 may increase the transparency of the virtual object instead of reducing its display size. In this case, the display control unit 23, described later, will display the virtual object semi-transparent.

[0063] Furthermore, if a virtual object has attributes that allow it to be placed in the air (for example, if it has wings or is riding on a means of flight), the display position determination unit 21 may select an aerial position with a low probability of collision with a moving object as the avoidance position. For example, if a virtual object that can be placed in the air cannot select an aerial position on the ground with a low probability of collision with a moving object, the aerial position may be selected as the avoidance position.

[0064] Refer to Figure 1. The display control unit 23 displays the virtual object on the HMI 17's display device so that the virtual object appears superimposed on the avoidance position determined by the display position determination unit 21. If the display device is an instrument panel, navigation device, IVI device-mounted display device, or video see-through type HMD, it displays an image in which the virtual object is superimposed on the avoidance position within the image of the vehicle 1's surroundings.

[0065] Furthermore, if the display format change unit 22 changes the display format of a virtual object, the display control unit 23 displays the virtual object after the change. On the other hand, if reducing the display size of the virtual object does not allow for the selection of a ground avoidance position where the virtual object is unlikely to collide with the moving object, the display control unit 23 will superimpose the virtual object onto the standard position. In this case, the virtual object will be displayed in a position where it is unlikely to collide with the moving object.

[0066] Furthermore, if a real-world moving object collides with a virtual object, the display control unit 23 displays an image of the virtual object performing a predetermined action (for example, the virtual object being thrown off by the moving object or the virtual object disappearing) to resolve the situation where the virtual object is superimposed on the moving object.

[0067] (operation) Figure 7 is a flowchart of an example of a display method in the embodiment. In step S1, the external sensor 11 detects environmental information of the environment surrounding the vehicle 1. In step S2, the collision probability prediction unit 20 estimates collision probability factors, which are factors that make it possible to collide with real-world moving objects other than vehicles, based on environmental information. In step S3, the collision probability prediction unit 20 predicts the movement of the moving object based on the collision probability factors.

[0068] In step S4, the collision probability prediction unit 20 calculates the likelihood of a collision between the moving object and the virtual object based on the predicted movement of the moving object. The collision probability prediction unit 20 calculates the areas where the virtual object is unlikely to collide with the moving object. In step S5, the display position determination unit 21 determines whether or not there is an area where the virtual object is unlikely to collide with the moving object.

[0069] If there is no area where the virtual object is unlikely to collide with the moving object (step S5:N), the process proceeds to step S7. If there is an area where the virtual object is unlikely to collide with the moving object (step S5:Y), the process proceeds to step S6. In step S6, the display position determination unit 21 selects an avoidance position in an area where the virtual object is unlikely to collide with the moving object. The display control unit 23 places the virtual object at the avoidance position determined by the display position determination unit 21 and displays the virtual object so that it is superimposed on the avoidance position. The process then proceeds to step S10.

[0070] In step S7, the display position determination unit 21 determines whether the virtual object can be placed in an area where collision with a moving object is unlikely by changing the display form of the virtual object (for example, by reducing the display size). If the virtual object cannot be placed (step S7:N), the process proceeds to step S9. If the virtual object can be placed (step S7:Y), the process proceeds to step S8.

[0071] In step S8, the display format change unit 22 changes the display format of the virtual object (for example, by reducing the display size). The process then proceeds to step S6. In step S9, the display control unit 23 places the virtual object at a standard position (i.e., in an area where it is likely to collide with a moving object). The display control unit 23 displays the virtual object so that it is superimposed on the standard position. The process then proceeds to step S10.

[0072] In step S10, the display control unit 23 determines whether a collision has occurred between a moving object in the real world and a virtual object. If no collision occurs (step S10:N), the process ends. If a collision occurs (step S10:Y), the process proceeds to step S11. In step S11, the display control unit 23 displays an image of the virtual object performing a predetermined action to resolve the situation in which the virtual object is superimposed on the moving object. The process then ends.

[0073] (Effects of the embodiment) (1) The content display device 10 displays mixed reality content or augmented reality content in which virtual objects are superimposed on video footage of the area around the vehicle or the real-world scenery around the vehicle. The controller 18 detects environmental information of the environment around the vehicle, estimates factors that may cause collision with real-world moving objects other than the vehicle based on the environmental information, predicts the movement of the moving objects according to the estimated factors, selects an avoidance position in real space that will not collide with the moving objects based on the predicted movement of the moving objects, and may superimpose and display virtual objects at the avoidance position.

[0074] This reduces the unnaturalness of virtual objects being placed in locations where they are likely to collide or interfere with real-world moving objects. Furthermore, by pre-determining the probability of collisions or interference between real-world moving objects and virtual objects, it is possible to reduce the frequency of failures in the calculation process (image recognition of moving objects and rendering of virtual objects) that would otherwise cause users to feel uneasy when virtual objects collide or interfere with fast-moving objects.

[0075] (2) The controller 18 may estimate an area in real space where a moving object is likely to pass near the vehicle as a factor to be estimated, and select a position outside of that area as an avoidance position. This reduces the unnaturalness of virtual objects being placed in locations where they are likely to collide or interfere with real-world moving objects.

[0076] (3) If the controller 18 cannot select a ground position where the virtual object placed on the ground is unlikely to collide with a moving object as an avoidance position, it may reduce the display size of the virtual object or display the virtual object as semi-transparent. This reduces the likelihood of virtual objects colliding with moving objects, and also reduces the unnatural appearance caused by virtual objects being superimposed on moving objects in the event of a collision.

[0077] (4) If the controller 18 cannot select a ground position as an avoidance position where the virtual object that can be placed in the air is unlikely to collide with the moving object, it may select an air position as an avoidance position where the virtual object is unlikely to collide with the moving object. This reduces the likelihood of virtual objects colliding with moving objects.

[0078] (5) When the moving object and the virtual object collide, the controller 18 may resolve the situation in which the virtual object is superimposed on the moving object by displaying the virtual object that performs a predetermined action. This avoids the unnatural appearance caused by virtual objects being superimposed on moving objects. [Explanation of Symbols]

[0079] 1...Vehicle, 10...Content display device, 11...External environment sensor, 12...Vehicle sensor, 13...Map database, 14...Positioning device, 15...Communication device, 16...In-vehicle camera, 17...Human-machine interface, 18...Controller, 20...Collision probability prediction unit, 21...Display position determination unit, 22...Display format change unit, 23...Display control unit

Claims

1. A mixed reality or augmented reality display method that superimposes virtual objects onto video footage of the area around a vehicle or the actual scenery around the vehicle and displays them on a display device, The system detects environmental information of the surrounding environment of the vehicle. Based on the aforementioned environmental information, the factors that may cause a collision with a real-world moving object other than the vehicle are estimated. The movement of the moving object is predicted according to the estimated factors, Based on the predicted movement of the moving object, select an avoidance position in real space that will not collide with the moving object. The virtual object is displayed superimposed on the avoidance position. A method of display characterized by the following features.

2. As the aforementioned estimated factors, the region in real space where the moving object is likely to pass near the vehicle is estimated. A position outside the aforementioned region is selected as the avoidance position. The display method according to feature 1.

3. The display method according to claim 1, characterized in that, if a ground position where the virtual object placed on the ground is unlikely to collide with the moving object cannot be selected as the avoidance position, the display size of the virtual object is reduced or the virtual object is displayed semi-transparently.

4. The display method according to claim 1, characterized in that, if a ground position where the virtual object that can be placed in the air is unlikely to collide with the moving object cannot be selected as the avoidance position, the display method according to claim 1 is characterized in that a position in the air where the likelihood of collision with the moving object is unlikely is selected as the avoidance position.

5. The display method according to claim 1, characterized in that when the moving object and the virtual object collide, the virtual object performs a predetermined action, thereby resolving the state in which the virtual object is superimposed on the moving object.

6. A mixed reality or augmented reality content display device that overlays virtual objects onto video footage of the area around a vehicle or onto the actual scenery around the vehicle, Display device and A sensor that detects environmental information of the environment surrounding the vehicle, Based on the environmental information, the factors that may cause a collision with a real-world moving object other than the vehicle are estimated, the movement of the moving object is predicted according to the estimated factors, and based on the predicted movement of the moving object, an avoidance position in real space that will not collide with the moving object is selected. A controller that superimposes the virtual object onto the avoidance position and displays it on the display device, A content presentation device characterized by comprising the following features.

Images