Immersive virtual display

Abstract

The present disclosure relates to immersive virtual displays. The present invention discloses a VR system for a vehicle, which can implement a method to solve the problem that a moving vehicle may cause motion sickness in passengers. The VR system can provide a virtual view that matches visual cues with the body movement experienced by the passenger. The VR system can provide an immersive VR experience by replacing the real world view with a virtual environment view. Active vehicle systems and / or vehicle control systems can be integrated with the VR system to provide physical effects for the virtual experience. When it is determined that the passenger is prone to motion sickness or is showing signs of motion sickness, the virtual environment can be changed to accommodate the passenger.

Description

[0001] This application is a divisional application of the invention patent application with application date of September 22, 2017, application number 201780057756.1, and name “Immersive Virtual Display”. Background Art

[0002] Virtual reality (VR) allows users to experience and / or interact with an immersive artificial environment so that the user feels as if they are physically in that environment. For example, a virtual reality system may display a stereoscopic scene to a user to create an illusion of depth, and a computer may adjust the scene content in real time to provide the illusion that the user is moving within the scene. When a user views the images through a virtual reality system, the user may thus feel as if they are moving within the scene from a first-person perspective. Similarly, mixed reality (MR) combines computer-generated information (referred to as virtual content) with real-world images or real-world views to enhance or add content to the user's view of the world. Summary of the invention

[0003] Various embodiments of methods and apparatus for providing an enhanced virtual reality (VR) and / or mixed reality (MR) experience to passengers in a vehicle are described. Embodiments of a VR system are described that may implement VR methods to, for example, address the problem that a vehicle in motion may cause motion sickness to passengers. Embodiments of a VR system may provide an immersive VR experience to passengers in a vehicle, such as by replacing a real-world view with any of a variety of types of virtual experiences and environments that a passenger may desire. Vehicle motion may be integrated into the virtual experience to help prevent motion sickness. In some embodiments, active vehicle systems within constraints and / or vehicle control systems may be integrated with the VR system to provide physical effects with a virtual experience, such as rushing out wind or heat through an HVAC system, surround sound and sound effects through an audio system, and acceleration or motion effects through a seat.

[0004] In addition to reducing motion sickness, embodiments of the VR system may also provide passengers in a moving vehicle with an enhanced immersive virtual experience that is not achievable in conventional fixed VR systems. Integrating a VR system with a moving vehicle provides an opportunity to enhance the virtual experience that cannot be experienced while using a fixed simulator or sitting in a room wearing an HMD. For example, acceleration and motion in the virtual experience may match or be enhanced by the acceleration and motion of the vehicle, and therefore do not have to be simulated using a gravity vector as in a fixed simulator.

[0005] In some embodiments, the VR system may include at least one VR controller that generates virtual content projected to the passenger, and at least one VR projection or display device, such as a headset, helmet, goggles, or glasses (referred to herein as a head-mounted display (HMD)), which includes a projector mechanism for projecting or displaying a frame including a left image and a right image to the passenger's eyes, thereby providing a 3D virtual view to the passenger. Alternatively, in some embodiments, the VR system may include at least one VR controller that generates virtual content projected to the passenger, and at least one projector mechanism for projecting or displaying the virtual frame to at least one window of the vehicle, thereby providing a 3D virtual view to the passenger. In either case, the 3D virtual view may include a view of the passenger's environment (including the environment outside the vehicle) enhanced with virtual content (e.g., virtual objects, virtual labels, etc.), or may provide an immersive VR environment that may include visual cues of the environment outside the vehicle.

[0006] Embodiments of the VR system may integrate inputs from multiple sources, including but not limited to vehicle internal and external sensors (e.g., depth cameras (e.g., LiDAR) and video cameras), vehicle and HMD inertial measurement units (IMUs), vehicle control systems (such as throttle control, braking, steering, navigation, and active suspension systems), world maps, 3D models, video, audio, and other information from external sources (such as cloud-based storage or web-based applications); video or audio input from the vehicle AV system and user devices such as laptops, tablets, or smartphones to generate immersive virtual content for output through an HMD or other VR projection system. In some embodiments, the VR system may also generate signals to the vehicle control systems (e.g., to control braking, acceleration, steering, or suspension / motion within constraints) and vehicle activity systems (e.g., audio and HVAC systems, and active seats) to provide physical effects synchronized with the projected virtual content, thereby further enhancing the passenger experience.

[0007] When using a VR system in a moving vehicle, some passengers may experience motion sickness more easily than other passengers. In some embodiments, the VR system may adjust the VR environment and experience to accommodate the preferences and tendencies of different passengers. For example, visual cues indicating passing passengers may be slowed down or accelerated when compared to the actual speed or acceleration of the vehicle to accommodate the preferences and tendencies of different passengers. In some embodiments, the VR system may monitor passengers for signs of discomfort or motion sickness. If signs of motion sickness are detected, the VR system may adjust the virtual environment to reduce motion sickness, such as by slowing down visual cues to have a 1:1 mapping ratio with the actual vehicle speed or acceleration. In addition to adjusting the mapping ratio of the passenger to help prevent motion sickness, one or more other visual and auditory techniques or cues (referred to as adjustments) may also be used in the VR experience to increase the comfort of passengers using the VR system in the vehicle and reduce their motion sickness. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A virtual reality (VR) system according to some embodiments is shown, which includes a head-mounted device (HMD) that can be used by passengers in a vehicle.

[0009] Figure 2 A VR system according to some embodiments is shown that projects VR content onto the windows of a vehicle for passengers to view.

[0010] Figure 3 Projected VR content according to some embodiments is shown so that the VR content appears to the viewer to be in the space in front of the vehicle.

[0011] Figure 4 Projected VR content according to some embodiments is shown so that it appears to the viewer that the VR content is in space within the vehicle's real-world view.

[0012] Figure 5 Projected VR content according to some embodiments is shown so that the VR content appears to the viewer to be in space in a simulated view of the vehicle.

[0013] Figure 6 A VR system in a vehicle is shown according to some embodiments.

[0014] Figure 7 is a block diagram illustrating components of a VR system in a vehicle according to some embodiments.

[0015] Figure 8 is a block diagram illustrating an activity system in a vehicle that may be used in a VR system to provide synchronized physical effects to passengers, according to some embodiments.

[0016] Figures 9 to 11Several exemplary VR experiences that may be provided by embodiments of the VR system are shown and are not intended to be limiting.

[0017] Fig.12 is a high-level flow chart of a method for providing a VR experience to passengers in a vehicle, according to some embodiments.

[0018] Fig.13 is a flowchart of a method for providing virtual content in a mixed reality view of an environment to passengers in a vehicle according to some embodiments.

[0019] Fig.14 is a flow chart of a method for providing an immersive VR experience with environmental effects to passengers in a vehicle according to some embodiments.

[0020] Figures 15 to 17 Graphically illustrates visual accommodations that may be used in a VR experience according to some embodiments.

[0021] Fig.18 A VR system for monitoring passengers using a VR system in a vehicle is shown according to some embodiments.

[0022] Fig.19 A VR application is shown that adapts the VR experience to a passenger based on passenger preferences and passenger sensor data according to some embodiments.

[0023] Fig. 20 is a flow chart of a method for adapting a VR experience to a passenger based on passenger preferences and passenger sensor data, according to some embodiments.

[0024] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0025] "Comprising". This term is open ended. As used in the appended claims, this term does not exclude additional structures or steps. Consider the following cited claim: "A device comprising one or more processor units..." Such a claim does not exclude the device from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

[0026] "Configured to". Various units, circuits, or other components may be described or recited as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component may be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware - such as circuits, memories storing executable program instructions to implement operations, etc. Reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke 35 USC §112(f) for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software and / or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0027] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0028] "Based on". As used herein, this term is used to describe one or more factors that influence a determination. This term does not exclude additional factors that influence a determination. That is, a determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B". In this case, B is a factor that influences the determination of A, and such a phrase does not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B. DETAILED DESCRIPTION

[0029] Various embodiments of methods and apparatus for providing an enhanced virtual reality (VR) experience for passengers in a vehicle are described herein. Although the embodiments herein generally refer to a VR system including a VR controller and a VR projection device that provides an enhanced VR experience for passengers in a vehicle, it is noted that some embodiments of the VR system may instead of or also provide a mixed reality (MR) experience for passengers in a vehicle.

[0030] As used herein, the term "vehicle" generally refers to a passenger vehicle. Similarly, as used herein, "passenger" generally refers to an occupant of a vehicle. Embodiments of the VR system as described herein may be implemented, for example, in an autonomous or "self-driving" vehicle in which all occupants are passengers. However, it should be noted that the embodiments may also be implemented in a conventional vehicle in which one of the occupants is a driver within the constraints. In addition, the embodiments may be implemented in other types of ground transportation (e.g., buses and trains), and may also be applicable to airplanes, helicopters, boats, ships, etc. As used herein, the term "vehicle" covers all of these use cases. It should also be noted that the embodiments are generally described as providing an enhanced VR experience for passengers in a moving vehicle. However, in some embodiments, the VR system may be used in a fixed vehicle to provide an interesting VR experience to the user. In addition, in some embodiments, the enhanced VR experience in a moving vehicle provided by the VR system may be recorded for later playback by the person having the initial experience or others, for example, when sitting in a living room, or when following a similar route that can be mapped to the experience. In addition, in some embodiments, two or more passengers in a vehicle, or one or more passengers in a vehicle and one or more people in other locations such as an office or home may use the VR system to participate in a virtual environment.

[0031] Many passengers in a vehicle may experience motion sickness. Typically, this does not happen to the driver. However, with the advent of autonomous vehicles, the driver becomes a passenger and may want to do something, such as while riding to work. For example, a passenger in a conventional or autonomous vehicle may want to read a book or work on a laptop. However, many passengers in a vehicle may experience motion sickness if they try to read a book or work on a computer because the vestibular sense of motion does not correspond to the visual motion experienced, for example, the vestibular organs report that the passenger is experiencing the acceleration of the vehicle, while the visual sense (primarily realized by the image of the book or computer screen) reports that the passenger is not moving or is moving at a speed that is very different from the vehicle. Most of their vision is filled with non-moving objects (e.g., books or computer screens), while their vestibular sense reports acceleration. Another physiological effect that may cause motion sickness is when the passenger's head tilts to the side, downward, or backward when going around a bend or during the rotation of other vehicles; the passenger may experience different and disorienting signals in response to the movement of the body's visual and vestibular systems.

[0032] With the advent of VR systems such as head-mounted displays (HMDs), passengers may wish to enjoy a VR experience while riding in a vehicle. However, conventional VR systems may also cause motion sickness in a moving vehicle because a person's body experiences motion that is different from the motion they see visually. In addition, many people experience motion sickness when sitting in the back seat of a vehicle for similar reasons, as their eyes may be focused on the interior of the car, and therefore the motion they experience may not be consistent with what they see visually because they cannot see the forward view outside the vehicle (if any). This problem may be more severe for passengers riding in rear-facing seats (such as may be provided in an autonomous vehicle). In addition, an autonomous vehicle may have limited or even no windows, and therefore the motion experienced by passengers in such a vehicle may not be consistent with the motion they see visually, thereby potentially causing motion sickness.

[0033] The present invention describes embodiments of a VR system that can implement VR methods to address these and other issues that a moving vehicle may cause motion sickness in passengers. Embodiments of the VR system can, for example, provide an augmented or virtual view that helps match visual cues to the motion that passengers in a moving vehicle are experiencing. In addition, embodiments of the VR system can project content that a passenger may want to watch or read (such as a computer screen, a book, or a video) as virtual content at a distance from the viewer (e.g., outside the vehicle or some distance in front of the vehicle), so that the virtual content appears as a stable or fixed distant object in the external environment, while the visual cues of the real environment move in the passenger's field of view, allowing the passenger to work, watch, or read comfortably without experiencing motion sickness that may occur if the passenger attempts to work or watch content on a physical screen on a portable computing device placed on their lap, or read a physical book or newspaper. Therefore, embodiments of the VR system can help improve productivity because passengers in a vehicle can perform work more comfortably while riding in the vehicle.

[0034] As described above, embodiments of the VR system may project virtual content for viewing at a distance from the passenger (e.g., outside the vehicle or at some distance in front of the vehicle), so that the virtual content appears as a stable or fixed distant object in the external environment, while the visual cues of the real environment move in the passenger's field of view. In some embodiments, the mapping ratio of these visual cues to the actual speed or acceleration of the vehicle can be transmitted at 1:1. However, in some embodiments, the visual cues can be slowed down or accelerated when compared to the actual speed or acceleration of the vehicle, which can, for example, help reduce motion sickness, or be used to provide other effects or sensations to passengers. For example, maintaining a 1:1 mapping ratio or slowing down the visual cues can help reduce motion sickness for passengers who suffer from motion sickness or show signs of motion sickness. As another example, for passengers who are not prone to or do not show signs of motion sickness, the visual cues can be accelerated to a mapping ratio of 1:2, for example, to provide passengers with an enhanced, more stimulating virtual experience.

[0035] In some embodiments, the VR system may include at least one VR controller that generates virtual content projected to the passenger, and at least one VR projection device, such as a headset, helmet, goggles or glasses (referred to herein as a head-mounted display (HMD)), which includes a projector mechanism for projecting or displaying a frame including a left image and a right image to the passenger's eyes, thereby providing a 3D virtual view to the passenger. If there are two or more passengers in the vehicle, the VR system may include more than one HMD; each passenger may wear an HMD controlled by a VR controller. Alternatively, in some embodiments, the VR system may include at least one VR controller that generates virtual content projected for the passenger, and at least one projector mechanism for projecting or displaying a frame including virtual content to the window of the vehicle, thereby providing a 3D virtual view to the passenger. In either case, the 3D virtual view may include a view of the passenger's environment (including the environment outside the vehicle) enhanced with virtual content (e.g., virtual objects, virtual labels, etc.), or may provide an immersive VR environment that may include visual cues of the environment outside the vehicle.

[0036] Embodiments of the VR system may integrate inputs from multiple sources, including but not limited to vehicle internal and external sensors (e.g., depth cameras (e.g., LiDAR) and video cameras), vehicle and HMD inertial measurement units (IMUs), vehicle control systems (such as throttle control, braking, steering, navigation, and active suspension systems), world maps, 3D models, video, audio, and other information from external sources (such as cloud-based storage or web-based applications); video or audio input from the vehicle AV system; input from applications on mobile multi-purpose devices (such as smartphones connected to the vehicle AV system); input from devices connected to the vehicle OBD-II port and user devices such as laptops, tablets, or smartphones to generate augmented or immersive virtual content for output through the HMD or other VR projection system. In some embodiments, the VR system may also generate signals to the vehicle control systems (e.g., controlling braking, acceleration, steering, or suspension / motion within constraints) and vehicle activity systems (e.g., audio and HVAC systems, and active seats) to provide physical effects synchronized with the projected virtual content, thereby further enhancing the passenger experience. When presenting virtual content for display, the VR system may synchronize the movement and acceleration of the virtual content with the movement and acceleration of the vehicle along a route in the real world based on input from one or more sources in the vehicle.

[0037] In addition to solving the problem of motion sickness for passengers in moving vehicles and providing benefits such as increased productivity for passengers in moving vehicles, embodiments of the VR system can provide immersive VR experiences to passengers in the vehicle, such as by replacing the real world view with any of the various types of virtual experiences and environments that passengers may desire. Vehicle motion can be integrated into the virtual experience to help prevent motion sickness or enhance the virtual experience. Integrating the VR system with a moving vehicle provides an opportunity to enhance the virtual experience that cannot be felt when using a fixed simulator or sitting in a room wearing an HMD. For example, the acceleration and motion in the virtual experience can be matched with or enhanced by the acceleration and motion of the vehicle. In some embodiments, active vehicle systems (e.g., HVAC systems, audio systems, and active seats) and / or vehicle control systems (e.g., brakes, throttles, steering, and active suspension systems) within the constraints can be integrated with the VR system to provide physical effects with virtual experiences, such as rushing out wind or heat through the HVAC system, generating surround sound and sound effects through the audio system, and acceleration or motion effects through the seat. Passengers can choose to enjoy a relaxing virtual experience while riding (such as floating in a river or soaring on a hang glider), or an exciting virtual experience (such as a car chase or driving through a post-apocalyptic wasteland attacked by zombies), or any virtual experience in between. Passengers can choose to enjoy a virtual experience of riding through other real locations (such as the streets of London, or through a fictional city or landscape). The virtual experience can be educational and interactive, such as allowing passengers to discover historical or other information about landmarks in the virtual view of the city they are experiencing. The virtual experience can be interactive in other ways, such as allowing passengers to pass other vehicles during a road race experience, or run past zombies in a post-apocalyptic landscape. For another example, if a vehicle is stopped at a red light or for other reasons while fleeing zombies in a post-apocalyptic landscape, the virtual experience can cause the vehicle to stop and not allow the car to restart until the light turns green to build suspense.

[0038] In some embodiments, virtual views of real or fictional characters may be integrated into the virtual experience provided by the VR system. For example, a virtual representation of an author or talk show host may appear to be sitting in a seat next to a passenger; the virtual author may be reading one of their books to the passenger, or a virtual talk show host may be sitting in a seat next to the passenger hosting the show, with their voice provided through the audio system. As another example, a passenger may experience riding on a flatbed truck to watch a band perform, with the band's music provided through the audio system.

[0039] In some embodiments, two or more passengers in a car can participate in the same virtual experience. For example, four passengers can each wear an HMD that projects a view as if the passengers were on a giant hang glider gliding over a landscape or city. The passengers' virtual experience can be interactive; for example, a passenger can drop a virtual water balloon onto a feature in the environment.

[0040] In some embodiments, an immersive VR experience can be used to increase productivity while riding in a vehicle while also providing an interesting VR experience for participants. For example, two or more people can hold a meeting around a table in a virtual environment (e.g., in a virtual room or on the bed of a flatbed truck) as an alternative. Alternatively, different participants can experience different VR environments for meetings; for example, for one participant, they may appear to be holding a meeting on the flatbed of a truck, while for another participant, they may appear to be holding a meeting in a room in an office. Participants can be in the same vehicle on their way to get off work, or in different vehicles. Or some participants may be in a vehicle on their way to get off work, while one or more other participants are already in the office, at home, or elsewhere. The view of the computer screen of one of the participants can be projected into the virtual environment for all participants to see. In some embodiments, participants have different virtual experiences based on their actual environment. For example, a participant who is already in the office can watch a meeting in a simulated office environment, while a participant who is still in his vehicle can watch a meeting in the bed of a virtual flatbed truck. For participants in different vehicles, the virtual experience can be different according to the different routes they follow in the real environment.

[0041] In some embodiments, VR systems in two or more autonomous vehicles can be used to provide interesting interactive experiences to passengers in different vehicles. For example, an autonomous vehicle can be taken to a track or a large open lot, and a VR system can be used to provide passengers with a virtual experience of the race. The VR system can guide the vehicle's control system (e.g., steering, braking, throttle, active suspension) within constraints to travel on the track or in the lot, while providing visual, auditory, and tactile VR experiences to passengers in the vehicle. For example, a passenger may visually, auditorily, and through the physical inputs generated by the VR system in the vehicle, appear to be sitting in a racing car, and other vehicles are displayed as other racing cars participating in the race. As another example, passengers may appear as if they are participating in a scene in an actual movie.

[0042] The foregoing provides only some examples of different enhanced or immersive virtual experiences that may be provided and used by a VR system as described herein and is not intended to be limiting.

[0043] Figure 1A virtual reality (VR) system according to some embodiments is shown, which includes a head-mounted device (HMD) that can be used by a passenger in a vehicle. In these embodiments, the VR system 100 in the vehicle includes a VR controller 110 (e.g., mounted under the dashboard) and a VR headset (HMD 112). The HMD 112 can implement any of various types of virtual reality projection technologies. For example, the HMD 112 can be a near-eye VR system that projects left and right images on a screen in front of the eyes of the user 190 viewed by the passenger 190, such as DLP (digital light processing), LCD (liquid crystal display), and LCoS (liquid crystal on silicon) technology VR systems. As another example, the HMD 112 can be a direct retinal projector system that scans the left and right images to the eyes of the passenger 190 pixel by pixel. In order to scan the image, the light beams generated by the left and right projectors are directed to the left and right reflective components (e.g., elliptical reflectors or holographic combiners) located in front of the eyes of the user 190; the reflective components guide the light beams to the user's eyes. To create a three-dimensional (3D) effect, virtual content 116 at different depths or distances in the 3D virtual view 114 is shifted left or right in both images as a function of distance triangulation, with closer objects shifting more than farther objects.

[0044] Passenger 190 may wear HMD 112 while working on, for example, user device 192 (e.g., notebook or laptop computer). Controller 110 and HMD 112 may be communicatively coupled via a wired (e.g., user may plug HMD 112 into a port (e.g., USB port) on a seat or console) or a wireless (e.g., Bluetooth) connection. Controller 110 and user device 192 may also be communicatively coupled via a wired (e.g., user may plug user device 192 into a port (e.g., USB port) on a seat or console) or a wireless (e.g., Bluetooth) connection. Video output from user device 192 may be provided to VR controller 110, which may then generate a video including a video provided to HMD 112. 112. HMD 112 may project the frame for passenger 190 to view. In some embodiments, the frame may be projected in a mixed or augmented reality view so that passenger 190 sees virtual content 116 in a view of the real environment, or alternatively may be projected into a virtual environment with other virtual content. If projected into a view of the real environment, virtual content 116 may appear fixed at a distance in front of the vehicle (e.g., a few meters, or even displayed as a giant display at the horizon) to mitigate effects that may cause motion sickness. If projected into a virtual environment, virtual content 116 may appear fixed at a distance in front of passenger 190, and visual cues (e.g., virtual markers moving past passenger 190) may be provided to mitigate effects that may cause motion sickness.

[0045] Figure 2A VR system according to some embodiments is shown that projects VR content onto the windows of a vehicle for viewing by passengers. In these embodiments, the VR system 200 in the vehicle includes a VR controller 210 (e.g., mounted below the dashboard) and a projector 220 system configured to project virtual content onto a window 208 (e.g., a windshield of the vehicle). In some embodiments, the window 208 may include technology such as waveguide technology or holographic combiner technology to improve the projection that can be achieved by projecting light onto a conventional glass window. Passenger 290 may work on a user device 292 (e.g., a notebook or laptop computer). The controller 210 and the projector 220 may be communicatively connected via a wired or wireless (e.g., Bluetooth) connection. The controller 210 and the user device 292 may also be communicatively coupled via a wired (e.g., a user may plug the user device 292 into a port (e.g., a USB port) on a seat or console) or a wireless (e.g., Bluetooth) connection. The video output from the user device 292 may be provided to the VR controller 210, which may then generate virtual content including the output of the device 292 (e.g., a window) in a frame provided to the projector 220. HMD 220 may project the frame onto window 208 for viewing by passenger 290. In some embodiments, the frame may be projected in a mixed or augmented reality view 214 so that passenger 290 sees virtual content 216 in a view of the real environment outside window 208, or alternatively may be projected into a virtual environment with other virtual content. If projected into a view of the real environment, virtual content 216 may appear fixed at a distance in front of the vehicle (e.g., a few meters, or even displayed as a giant display at the horizon) to mitigate effects that may cause motion sickness. If projected into a virtual environment, virtual content 216 may appear fixed at a distance in front of passenger 290, and visual cues (e.g., virtual markers moving past passenger 290) may be provided to mitigate effects that may cause motion sickness.

[0046] Although Figure 2 Projection onto window 208 (such as a windshield of the vehicle) is shown, but in some embodiments, one or more projectors 220 can be used to project the virtual content generated by the VR controller 210 onto one or more other windows in the vehicle (e.g., side windows or door windows, or rear windows), or onto all windows in the vehicle to provide a "surround" virtual experience. It should also be noted that in some embodiments, instead of or in addition to displaying virtual content on the windows of the vehicle, one or more display screens in the vehicle can be used to display the virtual content generated by the VR controller 210.

[0047] Figure 3Projected VR content 316 is shown according to some embodiments so that the viewer appears to be fixed in space in the real environment at a distance in front of the vehicle. Figure 3 As shown, virtual content 316 (e.g., a projection of a display or a display portion of a passenger's handheld computing device) may appear to the passenger to be fixed at a certain distance in front of the vehicle (e.g., several meters, or even displayed as a giant display at the horizon), for example to alleviate the effects that may cause motion sickness. In some embodiments, virtual content 316 may be partially transparent so that the passenger can view the scene behind content 316. Virtual content 316 may include, but is not limited to, various display windows (e.g., directories, browsers, web pages), productivity tools (e.g., word processors), email applications or email messages, messaging applications, game windows, videos (e.g., videos from video streaming applications), etc. Although embodiments are described as displaying virtual content 316 input from a passenger's personal device (such as a laptop), it is noted that content from other sources, such as a DVD or video from a vehicle AV system, may alternatively or also be displayed. player, or video or other content from an external (e.g., network-based) source.

[0048] Figure 4 Projected VR content is shown according to some embodiments so that it appears to the viewer that the VR content is in space within the vehicle's real-world view. Passengers 490 may be seated in seats 480 of the vehicle. Figure 1 or Figure 2 The VR system shown may project virtual content 416 into a real-world view 417 in front of or around the vehicle. In some embodiments, the real-world view 417 may be provided through a window of the vehicle. Alternatively, in some embodiments, the real-world view 417 may be provided by a camera on the vehicle; the VR system may composite the virtual content 416 with a video of the real world around or in front of the vehicle and provide the composite video to an HMD or projector for display to the passenger 490. Figure 4 As shown, virtual content 416 (e.g., a projection of a display of a passenger's handheld computing device 492) may appear to passenger 490 to be fixed at some distance in front of the vehicle (e.g., a few meters, or even displayed as a giant display at the horizon), for example to mitigate the effects of motion sickness that may be induced.

[0049] Figure 5 Projected VR content according to some embodiments is shown so that the viewer appears to be in space in a simulated view of the vehicle. Passengers 590 may be seated in seats 580 of the vehicle. Figure 1 or Figure 2The VR system shown in FIG. 5 can generate frames representing a virtual world 518 that includes virtual content 516 and replaces a real world view, and provide the frames to an HMD or projector for display to a passenger 590. Figure 5 As shown, virtual content 516 (e.g., a projection of a display of a passenger's handheld computing device 592) may appear to the passenger 590 to be fixed at a distance in front of the passenger (e.g., a few meters, or even displayed as a giant display at the horizon), for example to mitigate the effects that may cause motion sickness. In some embodiments, visual cues may be included in the virtual world (e.g., virtual objects that appear to be moving past the passenger 590) to help mitigate motion sickness, or to provide an interesting visual experience. Although not shown, in some embodiments, a virtual vehicle interior may be included in the virtual world; the virtual vehicle interior may be made to represent any type of vehicle that the passenger 590 wants to see, such as an exotic supercar, convertible, or luxury sedan. Other types of virtual vehicles may also be presented, such as motorcycles or helicopters, to provide an interesting or exciting experience to the passenger 590.

[0050] In some embodiments, virtual representations of the passenger 590's hands and user devices 592 (e.g., keyboards) may also be displayed in the virtual content 516 to assist the passenger 590 when using the devices 592. In some embodiments, virtual representations of the passenger 590 (including, but not limited to, human arms, hands, and legs) may also be displayed in the virtual content 516. The virtual representations of the passenger 590 may be presented to match a particular virtual experience. For example, if a user is experiencing a virtual tour of Rome while riding in a virtual chariot, in addition to showing the chariot and horses, the virtual representation may also show a person wearing a robe and sandals, with Roman jewelry on their arms and hands.

[0051] Figure 6 A VR system in a vehicle is shown according to some embodiments. Figure 6 It shows that Figure 1 The VR system shown includes HMD692; however, it should be noted that a similar configuration can be implemented as Figure 2 The VR system shown in FIG. Figure 6As shown, vehicle 600 (which may be but is not necessarily an autonomous vehicle) may include VR controller 610, vehicle systems 626 (e.g., vehicle control systems such as throttle, brake, steering, and active suspension systems, as well as navigation, HVAC, and AV systems), internal and external sensors (e.g., LiDAR for depth mapping, cameras for interior or exterior views, IMU, localization systems, etc.). Vehicle 600 may include one or more passenger seats 690. In this example, forward and rear-facing seats are used for exemplary purposes. Passenger 690A may sit in the rear-facing seat, while passenger 690B may sit in the forward-facing seat. Passengers 690A and 690B wear corresponding HMDs 692A and 692B.

[0052] In some embodiments, the VR controller 610 may obtain input from one or more sensors 624 and from one or more vehicle systems 626, may also obtain input from one or more external sources (e.g., cloud-based storage or web-based applications), and may also obtain input from a user device held by a passenger 690. The VR controller 610 may generate respective VR views 694A and 694B for passengers 690A and 690B based at least in part on the various inputs, and provide the VR views 694A and 694B to respective HMDs 692A and 692B for display to passengers 690A and 690B. Depending on the preferences of passengers 690A and 690B, the VR views 694A and 694B may be the same virtual or real environment, or may be different virtual environments. For example, passenger 694A may want to see a real-world view in front of the vehicle 600, so the VR controller 610 may generate a view from a forward-facing camera (which may, but does not necessarily, include synthetic virtual content), and provide the VR view 694A to the HMD 692A for display to passenger 690A. In some embodiments, the audio may be provided through headphones of HMDs 692A and 692B; alternatively, the audio may be provided through the audio system of vehicle 600. In some embodiments, VR controller 610 may also generate signals to control systems of vehicle 600 (e.g., to control braking, acceleration, steering, and / or suspension movement within constraints) and vehicle activity systems (e.g., audio and HVAC systems, and active seats) to provide physical effects synchronized with projected view 694, thereby further enhancing the passenger's experience.

[0053] As an exemplary use case, the VR system can be used to make rear-facing passenger 690A appear to be traveling forward. Some people tend to experience motion sickness when traveling backward. Using the VR system, passenger 690A can be shown either a real-world view of the front of the vehicle or a virtual view that they are traveling forward. For passenger 690, since they appear to be traveling forward when they are actually traveling backward, acceleration can be sensed as braking, braking can be sensed as acceleration, a right turn can be sensed as a left turn, and a left turn can be sensed as a right turn.

[0054] Figure 7 is a block diagram illustrating components of a VR system in a vehicle according to some embodiments. Figure 7 As shown, the vehicle 700 (which may be but is not necessarily an autonomous vehicle) may include a VR controller 710. The VR controller 710 may include one or more processors 712. The processor 712 may include one or more of various types of processors, CPUs, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), memories, and / or other components for processing inputs from various sources to generate VR content and other output signals. The VR controller 710 may also include a memory 713, which may, for example, store program instructions executable by the processor 712, and data that may be used by the program instructions, the program instructions being used to perform the functions of the VR controller 710 to process inputs from various sources and generate VR content and other output signals. The VR controller 710 may also include an interface 714 to various vehicle systems, external sources 790, VR projection devices 720, and passenger user devices 792. The interface 714 may include wired and / or wireless connections to various components.

[0055] One or more VR projection or display devices 720 (referred to herein as VR devices) may be coupled to the VR controller 710 via a wired or wireless communication connection. The VR projection device 720 may be, for example, Figure 1 The HMD shown, or may be Figure 2The projector system shown. In some embodiments, the VR projection device 720 may include one or more processors 722, a projector 724, a display 726, and one or more wired and / or wireless interfaces 729 for connecting to the VR controller 710, receiving input from the VR controller, and providing output to the VR controller. The processor 722 may include one or more of various types of processors, CPUs, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders (codecs), memories, and / or other components. It should be noted that the HMD may include two projectors 724 (one for each eye) that display or project virtual content to two displays 726 (e.g., two screens in a near-eye VR system or reflective lenses in a direct retinal projector system). In some embodiments, the VR projection device 720 may also include a memory 723, which may, for example, store program instructions that can be executed by the processor 722, and data that can be used by the program instructions, the program instructions being used to perform the functions of the VR projection device 720 to connect to the VR controller 710, communicate with the VR controller, and process input from the VR controller. In some embodiments, the VR projection device 720 may also include an IMU 728 for detecting the motion and orientation of the VR projection device 720 (e.g., HMD). In some embodiments, the VR projection device 720 may include or be connected to a personal audio output device 725, such as headphones or earbuds. If the VR projection device 720 is an HMD, the audio output 725 may be integrated into the HMD.

[0056] Although not shown, in some embodiments, the VR projection device 720 and / or the VR controller 710 may include one or more microphones for receiving voice input from a passenger; for example, the HMD may include a microphone for receiving voice input from a passenger wearing the device. Voice input may be used, for example, for voice control of the VR system, or for communicating with other passengers wearing the HMD, or for external communications such as phone calls and teleconferences through the VR controller 710 or the vehicle 700 system. In addition, in some embodiments, the VR projection device 720 and / or the VR controller 710 may include or be connected to an internal camera or other sensor for detecting the movements or gestures (e.g., hand, arm, or head gestures), eye movements, facial expressions, etc. of the passenger. In some embodiments, the detected gestures, movements, or expressions may be used as inputs to the VR system to affect the presentation and / or display of virtual content to individual passengers. In some embodiments, the detected gestures, movements, or expressions may be used as inputs to the VR system to present an animated avatar of the corresponding passenger in the virtual content. In addition, in some embodiments, the VR projection device 720 may include one or more external cameras, depth cameras, or other sensors that may be used as inputs to the VR system to affect the presentation and / or display of virtual content to the corresponding passenger.

[0057] In some embodiments, sensors and cameras in the VR projection device 720, at other locations in the vehicle 700, or on the passenger can be used to monitor the passenger for signs of motion sickness (e.g., pallor, sweating, fidgeting, swallowing, hiccups, pulse rate, breathing rate, eye movement, etc.). If early signs of motion sickness are detected, the VR system can be conservative and return to or maintain a 1:1 mapping to real-life motion and acceleration in the simulated view displayed to the passenger, and / or otherwise change the virtual experience presented to the passenger. If the passenger does not show signs of motion sickness, the VR system can be different from the 1:1 mapping, which makes the virtual experience more flexible. For example, visual and other cues that provide a sense of acceleration can be added to the virtual experience to make the passenger feel that they are accelerating at a faster speed than the vehicle actually accelerates, and / or visual and other cues that provide a sense of turning can be added to the virtual experience to make the passenger feel that they are turning at a faster speed than the vehicle actually turns. Figures 18 to 20 Further shown is monitoring passengers while using the VR system in a moving vehicle and adjusting the VR experience based on the passengers' physiological responses.

[0058] like Figure 7 As shown, the VR controller 710 may receive various inputs (e.g., positioning, acceleration, braking, steering, motion, orientation, video, depth map, etc.) from internal and external vehicle sensors and control systems 702. Internal and external vehicle sensors may include, but are not limited to, depth cameras (e.g., LiDAR), cameras, inertial measurement units (IMUs). Vehicle control systems may include, but are not limited to, throttle control, braking, steering, navigation, and active suspension systems. The VR controller 710 may also obtain input (e.g., video and / or audio) from one or more vehicle AV systems 706. The VR controller 710 may also obtain input (e.g., world map data, 3D models of the local environment, information about objects or features in the local environment, video streams, audio (e.g., radio or satellite broadcasts), etc.) from one or more external sources 790 (e.g., cloud-based storage or web-based applications). The VR controller 710 may also obtain input (e.g., video, audio, user input to a keyboard or other input device, etc.) from one or more user devices 792 such as a laptop or tablet / tablet device. The VR controller 710 may also obtain input from one or more VR projection devices 720 (eg, motion and / or orientation information from an IMU 728 of the HMD).

[0059] In some embodiments, the VR controller 710 may also obtain data collected during previous drives on the vehicle's route and use the data during the current drive on the route. For example, data for frequently driven routes may be stored to cloud storage and accessed by the VR controller 710 while the vehicle is traveling on the route. Thus, on a route that a passenger frequently drives, data from previous drives on the route may be used during the current drive on the route. In some embodiments, the VR controller 710 may also obtain data collected for the route by other vehicles that have driven on the route and also implement the VR system, and use the data as the vehicle drives on the route. For example, data for different vehicles driving on a route may be stored to cloud storage and accessed by the VR controller 710 while the vehicle is driving on the route.

[0060] The VR controller 710 may generate augmented virtual content for one or more VR projection devices 720 based at least in part on the various inputs, and provide the VR content as output to the devices 720 for display. In some embodiments, the VR controller 710 may also generate and / or receive audio for the virtual content, and provide the audio to the one or more VR projection devices 720 to be played while the virtual content is being displayed. For example, the virtual content may provide a passenger with a virtual view of a passenger's personal device (e.g., a computer display window) in a view of the real environment, such as Figure 3 and Figure 4 As shown, the content appears to the passenger to be fixed in space at a certain distance in front of the vehicle, which can, for example, help prevent motion sickness and thus allow the passenger to work while riding in the vehicle. Alternatively, the virtual content can provide an immersive VR environment and experience to the passenger, such as Figures 9 to 11 shown.

[0061] In some embodiments, the VR controller 720 may generate output signals to one or more vehicle control systems 702 (e.g., controlling braking, acceleration, steering, or suspension / motion within constraints) and vehicle activity systems (e.g., AV 706 and HVAC 708 systems and active seats 730) to provide physical effects synchronized with the virtual content, thereby further enhancing the passenger's virtual environment and experience.

[0062] Although embodiments are generally described in which the VR controller 710 renders virtual content and provides frames including the virtual content to one or more VR projection devices 720 for display or projection to respective passengers, in some embodiments, at least a portion of the rendering of the virtual content may be performed by the VR projection device 720. Additionally, in some embodiments, the VR controller 710 may be a component of the VR projection device 720, for example, the VR controller 710 may be built into the HMD and may communicate with vehicle systems and sensors via wired and / or wireless communication connections.

[0063] Figure 8 is a block diagram illustrating an activity system in a vehicle that may be used in a VR system to provide synchronized physical effects to passengers, according to some embodiments. Figure 8 An example is shown in which a passenger 790 wearing an HMD 720 sits in an active seat 730 of a stationary or moving vehicle including a VR controller 710; the passenger 790 may face forward or backward in the vehicle. Figure 8 As shown, in some embodiments, the VR controller 720 can generate signals to one or more vehicle control systems 702 (e.g., controlling braking, acceleration, steering, or suspension / motion within the constraints) and vehicle activity systems (e.g., AV 706 and HVAC 708 systems and active seats 730) to provide physical effects synchronized with the virtual content, thereby further enhancing the virtual environment and experience of the passengers. For example, the active seat 730 can be tilted to change the acceleration force felt by the front or rear passengers 790, such as to reduce or increase the acceleration sensation felt by the passenger 790. For example, tilting the active seat 730 backward can cause the front passenger 790 to feel a gravity vector pushing on his back, thereby causing or increasing the feeling of forward acceleration. For another example, in some embodiments, one or more vehicle control systems 702 (e.g., throttle, braking and / or active suspension control systems) can be controlled by the VR controller 710 within the constraints (possibly together with the active seat 730) to make the passengers 790 feel that they are sitting still in the room instead of moving in the vehicle. In some embodiments, VR controller 710 can predict bumps in the road (e.g., surface condition information from previously recorded vehicle travel routes, or surface information obtained from external sources while traveling the route), and move active seat 730 up or down and / or control one or more of the vehicle's control systems (e.g., throttle, brakes, and / or active suspension) within constraints to reduce or eliminate vibrations that passenger 790 may experience when the vehicle goes over bumps. For another example, VR controller 710 can instruct HVAC system 708 to increase its fan speed so that, for example, Fig.10 (B) shows an immersive VR environment such as a hang glider experience where passengers 790 feel the wind blowing over them, or increases the heat so that passengers 790 sitting in front of a virtual fireplace or campfire feel the heat from the fire. As another example, the VR controller 710 can play audio accompanying the VR experience through the vehicle's audio system 706 or through an audio system integrated into the HMD 720.

[0064] As an exemplary use case, the VR system can be used to make the rear-facing passenger 790 appear to be moving forward. Some people tend to experience motion sickness when moving backward. Using the VR system, the passenger 790 can be shown a real-world view of the front of the vehicle or a virtual view that they are moving forward. For the passenger 790, since they appear to be moving forward when they are actually moving backward, acceleration can be sensed as braking, braking can be sensed as acceleration, right turns can be sensed as left turns, and left turns can be sensed as right turns. In some embodiments, the active seat 730, HVAC system 708, audio system 706, and one or more vehicle control systems 702 (e.g., throttle, brakes, and / or active suspension control systems) can be controlled by the VR controller 710 within constraints to make the passenger 790 feel that they are moving forward when they are actually moving backward.

[0065] Figures 9 to 11 Several exemplary immersive VR experiences that can be provided by embodiments of the VR system are graphically illustrated, and are not intended to be limiting. Embodiments of the VR system can provide immersive VR experiences to passengers in a vehicle, such as by replacing the real world view with any of the various types of virtual experiences and environments that a passenger may desire. Vehicle motion can be integrated into the virtual experience, such as to help prevent motion sickness and enhance the virtual experience. Integrating the VR system with a moving vehicle provides an opportunity to enhance the virtual experience that cannot be felt when using a fixed simulator or sitting in a room wearing an HMD. For example, the acceleration and motion in the virtual experience can be matched with or enhanced by the acceleration and motion of the vehicle. In some embodiments, active vehicle systems within the constraints (e.g., HVAC systems, audio systems, and active seats) and / or vehicle control systems (e.g., brakes, throttles, steering, and active suspension systems) can be integrated with the VR system to provide physical effects with virtual experiences, such as rushing out wind or heat through the HVAC system, generating surround sound and sound effects through the audio system, and acceleration or motion effects through the seat.

[0066] Fig. 9 (A) and Fig. 9 (B) shows an example of an immersive VR experience that can be used to increase productivity while riding in a vehicle while also providing an entertaining VR experience for the participants. For example, two or more people can sit around a table in a virtual environment (e.g., Fig. 9 In the virtual room shown in (A) or in Fig. 9Alternatively, different participants may experience different VR environments for the meeting; for example, to one participant it may appear as if they are meeting on the flatbed of a truck, while to another participant they may appear to be meeting in a room in an office. The participants may be in the same vehicle on their way to get off work, or in different vehicles. Or some participants may be in a vehicle on their way to get off work, while one or more other participants are already in the office, at home, or elsewhere. The view of the computer screen of one of the participants may be projected into the virtual environment for all participants to see. The virtual content presented to each participant may be based on the situation that the particular participant is experiencing. For example, Fig. 9 As shown in B, a static view of a meeting in an office can be presented to participants sitting at home or in the office, while Fig. 9 As shown in A, a dynamic view of a meeting can be presented from the back of a truck to participants riding in the vehicle, where passing scenery and movement are integrated into the experience.

[0067] In some embodiments, passengers may choose to enjoy a relaxing virtual experience while riding, such as Fig.10 (A) Floating in a canoe or tube on a river or as shown Fig.10 (B) Soaring on a hang glider as shown. In some embodiments, the path taken by the passenger in the virtual environment may follow the actual path followed by the vehicle in the real world; for example, curves or turns on the actual route may be modeled as turns or bends on the river in which the passenger floats. In some embodiments, the VR system may anticipate bumps in the road the vehicle is traveling on or stops of the vehicle, which may present virtual content in the environment to provide context for the movement actually felt by the passenger; for example, a virtual canoe in which the passenger is riding may hit a virtual log in the river as the vehicle passes over a speed bump, or the virtual canoe may be temporarily stuck in a sandbar or eddy when the vehicle stops at a stop sign or traffic light. In some embodiments, the VR system may direct the vehicle's systems to provide physical effects as well as the virtual experience, such as by directing the HVAC system to blow air to the passenger during the hang gliding experience.

[0068] Fig.10(c) Shows that passengers can choose to enjoy a virtual experience of riding through other real locations (such as the streets of London, or through a fictional city or landscape). The virtual experience can be educational and interactive, for example allowing passengers to discover historical or other information about landmarks in a virtual view of the city they are experiencing. The virtual experience can be interactive in other ways, such as allowing passengers to overtake other vehicles during a virtual road race experience, or run over zombies in a post-apocalyptic landscape. In some embodiments, the VR system can select a route in a simulated real city that closely matches the actual route the vehicle is traveling, such as making the turns, bends, and stops in the virtualized view of the city closely match the turns, bends, and stops in the actual route the vehicle is traveling. It should be noted that the match between the route in the real city being simulated and the actual route traveled by the vehicle does not have to be a 1:1 match. For example, the simulated city block can be extended by adding one or two simulated storefronts or houses so that the simulated city block matches the length of the street on the actual route. In some embodiments, the VR system may anticipate a bump in the road the vehicle is traveling on or a stop in the vehicle, which may render virtual content in the environment to provide context for the motion that the passenger actually feels; for example, a virtual vehicle in which the passenger is riding may run past zombies in an apocalyptic landscape as the vehicle passes over a speed bump. In some embodiments, the VR system may direct the vehicle's systems to provide physical effects as well as the virtual experience, such as by directing the HVAC system to blow air to the passenger while the virtual view shows wind blowing across the virtual landscape.

[0069] like Fig.11 As shown, in some embodiments, virtual views of real or fictional characters can be integrated into the virtual experience provided by the VR system. Fig.11 As shown in (B), a virtual representation 1195 of an author or talk show host may appear to be sitting in a seat next to the passenger 1190; the virtual author may be reading one of their books to the passenger, or the virtual talk show host may be sitting in a seat next to the passenger hosting the show, with their voice provided through the audio system of the vehicle or HMD 1100. For another example, Fig.11 As shown in (A), passengers can experience riding on a flatbed truck and watching a virtual band's concert, where the band's music is provided through the audio system of the vehicle or HMD 1100.

[0070] Fig.12 is a high-level flow chart of a method for providing a VR experience to passengers in a vehicle, according to some embodiments. Fig.12 The method can be, for example, Figure 1 , Figure 2 , Figure 6 or Figure 7 This is achieved using the VR system shown in .

[0071] As shown at 1200, a VR system in a vehicle can obtain input from vehicle sensors (e.g., depth map information, video, etc.), vehicle systems (e.g., steering, braking, throttle, active suspension, positioning, and IMU systems), and external sources (e.g., world map data, 3D models, etc.). As shown at 1210, the VR system can generate virtual content based at least in part on the input from the vehicle sensors, vehicle systems, and external sources. As shown at 1220, the VR system can send the virtual content to at least one VR projection device, such as Figure 1 The HMD shown or Figure 2 As shown at 1230, the VR projection device displays the virtual content to the passenger. Although not shown, in some embodiments, the VR system can also generate signals to the vehicle control system (e.g., to control braking, acceleration, steering, or suspension / motion within constraints) and vehicle activity systems (e.g., audio and HVAC systems, and active seats) to provide physical effects synchronized with the projected virtual content, thereby further enhancing the passenger experience. As shown by the arrow returning from 1230 to 1200, Fig.12 The method can be a continuous process during the duration of the VR experience.

[0072] In some embodiments, the VR projection device (e.g., an HMD) may include a camera, a depth camera, and / or other sensors, and the VR system may replace the VR projection device or may also obtain input from the VR projection device, and may generate virtual content at least in part based on the input from the sensors on the VR projection device.

[0073] Fig.13 is a flowchart of a method for providing virtual content in a mixed reality view of an environment to passengers in a vehicle according to some embodiments. Fig.13 The method can be, for example, Figure 1 , Figure 2 , Figure 6 or Figure 7 This is achieved using the VR system shown in .

[0074] As shown at 1300, a VR device (e.g., Figure 1) and a user device (e.g., a laptop or notebook computer, a tablet or tablet device, etc.) may be connected to a VR system in a vehicle, for example using a wired or wireless connection. As shown at 1310, the VR system may obtain image input (e.g., a view of a display or one or more windows on a device display) from the user device, and may also obtain input from vehicle sensors (e.g., depth map information, video, etc.), vehicle systems (e.g., steering, braking, throttle, active suspension, positioning, and IMU systems), and external sources (e.g., world map data, 3D models, etc.). As shown at 1320, the VR system may generate virtual content including images input from the user device and based on input from vehicle sensors, vehicle systems, and external sources. As shown at 1330, the VR system may send the virtual content to a VR device (e.g., an HMD) for display to a passenger. As shown at 1340, the VR device displays the virtual content to the passenger so that it appears that the image input is fixed at a certain distance in front of the vehicle, for example, as shown in FIG. Figure 3 , 4 Or 5. As indicated by the arrow from 1340 back to 1310, Fig.13 The method can be a continuous process during the duration of the VR experience.

[0075] In some embodiments, the VR projection device (e.g., an HMD) may include a camera, a depth camera, and / or other sensors, and the VR system may replace the VR projection device or may also obtain input from the VR projection device, and may generate virtual content at least in part based on the input from the sensors on the VR projection device.

[0076] Fig.14 is a flow chart of a method for providing an immersive VR experience with environmental effects to passengers in a vehicle according to some embodiments. Fig.14 The method can be, for example, Figure 1 , Figure 2 , Figure 6 or Figure 7 This is achieved using the VR system shown in .

[0077] As shown at 1400, a VR device (e.g., Figure 1As shown at 1410, the VR system may obtain input from vehicle sensors (e.g., depth map information, video, etc.), vehicle systems (e.g., steering, braking, throttle, active suspension, positioning, and IMU systems), and external sources (e.g., world map data, 3D models, etc.). As shown at 1420, the VR system may generate virtual content representing an immersive VR environment based on input from vehicle sensors, vehicle systems, and external sources. As shown at 1430, the VR system may send the virtual content to a VR device (e.g., an HMD) for display to a passenger. As shown at 1440, the VR device displays the virtual content to provide an immersive VR experience to the passenger. As shown at 1450, the VR system may also send signals to one or more vehicle systems. As shown at 1460, one or more vehicle systems may provide passengers with physical effects that are synchronized with and enhance the VR experience. As shown by the arrow returning from 1460 to 1400, Fig.14 The method can be a continuous process during the duration of the VR experience.

[0078] In some embodiments, the VR projection device (e.g., an HMD) may include a camera, a depth camera, and / or other sensors, and the VR system may replace the VR projection device or may also obtain input from the VR projection device, and may generate virtual content at least in part based on the input from the sensors on the VR projection device.

[0079] In some embodiments, Figure 12 to Figure 14 In the method described in , in order to reduce or eliminate the delay time of presenting virtual content for display to passengers, the VR system can "look ahead", for example, using world map data, a 3D model of the environment, and vehicle sensor data to present virtual content and determine the physical effect of the location on the route the vehicle is traveling before the vehicle actually arrives at the specific location. This "look ahead" presentation helps ensure that the visual and physical experience of the passengers in the VR environment closely matches the actual physical movement of the vehicle on the route.

[0080] Adapting to passenger preferences and tendencies in VR environments

[0081] like Figures 1 to 14As shown, when participating in a VR environment presented by a VR system in a vehicle, some passengers may tend to experience motion sickness more easily than other passengers. Some passengers who participate in this way may rarely or never experience motion sickness. In some embodiments, a VR application executed on the VR system can adjust the VR environment and experience to accommodate the preferences and tendencies of different passengers. For example, in some embodiments, visual cues can be slowed down or accelerated when compared to the actual speed or acceleration of the vehicle to accommodate the preferences and tendencies of different passengers. Maintaining a 1:1 mapping ratio or slowing down visual cues can help reduce motion sickness for passengers who suffer from motion sickness or show signs of motion sickness. For passengers who are not prone to or do not show signs of motion sickness, the visual cues can be accelerated to a mapping ratio of 1:2, for example, to provide passengers with an enhanced, more stimulating virtual experience.

[0082] In addition to adjusting the mapping ratio for a passenger to help prevent motion sickness, several other visual and auditory techniques (referred to as adjustments) may be used in the VR experience to increase comfort and reduce motion sickness for passengers using the VR system in a vehicle. A guide describing these adjustments for passengers using a VR system in a vehicle may be provided to VR application developers. One or more of these adjustments may be integrated into a VR application executed in a VR system according to the guide, and the adjustments may be applied to the VR experience generated by the VR application. One or more adjustments may be applied to the VR experience by default, or one or more adjustments may be added to (or removed from) the VR experience based on the preferences of a particular passenger and / or based on feedback from a passenger monitoring system that detects signs of motion sickness or discomfort in a particular passenger.

[0083] While embodiments are generally described as providing adjustments in a VR experience for passengers using a VR system in a moving vehicle, in some embodiments, the VR experience may be recorded for later playback by the person having the initial experience or others when not in a moving vehicle (e.g., while sitting in a room). One or more adjustments described herein may be provided during playback of the VR experience so that viewers who are stationary (e.g., sitting in a room) and whose vestibular apparatus is not experiencing any acceleration do not experience symptoms of motion sickness when viewing the replayed mobile VR content.

[0084] Figures 15 to 17 Several visual adjustments that may be used in a VR experience to increase comfort and reduce motion sickness for passengers using a VR system in a vehicle are graphically illustrated in accordance with some embodiments. Figures 15 to 17 The adjustments shown in can be made, for example, in Figures 1 to 14 The VR system shown is implemented in the

[0085] like Fig.15As shown, in some embodiments, a virtual ground plane or platform 1504 can be provided below the passenger in the virtual environment 1500 (e.g., located in the virtual environment 1500 where the passenger's feet are located) to help avoid symptoms of motion sickness. This prevents the passenger from having the sensation of flying through space without a ground beneath them, which can be disconcerting to some passengers. In some embodiments, virtual representations of the passenger's arms and legs can also be displayed.

[0086] Looking down or to the side in a moving vehicle, especially when doing activities such as reading, may cause motion sickness symptoms in some passengers. Therefore, VR environment 1500 should encourage passengers to look forward and keep their heads upright while the vehicle is in motion. Fig.15 As shown, VR applications may provide compelling visual content 1502 directly in front of the passenger and at eye level to encourage the passenger to look forward and keep their head upright. VR applications should avoid displaying text or other compelling content that requires the passenger to look down to view the content when the vehicle is in motion. VR applications should avoid displaying text or other compelling content to the right or left of the passenger because when the vehicle is in motion, the optical flow cues provided by the peripheral virtual content are faster to the side, thus having the potential to cause motion sickness.

[0087] like Fig.15 As shown, anchored virtual content 1506 may be provided to help passengers know their direction in the real world while experiencing the virtual environment 1500 in a moving vehicle. In some embodiments, objects or aspects may be displayed in the virtual environment anchored to real-world directions (e.g., the sun or moon anchored to due west or due east, or tall buildings anchored to due north), so that when the vehicle follows a real-world route, passengers can more easily determine their position and direction in the real world. When the vehicle turns in the real world, the virtual world (including the eye-catching virtual content 1502) turns with the vehicle, but the anchored virtual content 1506 remains fixed relative to the real world.

[0088] In some embodiments, preview information about upcoming driving maneuvers and real-world terrain effects that can be physically felt (e.g., right turns, left turns, stops (e.g., at red lights or stop signs), climbs, slopes, speed bumps, downhill sections, etc.) can be provided to improve passenger comfort. In a VR experience without a real-world video perspective and without providing cues about maneuvers, passengers may feel uncomfortable because they cannot tell where they are in the real world and cannot predict upcoming driving maneuvers that will physically affect their vestibular / balance sense and thus support their bodies. Preview information for driving maneuvers and real-world terrain effects can be provided by visual, auditory, and / or tactile cues. For example, a visual cue (e.g., a directional arrow) can be displayed to indicate an upcoming left or right turn, climb, or descent. As another example, an audio cue can be played to indicate an upcoming maneuver; for example, a sound or tone can be played to the left ear to indicate an upcoming left turn, a sound or tone can be played to the right ear to indicate an upcoming right turn, and different tones or sounds can be played to both ears to indicate an upcoming descent or climb. As another example, tactile cues may be generated to indicate an upcoming maneuver; for example, the left side of the seat may vibrate to indicate an upcoming left turn, and the right side of the seat may vibrate to indicate an upcoming right turn. As another example, visual, auditory, and / or tactile cues may be provided to alert passengers of an upcoming tilt or bump in the road. Fig.16 As shown, in some embodiments, a virtual path or track 1608 may be displayed in the virtual environment 1600, which follows the path and contour of the real-world route that the vehicle is traveling, and through which the passenger can see the path and contour of the real-world route, including but not limited to upcoming turns, hills, bumps and inclines in the road. Navigation system and vehicle system data can be input into the VR system and used to provide preview information. Platform 1604 and anchored virtual content 1606 can also be displayed. Visual cues for upcoming maneuvers can also be displayed with track 1608; for example, a door 1610 or other visual cue (such as a stop sign or traffic light on the road) can be displayed to indicate an upcoming stop; the door 1610 can remain until the vehicle is ready to move again. Audio and / or tactile cues can also be provided.

[0089] In some embodiments, if the VR system detects that a passenger may begin to experience motion sickness, some content may be removed from the VR environment, thereby providing the passenger with fewer, more low-profile visual objects passing through the virtual world. Fig.17As shown, the low-stimulus, sparse visual environment 1700 can be used to provide a soothing, relaxing experience. In some embodiments, objects or aspects can be displayed in the virtual environment anchored to a real-world direction (e.g., a setting sun anchored to the west, or a tall building anchored to the north) so that passengers can more easily determine their location in the real world as the vehicle follows the real-world route. For example, a VR environment 1700 showing a sparse field of view of stars with a sunset anchor 1706 and a small number of low-profile visual objects passing by can help passengers determine their orientation in the real world and can provide a comfortable, relaxing VR experience.

[0090] Experiencing strong light flow in the virtual world for an uninterrupted period of time (i.e., driving without stopping) can cause an illusory perception of motion in the opposite direction (i.e., backward) when the vehicle stops. In some embodiments, the light flow / visual motion aftereffect can be mitigated to avoid such motion perception when the vehicle stops. For example, visual stimulation can be provided at or near the stop / braking point of the vehicle to reduce this effect. In some embodiments, the visual stimulation can be a motion cue that flows in the opposite direction (i.e., forward) to the perceived backward motion; the motion cue can offset the feeling of moving backward. For example, the motion cue can be, for example, an object that is still moving forward, passes by a passenger, moves slowly, and slows to a complete stop within a few seconds. This makes the passenger feel that they are drifting forward, which offsets the illusion of moving backward. As another example of a visual cue that can be used to mitigate the light flow / visual motion aftereffect, the contrast of the virtual world can be reduced because the illusion of backward motion may be more likely to have a higher contrast.

[0091] Passengers may feel uncomfortable due to inconsistencies between visual input and the real-world ambient noise they hear. Therefore, in some embodiments, audio (e.g., music, sound effects, white noise, etc.) may be provided to supplement the visual experience provided by the VR application, such as through headphones or earbuds that block or cancel road noise and other real-world ambient noise.

[0092] Figures 18 to 20 Monitoring passengers and adjusting the VR experience based on the passenger's preferences and physiological responses while using a VR system in a moving vehicle is shown. Figures 18 to 20 The method and apparatus shown in can be used, for example, in Figures 1 to 14 The VR system shown is implemented in the

[0093] Fig.18Monitoring of a passenger 1890 using a VR system in a vehicle is shown according to some embodiments. Virtual content 1840 and audio 1842 generated by a VR application executed on a VR controller 1810 and presented to the passenger 1890 via an HMD 1820 may be adapted to a particular passenger 1890 according to the passenger's preferences and / or according to passenger sensor data 1830 collected by sensors in the vehicle. In some embodiments, the VR application may initialize the VR environment for the passenger 1890 according to predefined or predetermined passenger preferences (e.g., passenger preference data maintained in a preference file stored on the controller 1810, or passenger preference data accessed from an external memory such as cloud storage). In some embodiments, the VR system may provide an interface that allows the passenger 1890 to specify their preferences for the VR environment to be experienced (e.g., a passenger may request a relaxing, normal, or stimulating experience).

[0094] In some embodiments, when using HMD 1820 to experience the VR world generated by the VR application on VR controller 1810, various sensors can be used to monitor the passenger 1890 for signs of discomfort or motion sickness (e.g., pallor, sweating, fidgeting, swallowing, hiccups, pulse rate, breathing rate, eye movement, etc.); passenger sensor data 1830 can be provided to VR controller 1810 via a wired or wireless connection. In some embodiments, the sensors may include sensors 1832 in or on HMD 1820, such as internal cameras that monitor the passenger's eyes or other parts of the passenger's face, external cameras that monitor other parts of the passenger's body (e.g., arms and hands), IMUs that detect and track passenger head movements, and / or sensors that monitor physiological reactions such as sweating, swallowing, and breathing rate. In some embodiments, the sensors may include sensors 1834 attached to or worn on other parts of the passenger's body, such as wristbands, armbands, or watches that monitor sweating, pulse rate, swallowing, or other physiological reactions and / or include IMUs that detect and track passenger body movements. In some embodiments, the sensors may include sensors 1836 in the seats that detect, for example, the movement and weight distribution of the passenger, which may be used, for example, to determine if the passenger is fidgeting. In some embodiments, the sensors may include cameras 1838 in the vehicle that capture video of the passenger, which may be used, for example, to detect the passenger's movement and reactions (e.g., swallowing, burping, etc.).

[0095] If signs of motion sickness are detected, for example, if the VR controller 1810's analysis of the passenger sensor data 1830 indicates that the passenger appears to be fidgeting, has his eyes closed, or is swallowing or burping frequently, the VR application executed on the VR controller 1810 may change the virtual environment generated by the application to accommodate the passenger and reduce motion sickness. For example, the application may return to or maintain a 1:1 mapping to real-life motion and acceleration in the simulated view displayed to the passenger. The VR application may also use one or more visual and audio adjustments as described above to change the virtual experience presented to the passenger, such as by displaying a virtual platform below the user, displaying eye-catching virtual content in front of the user, displaying anchoring content in the virtual environment, reducing content used to provide optical flow through the passenger, providing visual cues for upcoming vehicle maneuvers, displaying a virtual track, reducing the virtual world to a low-stimulus, sparse visual environment that can provide a soothing, relaxing experience, providing visual stimulation to offset optical flow / visual motion aftereffects and / or playing soothing audio to block road noise. If the passenger does not show signs of motion sickness, the VR system may be different from the 1:1 mapping, which allows for more flexibility in the virtual experience. For example, visual and other cues that provide a sense of acceleration may be added to the virtual experience to make the passenger feel that they are accelerating faster than the vehicle is actually accelerating, and / or visual and other cues that provide a sense of turning may be added to the virtual experience to make the passenger feel that they are turning faster than the vehicle is actually turning. In addition, based on the passenger's lack of motion sickness, preferences, and / or input to the VR system, one or more of the comfort and motion sickness reduction adjustments described above may not be displayed or may be removed from the environment.

[0096] Fig.19 An exemplary VR application that adjusts the VR experience to suit a passenger based on passenger preferences and passenger sensor data according to some embodiments is shown. VR application 1912 may include, but is not limited to, an adjustment decision module 1914, a VR rendering module 1916, and an audio module 1918. VR application 1912 may also encode or access the adjustment specification module 1914. Figures 15 to 17The adjustment decision module may set an initial VR environment for the passenger based on the passenger's preferences. For example, the passenger may be prone to motion sickness, so their preferences may specify that they prefer a calm, soothing virtual experience. Therefore, the adjustment decision module may determine one or more adjustments to be set for the passenger based on the adjustment guide 1920. Alternatively, the passenger may not be prone to motion sickness, so their preferences may specify that they prefer a stimulating virtual experience. Other preferences, such as audio preferences, may also be specified. The initial adjustment settings may be provided to the VR rendering module 1916. The VR rendering module 1916 may then generate virtual content 1940 representing an immersive VR environment suitable for the passenger based on the current adjustment settings and at least in part on navigation and terrain data obtained from vehicle sensors and systems and vehicle sensor data. When presenting the virtual content, the VR rendering module 1916 synchronizes the VR environment with the movement and acceleration (e.g., turning, accelerating, stopping, etc.) of the vehicle determined from the input. The audio module 1918 may also generate audio 1942 for the passenger suitable for the current adjustment settings.

[0097] The adjustment decision module 1914 can monitor passengers to detect signs of motion sickness, such as using Fig.18Passenger sensor data obtained by various sensors shown. If the adjustment decision module 1914 detects signs of motion sickness, the adjustment decision module 1914 may determine one or more adjustments to be set for the passenger based on the adjustment guide 1920. The adjustment settings may be changed to accommodate the passenger and reduce motion sickness, and the VR rendering module 1916 may begin generating and displaying virtual content and audio according to the changed settings and adjustments. For example, the VR rendering module 1916 may return to or maintain a 1:1 mapping to real-life motion and acceleration in the simulated view displayed to the passenger. The VR rendering module 1916 may also use one or more visual and audio adjustments as described above to change the virtual experience presented to the passenger, such as by displaying a virtual platform below the user, displaying eye-catching virtual content in front of the user, displaying anchoring content in the virtual environment, reducing content used to provide optical flow passing through the passenger, providing visual cues for upcoming vehicle maneuvers, displaying virtual tracks, reducing the virtual world to a low-stimulus, sparse visual environment that can provide a soothing, relaxing experience, providing visual stimulation to offset optical flow / visual motion aftereffects and / or playing soothing audio to block road noise. In some embodiments, if the passenger does not show signs of motion sickness, the adjustment decision module 1914 may change the settings to change the 1:1 mapping, which allows for greater flexibility in the virtual experience. For example, visual and other cues that provide a sense of acceleration may be added to the virtual experience to make the passenger feel that they are accelerating faster than the vehicle is actually accelerating, and / or visual and other cues that provide a sense of turning may be added to the virtual experience to make the passenger feel that they are turning faster than the vehicle is actually turning. In addition, based on the passenger's lack of motion sickness, preferences, and / or input to the VR application 1912, one or more of the comfort and motion sickness reduction adjustments described above may not be displayed or may be removed from the virtual environment.

[0098] Fig. 202010 is a flowchart of a method for adjusting a VR experience to accommodate a passenger based on passenger preferences and passenger sensor data according to some embodiments. As shown at 2010, a VR application executed on a VR controller of a VR system in a vehicle can set an initial VR environment for a passenger according to the passenger's preferences. For example, a passenger may be prone to motion sickness, so their preferences may specify that they prefer a calm, soothing virtual experience. Alternatively, a passenger may not be prone to motion sickness, so their preferences may specify that they prefer a stimulating virtual experience. Other preferences, such as audio preferences, may also be specified. As shown at 2020, the VR application can then generate virtual content and audio representing an immersive VR environment based on the current settings and at least in part based on inputs from vehicle sensors and systems including but not limited to navigation / route data, terrain contour data, and vehicle sensor data. When presenting the virtual content, the VR application synchronizes the VR environment with the movement and acceleration (e.g., turning, accelerating, stopping, etc.) of the vehicle determined from the input. As shown at 2030, the VR controller sends the virtual content and audio to the VR device used by the passenger (e.g., an HMD worn by the passenger).

[0099] As shown at 2040, the VR system can use, for example, Fig.18The various sensors shown monitor passengers to detect signs of motion sickness. At 2050, if motion sickness is detected, the VR environment may be changed to accommodate the passenger and reduce motion sickness, and the method returns to element 2020 to generate and display virtual content and audio according to the changed settings and adjustments. For example, the application may return to or maintain a 1:1 mapping to real-life motion and acceleration in the simulated view displayed to the passenger. The VR application may also use one or more visual and audio adjustments as described above to change the virtual experience presented to the passenger, such as by displaying a virtual platform below the user, displaying eye-catching virtual content in front of the user, displaying anchoring content in the virtual environment, reducing content for providing optical flow through the passenger, providing visual cues for upcoming vehicle maneuvers, displaying virtual tracks, reducing the virtual world to a low-stimulation, sparse visual environment that can provide a soothing, relaxing experience, providing visual stimulation to offset optical flow / visual motion aftereffects and / or playing soothing audio to block road noise. At 2050, if no signs of motion sickness are detected, the method returns to element 2020 to continue generating and displaying virtual content and audio according to the current settings. In some embodiments, if the passenger does not show signs of motion sickness, the VR system may deviate from a 1:1 mapping, which allows for greater flexibility in the virtual experience. For example, visual and other cues that provide a sense of acceleration may be added to the virtual experience to make the passenger feel that they are accelerating faster than the vehicle is actually accelerating, and / or visual and other cues that provide a sense of turning may be added to the virtual experience to make the passenger feel that they are turning faster than the vehicle is actually turning. Additionally, based on the passenger's lack of motion sickness, preferences, and / or input to the VR system, one or more of the comfort and motion sickness mitigation adjustments described above may not be displayed or may be removed from the environment.

[0100] Content of VR systems in vehicles

[0101] In some embodiments, virtual content developers can create VR content that can be used in a VR system in a vehicle as described herein. VR content can be provided by a VR system provider or a third party. For example, a developer can create a VR application that can be executed on a VR controller to provide passengers with a unique VR experience through a VR system in a vehicle. For example, an entertainment studio can develop a VR application that allows passengers to experience a virtual environment based on a movie or movie franchise made or owned by the studio while riding in a vehicle. The VR experience provided by the VR application may include modular elements or parts that can be matched with various trip durations and route configurations. VR applications can be purchased and downloaded to the VR system, for example, through an online store or site. For example, an online store can be provided by a VR system provider. An interface for an online store can be provided by a VR system that allows users to browse, select, purchase, and download VR applications from an online store. A VR system provider can receive a portion of each purchase through an online store.

[0102] In some embodiments, a VR system as described herein may allow a user to build or customize a virtual environment along one or more routes, such as along a frequently traveled route, which may then be experienced by the user, shared with others in the same vehicle, or provided to others (e.g., through an online VR environment sharing website), who may then experience the virtual environment in their own vehicles. For example, a VR system may allow a user to interact with a virtual environment using gestures, voice commands, or input to an input device to add, remove, or modify virtual content in the virtual environment. For example, a user may add a virtual building or other structure along a virtual representation of a route that the user frequently travels, scatter virtual seeds or place virtual trees along a virtual representation of the route, or otherwise customize or interact with a virtual representation of the route. When a user travels a route in a vehicle using a VR system to experience the VR environment, the user may see the results of their interactions, such as virtual plants that grow from virtual seeds they previously scattered, virtual structures or trees they place, and so on. Others traveling the same route may obtain a customized VR environment directly from a user, a website, or other source, load the VR environment into the VR system of their vehicle, and experience the customized VR environment themselves. If desired, the customized VR environment can also be adapted and experienced on other routes.

[0103] Autonomous Vehicles

[0104] As previously mentioned, embodiments of a VR system as described herein may be implemented, for example, in an autonomous or "self-driving" vehicle in which all occupants are passengers. The VR system may be used in safer, smaller, and less expensive autonomous vehicles. Windows in vehicles are inherently unsafe, structurally not soundproof, and increase the cost of the vehicle. By providing a virtual view of a real-world or simulated environment, a VR system may reduce or eliminate the need for windows in an autonomous vehicle, thereby allowing vehicle designs to use fewer and / or smaller windows, or no windows at all. In addition, the VR experience provided by the VR system may give passengers the feeling that the vehicle they are actually riding in is larger than the actual size of the autonomous vehicle, which may provide passengers with a more comfortable and safer experience when riding in a small autonomous vehicle.

[0105] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the present disclosure, even when a single embodiment is described only with respect to specific features. The feature examples provided in the present disclosure are intended to be illustrative, not limiting, unless otherwise stated. The scope of the present disclosure includes any feature or combination of features disclosed herein (explicitly or implicitly) or any generalization thereof, regardless of whether it alleviates any or all of the problems addressed herein. Therefore, new claims may be made for any such combination of features during the prosecution of this patent application (or a patent application claiming priority thereto). Specifically, with reference to the attached claims, the features of the dependent claims may be combined with the features of the independent claims, and the features from the corresponding independent claims may be combined in any appropriate manner rather than just by the specific combinations listed in the attached claims.

[0106] In different embodiments, the various methods described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the frames of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from the present disclosure, various modifications and changes can obviously be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. The boundaries between various components and operations are arbitrary to a certain extent, and specific operations are shown in the context of a specific exemplary configuration. Other allocations of functions are contemplated, and they may fall within the scope of the appended claims. Finally, the structure and functionality presented as discrete components in the exemplary configuration may be implemented as a combined structure or components. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

[0107] Additionally, embodiments of the present disclosure may be described in accordance with the following terms:

[0108] Clause: A system comprising:

[0109] A virtual reality (VR) controller in a vehicle, wherein the VR controller is configured to:

[0110] obtaining input from one or more sources in the vehicle;

[0111] obtaining display screen input from a user device in the vehicle;

[0112] displaying virtual content including the display screen input from the user device based at least in part on the input from the one or more sources in the vehicle; and

[0113] The virtual content is sent to a VR device for display to passengers in the vehicle in a 3D virtual view, wherein the display screen input from the user device is rendered and displayed so that the display screen input appears to the passenger to be at a fixed distance in front of the vehicle.

[0114] Clause 2: The system of clause 1, wherein the VR device comprises a head mounted device (HMD) worn by the passenger, the head mounted device displaying the virtual content for viewing by the passenger.

[0115] Clause 3: The system of clause 1, wherein the VR device comprises a projector that projects the virtual content onto a window of the vehicle for viewing by the passenger.

[0116] Clause 4: The system of clause 1, wherein the virtual content is displayed within a view of a real-world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

[0117] Clause 5: A system according to clause 1, wherein the virtual content input by the display screen represents a virtual environment that replaces the view of the real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues indicating to the passenger the movement of the vehicle in the virtual environment.

[0118] Clause 6: The system of clause 5, wherein the VR controller is further configured to:

[0119] determining whether the passenger is susceptible to or is showing signs of motion sickness; and

[0120] Upon determining that the passenger is susceptible to motion sickness or is showing signs of motion sickness, the virtual environment is altered according to one or more adjustments to reduce motion sickness.

[0121] Clause 7: The system of clause 6, wherein the VR controller is configured to determine whether the passenger is susceptible to motion sickness based on preference data of the passenger or input from the passenger.

[0122] Clause 8: The system of clause 6, wherein the VR controller is configured to determine whether the passenger is exhibiting signs of motion sickness based on passenger sensor data obtained from one or more sensors monitoring the passenger.

[0123] Clause 9: The system of clause 6, wherein the one or more adjustments include:

[0124] displaying a ground plane beneath the passenger in the virtual reality view;

[0125] reducing the number of visual cues indicating movement of the vehicle in the virtual environment;

[0126] slowing down the visual cue indicating movement of the vehicle in the virtual environment so that there is a 1:1 ratio between movement in the virtual environment and movement in the real environment;

[0127] Providing visual, audible, or tactile indication of upcoming vehicle maneuvers;

[0128] reducing the virtual content displayed in the virtual reality view to provide a sparse visual environment;

[0129] placing an anchor object that remains fixed relative to the real world in the virtual environment;

[0130] providing visual stimulation to counteract visual motion aftereffects while the vehicle is stopped; or

[0131] Provides audio output that blocks out real-world noise.

[0132] Clause 10: The system of clause 1, wherein the one or more sources include:

[0133] one or more vehicle sensors, wherein the input from the vehicle sensors comprises one or more of depth map information from external vehicle sensors, positioning information of the vehicle, video from one or more cameras on the vehicle, or motion and orientation information of the vehicle from an inertial measurement unit (IMU) of the vehicle; and

[0134] One or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

[0135] Clause 11: A system as described in clause 1, wherein the one or more sources include a source external to the vehicle accessed via a wireless connection, wherein the input from the external source includes one or more of world map data, a 3D model of the local environment, information about objects or features in the local environment, a video stream, or audio.

[0136] Clause 12: A system as described in clause 1, wherein the one or more sources include an inertial measurement unit (IMU) of the VR device, wherein input from the IMU includes movement and orientation information of the passenger's head.

[0137] Clause 13: The system of clause 1, wherein the vehicle is an autonomous vehicle.

[0138] Clause 14: A system according to clause 1, wherein when presenting the virtual content, the VR controller synchronizes the movement and acceleration of the virtual content with the movement and acceleration of the vehicle based on the input from the one or more sources in the vehicle.

[0139] Clause 15: A method comprising:

[0140] obtaining input from one or more sources in the vehicle via a virtual reality (VR) system in the vehicle;

[0141] Obtaining, through the VR system, a display screen input from a user device in the vehicle;

[0142] displaying, by the VR system, virtual content including the display screen input from the user device based at least in part on the input from the one or more sources in the vehicle; and

[0143] The virtual content in the 3D virtual view is displayed to passengers in the vehicle through the VR system, wherein the display screen input appears to the passengers to be at a fixed distance in front of the vehicle.

[0144] Clause 16: A method according to clause 15, wherein the VR system comprises a VR controller and a VR device, wherein the VR device is one of a head mounted device (HMD) or a projector, the head mounted device is configured to be worn by the passenger and display or project the virtual content for the passenger to watch, and the projector is configured to project the virtual content onto the window of the vehicle for the passenger to watch.

[0145] Clause 17: The method of clause 15, wherein the virtual content is displayed within a view of a real-world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

[0146] Clause 18: A method according to clause 15, wherein the virtual content input by the display screen represents a virtual environment, which replaces the view of the real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues indicating to the passenger the movement of the vehicle in the virtual environment.

[0147] Clause 19: The method according to clause 15, further comprising:

[0148] determining whether the passenger is susceptible to or is showing signs of motion sickness; and

[0149] Upon determining that the passenger is susceptible to or is showing signs of motion sickness, the virtual environment is altered based on one or more visual or audio adjustments to reduce motion sickness.

[0150] Clause 20: The method of clause 15, wherein presenting the virtual content comprises synchronizing movement and acceleration of the virtual content with movement and acceleration of the vehicle based on the input from the one or more sources in the vehicle.

Claims

1. A system comprising: A virtual reality (VR) controller in a vehicle, wherein the VR controller is configured to: obtaining input from one or more sources in the vehicle; obtaining display screen content from a user device of a passenger in the vehicle; presenting virtual content including the display screen content from the user device based at least in part on the input from the one or more sources in the vehicle; Based on the passenger's preference data, altering the virtual content based on one or more visual or audio adjustments to reduce motion sickness for the passenger; The virtual content is sent to a VR device for display to passengers in the vehicle in a 3D virtual view, wherein the display screen content from the user device is rendered and displayed so that the display screen content appears to the passengers to be at a fixed distance in front of the vehicle.

2. The system according to claim 1, wherein the VR device comprises: a head mounted device HMD worn by the passenger, the head mounted device displaying the virtual content for viewing by the passenger; or A projector projects the virtual content onto a window of the vehicle for viewing by the passenger.

3. The system of claim 1, wherein the user device comprises one or more of: a laptop, a mobile device, a smartphone, or a tablet, and wherein the vehicle comprises an autonomous vehicle.

4. The system of claim 1 , wherein the virtual content is displayed within a view of a real-world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

5. The system of claim 1 , wherein the virtual content, including the display screen content, represents a virtual environment that replaces a view of a real-world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues that indicate to the passenger the movement of the vehicle in the virtual environment.

6. The system of claim 5, wherein the VR controller is further configured to: determine that the passenger is susceptible to or is showing signs of motion sickness; and Upon determining that the passenger is susceptible to motion sickness or is showing signs of motion sickness, the virtual environment is altered according to one or more adjustments to reduce motion sickness.

7. The system of claim 6, wherein the VR controller is configured to determine that the passenger is susceptible to motion sickness based on the preference data of the passenger or input from the passenger.

8. The system of claim 6, wherein the VR controller is configured to determine that the passenger is exhibiting signs of motion sickness based on passenger sensor data obtained from one or more sensors monitoring the passenger.

9. The system of claim 6, wherein the one or more adjustments include: displaying a ground plane beneath the passenger in the virtual reality view; reducing the number of visual cues indicating movement of the vehicle in the virtual environment; slowing down the visual cue indicating movement of the vehicle in the virtual environment so that there is a 1:1 ratio between movement in the virtual environment and movement in a real environment; Providing visual, audible, or tactile indication of upcoming vehicle maneuvers; reducing the virtual content displayed in the virtual reality view to provide a sparse visual environment; placing an anchor object that remains fixed relative to the real world in the virtual environment; providing visual stimulation to counteract visual motion aftereffects while the vehicle is stopped; or Provides audio output that blocks out real-world noise.

10. The system of claim 1, wherein the one or more sources include: one or more vehicle sensors, wherein the input from the vehicle sensors comprises one or more of depth map information from external vehicle sensors, positioning information of the vehicle, video from one or more cameras on the vehicle, or motion and orientation information of the vehicle from an inertial measurement unit (IMU) of the vehicle; and One or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

11. The system of claim 1 , wherein the one or more sources include a source external to the vehicle accessed via a wireless connection, wherein input from the external source includes one or more of world map data, a 3D model of a local environment, information about objects or features in the local environment, a video stream, or audio.

12. The system of claim 1, wherein the one or more sources include an inertial measurement unit (IMU) of the VR device, wherein input from the IMU includes motion and orientation information of the passenger's head.

13. The system of claim 1 , wherein when presenting the virtual content, the VR controller synchronizes movement and acceleration of the virtual content with movement and acceleration of the vehicle based on the input from the one or more sources in the vehicle.

14. A system comprising: one or more processors; as well as a memory storing instructions that, when executed on or across the one or more processors, cause the one or more processors to: obtaining input for a vehicle following a route in a real environment at a location, wherein the input comprises three-dimensional (3D) model data of the real environment and sensor data from the vehicle; selecting a route in another location based on the route in the real environment; determining an upcoming change in altitude in the real environment based on the input; generating virtual content for providing a virtual view of a virtual environment, the virtual content including a simulation of the selected route in the other location, wherein movement and acceleration of the virtual content are synchronized with the movement and acceleration indicated in the input, and wherein the virtual content includes different visual cues indicating corresponding types of changes in the upcoming change in altitude; providing different audio cues indicating corresponding types of changes in the upcoming changes in altitude; as well as The virtual content is sent to a display device.

15. The system of claim 14, further comprising: A virtual reality (VR) controller is configured to render frames based on the virtual content.

16. The system of claim 14, wherein the instructions are executable to obtain at least a portion of the input from a user device of a passenger.

17. The system of claim 14, further comprising: One or more sensors coupled to the vehicle, the one or more sensors configured to provide the input.

18. The system of claim 14, wherein the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to: One or more signals are sent to one or more vehicle systems, the one or more signals causing the one or more vehicle systems to provide effects synchronized with the virtual content.

19. The system of claim 18, wherein the one or more vehicle systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

20. The system of claim 15, wherein the visual cue includes one or more directional arrows indicating a direction of an upcoming change in altitude.

21. A system comprising: A controller comprising one or more processors and a memory storing instructions that, when executed on or across the one or more processors, cause the one or more processors to: obtaining display content from a user device of a passenger in the vehicle; providing virtual content including the display content from the user device; Based on the passenger's preference data, altering the virtual content based on one or more visual or audio adjustments to reduce motion sickness for the passenger; The virtual content is sent to a virtual reality (VR) device for display to passengers in the vehicle in a 3D virtual view, wherein the display content from the user device is displayed so that the passenger appears to have the display content at a fixed distance in front of the vehicle.

22. A method comprising: Using one or more computing devices to perform: obtaining input for a vehicle following a route in a real environment at a location, wherein the input comprises three-dimensional (3D) model data of the real environment and sensor data from the vehicle; selecting a route in another location based on the route in the real environment; determining an upcoming change in altitude in the real environment based on the input; generating virtual content for providing a virtual view of a virtual environment, the virtual content including a simulation of the selected route in the other location, wherein movement and acceleration of the virtual content are synchronized with the movement and acceleration indicated in the input, and wherein the virtual content includes different visual cues indicating corresponding types of changes in the upcoming change in altitude; providing different audio cues indicating corresponding types of changes in the upcoming changes in altitude; as well as The virtual content is sent to a display device.

23. A method comprising: Using one or more computing devices to perform: obtaining input from one or more sources, wherein the one or more sources include sensors of a vehicle, wherein the input includes information about an environment external to the vehicle, and wherein the environment external to the vehicle includes other objects; generating virtual content based at least in part on the input, wherein the virtual content comprises: a virtual representation having motion and acceleration synchronized with the motion and acceleration of said vehicle indicated in said input; and other virtual representations of the object in an environment external to the vehicle, wherein motion and acceleration of the other virtual representations are based on the motion and acceleration indicated in the input and are different from the motion or acceleration of the virtual representation; and The virtual content is transmitted to a display device of a passenger in the vehicle.

Images