Adaptive head-up display

Abstract

The invention provides an adaptive head-up display. A virtual image plane is determined relative to a reference eye movement range. A virtual image projected into the virtual image plane is visible in the reference eye movement range. Occupant eye movement ranges are determined from sensor data. A first adjustment of the virtual image plane is performed based on the occupant eye movement range such that the virtual image projected into the virtual image plane is visible in the occupant eye movement range.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to augmented reality head-up displays in vehicles. BACKGROUND

[0002] Augmented reality head-up displays (AR HUDs) for vehicles can include a projector that projects a graphical image to be displayed on a semi-transparent windshield that serves as a combiner, or can include an array of holographic optical elements attached to the windshield. The projected graphical image can thus appear at a virtual image plane that is located in a forward direction relative to the vehicle. The graphical image appearing in the virtual image plane in front of the vehicle can thus appear to be in a plane that coincides with real objects that are visible through the semi-transparent windshield. SUMMARY

[0003] A system includes a computer including a processor and a memory storing instructions executable by the processor to determine a virtual image plane relative to a reference eyebox. A virtual image projected into the virtual image plane is visible in the reference eyebox. The instructions further include instructions to determine an occupant eyebox from sensor data. The instructions further include instructions to perform a first adjustment of the virtual image plane based on the occupant eyebox such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0004] The first adjustment can include translating the virtual image plane along at least one of a lateral axis that partially defines the virtual image plane and a vertical axis that partially defines the virtual image plane and extends perpendicular to the lateral axis.

[0005] The first adjustment can include translating the virtual image within the virtual image plane.

[0006] The instructions can further include instructions to perform an adjustment to a position of a seat occupied by an occupant such that the virtual image is visible in the occupant eyebox.

[0007] The instructions can further include instructions to perform a second adjustment of the virtual image plane based on occupant data of the occupant. The second adjustment can include translating the virtual image plane along a longitudinal axis extending normal to the virtual image plane. The instructions can further include instructions to input the occupant data of the occupant into a machine learning program that outputs an expected distance from the occupant eyebox to the virtual image plane. The second adjustment can include translating the virtual image plane along a longitudinal axis extending normal to the virtual image plane such that the virtual image plane is spaced apart from the occupant eyebox along the longitudinal axis by the expected distance. The instructions can further include instructions to perform the second adjustment based on weather data in addition to the occupant data. The instructions can further include instructions to perform an adjustment to a position of a seat occupied by the occupant such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0008] A method includes determining a virtual image plane relative to a reference eyebox. A virtual image projected into the virtual image plane is visible in the reference eyebox. The method further includes determining an occupant eyebox from sensor data. The method further includes performing a first adjustment of the virtual image plane based on the occupant eyebox such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0009] The first adjustment can include translating the virtual image plane along at least one of a lateral axis that partially defines the virtual image plane and a vertical axis that partially defines the virtual image plane and extends normal to the lateral axis.

[0010] The first adjustment can include translating the virtual image within the virtual image plane.

[0011] The method can further include performing an adjustment to a position of a seat occupied by the occupant such that the virtual image is visible in the occupant eyebox.

[0012] The method can further include performing a second adjustment of the virtual image plane based on occupant data of the occupant. The second adjustment can include translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane. The method can further include inputting the occupant data of the occupant into a machine learning program that outputs an expected distance from the occupant eyebox to the virtual image plane. The second adjustment can include translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane such that the virtual image plane is spaced apart from the occupant eyebox along the longitudinal axis by the expected distance. The method can further include performing the second adjustment based on weather data in addition to the occupant data. The method can further include performing an adjustment to a position of a seat occupied by the occupant such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0013] Also disclosed herein is a computing device programmed to perform any of the above-mentioned method steps. Also disclosed herein is a computer program product comprising a computer readable medium storing instructions executable by a computer processor to perform any of the above-mentioned method steps.

[0014] A vehicle can include a heads-up display (HUD) that can display information, such as information about the vehicle and / or objects surrounding the vehicle, to an occupant of the vehicle. The HUD can project an image onto a windshield of the vehicle. The HUD can provide content as an augmented reality (AR) image. The HUD can provide the AR image such that the AR image is overlaid by objects surrounding the vehicle when viewed by the occupant. Thus, the HUD can allow the occupant to view the image while also viewing the road along which the vehicle travels in a manner that displays the image. However, for various occupants viewing the virtual image plane, projecting the image to a single virtual image plane (e.g., on the windshield) for each occupant in the vehicle can result in misalignment and low quality between the image projected in the single virtual image plane and the occupant eyebox (e.g., based on various heights, seat positions, visual acuity, weather conditions, etc.). Such misalignment and low quality can result in an unsuitable appearance and blurriness of the AR image for the occupant.

[0015] As described herein, a vehicle computer can adjust a virtual image plane such that a virtual image projected into the virtual image plane is visible in an occupant eyebox. By adjusting the virtual image plane based on the occupant eyebox, the vehicle computer can enhance alignment and quality of the AR image relative to the occupant eyebox, which can provide a suitable appearance with a high quality AR image to the occupant. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1is a block diagram illustrating an example vehicle control system.

[0017] Figure 2 is a graphical illustration of an example virtual image plane determined with respect to a reference eyebox.

[0018] Figure 3 is a graphical illustration of a comparison of a virtual image plane to an example occupant virtual image plane determined with respect to an occupant eyebox.

[0019] Figure 4A is a graphical illustration of a comparison between a virtual image plane at a predetermined distance and a virtual image plane at an expected distance from an occupant eyebox.

[0020] Figure 4B is a graphical illustration of an example adjustment to a seat occupied by an occupant to account for a difference between a predetermined distance and an expected distance.

[0021] Figure 5 is an example neural network.

[0022] Figure 6 is an example flowchart of an example process for adapting a virtual image plane based on an occupant eyebox. DETAILED DESCRIPTION

[0023] REFERENCE Figures 1 to 5 , the example vehicle control system 100 includes a vehicle 105. A vehicle computer 110 in the vehicle 105 receives data from sensors 115. The vehicle computer 110 is programmed to determine a virtual image plane 200 with respect to a reference eyebox 205. A virtual image projected into the virtual image plane 200 is visible in the reference eyebox 205. The vehicle computer 110 is further programmed to determine an occupant eyebox 305 from sensor data. The vehicle computer 110 is further programmed to perform a first adjustment of the virtual image plane 200 based on the occupant eyebox 305 such that the virtual image projected into the virtual image plane 200 is visible in the occupant eyebox 305.

[0024] Turning now to Figure 1 , the vehicle 105 includes a vehicle computer 110, sensors 115, actuators 120 for actuating various vehicle components 125, and a vehicle communication module 130. The communication module 130 allows the vehicle computer 110 to communicate with a remote server computer 140 and / or other vehicles (e.g., via messaging or broadcast protocols such as Dedicated Short Range Communication (DSRC), cellular, and / or other protocols that can support vehicle-to-vehicle, vehicle-to-infrastructure, vehicle-to-cloud communications, etc., and / or via a packet network 135).

[0025] The vehicle computer 110 includes a processor and memory such as are known. The memory includes one or more forms of computer-readable media, and stores instructions executable by the vehicle computer 110 for performing various operations, including operations as disclosed herein. The vehicle computer 110 can also include two or more computing devices that operate in concert to implement vehicle 105 operations, including operations as described herein. Further, the vehicle computer 110 can be a general purpose computer having a processor and memory as described above, and / or can include an electronic control unit (ECU) or electronic controller, etc., for a particular function or set of functions, and / or can include special purpose electronic circuitry, including ASICs manufactured for particular operations (e.g., ASICs for processing sensor data and / or communicating sensor data). In another example, the vehicle computer 110 can include an FPGA (field programmable gate array), which is an integrated circuit manufactured to be configurable by the user. Typically, a hardware description language such as VHDL (very high speed integrated circuit hardware description language) is used in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided prior to manufacture, while logic components within a FPGA can be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuit). In some examples, a combination of processor, ASIC, and / or FPGA circuitry can be included in the vehicle computer 110.

[0026] The vehicle computer 110 can include programming to operate one or more of vehicle 105 propulsion, steering, transmission, climate control, interior and / or exterior lights, horn, doors, etc., and determine whether and when the vehicle computer 110 (rather than a human operator) controls such operations.

[0027] The vehicle computer 110 can include more than one processor (e.g., included in electronic controller units (ECUs), etc., included in the vehicle 105) or be communicatively coupled to more than one processor (e.g., via a vehicle communication network such as a communication bus, as further described below, such as a controller area network (CAN), etc.) for monitoring and / or controlling various vehicle components 125 (e.g., transmission controllers, steering controllers, etc.). The vehicle computer 110 is generally arranged for communication on a vehicle communication network, which can include a bus in the vehicle 105, such as a controller area network (CAN), etc., and / or other wired and / or wireless mechanisms.

[0028] Via the vehicle 105 network, the vehicle computer 110 can transmit messages (e.g., CAN messages) to and / or receive the messages from various devices in the vehicle 105 (e.g., sensors 115, actuators 120, ECUs, etc.). Alternatively or additionally, where the vehicle computer 110 actually comprises multiple devices, the vehicle communication network can be used to represent communication between devices represented as the vehicle computer 110 in this disclosure. Further, as mentioned below, various controllers and / or sensors 115 can provide data to the vehicle computer 110 via the vehicle communication network.

[0029] The vehicle 105 sensors 115 can include a variety of devices known to provide data to the vehicle computer 110. For example, the sensors 115 can include a light detection and ranging (lidar) sensor 115 disposed on the roof of the vehicle 105, behind the front windshield of the vehicle 105, around the vehicle 105, etc., that provides relative position, size, and shape of objects around the vehicle 105. As another example, one or more radar sensors 115 fixed to the bumpers of the vehicle 105 can provide data to provide position of objects, second vehicles, etc., relative to the position of the vehicle 105. Alternatively or additionally, the sensors 115 can also include, for example, camera sensors 115 (e.g., forward looking, side looking, etc.) that provide images from areas around the vehicle 105. In the context of this disclosure, an object is a physical (i.e., material) article that has mass and can be represented by a physical phenomenon (e.g., light or other electromagnetic waves or sound, etc.) that can be detected by a sensor 115. Thus, the vehicle 105, as well as other articles including as discussed below, all fall within the definition of “object” herein.

[0030] The vehicle computer 110 is programmed to receive data from one or more sensors 115 substantially continuously, periodically, and / or upon direction from the remote server computer 140, etc. The data can include, for example, a location of the vehicle 105. The location data specifies one or more points on the ground and can be in a known form (e.g., geographic coordinates such as latitude and longitude coordinates obtained via a navigation system using a global positioning system (GPS) as is known). Additionally or alternatively, the data can include locations of objects (e.g., vehicles, signs, trees, etc.) relative to the vehicle 105. As one example, the data can be image data of the environment surrounding the vehicle 105. In such an example, the image data can include one or more objects and / or signs (e.g., lane markers) on or along a roadway. Image data herein means digital image data (e.g., including pixels having intensity and color values) that can be acquired by a camera sensor 115. The sensor 115 can be mounted to the vehicle 105 in or on any suitable location (e.g., mounted on a bumper of the vehicle 105, on top of the vehicle 105, etc.) to collect images of the environment surrounding the vehicle 105.

[0031] The vehicle 105 actuators 120 are implemented via circuitry, chips, or other electronic and / or mechanical components that can actuate various vehicle subsystems according to appropriate control signals as is known. The actuators 120 can be used to control components 125, including propulsion and steering of the vehicle 105.

[0032] In the context of the present disclosure, a vehicle component 125 is one or more hardware components suitable for performing a mechanical or electromechanical function or operation, such as moving the vehicle 105, slowing or stopping the vehicle 105, steering the vehicle 105, etc. Non-limiting examples of components 125 include propulsion components (which include, for example, an internal combustion engine and / or an electric motor, etc.), transmission components, steering components (which can include, for example, one or more of a steering wheel, a steering rack, etc.), suspension components (which can include, for example, one or more of a damper (e.g., a shock absorber or a strut), a bushing, a spring, a control arm, a ball joint, a link, etc.), parking assist components, adaptive cruise control components, adaptive steering components, etc.

[0033] The vehicle 105 also includes a human-machine interface (HMI) 118. The HMI 118 includes user input devices such as knobs, buttons, switches, pedals, joysticks, touchscreens, and / or microphones, among others. The input devices can include sensors 115 to detect user input and provide user input data to the vehicle computer 110. That is, the vehicle computer 110 can be programmed to receive user input from the HMI 118. A passenger can provide user input via the HMI 118 (e.g., by selecting a virtual button on a touchscreen display, by providing a voice command, etc.). For example, a touchscreen display included in the HMI 118 can include sensors 115 to detect a passenger selecting a virtual button on the touchscreen display to, for example, select or deselect an operation, which input can be received in the vehicle computer 110 and used to determine a selection of the user input.

[0034] The HMI 118 also typically includes output devices to output signals or data to a passenger such as displays (including touchscreen displays), speakers, and / or lights, among others. The HMI 118 is coupled to the vehicle communication network and can send and / or receive messages to / from the vehicle computer 110 and other vehicle subsystems.

[0035] The vehicle 105 also includes an SLM 150. An “SLM” is an object that imposes a spatially varying modulation on a light beam. The SLM 150 is arranged to receive light from a projector 155 and modulate the light according to a pixel-by-pixel phase matrix (as discussed below) to output the light onto the windshield 210 to provide an augmented reality image that can appear to be outside of the vehicle 105. The SLM 150 is coupled to the vehicle communication network and can send and / or receive messages to / from the vehicle computer 110 and other vehicle subsystems.

[0036] The vehicle 105 also includes a projector 155. The projector 155 can be arranged to display an image in the field of view of a passenger of the vehicle 105. The projector 155 can be arranged to display an image in front of the vehicle of the passenger to provide information about the environment around the vehicle, operation of the vehicle, etc. For example, the projector 155 can project light onto the windshield 210. In particular, the projector 155 can project light through the SLM 150 onto the windshield 210. The light is reflected by the windshield 210 to provide an augmented reality image in the line of sight of the passenger so that the passenger can view and understand. While the augmented reality image is projected onto the windshield 210, the augmented reality image appears to be outside of the vehicle 105 to the passenger to provide an augmented reality display of the environment around the vehicle 105. In particular, the augmented reality image appears to be in a virtual image plane 200 in front of the vehicle 105, as discussed below. The projector 155 is coupled to the vehicle communication network and can send and / or receive messages to / from the vehicle computer 110 and other vehicle subsystems.

[0037] Additionally, the vehicle computer 110 can be configured for communicating with devices outside of the vehicle 105 via the vehicle-to-vehicle communication module 130 or interface (e.g., communicating with another vehicle and / or a remote server computer 140 (typically via direct radio frequency communication) through vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2X) wireless communication (cellular and / or short-range radio communication, etc.)). The communication module 130 can include one or more mechanisms that the vehicle’s computer can utilize to communicate, such as a transceiver, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or multiple topologies when multiple communication mechanisms are utilized). Exemplary communications provided via the communication module 130 include cellular, Bluetooth, IEEE 802.11, dedicated short-range communications (DSRC), cellular V2X (CV2X), and / or wide area network (WAN), including the Internet, providing data communication services. The label “V2X” is used herein for communications that can be vehicle-to-vehicle (V2V) and / or vehicle-to-infrastructure (V2I) and can be provided by the communication module 130 according to any suitable short-range communication mechanism (e.g., DSRC, cellular, etc.).

[0038] The network 135 represents one or more mechanisms that the vehicle computer 110 can utilize to communicate with remote computing devices (e.g., the remote server computer 140, another vehicle computer, etc.). Thus, the network 135 can be one or more of various wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or multiple topologies when multiple communication mechanisms are utilized). Exemplary communication networks include wireless communication networks (e.g., using Bluetooth®, Bluetooth Low Energy (BLE), IEEE 802.11, vehicle-to-vehicle (V2V) such as dedicated short-range communications (DSRC), etc.), local area networks (LANs), and / or wide area networks (WANs), including the Internet, providing data communication services. The network 135 represents one or more mechanisms that the vehicle computer 110 can utilize to communicate with remote computing devices (e.g., the remote server computer 140, another vehicle computer, etc.). Thus, the network 135 can be one or more of various wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any desired network topology (or multiple topologies when multiple communication mechanisms are utilized). Exemplary communication networks include wireless communication networks (e.g., using Bluetooth®, Bluetooth Low Energy (BLE), IEEE 802.11, vehicle-to-vehicle (V2V) such as dedicated short-range communications (DSRC), etc.), local area networks (LANs), and / or wide area networks (WANs), including the Internet, providing data communication services.

[0039] The remote server computer 140 can be a conventional computing device (i.e., including one or more processors and one or more memories) programmed to provide operations such as disclosed herein. Further, the remote server computer 140 can be accessible via the network 135 (e.g., the Internet, a cellular network, and / or some other wide area network).

[0040] Figure 2is a diagram illustrating a virtual image plane 200 for providing a virtual image to a reference eyebox 205. The eyebox is a free space viewing plane positioned within a passenger cabin 215 of a vehicle 105. The reference eyebox 205 is described by contextual information including four corners expressed as x, y, and z coordinates relative to a vehicle coordinate system (e.g., a Cartesian coordinate system with an origin at a predetermined point above and / or in the vehicle 105). The reference eyebox 205 is defined by a y-axis (i.e., extending in a lateral (or vehicle lateral) direction) and a z-axis (i.e., extending in a vertical direction). That is, the four corners of the reference eyebox 205 have the same x-coordinate value.

[0041] The reference eyebox 205 can be determined empirically (e.g., based on determining average (or some other statistical measure) height and average (or some other statistical measure) vehicle seat position (e.g., specified according to a vehicle coordinate system) for various occupants via testing and / or simulation). The reference eyebox 205 can be stored (e.g., in a memory of the vehicle computer 110). As another example, the remote server computer 140 can transmit the reference eyebox 205 to the vehicle computer 110 (e.g., via the network 135).

[0042] The virtual image plane 200 is a plane determined relative to the reference eyebox 205. That is, the virtual image plane 200 is determined such that if an observer’s eye is in the reference eyebox 205, then a virtual image projected onto the virtual image plane 200 is visible to the observer. The virtual image plane 200 is described by contextual information including four corners expressed as x, y, and z coordinates relative to a vehicle coordinate system. The virtual image plane 200 is defined by a y-axis (i.e., extending in a lateral (or vehicle lateral) direction) and a z-axis (i.e., extending in a vertical direction). That is, the four corners of the virtual image plane 200 have the same x-coordinate value.

[0043] The virtual image plane 200 can be determined such that the corners of the virtual image plane 200 correspond to the corners of the reference eyebox 205 projected a predetermined distance P along the x-axis (i.e., extending in the longitudinal direction). The coordinates of the corners of the reference eyebox 205 can be converted to the coordinates of the corners of the virtual image plane 200 based on the following parameters: the distance from the origin to the windshield 210 of one or more vehicle components 125 (e.g., the origin of the light from the projector 155), the diffraction angle to direct the light to the eyebox of the vehicle occupant, etc. The predetermined distance can be stored (e.g., in memory of the vehicle computer 110). The predetermined distance P can be determined empirically (e.g., based on testing various occupant eyeboxes against various virtual image planes to determine an average (or some other statistical measure) distance that provides a desired appearance virtual image projected into the virtual image plane 200 to various occupants (e.g., based on user input classifying the appearance of various virtual images projected into virtual image planes at various distances from the occupant)). The virtual image plane 200 can be stored (e.g., in memory of the vehicle computer 110). As another example, the vehicle computer 110 can receive the virtual image plane 200 from a remote server computer 140 (e.g., via the network 135).

[0044] A pixel-by-pixel phase matrix is determined based on the virtual image plane 200. The pixel-by-pixel phase matrix is a matrix that identifies pixels in the virtual image and specifies a pixel phase for each pixel. A “pixel phase” is an adjustment to the point in time at which a sample is taken in an analog-to-digital conversion. The pixel phase allows the pixel (or dot) clocks of the vehicle computer 110 and the projector 155 to be synchronized. A pixel clock is the speed at which pixels are transmitted so that a full frame of pixels fits into one refresh cycle. Unsynchronized pixel clocks can result in pixel banding (i.e., multiple pixels ending at the same pixel coordinate), which reduces the resolution of the virtual image. The projector 155 and the SLM 150 are actuated to output the virtual image into the virtual image plane 200 based on the pixel-by-pixel phase matrix. The pixel-by-pixel phase matrix can be determined empirically (e.g., based on testing and / or simulation to determine the pixel phase for each pixel in the virtual image that allows the virtual image to be projected into the virtual image plane 200). The pixel-by-pixel phase matrix can be stored (e.g., in memory of the vehicle computer 110). As another example, the vehicle computer 110 can receive the pixel-by-pixel phase matrix from a remote server computer 140 (e.g., via the network 135).

[0045] Figure 3is a diagram illustrating an offset between a reference eye movement range 205 and an occupant eye movement range 305 of a vehicle 105 occupant. The vehicle computer 110 can detect an occupant in the passenger cabin 215 based on sensor 115 data. For example, the vehicle computer 110 can receive sensor 115 data from a seat occupancy sensor 115 indicating that there is an occupant in the seat 220. The seat occupancy sensor 115 can be programmed to detect occupancy of the seat 220. The seat occupancy sensor 115 can be, for example, a contact rear sensor such as a pressure sensor and a contact switch. As another example, the seat occupancy sensor 115 can be a sensor that records values of an electrical variable (i.e., a variable whose value specifies some electrical quantity (e.g., voltage, current, resistance, etc.)) in an electrical circuit that includes electrical circuit elements in the seat 220 (e.g., a voltmeter, ammeter, ohmmeter, etc.). Values of the electrical variable corresponding to an open circuit can be classified as the seat 220 being unoccupied, and values of the electrical variable corresponding to a closed circuit can be classified as the seat 220 being occupied.

[0046] Additionally or alternatively, the vehicle computer 110 can receive sensor 115 data (e.g., image data) from a sensor 115 positioned to face the passenger cabin 215. The sensor 115 data can include one or more objects in the passenger cabin 215. The vehicle computer 110 can identify an occupant from the sensor 115 data. For example, object recognition techniques can be used (e.g., in the vehicle computer 110 based on lidar sensor 115 data, camera sensor 115 data, etc.) to identify the type of object (e.g., occupant, user device, package, etc.) and physical characteristics of the object.

[0047] The sensor 115 data can be interpreted using any suitable technique. For example, camera and / or lidar image data can be provided to a classifier that includes programming for utilizing one or more conventional image classification techniques. For example, the classifier can use machine learning techniques in which data known to represent various objects is provided to a machine learning program for training the classifier. Once trained, the classifier can accept vehicle sensor 115 data (e.g., images) as input and then provide, as output, an identification of an occupant for each of one or more respective relevant regions in the image or an indication that there is no occupant in the respective relevant region. Further, a coordinate system (e.g., polar or Cartesian) applied to the region proximate the vehicle 105 can be used to specify the location and / or region of an occupant identified from the sensor 115 data (e.g., converted to global latitude and longitude geographic coordinates, etc., according to the vehicle 105 coordinate system). Further, the vehicle computer 110 can employ various techniques to fuse (i.e., incorporate into a common coordinate system or frame of reference) data from different sensors 115 and / or multiple types of sensors 115, such as lidar, radar, and / or optical camera data.

[0048] Upon detecting an occupant in the passenger cabin 215 of the vehicle 105, the vehicle computer 110 can determine an occupant eyebox 305 based on the actual pose of the occupant. The occupant eyebox 305 is described by context information including four corners expressed as x, y, and z coordinates relative to a vehicle coordinate system. The occupant eyebox 305 is a free space view plane corresponding to the actual pose of the occupant’s eyes.

[0049] The vehicle computer 110 can determine the actual pose of the occupant based on sensor 115 data (e.g., image data, seat 220 position data (e.g., longitudinal and vertical positions of the seat bottom relative to the x and z axis lines of the vehicle coordinate system and the pitch of the seat back relative to the y axis line of the vehicle coordinate system, respectively), radar data, etc.) using any suitable technique. For example, when the occupant is seated inside the vehicle 105, the vehicle computer 110 can obtain an image from an image sensor 115 positioned to face the occupant. The vehicle computer 110 can then input the image to a machine learning program that identifies key points. The machine learning program can be a conventional neural network (e.g., OpenPose, Google Research and Machine Intelligence (G-RMI), DL-61, etc.) trained to process images. For example, OpenPose receives an image as input and identifies key points in the image that correspond to body parts (e.g., hands, feet, joints, head, etc.). OpenPose inputs the image to a plurality of convolutional layers that identify the key points in the image based on being trained with a reference dataset such as an a-pose and outputs the key points. The key points include depth data that is not included in the image alone, and the vehicle computer 110 can use the machine learning program (such as OpenPose) to determine the depth data to identify the actual pose of the occupant in the image. That is, the machine learning program outputs the key points as a set of three values: a length along a first axis of a 2D coordinate system in the image; a width along a second axis of the 2D coordinate system in the image; and a depth from the image sensor 115 to the vehicle occupant, which is typically a distance along a third axis that is perpendicular to a plane defined by the first and second axis lines of the image. The vehicle computer 110 can then connect the key points (e.g., using data processing techniques) to determine the actual pose of the occupant.

[0050] After determining the pose of the occupant’s head (e.g., specifying the vehicle coordinates of the keypoints corresponding to the occupant’s head), the vehicle computer 110 can determine the coordinates of the occupant’s eyes relative to the occupant’s head (i.e., in a coordinate system having an origin at the keypoint corresponding to the occupant’s head) (e.g., using known facial feature recognition algorithms). The vehicle computer 110 can then transform the coordinates of the occupant’s eyes relative to the occupant’s head into the vehicle coordinate system based on the pose of the occupant’s head (e.g., according to known coordinate transformation techniques). After determining the coordinates of the occupant’s eyes in the vehicle coordinate system, the vehicle computer 110 determines the coordinates of the corners of the occupant’s eyebox 305 such that the occupant’s eyes are positioned within (e.g., centered in) the occupant’s eyebox 305.

[0051] The vehicle computer 110 is programmed to determine whether a first adjustment is needed on the virtual image plane 200 based on the occupant’s eyebox 305. The vehicle computer 110 can transform the coordinates of the corners of the occupant’s eyebox 305, e.g., based on the vehicle component 125 parameters, as discussed above, to determine the coordinates of the four corners of an occupant virtual image plane 300 (i.e., an image plane that diffracts light into the occupant’s eyebox 305). The vehicle computer 110 can then compare the coordinates of the corners of the occupant virtual image plane 300 to the corresponding coordinates of the corners of the virtual image plane 200. If the coordinates of the corners of the occupant virtual image plane 300 match the corresponding coordinates of the corners of the virtual image plane 200, the vehicle computer 110 determines that no first adjustment is needed. If the coordinates of the corners of the occupant virtual image plane 300 do not match the corresponding coordinates of the corners of the virtual image plane 200, the vehicle computer 110 determines that a first adjustment is needed.

[0052] To perform the first adjustment, the vehicle computer 110 can translate the virtual image plane 200 along at least one of the y-axis and the z-axis of the vehicle coordinate system. That is, the vehicle computer 110 can transform the coordinates specifying the virtual image plane 200 from a first set (e.g., y and z) of coordinates to a second set (e.g., y and z) of coordinates. For example, the vehicle computer 110 can translate the virtual image plane 200 along the y-axis and / or the z-axis such that the coordinates of the corners of the virtual image plane 200 match the coordinates of the corners of the occupant virtual image plane 300. In such an example, the vehicle computer 110 can be programmed to cause or allow the actuators 120 of the projector 155 and / or the SLM 150 to adjust the angle at which light is projected from the projector 155 and / or the SLM 150 toward the windshield 210 in order to translate the virtual image plane 200.

[0053] Additionally or alternatively, to perform the first adjustment, the vehicle computer 110 can translate the virtual image along at least one of the y-axis and the z-axis of the vehicle coordinate system. For example, the vehicle computer 110 can actuate the projector 155 and / or the SLM 150 to project the reference virtual image based on the pixel-by-pixel phase matrix. In such an example, at least a portion of the reference virtual image can not be visible within the occupant eyebox 305. The vehicle computer 110 can translate the reference virtual image along the y-axis and / or the z-axis such that the reference virtual image is visible within the occupant eyebox 305. In such an example, the vehicle computer 110 can determine an updated pixel-by-pixel phase matrix based on translating the reference virtual image along the lateral axis and / or the vertical axis (e.g., according to known matrix transformation techniques) such that the reference virtual image is visible within the occupant eyebox 305. The updated pixel-by-pixel phase matrix can be stored (e.g., in memory of the vehicle computer 110).

[0054] Additionally or alternatively, the vehicle computer 110 can be programmed to perform an adjustment of the seat 220 occupied by the occupant. In such an example, the vehicle computer 110 can actuate the actuator 120 of the seat 220 to move the seat 220 along the y-axis (e.g., relative to a seat track) and / or can actuate the actuator 120 of the seat 220 to rotate the seat back about the y-axis (e.g., relative to a seat bottom). Adjusting the seat 220 can reduce or eliminate the need for the vehicle computer 110 to translate the virtual image plane 200 and / or the virtual image in order for the virtual image projected into the virtual image plane 200 to be visible within the occupant eyebox 305. That is, moving the seat bottom along the y-axis and / or rotating the seat back about the y-axis can at least partially account for the mismatch in coordinates of the corners of the occupant eyebox 305 and the coordinates of the corners of the reference eyebox 205. As another example, the vehicle computer 110 can be programmed to actuate the HMI 118 to output an alert to the occupant to perform an adjustment of the seat 220 in order for the virtual image to be visible within the occupant eyebox 305.

[0055] Figure 4Ais a diagram showing a predetermined distance P from the reference eye movement range 205 to the virtual image plane 200 and a difference between an expected distance D between the occupant eye movement range 305 and the virtual image plane 200. The vehicle computer 110 can determine whether a second adjustment to the virtual image plane 200 is needed based on the expected distance D between the occupant eye movement range 305 and the virtual image plane 200. After determining the expected distance D between the occupant eye movement range 305 and the virtual image plane 200 (discussed below), the vehicle computer 110 can compare the expected distance D to the predetermined distance P. If the expected distance D is equal to the predetermined distance P, the vehicle computer 110 determines that no second adjustment is needed. If the expected distance D is not equal to the predetermined distance P, the vehicle computer 110 determines that a second adjustment is needed.

[0056] The vehicle computer 110 can determine the expected distance D between the occupant eye movement range 305 and the virtual image plane 200 based on occupant data. As used herein, “occupant data” is data specific to the occupant. The occupant data may, for example, identify the occupant’s acuity (i.e., a measure of the spatial resolution of the occupant’s visual processing capabilities, i.e., the spatial resolution at which the occupant can reasonably view content relative to the eye movement range). The vehicle computer 110 may, for example, determine the occupant data based on user input. For example, the vehicle computer 110 can actuate and / or direct the HMI 118 to display virtual buttons corresponding to various acuities that the occupant can select to specify the occupant’s acuity. In other words, the HMI 118 can initiate sensors capable of detecting the occupant selecting a virtual button to specify the occupant’s acuity. Upon detecting the user input, the HMI 118 can provide the user input to the vehicle computer 110, and the vehicle computer 110 can determine the occupant’s acuity based on the user input. Additionally or alternatively, the vehicle computer 110 can determine the occupant data based on the sensor 115 data. For example, the vehicle computer 110 can determine the occupant’s eyeglass prescription based on the image data. In such an example, the classifier can be further trained to accept image data including the occupant’s eyeglasses as input and output the eyeglasses’ prescription.

[0057] The occupant data can also identify an expected distance D between the occupant eyebox 305 and the virtual image plane 200. In such an example, the vehicle computer 110 can actuate and / or direct the HMI 118 to display virtual buttons corresponding to various expected distances Ds at which to position the virtual image plane 200 from the occupant, which the occupant can select to specify a desired distance D between the occupant eyebox 305 and the virtual image plane 200. In other words, the HMI 118 can initiate a sensor capable of detecting an occupant selection of a virtual button to specify an expected distance D between the occupant eyebox 305 and the virtual image plane 200. Upon detecting the user input, the HMI 118 can provide the user input to the vehicle computer 110, and the vehicle computer 110 can determine the expected distance D between the occupant eyebox 305 and the virtual image plane 200 based on the user input.

[0058] The occupant data can also identify a position of the seat 220 occupied by the occupant. The vehicle computer 110 can determine, for example, a position of the seat 220 relative to the x-axis and / or z-axis of the vehicle coordinate system based on a seat position sensor 115 that specifies a relative position of a seat bottom along a seat track and / or a relative height of the seat bottom relative to the seat track. As another example, the vehicle computer 110 can determine a seat back rotation position about the y-axis of the vehicle coordinate system based on a seat back sensor that specifies an angle of inclination of a seat back relative to a seat bottom.

[0059] The vehicle computer 110 can store occupant data for each occupant. For example, the vehicle computer 110 can maintain a lookup table or the like that associates various occupant data with various occupants. The vehicle computer 110 can identify an occupant based on sensor 115 data, for example, via known facial recognition algorithms. The vehicle computer 110 can then access the lookup table to determine occupant data associated with the identified occupant.

[0060] In one example, as discussed above, the vehicle computer 110 can determine an expected distance D between the virtual image plane 200 and the occupant eyebox 305 based on an expected distance D specified by user input. As another example, the vehicle computer 110 can determine an expected distance D between the virtual image plane 200 and the occupant eyebox 305 by inputting occupant data and / or weather data to a neural network, such as a deep neural network (DNN) 500 (see Figure 5The DNN 500 can be trained (as discussed below) to accept occupant data and / or weather data as input and generate output specifying an expected distance D between the occupant eyebox 305 and the virtual image plane 200. The weather data is typically collected by the vehicle 105 sensors 115, but alternatively or additionally, can be provided from a source external to the vehicle 105 (e.g., a remote server computer 140) based on one or more times that the vehicle 105 was at or traveled through a specified location. Determining the expected distance based on the occupant data and / or weather data allows the distance between the occupant eyebox 305 and the virtual image plane 200 to be adapted to account for various conditions that can affect the appearance of the AR image (e.g., various acuities of various occupants, various meteorological visibility metrics (i.e., the distance at which objects or lights can be discerned under given weather conditions), etc.).

[0061] To perform the second adjustment, the vehicle computer 110 can translate the virtual image plane 200 along the x-axis of the vehicle coordinate system. For example, the vehicle computer 110 can translate the virtual image plane 200 along the x-axis by adding (or subtracting) the difference between the expected distance D and the predetermined distance P to the x-coordinate of the corner of the virtual image plane 200. That is, the vehicle computer 110 can translate the virtual image plane 200 such that the virtual image plane 200 is spaced apart from the occupant eyebox 305 by the expected distance D. In such an example, the vehicle computer 110 can also be programmed to update the pixel-by-pixel phase matrix (e.g., according to known matrix transformation techniques) such that the virtual image is projected into the virtual image plane 200 after the second adjustment is performed.

[0062] Additionally or alternatively, the vehicle computer 110 can be programmed to perform an adjustment of the seat 220 that is occupied by the occupant. In such an example, the vehicle computer 110 can actuate the actuator 120 of the seat 220 to move the seat 220 along the x-axis (e.g., along a seat track) and / or can actuate the actuator 120 of the seat 220 to rotate the seat back about the y-axis (e.g., relative to the seat bottom). For example, as discussed above, the vehicle computer 110 can be programmed to determine the expected distance D between the occupant eyebox 305 and the virtual image plane 200 based on the occupant data and / or the weather data. The vehicle computer 110 can then actuate the actuator 120 of the seat 220 to move the seat 220 along the x-axis and / or to rotate the seat back about the y-axis such that the seat 220 is spaced apart from the virtual image plane 200 by the expected distance D. Figure 4BAs shown in the middle, the vehicle computer 110 can actuate the seat 220 to move along the seat tracks such that the expected distance D is equal to the predetermined distance P. Adjusting the seat 220 can reduce or eliminate the need for the vehicle computer 110 to translate the virtual image plane 200 in order to make the virtual image projected into the virtual image plane 200 visible to the occupant eyebox 305. That is, moving the seat bottom along the x-axis and / or rotating the seat back about the y-axis can account for at least a portion of the difference between the expected distance D and the predetermined distance P. As another example, the vehicle computer 110 can be programmed to actuate the HMI 118 to output an alert to the occupant to perform an adjustment to the seat 220 in order to make the virtual image visible within the occupant eyebox 305.

[0063] After performing the first adjustment and / or the second adjustment, the vehicle computer actuates the projector 155 to provide the virtual image to the SLM 150. Providing the virtual image to the SLM 150 allows the vehicle computer 110 to output an augmented reality image onto the windshield 210. Specifically, the vehicle computer 110 actuates the SLM 150 to output the virtual image into the virtual image plane 200 based on the updated pixel-by-pixel phase matrix (e.g., after the first adjustment and / or the second adjustment). That is, the SLM 150 receives the virtual image as input and spatially modulates the virtual image according to the updated pixel-by-pixel phase matrix to output an augmented reality image. The augmented reality image is provided in the virtual image plane 200 and is visible in the occupant eyebox 305 for visibility by the occupant. The vehicle computer 110 can determine a respective occupant eyebox 305 for each detected occupant in the vehicle 105 and can output a respective augmented reality image in a respective virtual image plane after performing a respective first adjustment and / or a second adjustment on the respective virtual image plane, the respective augmented reality image being visible in a corresponding occupant eyebox 305 for visibility by the respective occupant.

[0064] Figure 5 is an illustration of a deep neural network (DNN) 500 that can be trained to determine the expected distance D between the occupant eyebox 305 and the virtual image plane 200. For example, the DNN 500 can be a software program that is loadable into memory and executed by a processor included in a computer. In example implementations, the DNN 500 can include, but is not limited to, a convolutional neural network (CNN), a R-CNN (Region-CNN), a Fast R-CNN, and a Faster R-CNN. The DNN includes a plurality of nodes and the nodes are arranged such that the DNN 500 includes an input layer, one or more hidden layers, and an output layer. Each layer of the DNN 500 can include a plurality of nodes 505. Although the DNN 500 is illustrated as including a single input layer, a single output layer, and a single hidden layer, the DNN 500 can include multiple input layers, multiple output layers, and multiple hidden layers. For example, the DNN 500 can include a first hidden layer and a second hidden layer. The first hidden layer can be arranged between the input layer and the second hidden layer, and the second hidden layer can be arranged between the first hidden layer and the output layer. The DNN 500 can include any number of hidden layers. Figure 3Three (3) hidden layers are shown, but it is understood that the DNN 500 can include more or fewer hidden layers. The input and output layers can also include more than one (1) node 505.

[0065] Because the nodes 505 are designed to mimic biological (e.g., human) neurons, they are sometimes referred to as artificial neurons 505. The input set for each neuron 505 (indicated by the arrows) is each multiplied by a respective weight. The weighted inputs can then be summed in an input function to provide a net input, which can be adjusted by a bias, if any. The net input can then be provided to an activation function, which in turn provides an output for the connected neuron 505. The activation function can be a variety of suitable functions, often selected based on empirical analysis. As shown by the arrow in Figure 5 The output of the neuron 505 can then be provided to be included in the input set of one or more neurons 505 in the next layer, as shown by the arrow in

[0066] As one example, the DNN 500 can be trained by ground truth data (i.e., data about real world conditions or states). For example, the DNN 500 can be trained by ground truth data and / or updated by a processor of the remote server computer 140 with additional data. For example, the weights can be initialized by using a Gaussian distribution, and the bias of each node 505 can be set to zero. Training the DNN 500 can include updating the weights and biases via suitable techniques, such as backpropagation with optimization. The ground truth data can include, but is not limited to, data specifying objects (e.g., occupants, vehicles, etc.) within an image or data specifying physical parameters. For example, the ground truth data can be data representing objects and object labels. In another example, the ground truth data can be data representing an expected distance at which a virtual image is visible to various occupants in the virtual image plane 200 given corresponding occupant visual acuity and weather conditions.

[0067] During operation, the vehicle computer 110 determines occupant data and / or weather data (as discussed above) and provides the occupant data and / or weather data to the DNN 500. The DNN 500 generates a prediction based on the received input. The output is an expected distance D between the occupant eyebox 305 and the virtual image plane 200 given the occupant data and / or weather data.

[0068] Figure 6 is a diagram of an example process 600 for operating a vehicle 105. The process 600 starts in block 605. The process 600 can be implemented by a vehicle computer 110 included in the vehicle 105 executing program instructions stored in its memory.

[0069] In block 605, the vehicle computer 110 determines whether an occupant is present in the passenger cabin 215 of the vehicle 105. As discussed above, the vehicle computer 110 can detect the presence of an occupant based on sensor 115 data. If the vehicle computer 110 detects the presence of an occupant, the process 600 continues in block 610. Otherwise, the process 600 remains in block 605.

[0070] In block 610, the vehicle computer 110 determines an occupant eyebox 305 for the occupant based on the actual pose of the occupant, as discussed above. Additionally, the vehicle computer 110 can determine an occupant virtual image plane 300 based on the occupant eyebox 305, as discussed above. The process 600 continues in block 615.

[0071] In block 615, the vehicle computer 110 determines whether a first adjustment of the virtual image plane 200 is needed based on the occupant eyebox 305. Prior to performing the first adjustment, the virtual image plane 200 corresponds to the reference eyebox 205, as discussed above. The vehicle computer 110 can compare the occupant virtual image plane 300 to the virtual image plane 200, as discussed above. If the virtual image plane 200 does not match the occupant image plane 300, as discussed above, the vehicle computer 110 determines that the first adjustment is needed. If the vehicle computer 110 determines that the first adjustment is needed, the process 600 continues in block 620. Otherwise, the process 600 continues in block 625.

[0072] In block 620, the vehicle computer 110 performs the first adjustment. For example, the vehicle computer 110 can translate the virtual image plane 200 along at least one of the y-axis and z-axis of the vehicle coordinate system to match the occupant virtual image plane 300, as discussed above. As another example, the vehicle computer 110 can translate virtual images projected into the virtual image plane 200 relative to the virtual image plane 200 so as to be visible within the occupant eyebox 305, as discussed above. Additionally or alternatively, the vehicle computer 110 can perform an adjustment of the seat 220 occupied by the occupant so as to make virtual images projected into the virtual image plane 200 visible within the occupant eyebox 305, as discussed above. The process 600 continues in block 625.

[0073] In block 625, the vehicle computer 110 determines whether a second adjustment to the virtual image plane 200 is needed based on at least one of the occupant data and the weather data. As discussed above, the vehicle computer 110 determines an expected distance D between the occupant eyebox 305 and the virtual image plane 200 based on the occupant data and / or the weather data. If the expected distance D is not equal to the predetermined distance P, as discussed above, the vehicle computer 110 determines to perform a second adjustment. If the vehicle computer 110 determines to perform the second adjustment, the process 600 continues in block 630. Otherwise, the process 600 continues in block 635.

[0074] In block 630, the vehicle computer 110 performs the second adjustment. For example, as discussed above, the vehicle computer 110 can translate the virtual image plane 200 along the x-axis of the vehicle coordinate system based on a difference between the expected distance D and the predetermined distance P. Additionally or alternatively, the vehicle computer 110 can perform an adjustment of the seat 220 occupied by the occupant, thus making the virtual image projected into the virtual image plane 200 visible within the occupant eyebox 305, as discussed above. The process 600 continues in block 635.

[0075] In block 635, the vehicle computer 110 actuates the projector 155 and the SLM 150 to output the virtual image into the virtual image plane 200, as discussed above. After block 635, the process 600 ends.

[0076] The systems and methods described herein can be modified and / or omitted depending on the circumstances, situations, and applicable rules and regulations. Furthermore, the user should use good judgment and common sense when operating a vehicle, regardless of the actions that the vehicle (such as a computer controlling the speed and / or acceleration of the vehicle) can take. The operations described herein should always be implemented and / or performed in accordance with the user manual and safety guidelines.

[0077] Generally, the computing systems and / or devices described can employ any one of a number of computer operating systems, including, but in no way limited to, versions and / or varieties of the Ford SYNC® application; AppLink / Smart Device Link middleware; Microsoft Windows® operating system; Microsoft Windows® operating system; Unix operating systems (e.g., the operating systems); the AIX UNIX operating system distributed by International Business Machines Corporation in Armonk, New York; the Linux operating system; the Mac OS and iOS operating systems distributed by Apple Inc. in Cupertino, California; the BlackBerry operating system distributed by BlackBerry Limited in Waterloo, Canada; and the Android operating system developed by Google and the Open Handset Alliance, or the CAR infotainment platform. Examples of computing devices include, but are not limited to, an onboard first computer, a computer workstation, a server, a desktop computer, a notebook computer, a laptop computer, or a handheld computer, or some other computing system and / or device.

[0078] Computers and computing devices generally include computer-executable instructions, where the instructions can be executable by one or more computing devices, such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages and / or technologies, including, without limitation, and either alone or in combination, Java TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Perl, HTML, and / or the like. Some of these applications can be compiled and executed on a virtual machine, such as the Java Virtual Machine, Dalvik Virtual Machine, and / or the like. Generally, a processor (e.g., a microprocessor) receives instructions, e.g., from a memory, computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer-readable medium, such as a storage medium, random access memory, and / or the like.

[0079] Memory can include computer-readable media (also referred to as processor-readable media) including any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to nonvolatile media and volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memory. Volatile media can include, for example, dynamic random access memory (DRAM), which typically constitutes a main memory. Such instructions can be transmitted by one or more transmission media including coaxial cables, copper wire, and fiber optics, including the wires that comprise a system bus coupled to a processor of a processing machine. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a solid state drive, a magnetic tape, or any other magnetic medium, a punch card, a paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH- EPROM, any other memory chip or cartridge, a cartridge, a cassette, any other computer- readable medium from which a computer can read.

[0080] A database, data repository, or other data store described herein can include various mechanisms for storing, accessing, and retrieving a variety of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), and others. Each such data store is generally included within a computing device employing a computer operating system such as one of those mentioned above, and is accessed via a network in any one or more of a variety of manners. A file system can be accessed from the computer operating system, and can include files stored in various formats. An RDBMS, typically, employs the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures such as the PL / SQL language described above.

[0081] In some examples, system elements can be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media (e.g., disks, memories, etc.) associated therewith. A computer program product can include such instructions stored on a computer-readable medium for implementing the functionality described herein.

[0082] With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring in a certain order, such processes could be practiced with the steps taken in an order different than described herein without departing from the spirit and scope of such a process. It should also be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and claim should be interpreted only in the light of the appended claims.

[0083] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent to those of skill in the art upon reading the above description. The scope of the application should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the present application is capable of modification and variation and is limited only by the following claims.

[0084] Unless specifically indicated otherwise, all terms used in the claims are intended to have the ordinary and customary meaning as understood by one of skill in the art. Specifically, the use of singular articles such as “a,” “the,” and “said” should be read to connote one or more of the indicated elements, unless the context clearly indicates otherwise.

[0085] According to the present application, there is provided a system having a computer having a processor and a memory storing instructions executable by the processor to determine a virtual image plane relative to a reference eye movement range, wherein a virtual image projected into the virtual image plane is visible in the reference eye movement range; determine an occupant eye movement range from sensor data; and perform a first adjustment of the virtual image plane based on the occupant eye movement range such that the virtual image projected into the virtual image plane is visible in the occupant eye movement range.

[0086] According to an embodiment, the first adjustment comprises translating the virtual image plane along at least one of a lateral axis partially defining the virtual image plane and a vertical axis partially defining the virtual image plane and extending perpendicular to the lateral axis.

[0087] According to an embodiment, the first adjustment comprises translating the virtual image within the virtual image plane.

[0088] According to an embodiment, the instructions further comprise instructions to perform an adjustment of a position of a seat occupied by an occupant such that the virtual image is visible in the occupant eye movement range.

[0089] According to an embodiment, the instructions further comprise instructions to perform a second adjustment of the virtual image plane based on occupant data of an occupant.

[0090] According to an embodiment, the second adjustment comprises translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane.

[0091] According to an embodiment, the instructions further comprise instructions to input the occupant data of the occupant into a machine learning program that outputs an expected distance from the occupant eyebox to the virtual image plane.

[0092] According to an embodiment, the second adjustment comprises translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane such that the virtual image plane is spaced apart from the occupant eyebox along the longitudinal axis by the expected distance.

[0093] According to an embodiment, the instructions further comprise instructions to perform the second adjustment based on weather data in addition to the occupant data.

[0094] According to an embodiment, the instructions further comprise instructions to perform an adjustment to a position of a seat occupied by the occupant such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0095] According to the invention, a method comprises determining a virtual image plane relative to a reference eyebox, wherein a virtual image projected into the virtual image plane is visible in the reference eyebox; determining an occupant eyebox from sensor data; and performing a first adjustment to the virtual image plane based on the occupant eyebox such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

[0096] In one aspect of the invention, the first adjustment comprises translating the virtual image plane along at least one of a lateral axis that partially defines the virtual image plane and a vertical axis that partially defines the virtual image plane and extends perpendicular to the lateral axis.

[0097] In one aspect of the invention, the first adjustment comprises translating the virtual image within the virtual image plane.

[0098] In one aspect of the invention, the method comprises performing an adjustment to a position of a seat occupied by an occupant such that the virtual image is visible in the occupant eyebox.

[0099] In one aspect of the invention, the method comprises performing a second adjustment to the virtual image plane based on occupant data of an occupant.

[0100] In one aspect of the invention, the second adjustment comprises translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane.

[0101] In one aspect of the application, the method includes inputting the occupant data of the occupant into a machine learning program that outputs an expected distance from the occupant eyebox to the virtual image plane.

[0102] In one aspect of the application, the second adjustment includes translating the virtual image plane along a longitudinal axis extending perpendicular to the virtual image plane such that the virtual image plane is spaced apart from the occupant eyebox along the longitudinal axis by the expected distance.

[0103] In one aspect of the application, the method includes performing the second adjustment based on weather data in addition to the occupant data.

[0104] In one aspect of the application, the method includes performing an adjustment to a position of a seat occupied by the occupant such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

Claims

1. A method comprising: determining a virtual image plane relative to a reference eyebox, wherein a virtual image projected into the virtual image plane is visible in the reference eyebox; determining an occupant eyebox from sensor data; and performing a first adjustment of the virtual image plane based on the occupant eyebox such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

2. The method of claim 1, wherein the first adjustment comprises translating the virtual image plane along at least one of a lateral axis that partially defines the virtual image plane and a vertical axis that partially defines the virtual image plane and extends perpendicular to the lateral axis.

3. The method of claim 1, wherein the first adjustment comprises translating the virtual image within the virtual image plane.

4. The method of claim 1, further comprising performing an adjustment to a position of a seat occupied by an occupant such that the virtual image is visible in the occupant eyebox.

5. The method of claim 1, further comprising performing a second adjustment of the virtual image plane based on occupant data of an occupant.

6. The method of claim 5, wherein the second adjustment comprises translating the virtual image plane along a longitudinal axis that extends perpendicular to the virtual image plane.

7. The method of claim 5, further comprising inputting the occupant data of the occupant into a machine learning program that outputs an expected distance from the occupant eyebox to the virtual image plane.

8. The method of claim 7, wherein the second adjustment comprises translating the virtual image plane along a longitudinal axis that extends perpendicular to the virtual image plane such that the virtual image plane is spaced apart from the occupant eyebox along the longitudinal axis by the expected distance.

9. The method of claim 5, further comprising performing the second adjustment based on weather data in addition to the occupant data.

10. The method of claim 5, further comprising performing an adjustment to a position of a seat occupied by the occupant such that the virtual image projected into the virtual image plane is visible in the occupant eyebox.

11. A computer programmed to perform the method of any one of claims 1-10.

12. A computer program product comprising instructions for performing the method of any one of claims 1-10.

13. A vehicle comprising a computer programmed to perform the method of any one of claims 1-10.

Images