Method for controlling the operation of a vehicle based on user input from a user of the vehicle into a graphic projection display

Abstract

A method for controlling operation of a vehicle (100) based on a user input (312) from a user (10, 910) of the vehicle (100) into a graphic projection display (250, 900) that displays an aligned graphic (920, 922, 930, 932) on a view ahead of the vehicle (100), the method comprising: Projecting the aligned graphic (920, 922, 930, 932) onto the view in front of the vehicle (100) on the graphic projection display (250, 900), comprising: monitoring a command selection to be presented to the user (10, 910) of the vehicle (100); the view in front of the vehicle (100) is monitored; a location of the eyes of the user (10, 910) of the vehicle (100) is monitored; and the aligned graphic (920, 922, 930, 932) is determined based on the command selection, the view in front of the vehicle (100) and the location of the eyes of the user (10, 910) of the vehicle (100); where monitoring command selection includes: Monitoring information regarding the operation of the vehicle (100) by an autonomous method of operating the vehicle (100) or by a semi-autonomous method of operating the vehicle (100); Determining configurable options for operating the vehicle (100) by the autonomous method for operating the vehicle (100) or by the semi-autonomous method for operating the vehicle (100) based on the monitored information; and Determining the command selection to be presented to the user (10, 910) of the vehicle (100) based on the determined configurable options; Monitoring the user input (312) to the graphic projection display (250, 900), comprising monitoring a hand gesture specified for the graphic projection display (250, 900) and a location of a hand relative to the eyes of the user (10, 910); and Controlling the operation of the vehicle (100) described by a vehicle speed, an engine speed and a wheel slip based on the user input (312) to the graphic projection display (250, 900), wherein the vehicle (100) is operated by an autonomous method of operating the vehicle (100) or by a semi-autonomous method of operating the vehicle (100).

Description

TECHNICAL FIELD

[0001] This disclosure relates to a method for controlling the operation of a vehicle based on user input from a user of the vehicle to a graphic projection display. BACKGROUND

[0002] Presenting information to the vehicle operator in an effective manner is desirable and reduces operator strain. Head-up displays project light onto a screen, converting the light into a visible image on the screen. Head-up displays are known to present information to the vehicle operator in an effective manner, reducing operator strain while allowing the operator to remain focused on driving.

[0003] For example, US 2007 / 0 057 781 A1 discloses the ability to select different systems, such as navigation, internet, heating, or telephone, via a head-up display. A comparable system of this type is described in DE 10 2009 046 376 A1.

[0004] Vehicle systems that utilize driving aids such as adaptive cruise control (ACC), automatic lateral control, collision avoidance or preparation systems, and lane-keeping aids monitor and process information regarding the operating environment surrounding the vehicle. Information is available from a variety of sources to locate the vehicle relative to a 3D map database, plan a route for the vehicle to a destination, and correlate that route with available information regarding the route. Onboard vehicle systems provide a wide variety of information that can be used to enhance vehicle control. Vehicle-to-vehicle communications are known to utilize data collected within one vehicle when communicating with vehicles elsewhere on the road.It is known that infrared sensors detect features in a view based on the difference in infrared light emissions.

[0005] DE 10 2008 054 641 A1 describes a control device by means of which a vehicle can be shown in different scales on a display to facilitate a parking process.

[0006] Input from a vehicle operator or passenger can be monitored or received from a number of sources. Buttons, levers, steering wheels, pedals, and other similar input devices allow a person to input control settings into a vehicle. Input to a graphical device such as a touchscreen display is known. SUMMARY

[0007] A method is proposed for controlling the operation of a vehicle based on a user input from a user of the vehicle to a graphic projection display that presents an aligned graphic on a view in front of the vehicle, the method comprising the features of claim 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] One or more embodiments are described by way of example with reference to the accompanying drawings, in which: Fig. 1 illustrates an exemplary vehicle equipped with an enhanced image recognition system in accordance with the present disclosure; Fig. 2 illustrates exemplary embodiments of the graphics projection display according to the present disclosure; Fig. 3 illustrates a substantially transparent display according to the present disclosure; Fig.4 illustrates an exemplary user input module for receiving user inputs specified for a portion of the graphics projection display, in accordance with the present disclosure; Fig. 5 illustrates an exemplary information flow for dynamically aligning a graphic to a driving scene using the graphic projection display including an EVS system manager that monitors inputs to generate display requests, in accordance with the present disclosure; Fig. 6 illustrates an exemplary information flow for determining a graphic based on inputs to an EVS system manager in accordance with the present disclosure; Fig. 7 illustrates an exemplary radiation source capable of emitting light into a limited field of view, in accordance with the present disclosure; Fig.8 illustrates an exemplary process for creating the necessary pattern of radiation sources aligned with a polymer substrate to enable viewing in a limited field of view, in accordance with the present disclosure; Fig. 9 illustrates an exemplary process for operating the methods disclosed herein in accordance with the present disclosure; and Fig. 10 illustrates an exemplary view through a see-through windshield graphics projection display and user input to aligned graphics on the display in accordance with the present disclosure. DETAILED DESCRIPTION

[0009] In the drawings, in which the representations serve only to illustrate certain exemplary embodiments, Fig.1 shows an exemplary vehicle equipped with an enhanced image recognition system. An exemplary EVS system is disclosed in US application US 2010 / 0 253 593 A1, which corresponds to DE 10 2010 013 534 A1. The exemplary vehicle 100 includes an EVS system manager 110, a user input module (UIM) 515, vehicle sensor systems (including a camera system 120 and a radar system 125), vehicle operating sensors (including a vehicle speed sensor 130), information systems (including a GPS device 140 and a wireless communication system 145), a head-up display (HUD) 150, an EVS graphics system 155, a graphics projection system 158, and an occupant eye location detection system 160. The EVS system manager 110 can communicate directly with various systems and components or the EVS system manager can alternatively or additionally communicate via a LAN / CAN system 115.The EVS system manager 110 communicates with the EVS graphics system 155 to project graphics onto the HUD 150. However, the EVS system manager 110 may communicate with the EVS graphics system 155 to instead or additionally project graphics onto other surfaces within the vehicle to perform methods disclosed herein. As will be described in more detail with reference to FIG. Fig.4, the UIM 515 may receive user inputs specified for a portion of a display device, with the user inputs being monitored by the EVS system manager 110. A camera system 120 includes a camera or image capture device that periodically or sequentially captures images representing a view from within the vehicle. A radar system 125 includes a device known in the art that utilizes electromagnetic radiation to detect other vehicles or objects in proximity to the vehicle. Within a vehicle, a number of known on-board sensors are widely used to monitor vehicle speed, engine speed, wheel slip, and other parameters describing vehicle operation.Vehicle speed sensor 130 represents one such vehicle-integrated sensor, but the scope of the disclosure includes any such sensors for use by the EVS. A GPS device 140 and a wireless communication system 145 communicate with resources external to the vehicle, such as a satellite system 180 and a cellular telecommunications tower 190. GPS device 140 may be used in conjunction with a 3D map data database containing detailed information regarding a global coordinate received by GPS device 140 regarding the current location of the vehicle. Information from the vehicle sensor systems and vehicle operational sensors may be used by EVS system manager 110 to monitor the current orientation of the vehicle.The HUD 150 includes a windshield equipped with features capable of displaying an image projected onto it while remaining transparent or substantially transparent, allowing vehicle occupants to clearly observe the vehicle's exterior environment through the windshield. The HUD 150 may include a portion of the windshield, such as a portion including the driver's side half of the windshield, or the HUD 150 may include substantially the entire windshield. It is understood that while the HUD 150 includes the windshield at the front of the vehicle, other surfaces within the vehicle, including side windows and a rear window, could be used for projection. Additionally, the view on the front windshield could be continued as a continuous image onto the vehicle's front "A-pillars" and side windows.The EVS graphics engine 155 includes display software or programming that translates information display requests from the EVS system manager 110 into graphical representations of the information. The EVS graphics engine 155 includes programming to compensate for the curved and sloped surface of the windshield and any other surfaces (e.g., the windshield shown in FIG. Fig.2) onto which the graphics are to be projected. The EVS graphics engine 155 controls the graphics projection system 158, including a laser or projector device that generates excitation light for projecting the graphics. An exemplary occupant eye location detection system 160 includes sensors known in the art for approximating a location of an occupant's head and also the orientation or location toward which the occupant's eyes are facing. Based on the output of the occupant eye location detection system 160, the current location and orientation of the vehicle 100, and a user input location, the EVS system manager 110 can dynamically accurately align the graphics with the HUD 150 so that the occupant sees the images overlaid with visual images through the windshield.The dynamic alignment of graphics is achieved using . Fig. 5 discussed in more detail.

[0010] Information may be presented to a user of a vehicle according to a number of exemplary embodiments. A number of video devices may be used to present information to the user. However, presenting the information within a user's context of a view of the vehicle's operating environment reduces the visual complexity for controlling the vehicle. A graphics projection display may be used to display graphics in the context of a view in front of the vehicle. A graphics projection display 250 and the associated graphics may be used according to a number of exemplary embodiments. Fig.Figure 2 illustrates exemplary embodiments of the graphics projection display, including a substantially transparent windshield HUD 150, a touchscreen display 260, a human-machine interface (HMI) 151 including a display, opaque components 280, and a display monitor 270 installed within the vehicle interior. It should be appreciated that any of the illustrated embodiments, including graphics projection display 250, may project graphics onto the display corresponding to and aligned with a view of the vehicle's driving scene.

[0011] According to one embodiment, the graphic projection display 250 may include a HUD 150 that presents information to the user in the context of the view through the HUD. The HUD is a transparent windshield with small luminous particles or structures embedded within the windshield. As a result, the user can still see through the windshield and, for example, observe another vehicle in traffic in front of the user's vehicle. When the laser excitation light is projected onto the windshield, the luminous particles or structures emit visible light rearward toward the user. However, this emitted light does not interfere with the user's view through the windshield.For example, the user may continue to see the vehicle in front of the user's vehicle, and the EVS system manager may instruct a graphic containing a box to be drawn at a location on the windshield and at a proper size so that the user sees a dynamically oriented box drawn around the vehicle in front of the user's vehicle. As a result, the HUD 150 may act as a graphic projection display 250, projecting graphics overlaid on, or corresponding to, a view of the vehicle's driving scene.

[0012] Fig.3 illustrates a substantially see-through display. Viewer 10 can view any object (e.g., a cube 12) through a substrate 14. Substrate 14 can be see-through or substantially see-through. While viewer 10 views any object 12 through substrate 14, the viewer can also view images (e.g., a circle 15 and a triangle 16) formed on substrate 14. Substrate 14 can be part of a vehicle windshield, a glass substrate, a plastic substrate, a polymer substrate, or other see-through (or substantially see-through) medium that one of ordinary skill in the art will recognize. Other substrates can complement substrate 14 to provide tinting, substrate protection, light filtering (e.g., external ultraviolet light filtering), and other functions.

[0013] Fig.Figure 3 illustrates illumination of substrate 14, which is illuminated with excitation light (e.g., ultraviolet light or infrared light) from light sources (e.g., a projector or laser) represented by device 20. The received excitation light may be absorbed by light-emitting material on substrate 14. When the light-emitting material receives the excitation light, the light-emitting material may emit visible light. Accordingly, by selectively illuminating substrate 14 with excitation light, images (e.g., a circle 15 and a triangle 16) may be formed on substrate 14.

[0014] In one embodiment, the excitation light is output from a device 20 including a projector. The projector may be a digital projector. The projector may be a micromirror array (MMA) projector (e.g., a digital light processing (DLP) projector). An MMA projector that outputs ultraviolet light may be similar to an MMA projector that outputs visible light, except that the color wheel has light filters adapted to the ultraviolet light spectrum. The projector may be a liquid crystal display (LCD) projector. The projector may be a liquid crystal on silicon (LCOS) projector. The projector may be an analog projector (e.g., a slide film projector or a film projector). Those of ordinary skill in the art will recognize other types of projectors that may be used to project ultraviolet light onto the substrate 14.

[0015] In another embodiment, an excitation light is output from device 20, which includes a laser. The intensity and / or motion of a laser beam output from device 20 can be modulated to create an image in substrate 14. In down-conversion embodiments, the output from the laser can be ultraviolet light. In up-conversion embodiments, the output from the laser can be infrared light.

[0016] Fig.Figure 3 illustrates light-emitting material (e.g., light-emitting particles 22) dispersed in a substantially transparent substrate. When excitation light is absorbed by the light-emitting particles 22, the light-emitting particles emit visible light. Accordingly, in down-conversion embodiments, when ultraviolet light is absorbed by the light-emitting particles 22, visible light is emitted by the light-emitting particles. Likewise, in up-conversion embodiments, when infrared light is absorbed by the light-emitting particles 22, visible light is emitted by the light-emitting particles.

[0017] Fig.Figure 3 illustrates light-emitting material containing light-emitting particles 22 dispersed in a substantially transparent substrate. These light-emitting particles 22 may be substantially similar particles throughout, or the composition of the particles may be as shown in Fig.3. When excitation light is absorbed by the light-emitting particles 22, the particles emit visible light. Accordingly, in down-conversion embodiments, when ultraviolet light is absorbed by the light-emitting particles, visible light is emitted by the light-emitting particles. Likewise, in up-conversion embodiments, when infrared light is absorbed by the light-emitting particles, visible light is emitted by the light-emitting particles. Each light-emitting particle may be a different type of light-emitting material that emits a different range of wavelengths of visible light in response to a different range of wavelengths of excitation light (e.g., ultraviolet or infrared light).

[0018] The light-emitting particles 22 may be dispersed throughout the substrate 14. As in Fig.3, the particles may alternatively be disposed on a surface of the substrate 14. Light-emitting particles 22 may be integrated into the substrate 14 by being coated onto the substrate 14. Light-emitting material may be fluorescent material that emits visible light in response to the absorption of electromagnetic radiation (e.g., visible light, ultraviolet light, or infrared light) having a different wavelength than the emitted visible light. The size of the particles may be smaller than the wavelength of visible light, which may reduce or eliminate scattering of visible light by the particles. Examples of particles smaller than the wavelength of visible light are nanoparticles or molecules. Each of the light-emitting particles may have a diameter less than about 400 nanometers.According to embodiments, each of the light-emitting particles may have a diameter less than about 300 nanometers, less than about 200 nanometers, less than about 100 nanometers, or less than about 50 nanometers. The light-emitting particles may be single molecules.

[0019] According to another embodiment of the graphic projection display 250, a screen may be used to present information to the user in the context of the view. An exemplary HMI device 151 includes a display associated therewith for receiving a projected image of a view corresponding to the view of a driving scene of the vehicle and displaying information to the user overlaid or coordinated with the view. The HMI 151 includes a computing device where the operator of the vehicle can input commands for controlling various systems of the vehicle in signal communication with the HMI 151 and for receiving desired information. For example, the user can provide input using the HMI 151 based on the scene visible through or on the display. Methods of projection onto HMI displays are known in the art, and this disclosure is not intended to be limited to any projection method.The HMI 151 can act as a graphics projection display 250 that overlays or projects graphics onto a view of the vehicle's driving scene.

[0020] According to another embodiment of the graphic projection display 250, a touchscreen device 260 may be utilized. Touchscreen devices are well known in the art and will not be discussed in detail here. A touchscreen device 260 may act as the graphic projection display 250, overlaying or projecting graphics onto a view of the vehicle's driving scene.

[0021] According to another embodiment of the graphics projection display 250, the graphics projection display may include non-transparent components 280 for receiving a projected image of a view corresponding to the vehicle's driving scene and for displaying a graphic overlaid or corresponding to the view. The non-transparent components 250 may include an instrument panel, seats, headrests, structural pillars, roofs, sun visors, and steering wheels. The non-transparent components 250 may include a transparent phosphor layer responsive to an excitation projector for displaying graphics. Projection methods are discussed above with respect to the substantially transparent windshield HUD 150 and are not discussed with respect to the non-transparent component 280. US application US 2010 / 0 253 596 A1 discloses projecting images and graphics onto non-transparent components.In alternative embodiments, the opaque component 250 may have the features described below with reference to FIG. Fig. 7 and Fig. 8 discussed architecture with limited pixelated field of view when the projection of dynamically aligned graphics onto the HUD 150 and alternatively onto the opaque components 280 is discussed.

[0022] Methods for displaying information to a user of a vehicle via a graphic projection display 250 may provide details to the user regarding the operating environment of the vehicle and the operation of the vehicle. For example, the graphic projection display 250 may present navigation details to the user regarding a planned route. If the vehicle is traveling on a highway and the planned route includes the vehicle using an upcoming merge lane, a graphic may be displayed on the graphic projection display 250 highlighting merge lane signs, highlighting lane markings for the merge lane, prompting a lane change in preparation for using the merge lane, and utilizing a graphic, such as an arrow indicator aligned with the surface of the merge lane, to indicate to the user where to go.In another example, if information from a wireless connection to the Internet includes heavy or stopped traffic along a planned route, multiple options including, for example, text information, highlighted characters, and an inset map may be displayed in the graphic projection display 250. In another example, the user may initiate a query regarding potential points of interest, including, for example, gas stations, restaurants, shopping plazas, or other similar destinations. When the vehicle approaches a certain distance, information regarding these points may be displayed, including prices, reviews or ratings by other advertiser customers (e.g., displaying an average star rating out of five), aligned graphics depicting the location or where the parking space is located, or other information.In another example, details regarding the operation of the vehicle may be displayed, e.g., information regarding operation at a current speed, including the time until a planned target is reached, and specific fuel consumption, and estimated optimal operation at a different speed for improved specific fuel consumption. A number of embodiments of the information that may be displayed are conceivable.

[0023] In addition to information regarding a route the vehicle can take or operating information of the vehicle driven by the user, information regarding a number of autonomous or semi-autonomous methods for operating a vehicle can be displayed on the graphic projection display 250. Example autonomous or semi-autonomous methods utilize sensor inputs to control the speed and / or steer the vehicle. The methods include adaptive cruise control, lane keeping, lane changing, and collision preparation and avoidance. Example autonomous and semi-autonomous methods are disclosed in detail in the above-mentioned US application US 2010 / 0 253 593 A1. Information regarding the operation of autonomous and semi-autonomous methods, e.g.A current distance or a desired distance the vehicle is attempting to achieve; highlighting lanes currently being followed, including a metric representing how close the vehicle is to the center of the current lane; indicating an impending lane change to the user; and hazard assessments representing the relative positions and trajectories of surrounding vehicles, pedestrians, and other objects may be displayed. A number of autonomous and semi-autonomous methods for controlling a vehicle, as well as related information that can be communicated to a user or vehicle occupants, are conceivable.

[0024] Various types of information may be presented to a user of a vehicle on a graphic projection display 250. Much of the information that may be presented includes options or may prompt for input from the user. For example, a user may wish to change priorities or alter a planned route en route to a planned destination. Known methods for changing a route include the user focusing on a map display or entering text information into a field. These methods require the user to shift their viewpoint away from the vehicle's driving scene, for example, paying less attention to other vehicles in the vicinity of the user's vehicle. Furthermore, this information, which is disconnected from the driving scene, may be difficult to quickly interpret.A method that monitors user input via the graphic projection display 250 and determines a control command based on the user input may enable the user to provide the input while maintaining focus on the driving scene and with enhanced context for the driving scene.

[0025] The EVS system manager 110 includes a programmable processor that provides programming for monitoring a user input specified for a portion of the graphic projection display 250 (see Fig. 2) and determines a graphic representing operating characteristics based on the monitored user input. It should be noted that the graphics displayed on the display can be used to provide control inputs to the display, the vehicle control systems, or another associated system.

[0026] Based on Fig.4 illustrates an exemplary user input module (UIM) 515 for receiving user inputs specified for a portion of the graphic projection display 250. The user inputs are monitored by the EVS system manager 110 and may be used to display graphics on the graphic projection display 250 according to the driving scene. The UIM 515 includes embodiments of user input devices and modules, and the exemplary vehicle may include only one of the input devices and modules included within the UIM 515 for receiving and monitoring a user input specified for a portion of the graphic projection display 250. However, embodiments are contemplated where the vehicle may include variations of the user input devices and modules included within the UIM 515, and selected ones may be enabled based on a vehicle user's preference.The example UIM 515 includes an imaging device 161, a touchscreen input module (TSIM) 149, a controller device 153, and a keyboard device 169.

[0027] The TSIM 149 may be configured to receive touch gestures from a user's hand or finger on the module, which are specified for a portion of the graphics projection display corresponding to the driving scene. Additionally, the TSIM 149 may include a display corresponding to the vehicle's driving scene and configured to dynamically align graphics thereon. It will be appreciated that the received touch gestures from the user's hand provide a touch-based user interface for determining the selection of options via the display.

[0028] Various devices are known in the art that can receive touch gestures from a user's hand or finger. The TSIM 149 may be a touchscreen device that includes a screen that can identify the occurrence, location, and position of a touch within a region of the display. According to one embodiment, the display may include a view of the vehicle's driving scene. According to another embodiment, when providing, for example, alternative options for selection, the display may include selected images for selection. For example, the display providing a navigation selection could have one half of the screen with a text message providing, for example, an option to maintain a currently selected route, and the other half of the screen with a segment of the driving scene view including a sign showing a street name or exit number thatwhich provides the user with the option to select the displayed road sign to change the planned route. The TSIM 149 can be displayed within the HMI device 151 (see . Fig.2) be integrated. The TSIM 149 may be embodied as a handheld device such as a Personal Digital Assistant (PDA) or a cellular phone, all having displays containing the vehicle's driving scene. The TSIM 149 allows a user to interact with what is displayed on the display (i.e., the driving scene) rather than indirectly with a mouse or touchpad, and lets the user perform touch gestures without the need for such intermediary devices. Such methods for displaying options for selection may include graphical prompts such as an icon or flashing box indicator around a displayed item, audio prompts, or any number of other features known in the art for quickly conveying information to the user to enable selection of the options being presented.

[0029] Furthermore, various methods for monitoring and receiving touch gestures from a user's hand or finger are known in the art. In one non-limiting embodiment, the TSIM 149 may be a resistive touchscreen display consisting of a standard glass plate coated with a thin conductive and a resistive metal plate, with electrical current flowing between the two layers and monitoring the touch gesture when the two layers come into contact at the same location, thereby interrupting the electrical current. In another non-limiting embodiment, the TSIM 149 may be a surface acoustic wave touchscreen display that uses ultrasonic waves to process user input from the screen.In this embodiment, the surface acoustic wave touchscreen display utilizes a transducer that identifies whether a signal has been altered by a touch or any movement, with ultrasonic waves flowing across the entire screen when the user touches the display in a specific location. In yet another non-limiting embodiment, the TSIM 149 may be a capacitive touchscreen display consisting of an indium tin oxide plate that stores electrical charge and is positioned on a glass layer of the display. When the user's hand or finger touches the display, a small amount of charge is transferred to the user's hand or finger, reducing the amount of charge on the capacitive plate.An associated processor calculates the comparative change and determines the exact location where the touch gesture occurs, where the location corresponds to the location on the vehicle's driving scene. In yet another non-limiting embodiment, the TSIM 149 may include infrared touchscreen displays. Infrared touchscreen displays may respond to infrared or thermal waves (i.e., heat). Infrared touchscreen displays may further utilize vertical and horizontal infrared sensors around the perimeter to create a grid and thus accurately detect the exact location where the screen has been touched by the user's hand or finger.

[0030] It should be appreciated that the TSIM 149 is not limited to any of the devices and methods discussed above, and that the TSIM 149 may only receive touch gestures of a user's hand or finger on the module 149 indicated for a portion of the graphic projection display 250 corresponding to the driving scene. The controller device 153 may include a mechanism such as a joystick, mouse, lever, stylus, touchpad, or other accessory for receiving user manipulations indicated for a portion of the graphic projection display 250.

[0031] The imaging device 161 may include capturing images of the user's hand and utilizing associated software to monitor the location and orientation of the user's hand to provide an optical image. The images may be captured using infrared and / or ultraviolet technology. The optical image may be processed to generate a three-dimensional (3D) image of the user's hand movement. As will be appreciated, the optical image of the hand movement provides a gesture-based user interface. Thus, user hand movements (i.e., gestures) detected by the imaging device 161 may be used in place of touch interface elements such as a mouse, lever, joystick, button, or other accessory.According to an exemplary embodiment, the imaging device 161 requires capturing a sequence of images, with the location and orientation of the user's hand being monitored for each image to provide the optical image of the user's hand movement. The software associated with the imaging device 161 may be integrated within the imaging device 161 and in signal communication with the EVS system manager 110 through suitable dedicated circuitry. Furthermore, the software associated with the imaging device 161 may be integrated within the EVS system manager 110.

[0032] Various methods are known in the art that use imaging devices for processing optical images based on the monitored location and orientation of an object (i.e., a hand) to generate a 3D map of the object. PCT International Publication WO 2007 / 043036 A1 discloses a method and system based on projecting a laser speckle pattern onto an object and then analyzing an image of the pattern on the object. US Patent US 6 100 517 A discloses a method for using pulsed modulation to measure the time of flight of light from an object to a detector, wherein a camera generates an image indicative of distances to points in objects in a scene. US 2010 / 0 020 078 A1 discloses a method for illuminating an object with beams of rays and generating an intensity-based 3D map of the object.The present disclosure is not limited to any of the methods discussed above and may include any device that can monitor hand movement to provide gesture-based input.

[0033] As an alternative or additional input source, the keyboard device 169 may receive user keystrokes on the keyboard device specified for a portion of the graphic projection display 250. The keyboard device 169 may be used in conjunction with other devices, such as the controller device, which may include a joystick, or the HMI. Keyboard devices may include alphanumeric input devices and may include, for example, a miniature keyboard used with a computer or cellular phone device. In another embodiment, the keyboard may include a selected number of keys, for example, five keys labeled A through E or different colored keys.

[0034] The displays presented for selection could include icons corresponding to the labels of the keys on the keyboard device 169. As can be appreciated, the received user keystrokes on the keyboard device 169 provide a key-based interface.

[0035] The HMI device 151 can be used to receive user input. Methods for detecting a touch on a surface of the HMI device 151 are known. Alternatively, buttons associated with the HMI device 151, which are graphically labeled with selections displayed on the HMI device, can be used.

[0036] Fig.5 illustrates an exemplary information flow for dynamically aligning a graphic to a driving scene using the graphic projection display 250, which includes an EVS system manager 110 that monitors inputs 360 from the occupant eye location detection system 160, inputs 371 from a vehicle location and orientation module 370, and inputs 312 from the UIM 515 to thereby generate display requests 330. By monitoring and combining information from various sources, such as inputs 371, 360, and 312, the EVS system manager 110 can operate one or more methods for generating dynamically aligned information regarding the operation of the vehicle and its operating environment on a display device or on a windshield HUD device that includes an operator input device to the graphic display.

[0037] Based on Fig. 1 and Fig.5, the occupant eye location detection system 160 includes sensors known in the art for approximating an occupant's head location and further the orientation or location toward which the occupant's eyes are looking. An occupant may be an operator or user of the vehicle or a passenger within the vehicle. Head and eye detection devices are known in the art and will not be discussed in further detail here. For this disclosure, a camera-based device is utilized in conjunction with image recognition software to estimate, based on the image recognition programming, a three-dimensional head location within the vehicle, which can be coordinated with a vehicle coordinate system, and a gaze direction of the occupant. An eye location input 360 is input to the EVS system manager 110.

[0038] The UIM 515 may be configured to monitor the user input location specified for a portion of the graphic projection display corresponding to the driving scene. A location of a view of an object by a HUD device may be calculated. One such method, which determines a location of a view of an object on a HUD device based on a location of the object and the location of the operator / user's eyes, is disclosed in US application US 2010 / 0 253 489 A1. The user input location refers to the location on the display or HUD device toward which the user's gesture is directed. In some embodiments, the location of the display corresponding to a user input location may include text or a graphic indicating a selection that may be made.In some embodiments, an object visible through the HUD device defines the location or portion of the HUD device corresponding to a user input location that can be pointed to to indicate selection of the object. For example, if a potential target is visible through a center-left portion of the HUD device, a user hand gesture may be interpreted as a user input location corresponding to the potential target based on the location of the user's eyes relative to the HUD device and a particular location of the view of the potential target on the HUD device pointing to the center-left portion of the HUD device. In another exemplary embodiment, a user input location may be determined by the location of the gaze, where, for example,An exemplary selector knob on a steering wheel allows the user to indicate the selection for a portion of the display based on the location the user is looking at when the selector knob is depressed. By determining where a hand gesture is indicated on a HUD device based on the user's location, a user input location 312 can be generated and used as a control input, which is entered into the EVS system manager 110.

[0039] Returning to Fig.5, according to an exemplary embodiment, the EVS system manager 110 monitors the eye location input 360, the vehicle location and orientation input 371, and the user input location 312 to thereby generate the display requests 330 for dynamically aligning the graphic on the graphic projection display. If the graphic includes an object or a portion of a view in the vehicle's operating environment, the EVS system manager 110 additionally requires the sensor data representing the object or view. It should be appreciated that if the graphic includes a feedback graphic, representing, for example, feedback from the location of gaze, a gesture, or a selection control, only the eye location input 360 and the user input location 371 need be monitored to generate the display requests 330 for dynamically aligning the feedback graphic on the graphic projection display 250.

[0040] Fig.6 illustrates an exemplary information flow for determining a graphic based on inputs to the EVS system manager 110. The illustrated process includes an EVS system manager 110 monitoring information from various sources and generating display requests 430, a graphics module 455 monitoring the display requests 430 from the EVS system manager and generating graphic commands 440, and, for example, a graphics projection display 250 incorporating the exemplary display embodiments disclosed herein. The EVS system manager 110 includes a command selection module 405 that integrates user input 512 with other information available to the EVS system manager to provide any graphics required for command selection, including selection prompts and feedback via display requests 430, and outputting vehicle commands representing selections made by the user as a command signal 420.User input 512 may include a user input location 312 or any other input that may convey a command selection or a desire to make a command selection. An exemplary UIM 515 is shown, including an imaging device 161, a TSIM 149, a controller device 153, and a keyboard device 169.

[0041] The exemplary operation of the command selection module 405 may include a number of embodiments. According to one exemplary embodiment, inputs from a navigation module provide module 405 with a currently planned route. Input from vehicle sensors, from a wireless connection, or from another input source may indicate to the navigation module that the road is congested along the planned route. The navigation module plots alternative routes, providing two alternative suggested routes to module 405. Module 405 combines the data from the navigation module with visual data from camera sensors and determines graphics that can best represent the options that need to be presented for the user to select.In an exemplary condition where three options include 1) taking the next right turn, 2) taking a second right turn, and 3) remaining on the current route despite the congestion, the module may determine the display of option 1) in a right third of the graphic projection display 250, which includes a graphic depicting a sharp right turn, option 2) in a middle third of the graphic projection display 250, which includes a graphic depicting a right turn that is less sharp than that depicted for option 1), and option 3) in a left third of the graphic projection display 250, which includes a graphic depicting a straight arrow pointing forward. Text may be displayed representing the congestion condition ahead and representing each of the options. Input from the user may be monitored and entered into the module 405.Module 405 may determine feedback to the user input and display it on the graphic projection display 250. When a user selection is made, the updated navigation route or upcoming navigation maneuvers may be displayed on the graphic projection display 250. The operation of module 405 is a non-limiting example, and a number of other exemplary module operations may be similarly expressed.

[0042] The limited pixelated field of view architecture enables the display of graphics for one vehicle occupant without displaying the graphics for another occupant. An exemplary embodiment of a limited pixelated field of view architecture that enables image viewing from a limited direction includes the use of microstructures or an array of particles that accept excitation light and emitted light in a limited direction. US application US 2010 / 0 253 918 A1 discloses a limited pixelated field of view architecture. Fig. 7 and Fig. 8 illustrate an example use of an architecture with a limited pixelated field of view. Fig.Figure 7 illustrates an exemplary radiation source capable of emitting light to a limited field of view. The exemplary radiation source includes a UV-transparent encapsulation 702, made, for example, from SiO2, filled with a LIF material 704 that fluoresces at visible wavelengths when irradiated with ultraviolet radiation, with a parabolic narrowband multilayer reflective structure 706. In this exemplary embodiment, a thin layer of these radiation sources is deposited on a polymer. In preparation for the layer, depressions similar to the parabolic shapes formed in the radiation sources are embossed in the polymer material. The radiation sources are deposited on the polymer substrate by chemical vapor deposition, filling the parabolic depressions with radiation sources. Fig.Figure 8 illustrates an exemplary process for creating the necessary structure for radiation sources aligned with a polymer substrate to enable limited-field vision. Using an exemplary process such as etching, freestanding parabolas are created by dissolving them from the substrate and filled with emissive material. Removal from the polymer substrate can also be accomplished by dissolving the plastic substrate with a suitable solvent. The freestanding parabolas are then inserted into probes created in the glass substrate by photolithographic processes or embossing.The process of fitting the parabola to the sode can be carried out by a process such as fluid self-assembly similar to that implemented by Alien Technology, in which the parabolas are flowed over the substrate and parabola-sode fitting takes place in a statistical manner.

[0043] Based on the procedures discussed above, including the Fig.According to the method for dynamically aligning a graphic discussed in FIG. 5, a dynamically aligned graphic may be projected onto a graphics projection display 250 embodied in a substantially transparent windshield head-up display 150 including a limited pixelated field of view architecture, wherein the graphics correspond to the driving scene of the vehicle. The pixelated field of view enables the graphic to be viewed from a limited direction (e.g., from a first limited direction and / or from a second limited direction). The dynamically aligned graphic may include a first dynamically aligned graphic based on the location of a first viewer and a second dynamically aligned graphic based on the location of a second viewer. It should be noted that both the first and second dynamically aligned graphics may be projected onto the substantially transparent windshield HUD.Furthermore, the first dynamically oriented graphic, which is based on the location of the first viewer, further relies on the use of the pixelated field of view, which enables the graphic to be viewed from a first limited direction oriented towards the location of the first viewer. It should be noted that the location of the first viewer may correspond to the driver of the vehicle. Likewise, the second dynamically oriented graphic, which is based on the location of the second viewer, further relies on the use of the pixelated field of view, which enables the graphic to be viewed from a second limited direction oriented towards the location of the second viewer. It should be noted that the location of the second viewer may correspond to one of the passenger seats in the vehicle. Likewise, the location shown in . Fig.5 may further include monitoring data related to the eye location of a second observer, wherein the eye location of a second observer corresponds to one of the occupants at the second observer's location. It should be noted that monitoring the eye location of an occupant may include monitoring data related to the eye location of a first observer and / or the eye location of a second observer.

[0044] The limited pixelated field of view architecture method discussed above is applied to the substantially transparent windshield HUD 150. However, this method may additionally be applied to the non-transparent components 280. For example, a portion of the instrument panel or support pillars could include a limited pixelated field of view architecture so that graphics could be presented to the user / operator of the vehicle, but not to the passenger in the other front seat of the vehicle.

[0045] The above methods represent ways in which user input can be used to select commands for the vehicle. For example, methods include interpreting user gestures made to the windshield as commands. Any number of methods can be used to present selectable commands to the user, and any number of methods can be used to interpret user input as desired commands. A textual representation or graphic representation can be projected onto the graphic projection display 250, indicating a selection that can be made for that text or graphic.In such an example, a user indication for the region or for the text or for the graphic may be used based on an aligned location of the text or graphic and a particular intersection point between the user's eyes, the graphic, and the location of the gesture by the user to determine the desired input from the user. Such a supervised gesture may be input directly as a command. In another exemplary embodiment, such a supervised gesture may generate a prompt for the user, e.g., representing a suggested command and requiring confirmation from the user to generate an actual command based on the suggested command. Such confirmation may include a gesture for a confirmation graphic; a button located, e.g., on the steering wheel of the vehicle; or a verbal command.A gesture can be used to select commands presented to the user. Additionally, or alternatively, user gestures can prompt the system to respond. For example, if a user points to an object visible on the graphic projection display 250, the system can provide information about the object or determine whether appropriate command options related to the object can be presented to the user. For example, if the user points to a parking garage entrance, the user can be presented with a prompt representing parking fees for the building and / or a prompt asking whether the vehicle should be instructed to enter the parking garage. A number of embodiments for presenting and selecting options through user gestures are conceivable.

[0046] The above methods may be integrated within an EVS system manager. Such an EVS system manager may reside within a single control module or within a single physical device, or alternatively, may extend across a number of modules or physical devices. The EVS system manager may take on a number of different embodiments, and the disclosure is not intended to be limited to the particular exemplary embodiments of the EVS system manager disclosed herein. Control module, module, control, controller, control unit, processor, and similar terms mean any one or various suitable combinations of one or more application-specific integrated circuit(s) (ASIC), electronic circuit(s), central processing unit(s) (preferably microprocessor(s)), and associated memory and storage (read-only, programmable read-only, read-write, hard disk, etc.).) executing one or more software or firmware programs, combinational logic circuit(s), input / output circuit(s) and devices, suitable signal conditioning and buffering circuitry, and other suitable components to provide the described functionality. The control module has a set of control algorithms including resident software program instructions and calibrations stored in memory and executed to provide the desired functions. The algorithms are preferably executed during preset loop cycles. The algorithms are executed, for example, by a central processing unit and are operable to monitor inputs from sensing devices and other networked modules and to execute control and diagnostic routines to control the operation of actuators.The loop cycles can be executed at regular intervals during continuous engine and vehicle operation, e.g., every 3.125, 6.25, 12.5, 25, and 100 milliseconds. Alternatively, the algorithms can be executed in response to the occurrence of an event.

[0047] Fig. Figure 9 illustrates an exemplary process 800 for practicing the methods disclosed herein. Exemplary steps for process 800 are shown in the following table. Table 1 Step Description 802 Monitor a command selection to be displayed 804 Monitor a view in front of the vehicle 806 Monitor a location of the eyes of a vehicle user 808 Determine an aligned graphic that describes the command selection to be projected 810 Project the aligned graphic 812 Monitor a user input to a location on the display 814 Determine an activation of the command selection based on the monitored user input 816 Control the vehicle based on the activation of the command selection

[0048] The process 800 begins in step 802 by monitoring a command selection to be presented to the user of the vehicle. Such a command selection may be prompted according to a number of exemplary embodiments. For example, the command selection is prompted by a missing instruction from the user when a navigation choice needs to be made (the planned route includes a route that is now closed). According to another embodiment, the command selection may be prompted by an option identified by the EVS system manager reaching a threshold probability that the user may be interested in the identified option. For example, if the vehicle has a full tank of gas, the user is unlikely to select an identified gas station in the view as a new destination. Conversely, if the vehicle's gas tank is nearly empty, such a selection is more likely.A number of methods for determining such probabilities and for setting the probability threshold(s) are known in the art. According to another exemplary embodiment, command selections may be requested by the user. For example, the EVS system manager may be awakened to display a number of probable or configurable options that can be selected by the user if the user makes a gesture toward the display. A number of methods for requesting and determining the content of command selections are conceivable.

[0049] Step 804 monitors a view in front of the vehicle. Step 806 monitors a location of the user's eyes. The command selection of step 802 can be contextualized with a graphic aligned with the view based on the user's eye location. Step 808 determines, based on the command selection, the monitored view, and the monitored location of the user's eyes, an appropriate aligned graphic or graphics for display on the graphic projection display to convey the command selection available to the user. In step 810, the determined graphic is projected onto the graphic projection display. In step 812, user input to a portion or location of the graphic projection display is monitored.In an embodiment utilizing hand gestures, the location of the user's hand relative to the user's eyes is used to determine, based on the user's perception, which portion of the display the hand gestures are directed to. In step 814, the user input is interpreted to selectively activate the command selection. In step 816, the activation of the command selection is used to control the vehicle. Process 800 is an exemplary, non-limiting embodiment for utilizing the methods disclosed herein.

[0050] Fig.10 illustrates an exemplary view through a transparent windshield graphic projection display 900 and user input to the aligned graphic on the display. The view visible via display 900 may either be the actual view visible through a substantially transparent display or may be projected as part of the displayed graphics. A roadway 902 is shown. The visible horizon 904 is also shown. An exit 906 and a road sign 908 are shown. An exemplary user pointing at display 910 is shown. An aligned graphic 920 aligned with a current lane on roadway 902 is shown. Additionally, text 922 may be displayed accompanying graphic 920. Similarly, an aligned graphic 930 for exit 906 and optional accompanying text 932 aligned with exit 906 are displayed.A user pointing at the display 910 is monitored, and based on the location of the input, a selection between options represented by graphics 920 and 930 can be determined.

Claims

[1] A method for controlling operation of a vehicle (100) based on user input (312) from a user (10, 910) of the vehicle (100) into a graphic projection display (250, 900) that displays an aligned graphic (920, 922, 930, 932) on a view ahead of the vehicle (100), the method comprising: Projecting the aligned graphic (920, 922, 930, 932) onto the view in front of the vehicle (100) on the graphic projection display (250, 900), comprising: monitoring a command selection to be presented to the user (10, 910) of the vehicle (100); the view in front of the vehicle (100) is monitored; a location of the eyes of the user (10, 910) of the vehicle (100) is monitored; and the aligned graphic (920, 922, 930, 932) is determined based on the command selection, the view in front of the vehicle (100) and the location of the eyes of the user (10, 910) of the vehicle (100); where monitoring command selection includes: Monitoring information regarding the operation of the vehicle (100) by an autonomous method of operating the vehicle (100) or by a semi-autonomous method of operating the vehicle (100); Determining configurable options for operating the vehicle (100) by the autonomous method for operating the vehicle (100) or by the semi-autonomous method for operating the vehicle (100) based on the monitored information; and Determining the command selection to be presented to the user (10, 910) of the vehicle (100) based on the determined configurable options; Monitoring the user input (312) to the graphic projection display (250, 900), comprising monitoring a hand gesture specified for the graphic projection display (250, 900) and a location of a hand relative to the eyes of the user (10, 910); and Controlling the operation of the vehicle (100) described by a vehicle speed, an engine speed and a wheel slip based on the user input (312) to the graphic projection display (250, 900), wherein the vehicle (100) is operated by an autonomous method of operating the vehicle (100) or by a semi-autonomous method of operating the vehicle (100). [2] The method of claim 1, wherein the graphic projection display (250, 900) is a see-through windshield head-up display over a full windshield. [3] The method of claim 1, wherein monitoring the command selection comprises monitoring a selection to be presented with respect to a navigation route of the vehicle (100). [4] The method of claim 1, wherein monitoring instruction selection comprises: monitoring a user input specified for an object in the view in front of the vehicle (100); Retrieving information regarding the object via a wireless connection; and Determine the command selection to be presented based on the user input specified for the object and the retrieved information. [5] The method of claim 1, wherein the operation of the vehicle (100) by the autonomous method of operating the vehicle (100) or by the semi-autonomous method of operating the vehicle (100) comprises operating the vehicle (100) by an adaptive cruise control method or by a lane keeping method or by a lane change method or by a collision preparation method or by a collision avoidance method. [6] The method of claim 1, wherein monitoring the hand gesture comprises using an imaging device (161) to generate a three-dimensional image of the hand gesture; and determining the hand gesture to indicate the aligned graphic on the three-dimensional image of the hand gesture.

Images