METHOD FOR CONTROLLING A VEHICLE USING VIRTUAL VEHICLE CONTROLS

Abstract

Method for controlling a vehicle (12) using virtual vehicle controls, the method comprising: Establishing a short-range wireless communication (SRWC) connection to an augmented reality / virtual reality (AR / VR) device (14, 16) which includes a display and a camera and is configured to capture image or video data, wherein the AR / VR device (14, 16) includes a short-range wireless communication (SRWC) circuit used to send and receive SRWC signals; Receiving a virtual vehicle control graphics request from the AR / VR device (14, 16) on the vehicle (12); Sending a virtual vehicle control graphic response, including a virtual vehicle control graphic, from the vehicle (12) to the AR / VR device (14, 16) via the established SRWC connection, wherein the AR / VR device (14, 16) is further configured to display the virtual vehicle control graphic on the display of the AR / VR device (14, 16) such that the virtual vehicle control graphic is displayed over image or video data captured by the camera, wherein the virtual vehicle control graphic includes one or more vehicle user interface components; and Receiving a vehicle control input from a user of the AR / VR device (14, 16); characterized by the fact that is determined whether the AR / VR device (14, 16) is authorized and / or authenticated to receive and / or use virtual vehicle control graphics; that the vehicle user interface component corresponds to a capacitive touch sensor array (50) integrated into the vehicle electronics of the vehicle (12) and installed on the rear (122) of the driver's or front passenger's seat (120) of the vehicle (12), wherein the AR / VR device (14, 16) is further configured to display the virtual vehicle control graphics on the display of the AR / VR device (14, 16) over the capacitive touch sensor array (50); and that Receiving the vehicle control input further includes receiving sensor position inputs from the capacitive touch sensor array (50), over which the virtual vehicle control graphic is displayed on the AR / VR device display (14, 16), indicating a position within the capacitive touch sensor array (50) that is being touched by the user.

Description

INTRODUCTION

[0001] The present invention relates to the use of augmented reality / virtual reality (AR / VR) devices for displaying a virtual vehicle surface over images depicting a vehicle, and in particular to a method according to the preamble of claim 1 for controlling a vehicle using virtual vehicle controls, as is essentially known from US 2015 / 0 202 962 A1.

[0002] Further methods for controlling a vehicle using augmented reality / virtual reality (AR / VR) devices are also disclosed in the documents DE 10 2010 062 282 A1, DE 10 2013 210 746 A1, DE 10 2015 208 494 A1 and US 9 019 128 B1.

[0003] Vehicles include hardware and software capable of performing various multimedia functions, including playing audio received from radio stations, as well as mobile devices (smartphones, tablets), many of which utilize wireless communications, such as short-range wireless communications (SRWC). In addition, vehicles include numerous vehicle system modules (VSMs) that can be used to provide information to other electronic computing devices, such as personal mobile devices. SUMMARY

[0004] According to the invention, a method for controlling a vehicle using virtual vehicle controls is presented, which is characterized by the features of claim 1.

[0005] According to one or more different embodiments, this method may include one or more of the following features in any technically possible combination of these features: • the virtual vehicle control graphic response, which includes virtual vehicle control graphic position information indicating a vehicle component on which the virtual vehicle control graphics are to be displayed or indicating a position in the display of the AR / VR device at which the virtual vehicle control graphics are to be displayed; • the vehicle control input indicating a selection or modification of the one or more vehicle user interface components, and wherein the selection or modification is determined based on the sensor position input; • the vehicle control inputs received by the AR / VR device; • the vehicle control inputs initially received at the AR / VR device using a controller of the ARNR device; • the vehicle control inputs initially received at the camera of the AR / VR device, and wherein the vehicle control inputs are communicated from the user to the AR / VR device through one or more gestures created by the user in a field of view of the camera; • Authorizing the vehicle control inputs based on authorization information contained in the vehicle control input and / or information contained in the request for the virtual vehicle control graphic; • modifying the operation of one or more vehicle system modules (VSMs) in response to receiving the vehicle control inputs, and wherein the modifying step is performed based on the vehicle control input; • after modifying the operation of one or more vehicle system modules (VSMs), sending a vehicle control input response that includes information for rendering updated virtual vehicle control graphics; and / or • the information used to render the updated virtual vehicle control graphics including an instance of the updated virtual vehicle control graphics, and wherein the updated virtual vehicle control graphics are generated on the vehicle based on the vehicle control input and / or the modified operation of the one or more VSMs.

[0006] Further described is another method for operating a vehicle using virtual vehicle controls, the method including: capturing image or video data from an area within a vehicle interior; sending a virtual vehicle control graphic request to a vehicle; receiving a virtual vehicle control graphic response from the vehicle, wherein the virtual vehicle control graphic response includes virtual vehicle control graphics; and presenting the captured image or video data and the virtual vehicle control graphics on a display of the AR / VR device such that the virtual vehicle control graphics are presented over the captured image or video data, wherein the virtual vehicle control graphic includes one or more vehicle user interface components.

[0007] According to one or more different embodiments, this method may include one or more of the following features in any technically possible combination of these features: • the captured image or video data captured by a camera arranged on the AR / VR device, and wherein the camera of the AR / VR device faces an area in front of a user when the display of the AR / VR device is positioned opposite the user; • the AR / VR device, which is a pair of ARNR glasses configured to visually present augmented reality or virtual reality media content to a user and which can be worn by the user, and, when the AR / VR device is properly worn by the user, the display of the AR / VR device faces the user and the camera of the AR / VR device captures an area in front of the user; • Receiving a virtual vehicle control graphic advertisement indicating that virtual vehicle control graphics are available for use by the AR / VR device; • Establishing a connection between the AR / VR device and a wireless communication device integrated into the vehicle's electronics using Bluetooth™, Wi-Fi™ or other IEEE 802.11 technology, and receiving the virtual vehicle control graphics via the established connection; • the virtual vehicle control graphic response, which includes virtual vehicle control graphic position information indicating a vehicle component on which the virtual vehicle control graphics are to be displayed or indicating a position on the display of the AR / VR device at which the virtual vehicle control graphics are to be displayed; • Receiving vehicle control inputs from a user at the AR / VR device and sending the vehicle control inputs to the vehicle, wherein the vehicle is configured to perform one or more actions based on the vehicle control input; • Receiving a second virtual vehicle control graphic response from the vehicle after sending the vehicle control input to the vehicle, and wherein the second virtual vehicle control graphic includes updated virtual vehicle control graphics, and displaying the updated virtual vehicle control graphics on the display of the AR / VR device; and / or • the virtual vehicle control graphics, which are updated based on the vehicle control input. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] One or more embodiments of the invention are described below in conjunction with the accompanying drawings, wherein like designations refer to like elements, and wherein: Fig. 1 is a block diagram illustrating one embodiment of a communication system capable of using the method disclosed herein; Fig. 2 is a perspective view illustrating one embodiment of a vehicle interior; and Fig. 3 is a flowchart of one embodiment of a method for operating a vehicle using virtual vehicle controls. DETAILED DESCRIPTION

[0009] The system and method described below enables an augmented reality / virtual reality (AR / VR) device to display virtual vehicle controls on a video or image such that the virtual vehicle controls are displayed over a surface corresponding to a vehicle or vehicle component. AR / VR devices can use cameras to capture an image or video of an area around a user and can then render graphics over specific portions of the captured image or video for display on a display of the AR / VR device. In this way, the AR / VR device can display graphics over real-world images or videos of a user's environment such that the graphics complement the user's visual perception of the environment, thereby providing both aesthetically pleasing graphics and information for the user.Additionally, in many embodiments, the graphics may be used to represent virtual vehicle controls with which a user may interact to adjust vehicle settings, such as temperature or climate settings and multimedia control settings.

[0010] For example, virtual vehicle graphics may be developed and displayed by AR / VR devices in conjunction with videos or images of the user's environment to provide an augmented reality (AR) view. AR / VR devices may utilize a camera that, when worn by a user, faces an area in front of the user (the field of view of the "AR / VR device"). In one embodiment, the AR / VR device is a pair of AR or VR goggles configured to fit around a user's head and including a display that, when properly worn, is positioned in front of the user's eyes or field of view.The AR / VR device can download virtual vehicle control graphics associated with the vehicle, and then the AR / VR device can display the virtual vehicle control graphics on a captured image or video in an area corresponding to the vehicle or a specific component of the vehicle. In many embodiments, the method and system discussed below can be used to magnify an area within the vehicle, such as an interior cabin of a vehicle.

[0011] In some embodiments, virtual vehicle control graphics may be developed based on a layout or design of a particular vehicle, and then the virtual vehicle control graphics may be sent to AR / VR devices on or near the vehicle so that the AR / VR devices can display the virtual vehicle control graphics over the vehicle. Because the virtual vehicle control graphics correspond to a layout of the vehicle, the virtual vehicle control graphics may be displayed at locations in the vehicle suitable for displaying user interfaces, such as solid and / or light surfaces. As illustrated below, virtual vehicle control graphics may be displayed over a portion of the back of a vehicle seat.The back of the vehicle seat, on which the virtual vehicle control graphics are displayed, includes a capacitive touch sensor array that detects when a user touches the seat and the location where the user touched the seat. The vehicle may also have virtual vehicle control information stored in memory that includes information about the location of various controls within the virtual vehicle control graphics. By using this virtual vehicle control information, as well as the user's touch position, the vehicle can determine which vehicle controls are being adjusted (or touched) by a user, and the vehicle can then adjust the appropriate vehicle settings.

[0012] With reference to Fig. 1, an operating environment is illustrated that includes a communication system 10 and that can be used to implement the method disclosed herein. The communication system 10 generally includes a vehicle 12 with a wireless communication device 30 and VSMs 22-58, augmented reality / virtual reality (AR / VR) devices 14 and 16, a constellation of Global Navigation Satellite System (GNSS) satellites 60, one or more wireless carrier systems 70, a fixed line communication network 76, a computer or server 78, and a vehicle backend service facility 80. It is understood that the disclosed method can be used with any number of different systems and is not specifically limited to the operating environment shown herein. Also, the architecture, construction, setup, and general operation of the system 10 and its individual components are well known in the art.Thus, the following paragraphs provide only a brief overview of such a communication system 10; however, other systems not shown here could also employ the disclosed methods.

[0013] The wireless carrier system 70 may be any suitable cellular system. The carrier system 70 is illustrated with a cellular tower 72; however, the carrier system 70 may include one or more of the following components (e.g., depending on the cellular technology): cellular towers, base transmission stations, mobile switching centers, base station controllers, developed nodes (e.g., eNodeBs), mobility management units (MMEs), serving and PGN gateways, etc., as well as any other network components required to connect the wireless carrier system 70 to the fixed network 76 or to connect the wireless carrier system to the user equipment (UEs, which may, for example, include the telematics equipment in the vehicle 12). The carrier system 70 may implement any suitable communications technology, including GSM / GPRS technology, CDMA or CDMA2000 technology, LTE technology, etc.In general, wireless carrier systems 70, their components, the arrangement of their components, the interaction of the components, etc., are largely known in the art.

[0014] In addition to using the wireless carrier system 70, a different wireless carrier system in the form of satellite communications may be used to provide unidirectional or bidirectional communications with the vehicle. This may be accomplished using one or more communications satellites (not shown) and an uplink transmitting station (not shown). Unidirectional communications may, for example, be satellite radio services, where programmed content data (news, music, etc.) is received from the uplink transmitting station, packaged for upload, and then transmitted to the satellite, which broadcasts the programming to the subscribers. Bidirectional communications may, for example, be satellite telephony services using the one or more communications satellites to relay telephone communications between the vehicle 12 and the uplink transmitting station.When used, this satellite telephony can be used either in addition to or instead of the wireless carrier system 70.

[0015] The fixed network 76 may be a conventional land-based telecommunications network connected to one or more landline telephones and connecting the wireless carrier system 70 to the remote location 80. For example, the fixed network 76 may include a public switched telephone network (PSTN) such as that used to provide landline telephony, packet-switched data communications, and Internet infrastructure. One or more segments of the fixed network 76 could be implemented using a standard landline network, a fiber optic or other fiber optic network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs), or networks providing broadband wireless access (BWA), or any combination thereof.

[0016] Computers 78 (not shown) may be any of a number of computers accessible via a private or public network, such as the Internet. Each of these computers 78 may be used for one or more purposes, such as purchasing or selecting skins or themes for use with the virtual vehicle control graphics displayed with an AR / VR device.Other such accessible computers 78 may include, for example, a computer at a customer service center to which diagnostic information and other vehicle data may be uploaded from the vehicle; a client computer used by the vehicle owner or another subscriber for such purposes as accessing or receiving vehicle data, or for setting or configuring subscriber preferences or controlling vehicle functions; a car sharing server that coordinates reservations and / or registrations by multiple users requesting use of a vehicle as part of a car sharing service; or a third-party storage location to or from which vehicle data or other information is provided by communicating with either the vehicle 12 or the remote device 80, or both.A computer 78 may also be used to provide Internet connectivity, such as DNS services, or as a network address server using DHCP or another suitable protocol to assign an IP address to the vehicle 12. In one embodiment, computers 78 may be used to perform the method described herein; in other embodiments, the method may be performed by servers or other computing devices at the remote facility 80, as explained in more detail below; and in yet another embodiment, the method may be performed by a combination of computers 78 and servers at the remote facility 80.

[0017] The remote device 80 may be designed to provide a variety of system backend functions to the vehicle electronics 20 and / or the AR / VR devices 14, 16 using one or more electronic servers, and in many cases, it may be a vehicle-related backend service device that provides vehicle-related backend functions. The remote device 80 includes servers (vehicle-related backend service servers) 82 and databases 84 that may be stored on a variety of storage devices. The remote device 80 may also include one or more network switches, live advisors, and an automated voice response system (VRS), all of which are familiar to those skilled in the art.The remote device 80 may include any or all of the various components, with all of the various components preferably being coupled together via a wired or wireless local area network. The remote device 80 receives and transmits data via a modem connected to the fixed network 76. Data transmissions may also occur through wireless systems, such as IEEE 802.11x, GPRS, and the like. Those skilled in the art will recognize that, although only one remote device 80 and one computer 78 are shown in the illustrated embodiment, numerous remote devices 80 and / or computers 78 may be used.

[0018] The servers 82 may be computers or other computing devices that include at least a processor and memory. The processor may be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, controllers, vehicle communication processors, and application-specific integrated circuits (ASICs). The processors may be dedicated processors used only for the servers 82 or may be shared with other systems. The at least one processor may execute various types of digitally stored instructions, such as software or firmware programs stored in memory (e.g., EEPROM, RAM, ROM), which enable the servers 82 to provide a variety of services.For example, the at least one processor may execute programs or process data to perform at least a portion of the method discussed herein. For network communication (e.g., intra-network communication, inter-network communication, including Internet connections), the servers may include one or more network interface cards (NICs) (including wireless NICs (WNICs)) that may be used to transport data to and from the computers. These NICs may enable one or more servers 82 to connect to each other, to databases 84, or to other network devices, including routers, modems, and / or switches. In a particular embodiment, the NICs (including WNICs) of the servers 82 may enable the establishment of SRWC connections and / or Ethernet ports (IEEE 802.11b).3) to which Ethernet cables can be connected, which can provide a data connection between two or more devices. The remote device 80 can include a series of routers, modems, switches, or other network devices that can be used to provide network functions, such as connecting to the landline network 76 and / or the cellular carrier system 70. In one embodiment, the servers 82 include an application that implements at least a portion of the system described below. Fig. 2 illustrated procedure.

[0019] The databases 84 may be stored on a variety of memories, such as RAM or other temporarily operating memory, as well as a non-transitory computer-readable medium (e.g., EEPROM) or any other electronic computer medium that stores some or all of the software for performing the various external device functions described herein. Likewise, one or more databases 84 in the remote device 80 may store account information as well as virtual vehicle control graphics. Likewise, a vehicle information database may be integrated that stores information about one or more vehicles, such as vehicle identification numbers (VINs), virtual vehicle control graphics, vehicle control graphic identifiers, vehicle layout types or features, and various other data related to the vehicle.The vehicle layout types or features may indicate the general layout of the vehicle or specific vehicle components and may correspond to a model year (e.g., 2018 Cruze®). In addition, certain features of a particular vehicle may be stored in databases 84, such as whether the vehicle in question includes one or more optional features.

[0020] The smartphone 14 is an augmented reality / virtual reality (AR / VR) device capable of presenting media content to a user on a display. As used herein, "AR / VR devices" are those devices configured to receive and present AR / VR media content and that have suitable hardware and processing capabilities for presenting AR / VR media content. Non-limiting examples of media control devices include a smartphone, a personal computer (PC), a laptop, a tablet, a smart TV, and ARNR glasses or displays. In some embodiments, the smartphone (media control device) 14 may include hardware, software, and / or firmware that enable cellular telecommunications and short-range wireless communications (SRWC), as well as other mobile device applications.As used herein, a personal AR / VR device is an AR / VR device that can be worn by a user, and where the wearability of the device is at least partially dependent on the user, such as a wearable device (e.g., a smartwatch, AR / VR glasses), an implantable device, or a handheld device (e.g., a smartphone, a tablet, a laptop). And, as used herein, a short-range wireless communication (SRWC) device is a device that is SRWC-capable and that includes the necessary SRWC circuitry to perform such SRWC.

[0021] In many embodiments, the hardware of the AR / VR devices, including the smartphone 14, may include a processor and memory (e.g., a non-transitory, computer-readable medium configured to operate with the processor) for storing software, firmware, etc. In one embodiment, the AR / VR devices may include SRWC circuitry, such as that described below with respect to the wireless communication device 30 of the vehicle 12. The processor and memory of the AR / VR device may enable various software applications that may be pre-installed or installed by the user (or manufacturer) (e.g., with a software application or graphical user interface (GUI)). An implementation of an application of the AR / VR device may enable the AR / VR device (e.g.,The AR / VR device 14 may enable a device (e.g., smartphone 14) to receive media content and virtual vehicle control graphics via the landline 76 and / or the cellular carrier system 70, and to play or otherwise present the media content and control graphics to a user via one or more user device interfaces on the AR / VR device. In one embodiment, the AR / VR device 14 may include a display that may be used to visually present media content to a user and an audio system that may be used to audibly present media content to a user. Additionally, the AR / VR device 14 may include a digital camera configured to capture images and / or videos of an area surrounding the AR / VR device. Other playback components or devices may be incorporated and / or controlled by the AR / VR device 14, such as a tactile or haptic device.

[0022] In some embodiments, an optical display head mount may be used in conjunction with an AR / VR device (e.g., a smartphone 14) to mount the device in front of the user's field of view while also delimiting the user's peripheral field of view. The optical head-mounted display may be a separately purchased and manufactured device that includes a slot or is otherwise configured to hold AR / VR devices in an orientation such that, when the device is inserted into the slot (or otherwise attached to the optical display head mount) and properly worn by the user, the media control device display is within the user's field of view. Furthermore, the AR / VR device may include a digital camera that, when the AR / VR device is held within a user's field of view, captures images or video of an area in front of the user.The captured images and / or videos can be presented on the AR / VR device's display along with other graphics, including virtual vehicle control graphics. Such an optical head-mounted display, used in conjunction with a suitable AR / VR device, can be used to transform a conventional AR / VR device (e.g., a smartphone) into an AR / VR headset. As those skilled in the art understand, although AR / VR headsets can be referred to as a pair of headsets, only a single display is required, rather than a pair of displays as the name might suggest.

[0023] The pair of augmented reality / virtual reality (AR / VR) glasses 16 is an AR / VR device that can be used to obtain, process, and present audiovisual content, such as virtual vehicle control graphics, to a user. The pair of AR / VR glasses 16 is a head-mounted display that includes a processor and memory, as well as computer instructions (e.g., software, firmware) for displaying AR / VR media content. And, in some embodiments, the AR / VR glasses 16 may be an AR / VR headset equipped with AR / VR glasses integrated with a head mount and including an audio speaker.The AR / VR headset 16 may also include other components, such as an SRWC circuit (similar to the SRWC circuit 32 described below), an accelerometer (or orientation sensors), one or more cameras or other lenses capable of receiving and digitizing light signals, a battery, speakers or headphones for displaying audio content, and various other hardware or software components. The AR / VR headset may be configured for use with applications developed according to a software development kit (SDK) specifically configured for AR / VR devices or that support AR / VR functionality.

[0024] Furthermore, as previously mentioned, the AR / VR headset may include speakers or headphones that can present audio content to the user via sound waves. This audio content may be part of audiovisual media content, and the audio and video output of the AR / VR headset may be synchronized. In further embodiments, the AR / VR headset may include only a video output and may connect to other user interfaces, such as those present in the vehicle, to output the audio content. And, as explained in more detail below, the AR / VR headset may present virtual vehicle control graphics.

[0025] The vehicle 12 is shown in the illustrated embodiment as a passenger car, however, it should be noted that any other vehicle, including motorcycles, trucks, off-road vehicles (SUVs), recreational vehicles (RVs), watercraft, aircraft, etc., may also be used. Portions of the vehicle electronics 20 are generally in Fig. 1 and include a global navigation satellite system (GNSS) module 22, body control unit (BCM) 24, other vehicle system modules (VSMs) 26, a wireless communication device 30, seat heaters 40, seat climate control devices 42, heating / ventilation / air conditioning 44, seat motors 46, interior lighting 48, and vehicle user interfaces 50-58. Some or all of the vehicle electronics may be interconnected for communication via one or more communication buses, such as bus 28. Communication bus 28 provides network connections to the vehicle electronics using one or more network protocols. Examples of suitable network connections include a controller area network (CAN), a media-oriented system transfer (MOST), a local area network (LIN), a local area network (LAN), and other suitable connections, such as Ethernet, or others that may be used, among others.comply with well-known ISO, SAE and IEEE standards and specifications.

[0026] The vehicle 12 may include numerous vehicle system modules (VSMs) as part of the vehicle electronics 20, such as the GNSS module 22, the BCM 24, the wireless communication device 30, temperature control modules 40-44, seat motors 46, interior lighting 48, and the vehicle user interfaces 52-58, as described in more detail below. The vehicle 12 may also include other VSMs 26 in the form of electronic hardware components located throughout the vehicle that receive input from one or more sensors and use the sensed inputs to perform diagnostic, monitoring, control, reporting, and / or other functions. Each of the VSMs 26 is preferably connected to the other VSMs and the wireless communication device 30 by the communication bus 28 and can be programmed to perform vehicle system and subsystem diagnostic tests.One or more VSMs 26 may have their software or firmware updated periodically or occasionally, and, in some embodiments, such vehicle updates may be over-the-air (OTA) updates received by a computer 78 or remote device 80 via the landline network 76 and communication devices 30. Those skilled in the art will recognize that the aforementioned VSMs are only examples of some of the modules that may be used in the vehicle 12, but numerous other modules are also possible.

[0027] The wireless communication device 30 is capable of transmitting data via short-range wireless communications (SRWC) and / or via satellite-based network communications. In the illustrated embodiment, a wireless communication device 30 includes an SRWC circuit 32, a cellular chipset 34, a processor 36, a memory 38, and antennas 40 and 50. In one embodiment, the wireless communication device 30 may be a stand-alone module, or in other embodiments, the device 30 may be integrated or included as part of one or more other vehicle system modules, such as a center stack module (CSM), a vehicle control module (BCM) 24, an infotainment module, a telematics unit, a head unit, and / or a gateway module.In some embodiments, the device 30 may be implemented as an OEM-installed (embedded) or as an aftermarket device installed in the vehicle.

[0028] The wireless communication device 30 may be configured for wireless communication according to one or more wireless protocols, including short-range wireless communication (SRWC), such as one of the IEEE 802.11 protocols, Wi-Fi™, WiMAX™, ZigBee™, Wi-Fi™ direct, Bluetooth™, Bluetooth™ Low Energy (BLE), or near-field communication (NFC). As used herein, Bluetooth™ refers to any of the Bluetooth™ technologies, such as Bluetooth Low Energy™ (BLE), Bluetooth™ 4.1, Bluetooth™ 4.2, Bluetooth™ 5.0, and other Bluetooth™ technologies that may be developed. As used herein, Wi-Fi™ or Wi-Fi™ technology refers to any of the Wi-Fi™ technologies, such as IEEE 802.11b / g / n / ac or any other IEEE 802.11 technology. Short-range wireless communication (SRWC) circuitry 32 enables wireless communication device 30 to transmit and receive SRWC signals, such as BLE signals.The SRWC circuitry may enable the connection between device 30 and another SRWC device. Furthermore, in some embodiments, the wireless communication device may include a cellular chipset 34, enabling the device to communicate using one or more cellular protocols, such as those used by cellular carrier system 70. In further embodiments, cellular chipset 34 may be integrated into another VSM, such as a separate telematics unit.

[0029] The wireless communication device 30 may enable the vehicle 12 to connect to one or more remote networks (e.g., one or more networks at the remote facility 80 or computer 78) via packet-switched data communication. This packet-switched data communication may be accomplished through the use of a non-vehicle wireless access point connected to a landline network via a router or modem. When used for packet-switched data communication, such as TCP / IP, the data transmission device 30 may be configured with a static IP address or may be configured to automatically receive an assigned IP address from another device on the network, such as a router or network address server.

[0030] Packet-switched data transmissions may also be performed using a cellular network accessible by device 30. Communication device 30 may transmit data using a cellular chipset 34 over wireless carrier system 70. In such an embodiment, radio transmissions may be used to establish a communication channel, such as a voice channel and / or a data channel, with wireless carrier system 70 so that voice and / or data transmissions may be sent and received over the channel. Data may be sent over either a data connection, such as packet data transmission over a data channel or over a voice channel, using techniques known in the art.For combined services that include both voice and data communications, the system can use a single call over a voice channel and switch between voice and data transmission over the voice channel as needed, again using techniques well known to those skilled in the art.

[0031] The processor 36 may be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, controllers, vehicle communication processors, and application-specific integrated circuits (ASICs). It may be a dedicated processor for the data transmission device 30, or it may be shared with other vehicle systems. The processor 36 executes various types of digitally stored instructions, such as software or firmware programs, stored in the memory 38, which enable the device 30 to provide a wide variety of services. For example, the processor 36 may execute programs or process data to perform at least a portion of the method described herein. The memory 38 may include volatile RAM or other temporarily operable memory, as well as a non-transitory computer-readable medium (e.g.,EEPROM) or any other electronic computing medium that stores some or all of the software for performing the various external device functions described herein. Similar components to those previously described (the processor 36 and / or memory 38, as well as the SRWC circuitry 32 and the cellular chipset 34) may be integrated into the AR / VR devices 14 and / or 16.

[0032] In one embodiment, the wireless communication device 30 can be operated in both an on state and an off state. As used herein, an "on state" is a state of the vehicle in which the vehicle's ignition or primary propulsion system is on, and, as used herein, a "off state" is a state of the vehicle in which the vehicle's ignition or primary propulsion system is not on. The operation or state of the wireless communication device 30 can be controlled by another vehicle system module, such as the vehicle's electrical system control module or an infotainment module. In the on state, the wireless communication device 30 can remain "on" at all times or can be powered by a vehicle battery or other power source.When powered off, the wireless communication device 30 may be maintained in a low power mode or periodically energized so that the device 30 can be activated and perform functions.

[0033] The vehicle may use the wireless communication device 30 to discover other SRWC devices, such as the AR / VR devices 14, 16. A connection between the wireless communication device 30 and one or more devices 14, 16 may enable the operation of various vehicle device functionalities. Vehicle device functionality (or function) refers to any function of the vehicle that can be augmented, enhanced, supported, or performed by an AR / VR device; any function of the AR / VR device that can be augmented, enhanced, supported, or performed by the vehicle; or any other function that can be performed by the vehicle and one or more mobile devices.For example, the vehicle device functionality may include using the AR / VR device 14 to provide the vehicle with a contact list that may be displayed on the visual display 58, audiovisual media content that may be played on the display 58 through the speakers 54, navigation information (e.g., start and / or end locations), and / or vehicle commands or instructions that direct the vehicle to perform certain operations.Further examples of vehicle device functionality include, for example, utilizing vehicle electronics to provide the AR / VR device with a hands-free experience, such as through the use of vehicle user interfaces to make, receive, and conduct voice calls; sending information to the AR / VR device, such as geographic information to the AR / VR device 14 (such as information obtained from the GPS module 22) or vehicle diagnostic information or codes; and executing commands (e.g., commands to perform certain vehicle functions) received from the content playback device on the vehicle.And, as explained in detail below, an established SRWC connection between the AR / VR device and the vehicle can be used to send virtual vehicle control graphics as well as virtual vehicle control information (including virtual vehicle control input information) between the vehicle and the AR / VR device.

[0034] The wireless communication device 30 may be configured to listen for one or more wireless signals, such as those that may be transmitted from the AR / VR device 14. In one embodiment, the wireless communication device 30 may receive a connection request message from the AR / VR device 14 or 16. Thereafter, the wireless communication device 30 may establish an SRWC connection with the requesting device via a four-way handshake and / or other connection establishment methods, such as those that establish a shared secret or one or more digital keys or other credentials used to secure the SRWC connection.

[0035] In one embodiment, the wireless communication device 30 may be placed in a discovery mode when the vehicle is placed in a powered-on state. As used herein, a discovery mode is a mode of operation for an SRWC device in which the SRWC device transmits wireless messages, typically intended to discover or recognize other SRWC devices using an SRWC protocol or technology. The discovery mode may involve periodically or intermittently transmitting messages or advertisements to elicit a response from other SRWC devices within range. In one embodiment, the wireless communication device 30 may transmit a Bluetooth™ Low Energy advertising packet (e.g., an advertisement), such as an ADV_IND (a non-directed advertisement), an ADV_DIRECT_IND (a directed advertisement), or an ADV_SCAN_IND (scannable non-directed advertisement).For example, wireless communication device 30 may send an ADV_IND message according to a predefined or predetermined time interval (e.g., 30 milliseconds (ms)). Or, device 30 may vary the time interval by randomizing the interval (e.g., randomly or pseudo-randomly selecting a time between 10 ms and 40 ms for each advertisement). This advertisement may be a generic message searching for other SRWC devices to connect to, or this advertisement could be a virtual vehicle control graphics advertisement that includes information about the availability of virtual vehicle control graphics for use with AR / VR devices.In further embodiments, a wired communication connection may be established between the AR / VR devices 14, 16 and the vehicle, for example, through the use of Universal Serial Bus (USB) cables and drivers integrated into the respective devices.

[0036] The global navigation satellite system (GNSS) module 22 receives radio signals from a constellation of GNSS satellites. In one embodiment, the GNSS module 22 may be a global positioning system (GPS) module that receives GPS signals from a constellation of GPS satellites 60. The GNSS module 22 may be used to provide navigation and other location-related services to the vehicle operator. Navigation information may be displayed on the display 58 (or another display within the vehicle) or presented verbally, such as in turn-by-turn navigation.The navigation services may be provided using an associated vehicle navigation module (which may be part of the GNSS module 22), or some or all of the navigation services may be provided via a telematics unit installed in the vehicle, with the position information being sent to a remote location for the purpose of equipping the vehicle with navigation maps, map annotations (points of interest, restaurants, etc.), route calculations, and the like. The position information may be transmitted to the remote location 80 or to another remote computer system, such as the computer 78, for other purposes, such as a car-sharing service. Furthermore, new or updated map data may be downloaded from the remote location 80 to the GNSS module 22 via a telematics unit installed in the vehicle.

[0037] The vehicle electronics 20 also includes a number of vehicle user interfaces that provide vehicle occupants with a means of providing and / or receiving information, including push button(s) 52, audio system 54, microphone 56, and visual display 58. As used herein, the term "vehicle user interface" broadly encompasses any suitable form of electronic device, including both hardware and software components located within the vehicle, that enables a vehicle user to communicate with or through a component of the vehicle. The push button(s) 52 enable manual user input to the communication device 30 to provide further data, responses, or control inputs. The audio system 54 provides audio output to a vehicle occupant and may be an associated standalone system or part of the primary vehicle audio system.According to the particular embodiment shown herein, the audio system 54 is operatively coupled to both the vehicle bus 28 and an entertainment bus (not shown) and may provide AM, FM, and satellite radio, CD, DVD, and other multimedia functionality. This functionality may be provided in conjunction with the infotainment module or independently. The microphone 56 provides audio input to the wireless communication device 30 to enable the driver or other occupants to provide voice commands and perform hands-free calling via the wireless carrier system 70. For this purpose, it may be connected to an integrated automatic speech processing unit utilizing human-machine interface (HMI) technology known to those skilled in the art.The visual display or touchscreen 58 is preferably a graphic display and can be used to provide a variety of input and output functions. The display 58 can be a touchscreen on the instrument panel, a heads-up display reflected from the windshield, or a projector capable of projecting graphics for viewing by a vehicle occupant. Various other vehicle user interfaces can also be employed, which interfaces in . Fig. 1 serves only as an example for a specific implementation.

[0038] The body control module (BCM) 24 is in the exemplary embodiment Fig. 1 as electrically connected to the communication bus 28. In some embodiments, the BCM 24 may be integrated with or as part of a center stack module (CSM) and / or may be integrated with the wireless communication device 30. Or, the BCM may be a separate device interconnected via the bus 28. The BCM 24 may include a processor and / or memory, which may be similar to the processor 36 and memory 38 of the wireless communication device 30, as discussed below. The BCM 24 may communicate with the wireless device 30 and / or one or more vehicle system modules, such as an engine control unit (ECU) (not shown), an audio system 54, or other VSMs 26.The BCM 24 may include a processor and memory so that the BCM may control one or more vehicle operations, including, for example, controlling the central locking system, the air conditioning system, the power mirrors, controlling the vehicle's primary drive (e.g., engine, primary propulsion system), and / or controlling various other vehicle modules. For example, the BCM 24 may send signals to seat heaters 40, seat AC devices 42, HVAC 44, interior lights 48, and / or seat motors 46. The BCM 24 may receive data from the wireless communication device 30 and then send the data to one or more vehicle modules.

[0039] Additionally, the BCM 24 may provide vehicle status information corresponding to the vehicle's condition or specific vehicle components or systems. For example, the BCM may provide information to the device 30 indicating whether the vehicle's ignition is on, what gear the vehicle is currently in (i.e., in-gear), and / or other information related to the vehicle. The BCM 24 may retrieve information from one or more other vehicle modules to obtain this information.

[0040] The capacitive touch sensor assembly 50 is a vehicle system module (VSM) that includes a two-dimensional array of capacitive touch sensors that can determine when a user has touched and / or changed the capacitance of the touch sensors. In further embodiments, other touch sensor assemblies may be used, such as resistive touch sensor assemblies that can detect a user's touch. The capacitive touch sensor assembly includes a plurality of capacitive touch sensor input units that include at least one capacitor and an insulator. In one embodiment, the capacitance of one or more capacitors may change when a user places their finger on the capacitive touch sensor assembly, thereby indicating which capacitive touch sensor input unit(s) the user has touched.By determining which capacitive touch sensor input units the user has touched, the vehicle 12 or the capacitive touch sensor assembly 50 can determine where the user's touch occurred.

[0041] The vehicle electronics 20 also includes numerous other VSMs 40-48, 52-58 that may be automatically controlled by the vehicle or that may be controlled via user inputs, such as those received through the use of virtual vehicle control graphics. The temperature control modules 40-44 include VSMs that are included as part of the vehicle electronics 20 and may be used to control the temperature or climate within the vehicle, such as within an interior cabin of the vehicle. The seat heaters 40 and / or the seat climate control (AC) device 42 may be coupled to a seat back and / or a seat cushion. These devices may heat and / or air condition these areas of the seat.Additionally, the vehicle may be equipped with an HVAC (heating / ventilation / AC) system (or an air conditioning system) that can regulate the ambient air temperature within the vehicle's cabin. Such a system may include an air conditioning (AC) unit or device that cools the vehicle's interior air, a vehicle heater that heats the vehicle's interior air, and a ventilation system with forced-air fans that can provide ventilation, as well as ducting for the cooled and / or heated air.

[0042] Additionally, the vehicle electronics 20 may include interior lights 48 that may be used to illuminate one or more areas within the vehicle cabin. The interior lights 48 may be light-emitting diodes that may be controlled by another VSM, such as an in-vehicle infotainment unit, and may be used to transmit light of different colors and intensities, as well as at different times. The power seat motors 46 may be automatically controlled by one or more vehicle systems (e.g., the BCM 24) and may be used to adjust the position, height, seatback recline, and / or other seat position or orientation characteristics. The power seat motors 46 may also be controlled via one or more handsets in the vehicle cabin.Various other types of VSMs may be incorporated into the system 10 and used with the method 200 below (. Fig. 3).

[0043] With reference to Fig. 2 illustrates an interior 100 of a vehicle 12 from the perspective of a rear-row passenger looking forward through the windshield 102 with an AR / VR device, such as the smartphone 14 or AR / VR headset 16. The vehicle interior 100 includes the windshield 102, the steering wheel 104, the center console or instrument panel 106 supporting the touchscreen display 58 and the wireless communication device 30, a touchscreen user interface 108 displayed on the touchscreen 58, a storage compartment 112, a gearshift 114, the seats 120 and 130, and the capacitive touch sensor assembly 50. The wireless communication device 30 communicates with the AR / VR devices within or near the vehicle interior 100, as represented by the wireless signals 110.According to the invention, the AR / VR devices 14, 16 display virtual vehicle control graphics 140, 142 over a specific vehicle component, namely over the capacitive touch sensor arrangement 50 on the rear side 122 of the driver seat 120.

[0044] As shown in the illustrated embodiment, the virtual vehicle control graphic 140 may be displayed via the capacitive touch sensor assembly 50. By displaying the virtual vehicle control graphic via a touch-based input device, such as the capacitive touch sensor assembly 50, the user of the AR / VR device may interact with the virtual vehicle control graphic by touching the touch-based input device to control or configure settings or perform actions of the vehicle. The virtual vehicle control graphic 140 may include menu items 141-146 as well as information 150. In further embodiments, the capacitive touch sensor assembly 50 may be included in other parts of the vehicle, such as on or near components 102-130.According to the invention, the vehicle includes a plurality of capacitive touch sensor assemblies 50 on the rear of each front-row vehicle seat. In other embodiments, other touch sensor assemblies (including resistive touch sensor assemblies) and / or other input mechanisms may be incorporated and distributed throughout the vehicle interior 100.

[0045] Other virtual vehicle control graphics 142 may be displayed in various areas of the AR / VR device's field of view, including via a corresponding vehicle input device. For example, virtual vehicle control graphics 142 show the current audio system volume level, and virtual vehicle control graphics 142 are displayed via touchscreen display 58, which can be used to control the vehicle's multimedia controls, including audio output levels.

[0046] With reference to Fig. 3 illustrates a method 300 for operating a vehicle with virtual vehicle controls. The method 200 may be performed by an AR / VR device, such as a smartphone 14 or AR / VR headset 16. In one embodiment, the method 200 (or portions of this method) may be implemented in a computer program product (or "application") embodied in a computer-readable medium and including instructions usable by one or more processors of one or more computers of one or more systems, such as the processor of the AR / VR device 14, 16. A computer program may include one or more software programs consisting of program instructions in source code, object code, executable code, or other format; one or more firmware programs; or hardware description language (HDL) files; and other program-related data.The data may include data structures, tables of values, or data in another suitable format. The program instructions may include program modules, routines, programs, objects, components, and / or the like. The computer program may be executed by one or more computers in conjunction with one another.

[0047] The program(s) may be executed on computer-readable media (such as memory that is part of the AR / VR device), which may be non-transitory and may include one or more storage devices, articles of manufacture, or the like. Examples of computer-readable media include computer system memory, e.g., RAM (random access memory), ROM (read-only memory); semiconductor memory, e.g., EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), flash memory; magnetic or optical disks or tapes; and / or the like. A computer-readable medium may also include computer-to-computer connections, for example, when data is transferred or delivered over a network or other communications link (wired, wireless, or a combination of both).All combinations of the above examples also fall within the scope of computer-readable media. It is therefore understood that the method may be performed, at least in part, by electronic items and / or devices capable of executing instructions according to one or more steps of the disclosed method. Furthermore, the conclusion of method 200, which is a method corresponding to method 200 but on-vehicle, may be represented on computer-readable media, such as memory 38 of wireless communication device 30, and according to one or more of the features discussed above with respect to method 200.

[0048] The method 200 begins with step 210, wherein a connection is established between the vehicle and an AR / VR device 16. A connection between the AR / VR device and the vehicle can be established via SRWC or via wired connectors. In the case of a wired connection, a user can connect a cable, such as a USB cable, to an appropriate port, such as a USB port near the center console 106, in the vehicle and in the AR / VR device. In other embodiments where an SRWC connection is established, the vehicle can transmit a light signal or advertisement via the SRWC circuitry 32. The advertisement can be the same message as an advertisement in the virtual vehicle control graphic or a separate message.The connection advertisement message may include data indicating that the vehicle is seeking or capable of connecting to an AR / VR device, or that the vehicle is capable of offering virtual vehicle control graphics services. This message may then be received by the AR / VR device, which may then generate a response, such as a message initiating a four-way handshake or other authentication and / or shared key / secrecy process. In further embodiments, the AR / VR device may send an advertisement or light signal message indicating to those SRWC devices within range that the AR / VR device is searching for devices capable of providing virtual vehicle control graphics services. Once the connection is established, the AR / VR device may send the request to the virtual vehicle control graphics (step 230) over the established connection.In some embodiments, no initial communication may be required before sending the virtual vehicle control graphic (step 230), and this virtual vehicle control graphic request may be the initial message between the two devices. After the connection is established, method 200 proceeds to step 220.

[0049] In step 220, the AR / VR device is used to capture images or videos with a camera. The camera may be included as part of the AR / VR device or as part of an AR / VR assembly on the user, such as a headset AR / VR assembly. The camera may be used to capture images or videos, after which the AR / VR device may present the captured images or videos on a display of the AR / VR device. The captured images or videos may be displayed immediately on the display of the AR / VR device or stored in memory for later viewing or other purposes. The method 200 continues with step 230.

[0050] In step 230, a virtual vehicle control graphic request is sent from the AR / VR device to the vehicle. The virtual vehicle control graphic request is a request for the virtual vehicle control graphic. The virtual vehicle control graphic request may include user credentials, such as a username / password pair or an authorized token (i.e., a token that authorizes virtual vehicle control graphics). In further embodiments, the credentials may be included as part of the initial connection establishment process (step 210). The virtual vehicle control graphic request may also include information related to the AR / VR device, which may include the manufacturer and / or model of the device, device processing attributes or capabilities, device release information, and version numbers of the device and the various components of the device.Additionally, the virtual vehicle control graphic request may include information related to the virtual vehicle control graphic, including a vehicle control graphic identifier that identifies specific virtual vehicle control graphics that may be used for display on the AR / VR device. Method 200 proceeds to step 240.

[0051] In step 240, a virtual vehicle control graphic response is received. The virtual vehicle control graphic response may be generated at the vehicle and in response to the virtual vehicle control graphic request sent by the AR / VR device (step 230). In one embodiment, the vehicle 12 may receive the virtual vehicle control graphic request over the established SRWC connection, and upon receipt, the vehicle 12 may verify the user credentials of the AR / VR device or user. After determining that the AR / VR device or user is authorized to use virtual vehicle control graphics, the vehicle may process the virtual vehicle control graphic request and then generate the virtual vehicle control graphic response.

[0052] In one embodiment, the virtual vehicle control graphic response also includes virtual vehicle control graphics that can be used for display on the display of the AR / VR device 16. The virtual vehicle control graphics can be selected for the virtual vehicle control graphic response based on information included in the virtual vehicle control graphic request, such as the vehicle control graphic identifier. For example, the virtual vehicle control graphic request can include information (e.g., the vehicle control graphic identifier) ​​indicating that the AR / VR device is requesting a specific user interface, such as a main screen of a vehicle controller or an infotainment module.Additionally or alternatively, the virtual vehicle control graphic response may include display information for the virtual vehicle control graphic, such as certain vehicle component information, certain graphic location information, and / or other information that may be used to locate a location of the virtual vehicle control graphics. The virtual vehicle control graphics may be stored in memory and / or displayed on the display of the AR / VR device and in a particular area. Method 200 proceeds to step 250.

[0053] In step 250, the virtual vehicle control graphics are presented on a display of the AR / VR device. The AR / VR device 16 may determine a display location on the display where the virtual vehicle control graphics should be located or displayed. This determination may be made based on information included in the virtual vehicle control graphics response, such as the designated vehicle component information or the designated graphic location information. The designated vehicle component information is information that designates a particular component of the vehicle (whether by name, reference label, etc.) as the component on which the virtual vehicle control graphics should be displayed or as a reference component used as a reference in locating the display position of the virtual vehicle control graphics.The designated graphic location information may include various information that can be used to locate the display position of the virtual vehicle control graphics. The graphics may be rendered over the captured images or videos, and as the user moves the AR / VR device, the graphics may be updated along with the captured images or videos so that the virtual vehicle control graphics appear as part of the vehicle interior 100. The method 200 then proceeds to step 260.

[0054] In step 260, the vehicle control input may be received. In one embodiment, the vehicle control input may be received at the AR / VR device, and in other embodiments, at the vehicle. Furthermore, in some embodiments, the vehicle control input may be received at both the AR / VR device and the vehicle. The vehicle control input may be input information used to control a vehicle function, used to navigate a user interface, or used to execute vehicle device functions. Various mechanisms may be used to receive the vehicle control inputs, as explained below.

[0055] In one embodiment, the vehicle control input may be received using detected user gestures or user speech. For example, the AR / VR device may use the camera to capture images and / or videos of the user, including the user's hands or arms, and then the AR / VR device may determine one or more vehicle control inputs based on the detected user movement and / or position. The AR / VR device may use processing techniques, such as image recognition and / or object recognition techniques, to determine information submitted by the user. In one case, a user may rotate their hand and forearm clockwise to increase the volume of the multimedia vehicle audio system 54, as indicated by graphic 142 in Fig. 2. In another example, the user may shake their head front to back and / or side to side. The AR / VR device may detect that a direction in which the user moves their head is based on the movement of the field of view of the AR / VR device's camera. Various gestures may be used to navigate through the user interface menus and / or select various vehicle settings or options. In yet another example, the AR / VR device may include a microphone configured to receive user speech. The user speech may either be processed via the AR / VR device using speech processing techniques, or the user speech may be sent to a remote server (or the vehicle) for speech processing.

[0056] In further embodiments, vehicle control input may be received through the use of vehicle system modules (VSMs), including capacitive touch sensor assembly 50, push button(s) 52, microphone 56, and / or touchscreen display 58. For example, the user may communicate via a user language that may be received via microphone 56. Or, the user may provide input by pressing a button 52 and touching a touchscreen display 58.

[0057] In further embodiments, the capacitive touch sensor assembly 50 can be used to receive vehicle control inputs. According to the invention, the capacitive touch sensor assembly is installed on the rear side 122 of a passenger or driver seat 120, as shown in Fig. 2. The capacitive touch sensor assembly can detect when a user touches an area of ​​the assembly, and additionally, the capacitive touch sensor assembly can detect the movement of a user's finger (or other body part) along the capacitive touch sensor assembly 50. The capacitive touch sensor can be configured to send this user input to another VSM (e.g., the wireless communication device 30 or the BCM 24), which can then determine a vehicle control input submitted by the user.In another embodiment, the vehicle 12 may have informed the AR / VR device 16 to present the virtual vehicle control graphics at a particular display position on the AR / VR device display, thereby enabling the vehicle to determine which controls the user is attempting to control based on the user's touch position on the capacitive touch sensor assembly 50 in conjunction with the position of certain adjustable or interactive controls within the virtual vehicle control graphics, such as the menu control items 141-146 of the virtual vehicle control graphics 140 (. Fig. 2). The method 200 continues with step 270.

[0058] In step 270, the vehicle control input is sent to the vehicle. The vehicle control inputs may be sent to the wireless communication device 30 of the vehicle 12 via the established connection or to another vehicle module via a wired connection. The vehicle control inputs may be stored in the memory of the AR / VR device for later use, for example, to perform step 280. The method 200 continues with step 280.

[0059] In step 280, the virtual vehicle control graphics are updated according to the vehicle control input and a vehicle response that occurs in response to the vehicle control input. In one embodiment, the virtual vehicle control graphic may be updated based on the vehicle control inputs received from the AR / VR device, such as in step 260. Alternatively or additionally, the virtual vehicle control graphic may be updated based on information received from a vehicle control input response, which is a response generated by the vehicle based on the vehicle control input. For example, a user may move their forearm and rotate it clockwise (of the vehicle control input), which may increase the volume, which may be determined using the virtual vehicle control graphic 142 ( Fig. 2). The vehicle control inputs may be received at the AR / VR device, and subsequently, the AR / VR device may send these vehicle control inputs to the vehicle via the established connection. Furthermore, the AR / VR device may display the virtual vehicle control graphics that reflect an increase in volume, such as displaying a volume of "8" in the virtual vehicle control graphics 142 ( Fig. 2).

[0060] In further embodiments, the AR / VR device may receive a vehicle control input response from the vehicle that is generated in response to the vehicle control input. This vehicle control input response may be generated and sent to the AR / VR device when the vehicle receives the vehicle control inputs from the AR / VR device or via one or more vehicle user interfaces. The vehicle control input responses may include updated virtual vehicle control graphics to be displayed by the AR / VR device that reflect an update to the vehicle control made by the vehicle in accordance with the vehicle control input. Method 200 then ends.

Claims

[1] A method for controlling a vehicle (12) using virtual vehicle controls, the method comprising: Establishing a short-range wireless communication (SRWC) connection to an augmented reality / virtual reality (AR / VR) device (14, 16) including a display and a camera and configured to capture image or video data, and wherein the AR / VR device (14, 16) includes short-range wireless communication (SRWC) circuitry used to transmit and receive SRWC signals; Receiving a virtual vehicle control graphics request from the AR / VR device (14, 16) on the vehicle (12); Sending a virtual vehicle control graphic response including a virtual vehicle control graphic from the vehicle (12) to the AR / VR device (14, 16) over the established SRWC connection, wherein the AR / VR device (14, 16) is further configured to display the virtual vehicle control graphic on the display of the AR / VR device (14, 16) such that the virtual vehicle control graphic is displayed over image or video data captured by the camera, wherein the virtual vehicle control graphic includes one or more vehicle user interface components; and Receiving a vehicle control input from a user of the AR / VR device (14, 16); characterized by , that determining whether the AR / VR device (14, 16) is authorized and / or authenticated to receive and / or use virtual vehicle control graphics; that the vehicle user interface component corresponds to a capacitive touch sensor assembly (50) integrated into the vehicle electronics of the vehicle (12) and installed on the rear side (122) of the driver or passenger seat (120) of the vehicle (12), wherein the AR / VR device (14, 16) is further configured to display the virtual vehicle control graphic on the display of the AR / VR device (14, 16) over the capacitive touch sensor assembly (50); and that receiving the vehicle control input further comprises receiving sensor position inputs from the capacitive touch sensor array (50) over which the virtual vehicle control graphic is presented on the display of the AR / VR device (14, 16) that indicate a position within the capacitive touch sensor array (50) that is touched by the user.

Images