METHOD FOR CONFIGURING AN AUGMENTED REALITY / VIRTUAL REALITY DEVICE FOR ADDING IMAGE OR VIDEO DATA USING A VIRTUAL VEHICLE SURFACE (SKIN)

Abstract

Method for configuring an augmented reality / virtual reality (AR / VR) device (14, 16) for augmenting image or video data using a virtual vehicle surface, wherein the method is performed by vehicle electronics (20) integrated in a vehicle (12), the method comprising: Establishing a connection to the AR / VR device (14, 16) using a wireless communication device (30) integrated into the vehicle electronics (20) of the vehicle (12); Receiving an initial virtual vehicle surface request from the AR / VR device (14, 16); Sending a second virtual vehicle surface request to one or more remote servers (82) in response to receiving the first virtual vehicle surface request; Receiving virtual vehicle surface access information; and Sending a virtual vehicle surface response to the AR / VR device (14, 16) via the established connection, wherein the AR / VR device (14, 16) is configured to receive image or video data and to display the virtual vehicle surface over the received image or video data, such that the virtual vehicle surface overlays a portion of the vehicle (12), and wherein the virtual vehicle surface response includes the virtual vehicle surface access information that can be used by the AR / VR device (14, 16) to receive the virtual vehicle surface from the remote server (82); wherein the virtual vehicle surface access information includes a virtual vehicle surface token which, when presented to the remote server (82), causes the remote server (82) to send the virtual vehicle surface to the AR / VR device (14, 16).

Description

INTRODUCTION

[0001] The present invention relates to the use of augmented reality / virtual reality (AR / VR) devices for displaying a virtual vehicle surface (skin) over images depicting a vehicle.

[0002] Vehicles contain hardware and software capable of performing various multimedia functions, including playing audio received from radio stations and mobile devices (smartphones, tablets), many of which utilize wireless communication, such as short-range wireless communication (SRWC). Furthermore, vehicles include numerous vehicle system modules (VSMs) that can be used to provide information to other electronic computing devices, such as personal mobile devices.

[0003] DE 10 2014 009 697 A1 discloses a method for operating a mobile virtual reality system in which a user, seated in a real vehicle, wears VR glasses that display a virtual vehicle interior. The configuration of this virtual interior is provided by an external data processing unit coupled to the vehicle and sent directly to the VR glasses. DE 10 2016 207 530 A1 discloses a system for displaying a virtual vehicle interior in which physical interior components such as seats and a steering wheel are arranged on a platform outside a vehicle and combined with digital glasses. US 2016 / 0019212 A1 describes an assistance system for technicians in the technical field of aircraft maintenance in which a wearable device with a camera identifies a component and, based on its identifier, overlays corresponding maintenance data onto an augmented reality display. SUMMARY

[0004] A method for configuring an augmented reality / virtual reality (AR / VR) device to augment image or video data using a virtual vehicle surface, wherein the method is performed by vehicle electronics located in a vehicle, and the method comprises: establishing a connection to the AR / VR device using a wireless communication device integrated into the vehicle electronics of the vehicle; receiving an initial virtual vehicle surface request from the AR / VR device; sending a second virtual vehicle surface request to one or more remote servers in response to receiving the initial virtual vehicle surface request; and receiving virtual vehicle surface access information.and sending a virtual vehicle surface response to the AR / VR device over the established connection, wherein the AR / VR device is configured to receive image or video data and to display the virtual vehicle surface over the received image or video data, such that the virtual vehicle surface overlays a portion of the vehicle, and wherein the virtual vehicle surface response includes the virtual vehicle surface access information that can be used by the AR / VR device to receive the virtual vehicle surface from the remote server. The virtual vehicle surface access information includes a virtual vehicle surface token which, when presented to the remote server, causes the remote server to send the virtual vehicle surface to the AR / VR device.

[0005] According to one or more different embodiments, this method can include one or more of the following features in any technically possible combination of these features: • the virtual vehicle surface configured to be inserted into captured image or video data, so that the virtual vehicle surface covers a section of the vehicle's interior cabin; • the AR / VR device, which is a pair of wearable electronic augmented reality / virtual reality (AR / VR) glasses configured to visually present augmented reality or virtual reality media content to a user; and / or • the AR / VR device, which is a personal AR / VR device.

[0006] According to a further unclaimed aspect of the invention, a method for configuring an augmented reality / virtual reality (AR / VR) device is provided to augment image or video data using a virtual vehicle surface, the method comprising: capturing image or video data using a camera located on the AR / VR device; receiving a virtual vehicle surface advertising message from a vehicle, wherein the vehicle uses a wireless communication device to transmit the virtual vehicle surface advertising message; in response to receiving the virtual vehicle surface advertising message, sending a virtual vehicle surface request, wherein the virtual vehicle surface request includes information from the virtual vehicle surface advertising message;and receiving a virtual vehicle surface response, wherein the virtual vehicle surface response includes a digital representation of the virtual vehicle surface or information that can be used by the AR / VR device to derive the digital representation of the virtual vehicle surface.

[0007] According to one or more different embodiments, this method can include one or more of the following features in any technically possible combination of these features: • the virtual vehicle surface request sent to the vehicle, and wherein the virtual vehicle surface response is sent by the vehicle; • the virtual vehicle surface response, which includes virtual vehicle surface access information, and wherein the method further comprises the following steps: sending a second virtual vehicle surface request to a remote server using the virtual vehicle surface access information; and receiving a second virtual vehicle surface response, which includes the digital representation of the virtual vehicle surface; • the virtual vehicle surface, which includes a plurality of virtual vehicle surface components, and wherein the method further includes the following steps: determining, in order to display at least one virtual vehicle surface component from the plurality of virtual vehicle surface components based on the image or video data; and reproducing at least a portion of the at least one virtual vehicle surface component on a display of the AR / VR device; • before rendering at least one part of the at least one virtual vehicle surface component on the display of the AR / VR device, modifying the at least one virtual vehicle surface component based on the one or more captured image or video data; • the virtual vehicle surface, which includes the virtual exterior environment graphics, and wherein the method further includes the following steps: Determining outdoor areas located outside the vehicle and within the captured image or video data; and rendering the virtual outdoor environment graphics within the outdoor areas; • Determining the orientation of the AR / VR device using an accelerometer located on the AR / VR device; • the vehicle that is configured to determine a position of the AR / VR device within the vehicle and then obtain the virtual vehicle surface based on the position of the AR / VR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] One or more embodiments of the invention are described below in conjunction with the accompanying drawings, wherein identical designations denote identical elements, and wherein the following applies: Fig. 1 is a block diagram representing an embodiment of a communication system capable of using the method disclosed herein; Fig. Figure 2 is a perspective view showing one embodiment of a vehicle interior; Fig. Figure 3 is a perspective view showing an embodiment of a vehicle interior with a virtual vehicle surface applied to the interior of the vehicle; Fig. Figure 4 is a flowchart of an embodiment of a method for transmitting a notice that virtual vehicle surfaces are available for use with augmented reality / virtual reality (AR / VR) devices; Fig. Figure 5 is a flowchart of an embodiment of a method for sending virtual vehicle surface information to an AR / VR device; and Fig. Figure 6 is a flowchart of an embodiment of a method for displaying the virtual vehicle surface over a corresponding area of ​​a vehicle using an AR / VR device. DETAILED DESCRIPTION

[0009] The system and method described below enables an augmented reality / virtual reality (AR / VR) device to display virtual vehicle surfaces on a video or image in such a way that the virtual vehicle surface is overlaid on a surface corresponding to a vehicle. AR / VR devices can use cameras to capture an image or video of an area around a user and then render graphics over specific parts of the captured image or video. In this way, the AR / VR device can display graphics over real-world images or videos of a user's surroundings, so that the graphics complement the user's visual perception of the environment. These complementary graphics can be used to create aesthetically pleasing visuals and provide information to the user.For example, a virtual vehicle skin can be developed and displayed by AR / VR devices in conjunction with videos or images of the user's environment to create the perception that the vehicle looks as depicted in the virtual vehicle surface. AR / VR devices can use a camera that, when worn by a user, faces an area in front of the user (the "AR / VR device's field of view"). In one embodiment, the AR / VR device is a pair of AR or VR glasses configured to fit around a user's head and incorporating a display that, when the glasses are worn correctly, is positioned in front of the user's eyes or field of view.The AR / VR device can download a virtual vehicle surface associated with the vehicle. Then, when the vehicle is within the AR / VR device's field of view, the device can display the virtual vehicle surface on a captured image or video in an area corresponding to the vehicle or a specific component of the vehicle. In many embodiments, the method and system described below can be used to magnify an area inside the vehicle, such as the interior cabin.

[0010] In some embodiments, a virtual vehicle interface can be developed based on a layout or design of a specific vehicle. This virtual vehicle interface can then be sent to AR / VR devices on or near the vehicle, allowing the AR / VR devices to display the virtual vehicle interface overlaid on the vehicle. Because the virtual vehicle interface corresponds to a vehicle layout, the virtual vehicle interfaces can be displayed while simultaneously transmitting the overall vehicle layout, enabling the user to still operate the vehicle even when the virtual vehicle graphics are superimposed. For example, a virtual vehicle interface can be developed for a vehicle's interior that includes various vehicle controls.The interface can be developed to achieve various aesthetic effects, within the constraints of the vehicle layout. This can also include designing the virtual vehicle surface to reflect the same physical shape and layout as the vehicle. Thus, when displaying the virtual vehicle surface over the vehicle's areas, the user can still see and use the vehicle's functional aspects, such as the various controls, and the overall layout can be maintained so that the user can still perceive the vehicle's physical boundaries.

[0011] In one embodiment, a user can access a virtual vehicle surface catalog and select various virtual vehicle surface types that the user wishes to assign to their vehicle. The user can select and / or purchase this virtual vehicle surface, and subsequently, the virtual vehicle surface can be assigned to a specific vehicle belonging to the user. The virtual vehicle surface can then be sent to the vehicle, which can store the surface and / or send it to AR / VR devices. In a particular embodiment, the vehicle can send a virtual vehicle surface availability message indicating to the AR / VR devices that a virtual vehicle surface is available for the vehicle.The vehicle can then receive a request for a specific surface and, in response, obtain the virtual vehicle surface and send it to the AR / VR device. In some embodiments, obtaining the virtual vehicle surface may involve sending a request for the surface to a remote device and then receiving the virtual vehicle surface from the remote device. In other embodiments, obtaining the virtual vehicle surface may involve retrieving the virtual vehicle surface from a storage device located in the vehicle.

[0012] As mentioned above, the vehicle can periodically send a message indicating the availability of a virtual vehicle surface. Upon receiving this notification, an AR / VR device can determine whether to obtain a specific virtual vehicle surface and, if so, send a virtual vehicle surface request to the vehicle or to a remote device. In one embodiment, the virtual vehicle surface request can be generated based on information (e.g., VIN, virtual vehicle surface ID) contained in the virtual vehicle surface availability message and then sent to a remote device. The remote device can respond by sending the requested virtual vehicle surface to the AR / VR device.In further embodiments, the virtual vehicle surface request can be sent to the vehicle, which can then receive the virtual vehicle surface and send it to the AR / VR device. After the AR / VR device has received the virtual vehicle surface, it can display the virtual vehicle surface on a captured image or video of the environment, so that graphics of the virtual vehicle surface are displayed over the vehicle or a vehicle component.

[0013] With reference to Fig. Figure 1 presents an operating environment comprising a communication system 10, which can be used to implement the method disclosed herein. The communication system 10 generally includes a vehicle 12 with a wireless communication device 30, augmented reality / virtual reality (AR / VR) devices 14 and 16, a constellation of global navigation satellite system (GNSS) satellites 60, one or more wireless carrier systems 70, a fixed-line communication network 76, a computer or server 78, and a remote facility 80. It is understood that the disclosed method can be used with any number of different systems and is not specifically limited to the operating environment shown herein. The architecture, construction, setup, and general operation of the system 10 and its individual components are also generally known in the art.Thus, the following paragraphs merely provide a brief overview of such a communication system 10; but other systems not shown here could also use the disclosed methods.

[0014] The wireless carrier system 70 can be any suitable mobile communication system. The carrier system 70 is shown with a mobile communication mast 72; however, the carrier system 70 can include one or more of the following components (e.g., depending on the mobile communication technology): mobile communication masts, base transmission stations, mobile switching centers, base station controllers, developed nodes (e.g., eNodeBs), mobility management units (MMEs), serving and PGN gateways, etc., as well as any other network components required to connect the wireless carrier system 70 to the fixed network 76 or to connect the wireless carrier system to user equipment (UEs, which may include, for example, the telematics equipment in the vehicle 12). The carrier system 70 can implement any suitable communication technology, including GSM / GPRS technology, CDMA or CDMA2000 technology, LTE technology, etc.In general, wireless carrier systems 70, their components, the arrangement of their components, the interaction of the components, etc. are largely known in the prior art.

[0015] Apart from using the wireless carrier system 70, a different wireless carrier system in the form of satellite communication can be used to provide unidirectional or bidirectional communication with the vehicle. This can be done using one or more communication satellites (not shown) and an uplink transmitting station (not shown). Unidirectional communication could, for example, be satellite radio services, in which programmed content data (news, music, etc.) is received from the uplink transmitting station, packaged for uploading, and then sent to the satellite, which broadcasts the programming to the subscribers. Bidirectional communication could, for example, be satellite telephony services, using the one or more communication satellites to relay telephone communications between the vehicle 12 and the uplink transmitting station.When used, this satellite telephony can be used either in addition to or instead of the wireless carrier system 70.

[0016] The fixed network 76 can be a conventional land-based telecommunications network connected to one or more landline telephones and linking the wireless carrier system 70 to the remote location 80. For example, the fixed network 76 can include a public switched telephone network (PSTN) as used to provide landline telephony, packet-switched data communications, and internet infrastructure. One or more segments of the fixed network 76 could be implemented using a standard landline, a fiber optic network or other fiber-optic network, a cable network, power lines, other wireless networks such as local area networks (WLANs) or networks providing wireless broadband access (BWA), or any combination thereof.

[0017] The 78 computers (not shown) could be some of a number of computers accessible via a private or public network, such as the Internet. Each of these 78 computers could be used for one or more purposes, such as purchasing or selecting different virtual vehicle skins for use with the 12-inch vehicle.Other such accessible computers 78 may include, for example: a computer in a customer service center where diagnostic information and other vehicle data can be uploaded from the vehicle; a client computer used by the vehicle owner or another participant for such purposes as accessing or receiving vehicle data or setting or configuring participant preferences or controlling vehicle functions; a car-sharing server that coordinates reservations and / or registrations of multiple users requesting to use a vehicle as part of a car-sharing service; or a third-party storage location to which or from which vehicle data or other information is provided either by communicating with the vehicle 12 or the remote facility 80 or both.A computer 78 can also be used to provide internet connectivity, such as DNS services, or as a network address server that uses DHCP or another suitable protocol to assign an IP address to the vehicle 12. In one embodiment, computers 78 can be used to perform the method described herein; in other embodiments, the method can be performed by servers or other computer devices in the remote facility 80, as explained in more detail below; and in yet another embodiment, the method can be performed by a combination of computers 78 and servers in the remote facility 80.

[0018] The remote facility 80 can be designed to provide a range of different system backend functions to the vehicle electronics 20 and / or the AR / VR devices 14, 16 using one or more electronic servers, and in many cases, it can be a vehicle-related backend service facility providing vehicle-related backend functions. The remote facility 80 includes servers (vehicle-related backend service servers) 82 and databases 84, which can be stored on a variety of storage devices. The remote facility 80 can also include one or more network switches, live advisors, and an automated speech output system (VRS), with which professionals in the field are familiar.The remote device 80 can include one or all of the various components, all of which are preferably interconnected via a wired or wireless local area network. The remote device 80 receives and transmits data via a modem connected to the fixed network 76. Data transmissions can also be carried out via wireless systems, such as IEEE 802.11x, GPRS, and the like. Those skilled in the art will recognize that, although only one remote device 80 and one computer 78 are shown in the illustrated embodiment, numerous remote devices 80 and / or computers 78 can be used.

[0019] The Server 82 can be computers or other computing devices that include at least one processor and memory. The processor can be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, controllers, vehicle communication processors, and application-specific integrated circuits (ASICs). The processors can be dedicated processors used only by the Server 82 or shared with other systems. The at least one processor can execute various types of digitally stored instructions, such as software or firmware programs stored in memory (e.g., EEPROM, RAM, ROM), enabling the Server 82 to provide a wide variety of services.For example, the at least one processor can execute programs or process data to perform at least part of the procedure described herein. For network communication (e.g., intranetwork communication, internetwork communication including Internet connections), the servers can include one or more network interface cards (NICs) (including wireless NICs (WNICs)) that can be used for data transport to and from the computers. These NICs can enable one or more servers 82 to connect to each other, to databases 84, or to other network devices, including routers, modems, and / or switches. In a particular embodiment, the NICs (including WNICs) of the servers 82 can enable SRWC connections and / or Ethernet ports (IEEE 802.1).3) include, to which Ethernet cables can be connected, which can provide a data connection between two or more devices. The remote facility 80 can include a number of routers, modems, switches, or other network devices that can be used to provide network functions, such as connecting to the fixed network 76 and / or the mobile carrier system 70. In one embodiment, the servers 82 include an application that performs at least part of what is described below. Fig. 4 illustrated procedures are carried out.

[0020] The databases 84 can be stored on a variety of storage media, such as RAM or other temporary storage, as well as non-volatile computer-readable media (e.g., EEPROM) or any other electronic computer medium that stores some or all of the software for performing the various external device functions described herein. In addition, one or more databases in the remote facility can store account information, as well as virtual vehicle surface mapping information and / or the virtual vehicle surface itself. As used herein, "virtual vehicle surface mapping" information refers to the information that maps a particular vehicle to a particular virtual vehicle surface. Digital representations of the virtual vehicle surface can be stored in the databases 84.Furthermore, a vehicle information database can be integrated, storing information about one or more vehicles, such as vehicle identification numbers (VINs), virtual vehicle skin identifiers, vehicle layout types or features, and various other vehicle-related data. The vehicle layout types or features can indicate the general layout of the vehicle and may correspond to a model year (e.g., 2018 Cruze®). Additionally, specific features of a particular vehicle can be stored in databases, such as whether the vehicle includes one or more optional features.

[0021] The Smartphone 14 is an augmented reality / virtual reality (AR / VR) device capable of delivering media content to one or more interfaces (either within the Smartphone 14 itself or in another device connected to the Smartphone 14). As used herein, “AR / VR devices” are those devices configured to receive and present AR / VR media content and possessing suitable hardware and processing capabilities for presenting AR / VR media content. Non-restrictive examples of media control devices include a smartphone, a personal computer (PC), a laptop, a tablet, a smart TV, and AR / VR glasses or a display.In some embodiments, the smartphone (media control device) 14 may include: hardware, software, and / or firmware enabling cellular telecommunications and short-range wireless communication (SRWC), as well as other mobile device applications. As used herein, a personal AR / VR device is an AR / VR device that can be worn by a user, wherein the wearability of the device is at least partially user-dependent, such as a wearable device (e.g., a smartwatch, AR / VR glasses), an implantable device, or a handheld device (e.g., a smartphone, a tablet, a laptop). And as used herein, a short-range wireless communication (SRWC) device is a device that is SRWC-enabled and includes the necessary SRWC circuitry to perform such SRWC.

[0022] In many embodiments, the hardware of the AR / VR devices, including the smartphone 14, can include a processor and memory (e.g., a non-volatile, computer-readable medium configured to operate with the processor) for storing the software, firmware, etc. In one embodiment, the AR / VR devices can include an SRWC circuit, such as the one described below with respect to the wireless communication device 30 of the vehicle 12. The processor and memory of the AR / VR device can enable various software applications that can be pre-installed or installed by the user (or manufacturer) (e.g., with a software application or graphical user interface (GUI)). An implementation of an application of the AR / VR device can enable the AR / VR device (e.g., the smartphone 14) to display media content (e.g., videos, videos, etc.).a virtual vehicle surface) via the fixed network 76 and / or the mobile carrier system 70 and to play back the media content or otherwise present it to a user via one or more user device interfaces on the AR / VR device. In one embodiment, the AR / VR device 14 may include a display that can be used to visually present media content to a user and an audio system that can be used to acoustically present media content to a user. Other playback components or devices, such as a tactile or haptic device, may be received and / or controlled by the AR / VR device 14.

[0023] In some embodiments, an optical head-mounted display can be used in conjunction with an AR / VR device (e.g., a smartphone) to mount the device in front of the user's field of vision while simultaneously limiting the user's peripheral vision. The head-mounted optical display can be a separately purchased and manufactured device that includes a slot or is otherwise configured to hold AR / VR devices in such a way that, when the device is inserted into the slot (or otherwise attached to the optical head-mounted display) and properly worn by the user, the display of the media control device is within the user's field of vision. Such a head-mounted optical display, used in conjunction with a suitable AR / VR device, can be used to control an ordinary AR / VR device (e.g., a smartphone).to turn a smartphone into AR / VR glasses. As experts understand, although AR / VR headsets can be described as glasses, only a single display is required, and not a pair of displays as the name might suggest.

[0024] The pair of augmented reality / virtual reality (AR / VR) glasses 16 represents an AR / VR device that can be used to receive, process, and present audiovisual content to a user, such as a virtual vehicle skin. The pair of AR / VR glasses 16 is a head-mounted display that includes a processor and memory, as well as computer instructions (e.g., software, firmware) for displaying AR / VR media content. In some embodiments, the AR / VR glasses 16 can be an AR / VR headset, which consists of AR / VR glasses with an integrated head mount and an audio speaker.The AR / VR glasses 16 can also include other components, such as an SRWC circuit (similar to the SRWC circuit 32 described below), an accelerometer (or orientation sensors), one or more cameras or other lenses capable of receiving and digitizing light signals, a battery, speakers or headphones for displaying audio content, and various other hardware or software components. The AR / VR glasses can be configured for use with applications developed using a Software Development Kit (SDK) specifically configured for AR / VR devices or that support AR / VR functionality.

[0025] Furthermore, as mentioned earlier, the AR / VR glasses can include speakers or headphones that can present audio content to the user via sound waves. This audio content can be part of audiovisual media, and the audio and video output of the AR / VR glasses can be synchronized. In other embodiments, the AR / VR glasses may only include a video output and can connect to other user interfaces, such as those present in the vehicle, to output the audio content. And, as explained in more detail below, the AR / VR glasses can send virtual vehicle surface requests to the vehicle to obtain one or more virtual vehicle surfaces for use when extending the AR / VR glasses' field of view to include an area of ​​the vehicle.These virtual vehicle surface requests can be aggregated into a single request that is initially sent by the AR / VR glasses (e.g., at the start of an immersive media content experience), or sent at a time that corresponds to the requirements of the desired vehicle function.

[0026] The vehicle 12 is shown in the illustrated embodiment as a passenger car; however, it should be noted that any other vehicle, including motorcycles, trucks, all-terrain vehicles (SUVs), recreational vehicles (RVs), watercraft, aircraft, etc., can also be used. Some vehicle electronics 20 are generally located in Fig. The vehicle electronics are shown in Figure 1 and include a global navigation satellite system (GNSS) module 22, a body control module (BCM) 24, other vehicle system modules (VSMs) 26, and a wireless communication device 30. Some or all of the vehicle electronics can be interconnected for communication via one or more communication buses, such as bus 28. The communication bus 28 provides network connections to the vehicle electronics using one or more network protocols. Examples of suitable network connections include a controller area network (CAN), a media-oriented system transfer (MOST), a local area network (LIN), a local area network (LAN), and other suitable connections, such as Ethernet, or others that conform to, among others, the known ISO, SAE, and IEEE standards and specifications.

[0027] The vehicle 12 can include numerous vehicle system modules (VSMs) as part of the vehicle electronics 20, such as the GNSS module 22, the BCM 24, the wireless communication device 30, and the vehicle user interfaces 52-58, as described in more detail below. The vehicle 12 can also include other VSMs 26 in the form of electronic hardware components located throughout the vehicle that receive input from one or more sensors and use the acquired input to perform diagnostic, monitoring, control, reporting, and / or other functions. Each of the VSMs 26 is preferably connected to the other VSMs and the wireless communication device 30 via the communication bus 28 and can be programmed to perform vehicle system and subsystem diagnostic tests.One or more VSMs 26 can have their software or firmware updated periodically or occasionally, and in some embodiments such vehicle updates can be over-the-air (OTA) updates received by a computer 78 or a remote facility 80 via the fixed network 76 and communication devices 30. Those skilled in the art will recognize that the aforementioned VSMs are only examples of some of the modules that can be used in the vehicle 12, but that numerous other modules are also possible.

[0028] The wireless communication device 30 is capable of transmitting data via short-range wireless communication (SRWC) and / or cellular communication using a cellular chipset 34, as illustrated in the embodiment shown. In the illustrated embodiment, a wireless communication device 30 includes an SRWC circuit 32, a cellular chipset 34, a processor 36, a memory 38, and antennas 40 and 50. In one embodiment, the wireless communication device 30 can be a standalone module, or in other embodiments, the device 30 can be integrated or included as part of one or more other vehicle system modules, such as a center-stack module (CSM), a body control module (BCM) 24, an infotainment module, a telematics unit, a head unit, and / or a gateway module.In some embodiments, the device 30 can be implemented as an OEM-installed (embedded) device or as an aftermarket device that is installed in the vehicle.

[0029] The wireless communication device 30 can be configured for wireless communication according to one or more wireless protocols, including short-range wireless communication (SRWC), such as any of the IEEE 802.11 protocols, Wi-Fi™, WiMAX™, ZigBee™, Wi-Fi™ Direct, Bluetooth™, Bluetooth™ Low Energy (BLE), or near field communication (NFC). As used herein, Bluetooth™ refers to any of the Bluetooth™ technologies, such as Bluetooth Low Energy™ (BLE), Bluetooth™ 4.1, Bluetooth™ 4.2, Bluetooth™ 5.0, and other Bluetooth™ technologies that may be developed. As used herein, Wi-Fi™ or Wi-Fi™ technology refers to any of the Wi-Fi™ technologies, such as IEEE 802.11b / g / n / ac or any other IEEE 802.11 technology. The short-range wireless communication (SRWC) circuit 32 enables the wireless communication device 30 to send and receive SRWC signals, such as BLE signals.The SRWC circuit can enable communication between the device 30 and another SRWC device. Furthermore, in some embodiments, the wireless communication device can include a cellular chipset 34, enabling the device to communicate using one or more cellular protocols, such as those used by the cellular carrier system 70. In other embodiments, the cellular chipset 34 can be integrated into another VSM, such as a separate telematics unit.

[0030] The wireless communication device 30 enables the vehicle 12 to be connected to one or more remote networks (e.g., one or more networks in the remote facility 80 or computer 78) via packet-switched data communication. This packet-switched data communication can be achieved using a non-vehicle-mounted wireless access point connected to a fixed network via a router or modem. When the data transmission device 30 is used for packet-switched data communication, such as TCP / IP, it can be configured with a static IP address or set up to automatically receive an assigned IP address from another device on the network, such as a router or network address server.

[0031] Packet-switched data transmissions can also be carried out using a mobile network to which the device 30 can access. The communication device 30 can transmit data via a mobile communication chipset 34 over the wireless carrier system 70. In such an embodiment, radio transmissions can be used to establish a communication channel, such as a voice channel and / or a data channel, with the wireless carrier system 70, so that voice and / or data transmissions can be sent and received over the channel. Data can be transmitted either via a data connection, such as packet data transmission over a data channel, or via a voice channel, using techniques known in the field.For combined services that include both voice and data communication, the system can use a single call over a voice channel and switch between voice and data transmission over the voice channel as needed; here too, techniques known to experts are used.

[0032] The processor 36 can be any type of device capable of processing electronic instructions, including microprocessors, microcontrollers, host processors, controllers, vehicle communication processors, and application-specific integrated circuits (ASICs). It can be a processor specifically designated for the data transmission device 30 or it can be shared with other vehicle systems. The processor 36 executes various types of digitally stored instructions, such as software or firmware programs stored in memory 38, which enable the device 30 to provide a wide variety of services. For example, the processor 36 can execute programs or process data to perform at least part of the procedure described herein. Memory 38 can be volatile RAM or other temporary storage, as well as a non-volatile, computer-readable medium (e.g., USB flash drive).EEPROM) or any other electronic computer medium that stores part or all of the software for performing the various external device functions described herein. Components similar to those described above (the processor 36 and / or the memory 38, as well as the SRWC circuit 32 and the mobile communication chipset 34) can be integrated into the AR / VR devices 14 and / or 16.

[0033] In one embodiment, the wireless communication device 30 can operate in both the switched-on and switched-off states. As used herein, a "switched-on state" is a state of the vehicle in which the vehicle's ignition or primary drive system is switched on, and, as used herein, a "switched-off state" is a state of the vehicle in which the vehicle's ignition or primary drive system is not switched on. The operation or state of the wireless communication device 30 can be controlled by another vehicle system module, such as the vehicle electrical system control unit or an infotainment module. In the switched-on state, the wireless communication device 30 can remain "switched on" at all times or be powered by a vehicle battery or other power source.When switched off, the wireless communication device 30 can be kept in a low-power mode or periodically supplied with energy so that the device 30 can be activated and perform functions.

[0034] The vehicle can use the wireless communication device 30 to detect other SRWC devices, such as the AR / VR devices 14, 16. A connection between the wireless communication device 30 and one or more devices 14, 16 can enable the operation of various vehicle device functionalities. Vehicle device functionality (or function) refers to any function of the vehicle that can be augmented, enhanced, supported, or performed by an AR / VR device; any function of the AR / VR device that can be augmented, enhanced, supported, or performed by the vehicle; or any other function that can be performed by the vehicle and one or more mobile devices.For example, the vehicle device functionality may include the use of the AR / VR device 14 to provide the vehicle with a contact list that can be displayed on the visual display 58, audiovisual media content that can be played back on the display 58 via the speakers 54, navigation information (e.g. start and / or end positions) and / or vehicle commands or instructions that direct the vehicle to perform certain operations.Other examples of vehicle device functionality include using the vehicle electronics to equip the AR / VR device with a hands-free system, for example by using vehicle user interfaces to make, receive and conduct voice calls; sending information to the AR / VR device, such as geographical information to the AR / VR device 14 (such as information obtained from the GPS module 22) or vehicle diagnostic information or codes; and executing commands (e.g. commands to perform certain vehicle functions) received from the content playback device in the vehicle.

[0035] The wireless communication device 30 can be configured to listen for one or more wireless signals, such as those that can be transmitted by the AR / VR device 14. In one embodiment, the wireless communication device 30 can receive a connection request message from the AR / VR device 14 or 16. The wireless communication device 30 can then establish an SRWC connection with the requesting device via a four-way handshake and / or other connection establishment methods, such as those that establish a shared secret or one or more digital keys or other credentials used to secure the SRWC connection.

[0036] In one embodiment, the wireless communication device 30 can be placed in a discovery mode when the vehicle is switched on. As used herein, a discovery mode is an operating mode for an SRWC device in which the SRWC device transmits wireless messages typically designed to detect or recognize other SRWC devices using an SRWC protocol or technology. The discovery mode may involve the periodic or intermittent transmission of messages or advertisements to elicit a response from other SRWC devices within range. In one embodiment, the wireless communication device 30 can transmit a Bluetooth™ Low Energy advertising packet (e.g., an advertisement), such as an ADV_IND (a non-directional advertisement), an ADV_DIRECT_IND (a directional advertisement), or an ADV_SCAN_IND (a scannable non-directional advertisement).For example, the wireless communication device 30 can send an ADV_IND message at a predefined or predetermined time interval (e.g., 30 milliseconds (ms)). Alternatively, the device 30 can vary the time interval by randomizing it (e.g., randomly or pseudo-randomly selecting a time between 10 ms and 40 ms for each advertisement). This advertisement can include information that virtual vehicle surfaces are available for use with AR / VR devices to supplement or extend a view of the vehicle 12. In other embodiments, a wired communication link can be established between the AR / VR devices 14, 16 and the vehicle, for example, by using Universal Serial Bus (USB) cables and drivers integrated into the respective devices.

[0037] The global navigation satellite system (GNSS) module 22 receives radio signals from a constellation of GNSS satellites. In one embodiment, the GNSS module 22 can be a global positioning system (GPS) module that receives GPS signals from a constellation of GPS satellites 60. The GNSS module 22 can be used to provide the driver with navigation and other position-related services. Navigation information can be displayed on the screen 58 (or another display within the vehicle) or presented verbally, as is the case, for example, with turn-by-turn navigation.Navigation services can be provided using an associated vehicle navigation module (which may be part of the GNSS module 22), or some or all navigation services can be provided via a telematics unit installed in the vehicle, in which position information is sent to a remote location for the purpose of equipping the vehicle with navigation maps, map annotations (points of interest, restaurants, etc.), route calculations, and the like. The position information can be transmitted to the remote location 80 or to another remote computer system, such as the computer 78, for other purposes, such as a car-sharing service. Additionally, new or updated map data can be downloaded from the remote location 80 to the GNSS module 22 via a telematics unit installed in the vehicle.

[0038] The vehicle electronics 20 also include a number of vehicle user interfaces that provide vehicle occupants with a means of providing and / or receiving information, including push button(s) 52, audio system 54, microphone 56, and optical display 58. As used herein, the term “vehicle user interface” broadly encompasses any suitable form of electronic device, including both hardware and software components located in the vehicle, that enables a vehicle user to communicate with or through a component of the vehicle. The push button(s) 52 enable manual user input into the communication device 30 to provide additional data, responses, or control inputs. The audio system 54 provides audio output to a vehicle occupant and may be an associated independent system or part of the primary vehicle audio system.According to the specific embodiment shown here, the audio system 54 is operationally coupled to both the vehicle bus 28 and an entertainment bus (not shown) and can provide AM, FM, and satellite radio, CD, DVD, and other multimedia functionality. This functionality can be provided in conjunction with the infotainment module or independently. The microphone 56 provides audio input to the wireless communication device 30 to enable the driver or other occupants to provide voice controls and perform hands-free calling via the wireless carrier system 70. For this purpose, it can be connected to an integrated automatic speech processing unit that uses human-machine interface (HMI) technology known to those skilled in the art.The optical display or touchscreen 58 is preferably a graphic display and can be used to provide a variety of input and output functions.

[0039] Display 58 can be a touchscreen on the dashboard, a heads-up display reflected from the windshield, or a projector capable of projecting graphics for viewing by a vehicle occupant. Various other vehicle user interfaces can also be used, as the interfaces of Fig. The numbers 1 serve only as an example of a specific implementation.

[0040] The body control module (BCM) 24 is, in the exemplary embodiment, made of Fig. 1 as electrically connected to the communication bus 28. In some embodiments, the BCM 24 can be integrated with or as part of a center stack module (CSM) and / or can be integrated with the wireless communication device 30. Or the BCM can be a separate device connected via the bus 28. The BCM 24 can include a processor and / or memory similar to the processor 34 and memory 36 of the wireless communication device 30, as explained below. The BCM 24 can communicate with the wireless device 30 and / or one or more vehicle system modules, such as an engine control unit (ECU) (not shown), an audio system 54, or other VSMs 26.The BCM 24 can include a processor and memory, enabling it to control one or more vehicle functions, including, for example, the central locking system, air conditioning, electric mirrors, the vehicle's primary drive (e.g., engine, primary drive system), and / or various other vehicle modules. The BCM 24 can receive data from the wireless communication device 30 and subsequently transmit the data to one or more vehicle modules.

[0041] Furthermore, the BCM 24 can provide vehicle status information relating to the vehicle's condition or specific vehicle components or systems. For example, the BCM can provide device 30 with information indicating whether the vehicle's ignition is switched on, the gear the vehicle is currently in (i.e., in gear state), and / or other vehicle-related information. The BCM 24 can retrieve information from one or more other vehicle modules to obtain this information.

[0042] With reference to the Fig. 2 and Fig. Figure 3 shows the interior 100 of a vehicle 12 from the perspective of a passenger in the rear seat row looking forward through the windshield 102 with an AR / VR device, such as a smartphone 14 or AR / VR glasses 16. Fig. 2 is a perspective view through the AR / VR device without the application of a virtual vehicle surface and Fig. Figure 3 is a perspective view through the AR / VR device using a virtual vehicle surface. The vehicle interior 100 includes the windshield 102, the steering wheel 104, the center console or dashboard 106, which holds the touchscreen display 58 and the wireless communication device 30, a touchscreen user interface 108 displayed on the touchscreen 58, a gearshift lever 112, the seats 120 and 130, and the A-pillars 140 and 142. The wireless communication device 30 communicates with the AR / VR devices inside or near the vehicle interior 100, as represented by the wireless signals 144 and 146.

[0043] As in the illustrated embodiment of Fig. As shown in Figure 3, a virtual vehicle surface can be displayed over the interior 100 of the vehicle 12, such as on specific vehicle components. For example, the virtual vehicle surface 148 comprises numerous virtual vehicle surface components 150-160. The virtual vehicle surface includes virtual trim surface graphics, shown at 154 and 160, as well as virtual seat surface graphics 156 and 158. In addition, virtual A-pillar surface graphics are shown at 150 and 152, respectively, which are displayed over the A-pillars 140 and 142. The virtual vehicle surface components can be configured to correspond to a specific vehicle component, such as the seats 120, 130, the pillars 140, 142, and / or other vehicle trim or components.The virtual vehicle surface components can be individually rendered and individually resized, repositioned, transformed, and / or modified based on the movement of the AR / VR device, as is recognized by experts. Furthermore, functional features of the vehicle, such as the cup holders 114, the display 58, and the gearshift lever 112, are not covered by the virtual vehicle surface to allow continued use and awareness of these functional vehicle components. In further embodiments, the method (as explained below) may involve determining which vehicle functional components are beyond the reach of the AR / VR device user, in which case these functional components may still be covered by the vehicle surface 148, since the AR / VR device user cannot reach them to access these components anyway.And although a single, checkerboard-like virtual vehicle surface is displayed, various designs, themes, scenes, colors, patterns and other visual effects can be used or integrated into the virtual vehicle surface.

[0044] With reference to Fig. Section 4 describes a method 300 for transmitting a notification that virtual vehicle surfaces are available for use with augmented reality / virtual reality (AR / VR) devices. Method 300 can be executed by one or more remote servers, such as a remote server 82 at the remote device 80. According to many embodiments, Method 300 enables a user to specify one or more virtual vehicle surfaces that can be used to augment graphics over the user's vehicle when viewed by a user with an AR / VR device. For example, a vehicle owner or operator can configure a client device (e.g., a VR headset) to display graphics over the user's vehicle.Computer 78, AR / VR Device 14, 16) to select one or more virtual vehicle surfaces that are aesthetically pleasing to him, such as a virtual vehicle surface that is treated as a popular superhero, such as Batman™ or Superman™. Other virtual vehicle surfaces can be developed to enhance the imagery of a vehicle, such as a jungle theme, a space theme, or a sports theme.

[0045] Prior to Procedure 300, a user with a computer, which may include an AR / VR device, can access a virtual vehicle surface catalog (i.e., a list of virtual vehicle surfaces). The user can browse different surfaces and enter vehicle information regarding the vehicle type or identity. As mentioned earlier, virtual vehicle surfaces can be positioned or designed to match a vehicle layout, so the variety of virtual vehicle surfaces can vary depending on the vehicle model or model year. Vehicle information about the type or identity of the vehicle can be entered into the virtual vehicle surface catalog, after which the virtual vehicle surfaces can be filtered based on the entered vehicle information.In one embodiment, the virtual vehicle surfaces can be purchased from the virtual vehicle surface catalog, and upon purchase, the virtual vehicle surfaces can be downloaded into the vehicle, an AR / VR device, or another device.

[0046] Procedure 300 begins with step 310, in which a request to assign a virtual vehicle interface is received. The request to assign a virtual vehicle interface can be a message indicating that a user wishes to assign a specific virtual vehicle interface to their vehicle or account. This request can be generated and / or sent by the client device with which the user views the virtual vehicle interface catalog. In one embodiment, as soon as a user acquires a virtual vehicle interface (or otherwise chooses to have a virtual vehicle interface assigned to the user's vehicle), the request to assign the virtual vehicle interface can be generated and sent to the remote server.

[0047] In one embodiment, the request to assign a virtual vehicle interface may include an identifier for the user's vehicle or other vehicle information that can be used to obtain a list of suitable virtual vehicle interfaces for the user's vehicle. In another embodiment, the request to assign a virtual vehicle interface may include a vehicle identification number (VIN), which can then be used to obtain specific information relating to the vehicle, including the vehicle model year and certain optional features integrated into the vehicle. The remote server can then use this specific vehicle information to determine whether the selected virtual vehicle interface is suitable for use with the vehicle (see step 320).

[0048] In further embodiments, a vehicle identifier (e.g., VIN, model year) can be sent to the remote server, and in response, the remote server can send information about various virtual vehicle interfaces to the client device (e.g., computer 78). This information can then be presented to the user, who can select a specific virtual vehicle interface. After selecting a specific virtual vehicle interface, the client device can generate a request to assign a virtual vehicle interface and send it to the remote server. In such an embodiment, the request to assign a virtual vehicle interface need not include any specific information about the user's vehicle 12, since this information has already been sent to the remote server.In further embodiments, the virtual vehicle interface can be assigned to a user's account and also to the account of a user with one or more vehicles. Method 300 can then proceed to step 320.

[0049] Step 320 determines whether a specific virtual vehicle skin is suitable or configured for use with the user's vehicle. In some embodiments, this step can be performed before receiving the request to assign the virtual vehicle skin. For example, the remote server may already possess information about a particular vehicle before receiving the request to assign the virtual vehicle skin, allowing the remote server to check at that time which skins are suitable for use with vehicle 12. In other embodiments, this determination can be performed after receiving the request to assign the virtual vehicle skin. After determining that the requested virtual vehicle skin is suitable or configured for use with the vehicle, procedure 300 proceeds to step 330.

[0050] In step 330, data records can be updated to reflect an association between the vehicle and the virtual vehicle surface. As mentioned above, databases can be maintained that store information relating to vehicles and virtual vehicle surfaces. These databases can be updated to reflect that a particular virtual vehicle surface is associated with a particular vehicle. In one embodiment, the databases 84 can include a virtual vehicle surface database containing information relating to the virtual vehicle surfaces as well as vehicle surface association information indicating that a particular vehicle or user is authorized to display a virtual vehicle surface from a captured image or video of the vehicle when using an AR / VR device.In further embodiments, this step can include assigning the requested virtual vehicle surfaces to a user account that is also assigned to specific vehicles. Method 300 continues with step 340.

[0051] In step 340, virtual vehicle surface information can be sent to the vehicle via the virtual vehicle surface specified in the virtual vehicle assignment request. In one embodiment, the virtual vehicle surface information can be sent via the fixed network 76 and the mobile carrier system 70. The vehicle can receive the virtual vehicle surface information using a wireless communication device 30, which may include a mobile communication chipset 34. In one embodiment, the virtual vehicle surface information can be sent to the vehicle immediately after the purchase of the virtual vehicle surface and / or its assignment to the respective vehicle.Alternatively, in other embodiments, the virtual vehicle surface information can be sent to the vehicle 12 when a request is received from the vehicle 12, such as a virtual vehicle surface request (see step 420 of procedure 400 (. Fig. 5)).

[0052] In one embodiment, the virtual vehicle surface information can include an identifier for the virtual vehicle surface. This identifier can then be used by the vehicle to request the virtual vehicle surface from a remote database, such as database 84. In other embodiments, the virtual vehicle surface information can include data representing the virtual vehicle surface, which may contain various graphics and other information, such as position information or other information that can assist an AR / VR device in correctly displaying the virtual vehicle surface over the vehicle.

[0053] After the virtual vehicle surface information is received, it can be stored in a storage device of the vehicle. In one embodiment, the virtual vehicle surface information can be stored in the memory 38 of the wireless communication device 30, or the vehicle can use other storage devices to store this information. Furthermore, the vehicle can use the virtual vehicle surface information to send or transmit virtual vehicle surface availability messages intended for receipt by one or more AR / VR devices. The virtual vehicle surface availability messages can indicate that one or more specific virtual vehicle surfaces are available for display in an augmented reality view of the vehicle.This transmission step, which is performed by the vehicle, is explained in more detail below, in particular in step 410 of procedure 400 (. Fig. 5).

[0054] With reference to ( Fig. Figure 5 describes an embodiment of Method 400 for transmitting virtual vehicle surface information to an AR / VR device. In some embodiments, Method 400 can be performed by the vehicle 12 using the wireless communication device 30. Method 400 begins with step 410, in which a virtual vehicle surface availability message is transmitted. The virtual vehicle surface availability message can be a wireless message indicating that virtual vehicle surfaces are available for download and / or display over the captured image or video of the vehicle using an AR / VR device. This indication can include specifying a service identifier (service ID) in the virtual vehicle surface availability message, indicating that a virtual vehicle surface service is available for use by the AR / VR device.Furthermore, in other embodiments, the virtual vehicle surface availability message can include information about specific virtual vehicle surfaces supported for use by an AR / VR device on the vehicle. For example, an AR / VR device can receive a virtual vehicle surface availability message from the vehicle and, in response, ask a user for feedback on whether the user wants to display a virtual vehicle surface over the vehicle. The user can then respond using one or more controls on their AR / VR device, as shown below in step 520 (. Fig. 6) will be explained in more detail.

[0055] In one embodiment, the availability message for the virtual vehicle surface can be transmitted by the vehicle at regular intervals using the advertising message described above. Alternatively, the message can be transmitted in response to a user who configures the vehicle to send a notification that virtual vehicle surfaces are available for download and / or display via a captured image or video of the vehicle with an AR / VR device. Method 400 continues with step 420.

[0056] In step 420, a virtual vehicle surface request is received at the vehicle. The virtual vehicle surface request can be sent by an AR / VR device, such as a smartphone 14 or AR / VR glasses 16. The virtual vehicle surface request can be sent via SRWC and / or via a wired connection, such as a Universal Serial Bus (USB) port. In one embodiment, an SRWC connection can be established after the availability message for the virtual vehicle surface has been received from the AR / VR device, and after the SRWC connection is established, the AR / VR device can send the request for the virtual vehicle surface to the vehicle via the established SRWC connection.

[0057] In the case of establishing a connection via a wired connection, a user can connect a cable, such as a USB cable, to a corresponding port in the vehicle and in the AR / VR device. In further embodiments where an SRWC connection is established, the vehicle can transmit a light signal or advertisement via the SRWC circuit 32. The advertisement can be the same message as the virtual vehicle surface availability message described in step 410, or it can be a separate message. The connection advertisement message can include data indicating that the vehicle intends to establish or can establish a connection to an AR / VR device, or that the vehicle can offer virtual vehicle surface services (i.e.,Virtual vehicle skin(s) are assigned to the vehicle, are available for download, and / or can be displayed via a captured image or video of the vehicle using an AR / VR device. This message can then be received by the AR / VR device, which can then generate a response, such as a message initiating a four-way handshake or other authentication and / or shared key / secrecy process. In further embodiments, the AR / VR device can send an advertising or light signal message indicating to those SRWC devices within range that the AR / VR device is searching for devices capable of providing virtual vehicle surface services. Once the connection is established, the AR / VR device can send the virtual vehicle surface request over the established connection.

[0058] In one embodiment, the virtual vehicle surface request can specify a particular virtual vehicle surface (the "requested virtual vehicle surface") that the AR / VR device wants to display to the user. In such an embodiment, the requested virtual vehicle surface can be specified by inserting a virtual vehicle surface identifier, which can then be used by the vehicle 12 or the remote server 82 to obtain the corresponding virtual vehicle surface.

[0059] In further embodiments, the virtual vehicle surface request can be a general virtual vehicle surface request and may not specify any particular virtual vehicle surfaces, but may indicate that the AR / VR device is interested, or may be interested, in receiving a virtual vehicle surface. In one embodiment, after receiving the general virtual vehicle surface request, the SRWC connection (as explained above) can be established. The vehicle can then send more specific information to the AR / VR device, such as the specific virtual vehicle surfaces the vehicle has to offer, which may include one or more virtual vehicle surface identifiers and / or representative graphics (i.e., a preview of the virtual vehicle surface).A user can then select a specific virtual vehicle surface using their AR / VR device. Furthermore, in some embodiments, the virtual vehicle surface request can also include location and position information of the AR / VR device, which can be used to establish a position of the AR / VR device 14, 16 within the vehicle interior 100 and / or an orientation of the AR / VR device. In one embodiment, the AR / VR device 14, 16 can use an accelerometer to determine an orientation of the AR / VR device. Method 400 continues to step 430.

[0060] In step 430, the vehicle receives the virtual vehicle skin selected by the user. If only a single virtual vehicle skin is available for use with the vehicle 12, it should be noted that if the user sends a general virtual vehicle skin request, the single virtual vehicle skin has been selected. In one embodiment, the vehicle may already contain virtual vehicle skin data stored in the memory representing the virtual vehicle skin, so that the vehicle can simply retrieve this data from memory, such as memory 38. In other embodiments, the vehicle may contain virtual vehicle skin information, such as that specified in step 340 ( Fig. 4) are received, but not the virtual vehicle interface itself. In this case, the vehicle can then query one or more remote servers, such as remote server 82, for the virtual vehicle interface. After receiving the virtual vehicle interface, the vehicle can store it in a memory location, such as memory location 38, for later use, for example, when receiving another virtual vehicle interface request.

[0061] Furthermore, after receiving the virtual vehicle surface, the vehicle can process the surface or various components of the surface based on the position and positional information of the AR / VR device. For example, the view to the AR / VR device may depend on the location and position of the AR / VR device, so the virtual vehicle surface can be modified before being sent to the AR / VR device, taking these attributes into account. Although the AR / VR device can handle most of the AR / VR processing, the vehicle can support the AR / VR device by providing only those components of the virtual vehicle surface that are visible to a user at the AR / VR device's location. For example, graphical components 156 and 158 of the back of seats 120 and 130 would most likely not be needed by a user sitting in seat 120 or 130.

[0062] In one embodiment, the user's position can be obtained by using various VSMs, such as a wireless communication device 30 and / or a vehicle occupancy detector. The wireless communication device 30 can receive information, such as information from a user indicating a seat in the vehicle where the user of the AR / VR device is located. Alternatively, the vehicle occupancy detector, coupled to a vehicle seat cushion, can determine that only a single user is in the vehicle besides the driver, allowing the vehicle to determine the position of the single user and the AR / VR device. In other embodiments, multiple antennas (e.g., directional antennas) can be used throughout the vehicle to obtain angle of incidence and / or angle of emission information, as well as signal strength information (e.g., relative received signal strength (RSSI)).Procedure 400 continues with step 440.

[0063] In step 440, the virtual vehicle surface is sent to the AR / VR device. As experts in the field understand, the virtual vehicle surface can be digitally represented as a series of bits that can be used to represent the graphics, metadata, and other information of the virtual vehicle surface. In one embodiment, the virtual vehicle surface can be sent from the vehicle to the AR / VR device via an established SRWC connection, such as the one described above. The data representing the virtual vehicle surface can be larger than a packet size used in the SRWC connection, in which case the virtual vehicle surface can be transmitted to the AR / VR device in multiple packets. After receiving the packets containing the virtual vehicle surface, the AR / VR device can then reconstruct the virtual vehicle surface.

[0064] In another embodiment, the vehicle can send virtual vehicle surface access information to the AR / VR device. This virtual vehicle surface access information can include information that allows the AR / VR device to obtain or derive the virtual vehicle surface. In one embodiment, this information can include a vehicle identifier (e.g., VIN), a virtual vehicle surface token, and / or a universal resource localizer (URL) (or universal resource identifier (URI)) that can be used to download the virtual vehicle surface from one or more remote servers, such as Remote Server 82. For example, the virtual vehicle surface token can be generated at the time of purchase or assignment of a particular virtual vehicle surface to a specific vehicle or user account (see step 330 in [reference]). Fig. 4), or the virtual vehicle surface token can be generated on the remote server upon receiving a virtual vehicle surface request. The virtual vehicle surface token can be sent to a remote server along with a vehicle identifier (e.g., VIN), which can then verify the token and the vehicle identifier. In response to the verification, the remote server can respond with the virtual vehicle surface. In some embodiments, the token and the vehicle identifier can be included in the URL or URI sent to the AR / VR device. Once the virtual vehicle surface has been obtained by the AR / VR device, Method 400 terminates.

[0065] With reference to Fig. 6 describes a method 500 for displaying virtual vehicle surfaces over a corresponding area of ​​a vehicle using an AR / VR device. The method 500 can be performed by an AR / VR device, such as a smartphone 14 and / or AR / VR glasses 16. The AR / VR device can include one or more digital cameras configured to capture images or videos of an area around the user, such as an area in front of the user. For example, if the AR / VR device is worn by the user, a camera located on the AR / VR device can be directed at an area in front of the user. The AR / VR device can then capture images or videos of this area and, additionally, if the vehicle or a component of the vehicle is within the field of view of the AR / VR device's camera (i.e.,Within the field of vision of the "AR / VR device," the AR / VR device can display the virtual vehicle surface over the vehicle or vehicle component. Furthermore, in other embodiments, the AR / VR device can receive image or video data captured from other devices or receive and / or display virtual background graphics, so that the user experiences a complete virtual reality (VR) visual effect.

[0066] Method 500 can begin with step 510, in which an indication that the vehicle supports virtual vehicle surfaces is received at the AR / VR device. In one embodiment, this indication can be an availability message for the virtual vehicle surface, as explained above (step 410 of Fig. 5) As mentioned above, this may include one or more virtual vehicle surface identifiers, a service ID, or other information that can be used to indicate that virtual vehicle surfaces are available for use with an AR / VR device, particularly for use when overlaying graphics onto an area of ​​the vehicle within an AR view. Upon receiving this message, Procedure 500 continues with step 520.

[0067] In step 520, a virtual vehicle surface request is sent from the AR / VR device to the vehicle. This request was previously explained with reference to step 420 of procedure 400 ( Fig. 5) As mentioned above, the request can include various pieces of information, such as the virtual vehicle surface identifier, which identifies which virtual vehicle surface the AR / VR device wants to access. In further embodiments, a vehicle user can configure their vehicle to activate / deactivate specific virtual vehicle surfaces, which may also include activating a single virtual vehicle surface at a specific time. Thus, in this case, the virtual vehicle surface request may not specify a particular virtual vehicle surface, but rather request the one that is currently capable of being used with AR / VR devices to augment the vehicle or vehicle components. Method 500 then proceeds to step 530.

[0068] In step 530, the virtual vehicle surface is received at the AR / VR device. The virtual vehicle surface can be embodied in various data files, which can be sent to the vehicle with one or more messages. After receiving the files, the AR / VR device can then assemble them to obtain the virtual vehicle surface. In one embodiment, the vehicle can store the virtual vehicle surface in a storage device included in the AR / VR device and can also send an acknowledgment message for the virtual vehicle surface to the vehicle (or to the remote server) confirming that the virtual vehicle surface has been received.

[0069] As mentioned above, the virtual vehicle surface can be received either by the vehicle or from a remote server. In the latter case, the AR / VR device can present a token, which the AR / VR device can receive from the vehicle, to the remote server, and in response, the remote server can send the virtual vehicle surface to the AR / VR device via the fixed network 76 and / or the mobile carrier system 70. In a particular embodiment, in response to receiving the virtual vehicle surface request, the vehicle can retrieve the requested (or specified) virtual vehicle surface from a remote server, and thereafter, the vehicle can forward the virtual vehicle surface to the AR / VR device via an SRWC connection or a wired connection. Method 500 then proceeds to step 540.

[0070] In step 540, the AR / VR device presents the virtual vehicle surface on its display. In many embodiments, the virtual vehicle surface is represented by images or videos captured by the AR / VR device and displayed within an area of ​​the captured images or videos where the vehicle or vehicle components are located. The AR / VR device can utilize various augmented reality techniques to accurately display the virtual vehicle surface on the screen. In one embodiment, the virtual vehicle surface can include numerous graphics. Each of these graphics can be associated with a specific component or area of ​​the vehicle.

[0071] In one embodiment, the virtual vehicle surface can include a multitude of virtual vehicle surface components, each corresponding to a specific area or component of the vehicle. For example, with reference to Fig.Three different vehicle components are depicted, including seats 120, 130, A-pillars 140, 142, and various trim components around the center console 106 and the gearshift lever 112. Virtual vehicle surface components 150-160 can be displayed over these components, and in some embodiments, the AR / VR device can use imaging and / or object recognition techniques, as well as other augmented reality (AR) techniques, to identify a vehicle component along with one or more attributes of the vehicle component or the AR / VR device's field of view. These virtual vehicle surface components 150-160 can then be presented based on this information. For example, the AR / VR device can identify the position and size of the driver's seat 120, as well as the AR / VR device's viewing angle.Subsequently, the AR / VR device can use this information to determine the position and size of the virtual vehicle component 156, which may be equal to or at least based on the position and size of the seat 120. The AR / VR device can then reproduce these graphics 156 in this specific area. Furthermore, the viewing angle of the AR / VR device can also be used as a basis for transforming and / or distorting the virtual vehicle component 156 so that it appears natural to the user.

[0072] In one embodiment, the AR / VR device can use an accelerometer and wireless communication with the vehicle to determine its position, orientation, and / or viewing angle within the vehicle. Additionally, or alternatively, the AR / VR device can use image processing techniques to detect specific objects within its camera's field of view, thereby confirming its position and / or orientation within the vehicle. For example, the vehicle interior may contain specific attributes or components (i.e., "virtual vehicle surface markers") that are easily identifiable using image processing techniques.Information about these virtual vehicle surface markers can be incorporated into the virtual vehicle surface data and serves to tell the AR / VR device where to display the various virtual vehicle surface components. Based on the component information integrated as part of the virtual vehicle surface, the AR / VR device can position the various virtual vehicle surface components within the display so that the AR components correspond to the components inside the vehicle. For example, the AR / VR device can determine a position within the video output (which is to be displayed on the AR / VR device's display) where a specific component should be located, and the AR / VR device can then include the AR component at that specific position within the video output.The video output can include real-world videos or images captured by a camera on the AR / VR device, as well as any virtual vehicle surface components that can be displayed on it. Furthermore, if the AR / VR device is repositioned or reoriented, it can update the video output to reflect an augmented reality (AR) output that corresponds to the new field of view of the AR / VR device.

[0073] In further embodiments, the virtual vehicle surface can include virtual exterior environment graphics, which add additional visual effects to areas outside the vehicle that are normally visible through a window. For example, the vehicle can include virtual exterior environment graphics as part of the virtual vehicle surface, whereupon the AR / VR device can: determine exterior areas within the AR / VR device's field of view; and then display the virtual exterior environment graphics within these areas, so that the user of the AR / VR device experiences a visual effect of being in a different or modified environment. For example, the virtual exterior environment graphics can depict outer space, giving the impression that the vehicle is driving in space. In one embodiment, the areas outside the vehicle can be entirely virtual graphics, i.e.,None of the captured images or videos of these outdoor areas are saved for viewing on the AR / VR device display. Procedure 500 then ends.

[0074] In one embodiment, Method 300, Method 400, and / or Method 500 (or parts of these various methods) can be implemented in a computer program (or an "application") embodied in a computer-readable medium and containing instructions that can be used by one or more processors of one or more computers of one or more systems. A computer program can include one or more software programs consisting of program instructions in source code, object code, executable code, or another format; one or more firmware programs; or files of a hardware description language (HDL); and other program-related data. The data can include data structures, value tables, or data in another suitable format. The program instructions can include program modules, routines, programs, objects, components, and / or the like.The computer program can be executed by one or more computers in conjunction with each other.

[0075] The program(s) may be executed on computer-readable media (such as memory that is part of the Remote Server 82 (Procedure 300), memory 38 (Procedure 400), memory on the AR / VR device (Procedure 500)), which may be non-volatile and may include one or more storage devices, manufacturing products, or the like. Examples of computer-readable media include computer system memory, e.g., RAM (random access memory), ROM (read-only memory); semiconductor memory, e.g., EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), flash memory; magnetic or optical disks or tapes; and / or the like.A computer-readable medium can also include computer-to-computer connections, for example, when data is transmitted or provided via a network or other communication link (wired, wireless, or a combination of both). All combinations of the foregoing examples also fall within the scope of computer-readable media. It is therefore understood that the method can be carried out, at least in part, by electronic articles and / or devices capable of executing instructions according to one or more steps of the disclosed method.

[0076] It is understood that the foregoing is a description of one or more embodiments of the invention. The invention is not limited to the particular embodiment(s) disclosed herein, but is defined exclusively by the following claims. Furthermore, the statements made in the foregoing description refer to specific embodiments and are not to be understood as limitations on the scope of the invention or the definition of the terms used in the claims, except where a term or expression has been expressly defined above. Various other embodiments and various changes and modifications to the disclosed embodiment(s) are obvious to those skilled in the art. All such other embodiments, changes, and modifications should be understood to fall within the scope of the appended claims.

[0077] As used in this specification and the claims, the terms "e.g.," "for example," "such as," and "as," and the verbs "comprising," "including," "possessing," and their other verb forms, when used in conjunction with a list of one or more components or other elements, are each to be interpreted as open, meaning that the list is not to be considered as excluding other additional components or elements. Other terms are to be interpreted in their broadest reasonable sense unless used in a context that requires a different interpretation. Additionally, the expression "and / or" is understood as an inclusive OR. Thus, the expression "A, B, and / or C" is to be understood as covering the following possibilities: "A"; "B"; "C"; "A and B"; "A and C"; "B and C"; and "A, B, and C."

Claims

[1] Method for configuring an augmented reality / virtual reality (AR / VR) device (14, 16) for augmenting image or video data using a virtual vehicle surface, wherein the method is performed by vehicle electronics (20) integrated in a vehicle (12), the method comprising: Establishing a connection to the AR / VR device (14, 16) using a wireless communication device (30) integrated into the vehicle electronics (20) of the vehicle (12); Receiving an initial virtual vehicle surface request from the AR / VR device (14, 16); Sending a second virtual vehicle surface request to one or more remote servers (82) in response to receiving the first virtual vehicle surface request; Receiving virtual vehicle surface access information; and Sending a virtual vehicle surface response to the AR / VR device (14, 16) via the established connection, wherein the AR / VR device (14, 16) is configured to receive image or video data and to display the virtual vehicle surface over the received image or video data, such that the virtual vehicle surface overlays a portion of the vehicle (12), and wherein the virtual vehicle surface response includes the virtual vehicle surface access information that can be used by the AR / VR device (14, 16) to receive the virtual vehicle surface from the remote server (82); wherein the virtual vehicle surface access information includes a virtual vehicle surface token which, when presented to the remote server (82), causes the remote server (82) to send the virtual vehicle surface to the AR / VR device (14, 16). [2] Method according to claim 1, wherein the virtual vehicle surface is configured for insertion into captured image or video data such that the virtual vehicle surface covers a section of an interior cabin of the vehicle (12). [3] Method according to claim 1, wherein the AR / VR device (14, 16) is a wearable electronic augmented reality / virtual reality (AR / VR) glasses (16) configured to visually present augmented reality or virtual reality media content to a user. [4] Method according to claim 1, wherein the AR / VR device (14, 16) is a personal AR / VR device.

Images