Synchronized rendering

Abstract

Current techniques for rendering content using data on multiple devices are typically invalid and / or inefficient. The present disclosure provides more efficient and / or efficient techniques for rendering such content. These techniques optionally complement or replace other methods for rendering content.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 349,063, “SYNCHRONIZED RENDERING,” filed on June 4, 2022. The contents of this application are hereby incorporated by reference in their entirety for all purposes. Background Art

[0003] Today, rendering content is often performed by a single device and then that device or another device displays whatever content is rendered. This architecture leverages the processing power of the device to provide a curated experience. However, the data required for rendering is not always on a single device and ensuring this can be inefficient. Therefore, there is a need for improved rendering techniques for systems with multiple devices. Summary of the invention

[0004] Current techniques for rendering content using data on multiple devices are often ineffective and / or inefficient. The present disclosure provides more effective and / or efficient techniques for rendering such content. These techniques optionally supplement or replace other methods for rendering content. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] For a better understanding of the various described examples, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0006] Figure 1 is a block diagram of an exemplary computing system.

[0007] Figure 2 is a block diagram illustrating a device having interconnected subsystems.

[0008] Figure 3 is a block diagram illustrating a carrier connected to a user device via a transmission.

[0009] 4A to 4H is a block diagram illustrating what is being displayed on a vehicle's display.

[0010] FIG. 5A to FIG. 5H is a flow chart illustrating different operations prior to operations performed by a vehicle and a user device.

[0011] Figure 6 is a flow chart illustrating a method for establishing a layout for synchronous rendering on multiple devices.

[0012] Figure 7 is a flow chart illustrating a method for time-based rendering synchronization.

[0013] Figure 8 is a flow chart illustrating a method for controlling rendering by another device.

[0014] Fig. 9 is a flow chart illustrating a method for rendering animations across multiple devices.

[0015] Fig.10 is a flow chart illustrating a method for customizing vehicle controls when connected to a user device.

[0016] Fig.11 is a flow chart illustrating a method for changing a layout used during synchronous rendering in the event of a connection loss. DETAILED DESCRIPTION

[0017] The following description sets forth exemplary methods, parameters, etc. However, it should be recognized that such description is not intended as a limitation on the scope of the present disclosure, but is provided as a description of examples.

[0018] In the method described herein where one or more steps depend on having met one or more conditions, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions of the steps depending on have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the skilled person will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on having met one or more conditions can be rewritten as a method for repeating until each condition described in the method is met. However, this does not require a system or computer-readable medium to declare that the system or computer-readable medium includes instructions for performing operations depending on the circumstances based on the satisfaction of one or more corresponding conditions, and therefore it is possible to determine whether possible situations have been met without explicitly repeating the steps of the method until all conditions of the steps in the decision method have been met. It will also be understood by those of ordinary skill in the art that, similar to the method with the steps depending on the circumstances, the system or computer-readable storage medium can repeat the steps of the method as needed for multiple times to ensure that all the steps depending on the circumstances have been performed.

[0019] Although the following description uses the terms "first", "second", etc. to describe various elements, these elements should not be limited by the terms. In some examples, these terms are used to distinguish various elements. For example, without departing from the scope of various described examples, the first device may be referred to as the second device, and similarly, the second device may be referred to as the first device. In some examples, the first device and the second device are two separate references to the same device. In some examples, both the first device and the second device are devices, but they are not the same device or the same type of device.

[0020] The terms used in the description of various examples described herein are only for the purpose of describing specific examples, and are not intended to be limiting. As used in the description of various examples described and in the appended claims, the singular forms "one", "a kind of" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "include" and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their grouping.

[0021] The term "if" is optionally understood to mean "when," "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]," or "in response to detecting [a stated condition or event]," depending on the context.

[0022] Now turn to Figure 1 , depicts a block diagram of a computing system 100. Computing system 100 is a non-limiting example of a computing system that can be used to perform the functionality described herein. It should be appreciated that other computer architectures of computing systems can be used to perform the functionality described herein.

[0023] In the illustrated example, computing system 100 includes a processor subsystem 110 coupled (e.g., wired or wirelessly) to memory 120 (e.g., system memory) and I / O interface 130 via interconnect 150 (e.g., a system bus, one or more memory locations, or other communication channels for connecting multiple components of computing system 100). In addition, I / O interface 130 is coupled (e.g., wired or wirelessly) to I / O device 140. In some examples, I / O interface 130 is included with I / O device 140 so that the two are a single component. It should be appreciated that there may be one or more I / O interfaces, each of which is coupled to one or more I / O devices. In some examples, multiple instances of processor subsystem 110 may be coupled to interconnect 150.

[0024] The computing system 100 can be any of various types of devices, including but not limited to a system on a chip, a server system, a personal computer system (e.g., an iPhone, iPad, or MacBook), a sensor, etc. In some examples, the computing system 100 is included with a physical component or coupled to a physical component for the purpose of modifying the physical component in response to an instruction (e.g., the computing system 100 receives an instruction to modify the physical component and causes the physical component to be modified in response to the instruction (e.g., through an actuator)). Examples of such physical components include acceleration controls, brakes, gearboxes, motors, pumps, refrigeration systems, suspension systems, steering controls, vacuum systems, and valves. As used herein, a sensor includes one or more hardware components that detect information about a physical environment near (e.g., surrounding) the sensor. In some examples, the hardware components of the sensor include a sensing component (e.g., an image sensor or a temperature sensor), a transmitting component (e.g., a laser or radio transmitter), a receiving component (e.g., a laser or radio receiver), or any combination thereof. Examples of sensors include angle sensors, chemical sensors, brake pressure sensors, contact sensors, contactless sensors, electrical sensors, flow sensors, force sensors, gas sensors, humidity sensors, image sensors (e.g., cameras), inertial measurement units, leak sensors, level sensors, light detection and ranging systems, metal sensors, motion sensors, particle sensors, photoelectric sensors, position sensors (e.g., global positioning systems), precipitation sensors, pressure sensors, proximity sensors, radio detection and ranging systems, radiation sensors, speed sensors (e.g., measuring the speed of an object), temperature sensors, time of flight sensors, torque sensors, and ultrasonic sensors. Although in Figure 1 Although a single computing system is shown in FIG. 1 , computing system 100 may also be implemented as two or more computing systems operating together.

[0025] In some examples, processor subsystem 110 includes one or more processors or processing units configured to execute program instructions to perform the functionality described herein. For example, processor subsystem 110 may execute an operating system, a middleware system, one or more applications, or any combination thereof.

[0026] In some examples, the operating system manages the resources of the computing system 100. Examples of the types of operating systems covered herein include batch operating systems (e.g., multiple virtual storage (MVS)), time-sharing operating systems (e.g., Unix), distributed operating systems (e.g., Advanced Interactive Executive (AIX)), network operating systems (e.g., Microsoft Windows Server), and real-time operating systems (e.g., QNX). In some examples, the operating system includes various programs, sets of instructions, software components, and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and for facilitating communication between various hardware and software components. In some examples, the operating system uses a priority-based scheduler that assigns priorities to different tasks to be executed by the processor subsystem 110. In such examples, the priority assigned to a task is used to identify the next task to be executed. In some examples, the priority-based scheduler identifies the next task to be executed when the previous task completes execution (e.g., the highest priority task runs to completion unless another higher priority task is ready).

[0027] In some examples, the middleware system provides one or more services and / or capabilities to applications (e.g., one or more applications running on the processor subsystem 110) outside of applications provided by the operating system (e.g., data management, application services, messaging, authentication, API management, etc.). In some examples, the middleware system is designed for heterogeneous computer clusters to provide hardware abstraction, low-level device control, implementation of common functionality, messaging between processes, package management, or any combination thereof. Examples of middleware systems include lightweight communication and marshaling (LCM), PX4, Robot Operating System (ROS), and ZeroMQ. In some examples, the middleware system uses a graphical architecture to represent processes and / or operations, where processing occurs in nodes that can receive, publish, and reuse sensor data, controls, states, plans, actuators, and other messages. In such examples, applications (e.g., applications executed on the processor subsystem 110 as described above) can be defined using a graphical architecture so that different operations of the application are included with different nodes in the graphical architecture.

[0028] In some examples, a publish-subscribe model is used to perform message transmission from a first node in a graphics architecture to a second node in the graphics architecture, where the first node publishes data on a channel to which the second node can subscribe. In such examples, the first node may store data in a memory (e.g., memory 120 or a local memory of the processor subsystem 110) and notify the second node that the data has been stored in the memory. In some examples, the first node notifies the second node that the data has been stored in the memory by transmitting a pointer (e.g., a memory pointer, such as an identifier of a memory location) to the second node, so that the second node can access the data from the location where the first node stores the data. In some examples, the first node will transmit the data directly to the second node, so that the second node will not need to access the memory based on the data received from the first node.

[0029] The memory 120 may include a computer-readable medium (e.g., a non-transitory or transitory computer-readable medium) that may be used to store program instructions that may be executed by the processor subsystem 110 to cause the computing system 100 to perform various operations described herein. For example, the memory 120 may store program instructions to implement the 4A to 4H , FIG. 5A to FIG. 5H and Figures 6 to 11 Functionality associated with any or all of the described processes.

[0030] The memory 120 may be implemented using different physical non-transitory memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM—SRAM, EDO RAM, SDRAM, DDRSDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), etc. The memory in the computing system 100 is not limited to a main storage device, such as the memory 120. Instead, the computing system 100 may also include other forms of storage devices, such as cache memory in the processor subsystem 110 and auxiliary storage devices (e.g., hard disk drives, storage arrays, etc.) on the I / O devices 140. In some examples, these other forms of storage devices may also store program instructions that can be executed by the processor subsystem 110 to perform the operations described herein. In some examples, the processor subsystem 110 (or each processor within the processor subsystem 110) includes a cache or other form of on-board memory.

[0031] The I / O interface 130 may be any of a variety of types of interfaces configured to couple to and communicate with other devices. In some examples, the I / O interface 130 includes a bridge chip (e.g., a south bridge) from a front-end bus to one or more back-end buses. The I / O interface 130 may be coupled to one or more I / O devices (e.g., an I / O device 140) via one or more corresponding buses or other interfaces. Examples of I / O devices include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or a wide area network), sensor devices (e.g., cameras, radars, lidars, ultrasonic sensors, GPS, inertial measurement devices, etc.), and auditory or visual output devices (e.g., speakers, lights, screens, projectors, etc.). In some examples, the computer system 100 is coupled to a network via a network interface device (e.g., configured to communicate via Wi-Fi, Bluetooth, Ethernet, etc.).

[0032] Figure 2 A block diagram of a device 200 having interconnected subsystems is depicted. In the illustrated example, the device 200 includes three different subsystems (i.e., a first subsystem 210, a second subsystem 220, and a third subsystem 230) that are coupled to each other (e.g., wired or wirelessly). Figure 1 (ie, computing system 100) as described in Figure 2 Examples of possible computer architectures for the subsystems included in Figure 2 Three subsystems are shown in FIG. 2 , but device 200 may include more or fewer subsystems.

[0033] In some examples, some subsystems are not connected to another subsystem (e.g., the first subsystem 210 may be connected to the second subsystem 220 and the third subsystem 230, but the second subsystem 220 may not be connected to the third subsystem 230). In some examples, some subsystems are connected via one or more wires, while other subsystems are wirelessly connected. In some examples, one or more subsystems are wirelessly connected to one or more computing systems external to the device 200, such as a server system. In such examples, the subsystems may be configured to communicate wirelessly with one or more computing systems external to the device 200.

[0034] In some examples, device 200 includes a housing that fully or partially encloses subsystems 210 to 230. Examples of device 200 include household appliances (e.g., a refrigerator or air conditioning system), robots (e.g., a robotic arm or a robotic vacuum cleaner), and vehicles. In some examples, device 200 is configured to navigate device 200 in a physical environment (with or without direct user input).

[0035] In some examples, one or more subsystems of the device 200 are used to control, manage, and / or receive data from one or more other subsystems of the device 200 and / or one or more computing systems remote from the device 200. For example, the first subsystem 210 and the second subsystem 220 may each be a camera that is capturing images for use by the third subsystem 230 to make decisions. In some examples, at least a portion of the device 200 is used as a distributed computing system. For example, a task may be split into different parts, with the first part being performed by the first subsystem 210 and the second part being performed by the second subsystem 220.

[0036] Now focus is on techniques for rendering content using multiple devices. The example of a vehicle and a user device is used for discussion purposes, but it should be understood that other types of devices and more devices (i.e., 3 or more) are within the scope of this disclosure and can benefit from the techniques described herein. For example, two different user devices (rather than a vehicle and a user device) can be used with the techniques described herein.

[0037] Figure 3 302 is a block diagram illustrating a vehicle 302 connected to a user device 320 via a transmission 330. This configuration may allow for a unified experience, bringing together user interface elements from both the vehicle 302 and the user device 320. For example, the user device 320 may be able to facilitate an experience through the vehicle 302 and integrate with the vehicle.

[0038] like Figure 3 As depicted, vehicle 302 includes vehicle process 304, vehicle renderer 306, integration process 308, integrated renderer 310, output device 312, vehicle sensor 314, and virtual assistant subsystem 316. The components of vehicle 302 are intended for explanation purposes and are not intended to be limiting. Vehicle 302 may include more or fewer components, including combinations of the components depicted or other components described with respect to computing system 100 or device 200.

[0039] As mentioned above, the vehicle 302 includes a vehicle process 304. In some examples, the vehicle process 304 is a software program (e.g., one or more instructions executed by one or more processors) of the vehicle 302 that is configured to manage operations performed by the vehicle 302. In such examples, the vehicle process 304 can be isolated from one or more other processes (e.g., the integration process 308) of the vehicle 302, such that at least some of its associated memory can be accessed only by the vehicle process 304, and communications to and / or from the vehicle process 304 are through a structured process of an interface (e.g., an application programming interface (API)) defined for the vehicle process 304.

[0040] The vehicle 302 also includes a vehicle renderer 306. In some examples, the vehicle renderer 306 is any hardware or software of the vehicle 302 that is used to generate (sometimes referred to as render) visual content (e.g., images (sometimes referred to as frames) or videos) from a model and / or one or more instructions. In such examples, the vehicle renderer 306 may be configured to be used only by the vehicle process 304 to generate visual content from data detected and / or determined by the vehicle 302. In some examples, the vehicle renderer 306 is configured to render content associated with the ecosystem of the vehicle 302, such as content stored locally only by the vehicle 302. For example, the vehicle renderer 306 may render content associated with a first set of vehicle instruments (e.g., vehicle speed in a heads-up display). The first set of vehicle instruments may be those vehicle instruments that do not interact with content rendered away from the vehicle renderer 306 (e.g., away from the vehicle 302), such as content that is visually independent and always appears in a fixed location (e.g., turn signal indicators and check engine indicators). In some examples, vehicle renderer 306 renders content from a process executing on vehicle 302 , such as a driver assistance system of vehicle 302 (eg, video from a backup camera).

[0041] The vehicle 302 also includes an integration process 308. In some examples, the integration process 308 is a software program (e.g., one or more instructions executed by one or more processors) of the vehicle 302 that is configured to manage operations based on data received from a device (e.g., user device 320) that is separate from the vehicle 302. In such examples, the integration process 308 can be isolated from one or more other processes (e.g., vehicle process 304) of the vehicle 302, such that at least some of its associated memory is accessible only to the integration process 308, and communications to and / or from the integration process 308 are through a structured process of an interface (e.g., API) defined for the integration process 308.

[0042] The vehicle 302 also includes an integrated renderer 310. In some examples, the integrated renderer 310 is any hardware or software of the vehicle 302 that is used to generate (sometimes referred to as render) visual content (e.g., images or video) from a model and / or one or more instructions. In such examples, the integrated renderer 310 may be configured to be used by the integration process 308 to generate and / or combine visual content from: (1) data detected, determined, and / or generated by the vehicle 302 (e.g., the vehicle renderer 306 or the integrated renderer 310), (2) data detected, determined, and / or received from the user device 320 (e.g., the user device renderer 322), or (3) any combination thereof. In some examples, the integrated renderer 310 renders content associated with a second set of vehicle instruments that is different from a first set of vehicle instruments rendered by the vehicle renderer 306. The second set of vehicle instruments may be those vehicle instruments that interact with content rendered remotely from the vehicle renderer 306 (e.g., remotely from the vehicle 302), such as content that is visually integrated or closely associated with content rendered by the user device renderer 322 (e.g., a speedometer, gear position, or cruise control indicator in a primary display of the vehicle 302). In some examples, the integrated renderer 310 renders notifications received from a process executing on the vehicle 302 (e.g., the vehicle process 304), such notifications may be a first set (e.g., a first type) of notifications associated with the vehicle 302 (e.g., check control messages).

[0043] In some examples, the vehicle 302 includes a system for validating information included with content that is not rendered by the vehicle renderer 306 (e.g., content rendered by the integrated renderer 310 or the user device renderer 322) to ensure that the content to be displayed is correct. The system can compare one or more values ​​included in such content with data detected by sensors of the vehicle 302 (e.g., the vehicle sensors 314).

[0044] The vehicle 302 also includes an output device 312. In some examples, the output device 312 is any hardware or software of the vehicle 302 that is used to output (e.g., transmit, display, transmit, or generate) data (e.g., visual, auditory, or tactile) from the vehicle 302. Examples of output devices 312 include display screens, touch-sensitive surfaces, projectors, and speakers. In one example, the output device 312 is a display screen that displays content rendered by each of the vehicle renderer 306, the integrated renderer 310, and the user device renderer 322.

[0045] The vehicle 302 also includes a vehicle sensor 314. In some examples, the vehicle sensor 314 is any hardware or software of the vehicle 302 used to detect data about the physical environment near (e.g., surrounding) the vehicle sensor 314, similar to that discussed above with respect to the computing system 100. Examples of vehicle sensors 314 include knobs, steering wheel buttons, touch-sensitive surfaces, cameras, microphones, and any other sensors discussed with respect to the computing system 100. In some examples, the vehicle sensor 314 detects user input. In such examples, the user input detected by the vehicle sensor 314 is transmitted to the vehicle process 304 and / or the integration process 308, as discussed further below.

[0046] In some examples, when the user input corresponds to content rendered by the vehicle renderer 306 or is related to a process of the vehicle 302 (e.g., cruise control, driver assistance systems, or volume control), the user input may be communicated to the vehicle process 304. When the user input is communicated to the vehicle process 304, the vehicle process 304 may determine what the result of the user input is and indicate a change in display through the vehicle renderer 306 or the integration process 308. In some examples, when the user input is handled by the vehicle process 304, the user input may not be communicated to the integration process 308, but rather the vehicle process 304 notifies the integration process 308 of any state (e.g., display) changes resulting from detecting the user input.

[0047] In some examples, when the user input is related to content rendered by the integrated renderer 310 or the user device renderer 322 (e.g., voice recognition activation, dashboard user interface controls, media, and actions related to phone calls), the user input can be transmitted to the integration process 308. When the user input is transmitted to the integration process 308, the integration process 308 can transmit the user input to: (1) the user device 320 to determine how to respond to the user input, or (2) the vehicle process 304. In some examples, when the user input is transmitted to the integration process 308, the user input is not transmitted to the vehicle process 304 at all, and any state changes caused by the user input are not transmitted to the vehicle process 304.

[0048] Vehicle 302 also includes a virtual assistant subsystem 316 (sometimes referred to as an artificial intelligence assistant or digital assistant). In some examples, virtual assistant subsystem 316 is a software program that performs one or more operations based on data detected by a sensor (such as a natural language voice command detected by a microphone of vehicle 302 or user device 320). In such examples, vehicle 302 may include a software program (i.e., the software program executes on one or more processors of vehicle 302) or an interface to the software program (e.g., the interface allows communication with one or more remote devices executing the software program). In some examples, vehicle 302 does not include virtual assistant subsystem 316. In such examples, audio detected by the microphone of vehicle 302 may be transmitted or transcribed and transmitted to user device 320 for processing by a virtual assistant subsystem (e.g., virtual assistant subsystem 326).

[0049] refer to Figure 3 , user device 320 is depicted as including a user device renderer 322, a user device sensor 324, and a virtual assistant subsystem 326. The components of user device 320 are intended for explanation purposes and are not intended to be limiting. User device 320 may include more or fewer components, including a combination of the depicted components or other components described for computing system 100 or device 200.

[0050] As mentioned above, user device 320 includes user device renderer 322. In some examples, user device renderer 322 is any hardware or software of user device 320 used to generate (sometimes referred to as render) visual content (e.g., images or videos) from a model and / or one or more instructions. In such examples, user device renderer 322 can be configured to generate visual content for display by vehicle 302 or user device 320 from data detected and / or determined by vehicle 302, user device 320, or any combination thereof. User device renderer 322 can also be configured to generate visual content for display by user device 320 instead of vehicle 302.

[0051] In some examples, the user device renderer 322 renders content associated with an application executing on the user device 320 (e.g., a map from a map application for a primary display of the vehicle 302 or map routing instructions for a heads-up display of the vehicle 302). In some examples, the user device renderer 322 renders a third set (e.g., of a different type) of vehicle instruments (different from the first set rendered by the vehicle renderer 306 and the second set rendered by the integrated renderer 310). In some examples, the user device renderer 322 renders notifications associated with the user device 320 (such as notifications issued by the operating system of the user device 320 or an application executing on the user device 320) and a second set of notifications associated with the vehicle 302 (different from the first set of notifications rendered by the integrated renderer 310, such as notifications received by the user device 320 from the vehicle 302 (e.g., low tire pressure)). In some examples, a notification received by user device 320 from vehicle 302 includes content for user device 320 to use in rendering a representation of the notification (e.g., a notification message, an icon, and optional parameters that may be associated with the notification, such as a format for presenting miles (%d miles)).

[0052] User device 320 also includes user device sensor 324. In some examples, user device sensor 324 is any hardware or software of user device 320 used to detect data about the physical environment near (e.g., surrounding) user device sensor 324, similar to that discussed above with respect to computing system 100. Examples of user device sensor 324 include touch-sensitive surfaces, cameras, microphones, and any other sensors discussed with respect to computing system 100. In some examples, user device sensor 324 detects user input. In such examples, user input detected by user device sensor 324 is received by a process executing on one or more processors of user device 320, which determines the operations to be performed, such as what content to render and transmit for display by vehicle 302.

[0053] User device 320 also includes a virtual assistant subsystem 328 (sometimes referred to as an artificial intelligence assistant or digital assistant). In some examples, virtual assistant subsystem 328 is a software program that performs one or more operations based on data detected by a sensor (such as a natural language voice command detected by a microphone of user device 320 or vehicle 302). In such examples, user device 320 may include the software program (i.e., the software program executes on one or more processors of user device 320) or an interface to the software program (e.g., the interface allows communication with one or more remote devices executing the software program).

[0054] In some examples, the virtual assistant subsystem 328 receives audio and / or transcribed content from the vehicle 328 to act upon, such as when the vehicle 302 does not include a virtual assistant subsystem (e.g., virtual assistant subsystem 318). In other examples, the virtual assistant subsystem 328 of the user device 320 works in conjunction (e.g., in collaboration or together) with the virtual assistant subsystem 318 of the vehicle 302 such that some operations are handled by the virtual assistant subsystem 318 of the vehicle 302 (e.g., such as operations based on data from the vehicle 302, operations that are more time-sensitive, or operations that require less processing) and other operations or portions of operations are handled by the virtual assistant subsystem 328 of the user device 302 (such as operations based on data from the user device 320 or from a device connected to the user device 320 instead of the vehicle 320).

[0055] refer to Figure 3 , vehicle 302 is depicted as communicating with user device 320 via a transmission 330. In some examples, transmission 330 is a communication channel between two or more devices for conveying data between the devices. Examples of transmission 330 include a wired (e.g., physically connected via one or more cables (such as USB or Lightning cables)) or wireless (e.g., an Internet connection, a WiFi connection, a cellular connection, short-range communication, radio signals, and any other wireless data connection or network to communicate data between devices) channel connecting, for example, vehicle 302 and user device 320. Transmission 330 can (1) enable vehicle 302 to communicate information to user device 320 for use by user device 320 to render content, and (2) enable user device 320 to communicate such rendered content, information, layout packages, or other data to vehicle 302.

[0056] In some examples, there are multiple different communication channels between the vehicle 302 and the user device 320, each communication channel being used for a different type of data. For example, a first communication channel may stream content (e.g., images or videos) from the user device 320 for display by the vehicle 302 (e.g., the content is encrypted by the user device 320 and decrypted by the vehicle 302), a second communication channel is used to transmit metadata and / or control information related to the streamed content (in some examples, the metadata and / or control information is transmitted via the first communication channel embedded in the content or along with the content), a third communication channel is used to transmit vehicle information to the user device 320 (e.g., vehicle information related to data detected by sensors of the vehicle 302), and a fourth communication channel is used to transmit data and information to set up the vehicle 302 to display content received from the user device 320 (e.g., a layout package having a layout used by the user device 320 and rendered content pre-installed on the vehicle 302 that can be modified by the vehicle 302 when it needs to be displayed by the vehicle 302).

[0057] In some examples, the vehicle 302 is paired with the user device 320 via a transmission 330. In such examples, the vehicle 302 may be paired with the user device 320 when a key for controlling (e.g., unlocking, locking, or starting) the vehicle 302 is established on the user device 320. However, the vehicle 302 may be paired with the user device 320 in any suitable manner. In examples where the vehicle 302 is paired with the user device 320 via a transmission 330 when a key is established on the user device 320, the pairing may be performed before the key is established and the key is established in response to the pairing. In other examples, the pairing may be performed without establishing a key on the user device 320 or after the key is established on the user device 320, such as when the key is established through a connection between the user device 320 and a device other than the vehicle 302. In some examples, establishing a key includes a pairing process that is different from the pairing process of the integrated feature described herein. In such examples, two pairing processes are used to establish secure communications between the vehicle 302 and the user device 320 using different key materials and can be performed in any order (e.g., key pairing can occur before integrated pairing). In some examples, key pairing and integrated pairing are included in a single pairing. In other examples, the vehicle 302 is paired with the user device 320 without establishing a key on the user device 320.

[0058] In some examples, pairing may allow the vehicle 302 to identify the user device 320 before a wireless connection is established between the two devices (e.g., via a Bluetooth beacon, via a key fob, or some data sent by the user device 320 before a wireless connection is established with the vehicle 302). By identifying the user device 320 before a wireless connection is established, the vehicle 302 may display (1) content received by the user device 320 during a previous connection or (2) content based on instructions received by the user device 320 during a previous connection. In some examples, the vehicle 302 defaults to a particular frame and / or layout based on a previous connection.

[0059] In some examples, the vehicle 302 preferentially establishes a first connection with a first wireless technology (e.g., Bluetooth) so that communication can occur more quickly, and then uses the first connection to establish a second connection with a second wireless technology (e.g., WiFi) to increase the bandwidth used for communication. In such examples, the second wireless technology may have more bandwidth and / or use more power than the first wireless technology. In one example, the vehicle 302 may use the first connection to perform one or more operations before or while establishing the second connection, such as the vehicle 302 may receive instructions from the user device 320 via the first connection to display content that has been stored and / or rendered by the vehicle 302 and / or start the engine of the vehicle 302 when it is detected that a door has been opened or closed.

[0060] 4A to 4H is a block diagram illustrating exemplary user interfaces according to some examples described herein. The user interfaces in these figures are used to illustrate the processes described below, including FIG. 5A to FIG. 5H and Figures 6 to 11 process.

[0061] 4A to 4H Each of the frames depicts a frame (i.e., frames 400a to 400h) displayed on a display (e.g., a touch-sensitive display, a heads-up display, or a display screen) of a vehicle (e.g., vehicle 302). The frame may include zero or more user interface elements rendered by the vehicle (e.g., some user interface elements rendered by vehicle renderer 306 and some user interface elements rendered by integrated renderer 310) and zero or more user interface elements rendered by one or more devices other than the vehicle (e.g., a user device (such as by user device renderer 322), a server, or any device separate from the vehicle). It should be understood that more or fewer user interface elements may be displayed with the vehicle. 4A to 4H The depicted frames are included together.

[0062] In some examples, the position and / or characteristics of user interface elements and / or what content is included in a frame is based on a layout (e.g., including a definition of the position of user interface elements within a frame, such as an initial position). In such examples, a device that renders at least a portion of a frame (e.g., one or more user interface elements or combines already rendered user interface elements) can use the layout to determine where, what, and how to render a particular user interface element. In some examples, the vehicle stores one or more layouts and selects between the one or more layouts based on information known to the vehicle. In such examples, another device (e.g., a user device) can control the selection process and select a layout for the vehicle to use.

[0063] FIG. 4A to FIG. 4B The process of transitioning from a frame rendered without input from a user device to a frame rendered with input from a user device is illustrated. Figure 4A , frame 400a is displayed at a first time before the vehicle is connected to a user device (e.g., user device 320). In some examples, frame 400a is rendered by the vehicle (e.g., by vehicle renderer 306 or integrated renderer 310) (e.g., rendering different user interface elements at specific locations and / or combining different rendered user interface elements to create frame 400a).

[0064] like Figure 4AAs depicted, frame 400a includes a plurality of user interface elements, including current time 402, speedometer 404, and fuel gauge 406. Such user interface elements may be rendered separately by the vehicle, a user device, another device (e.g., a different user device, a device associated with the manufacture of the vehicle, a server, or any device separate from the vehicle), or some combination thereof. For example, components of speedometer 404 (e.g., numbers and pointers) may have been rendered by a previously connected user device and stored by the vehicle, so that the vehicle may construct speedometer 404 to reflect current data detected by the vehicle (e.g., speed is 0). In such an example, the vehicle will render the current state of speedometer 404 using separate components to reflect that the speed is 0, and then combine speedometer 404 with other user interface elements included within frame 400a to produce frame 400a. It should be understood that frame 400a may include more or fewer user interface elements, including, for example, a background or other content.

[0065] Frame 400a is according to a first layout such that the first layout is used to determine the location of current time 402, speedometer 404, and fuel gauge 406 within frame 400a. In some examples, the first layout is selected for use by the vehicle, such as based on what layout was most recently used or the current context of the vehicle. In such examples, the first layout may be configured to be used when starting the vehicle, and the decision to use the first layout is based on information installed on the vehicle prior to connecting to any user device.

[0066] As mentioned above, frame 400a includes current time 402, which indicates the current time determined by the software or hardware component of the vehicle. The current time 402 is displayed in a digital format as a specific size and is updated as time passes. It should be understood that the current time 402 can use different formats such as analog formats to indicate the current time.

[0067] Frame 400a also includes a plurality of vehicle instruments, including a speedometer 404 and a fuel gauge 406. In some examples, the vehicle instrument is a user interface element that reflects data detected by a sensor (e.g., a sensor of the vehicle). The speedometer 404 indicates the current speed of the vehicle and is depicted in analog form with a gauge including a pointer pointing to the current speed. It should be understood that the speedometer 404 can indicate the current speed using different formats (such as a digital format with a number indicating the current speed instead of a pointer). The fuel gauge 406 indicates the current amount of fuel remaining in the vehicle and is depicted in analog form with a pointer pointing to the current amount. It should be understood that the fuel gauge 406 can indicate the current amount of fuel using different formats (such as a digital format with a number indicating the percentage remaining instead of a pointer).

[0068] Figure 4B Frame 400b is depicted and is Figure 4AIn some examples, frame 400b is displayed after a user device (e.g., user device 320) is connected to a vehicle. In such examples, frame 400b may be displayed in response to the user device being connected to the vehicle such that no user input is received by the user device or the vehicle after the connection and before displaying frame 400b.

[0069] In some examples, frame 400b is rendered by a vehicle (e.g., different user interface elements are rendered at particular locations and / or different rendered user interface elements are combined to create frame 400b). In such examples, the different user interface elements may have been received by the vehicle from a user device and then combined by the vehicle with other user interface elements to generate frame 400b, as discussed further below.

[0070] Frame 400b is according to a layout different from the first layout (i.e., a second layout). In some examples, the second layout is selected by the user device, different from the first layout. In such examples, the second layout can be selected based on a state of the vehicle communicated from the vehicle to the user device. In other examples, the second layout can be selected based on a previous layout used by the user device (e.g., a previous layout used by the user device for the vehicle or another vehicle). Figure 3 As depicted, the second layout includes two areas: a main area 408 and a side area 410. In some examples, the second layout may include only the main area 408 (not illustrated), which has one or more user interface elements rendered by the vehicle and one or more user interface elements rendered by the user device. For example, the second layout may correspond to a dashboard of a vehicle, where user interface elements that display more critical data (such as the current speed of the vehicle) are rendered by the vehicle, and user interface elements that display less critical data (such as the current time) are rendered by the user device. For another example, the second layout may correspond to a center console of the vehicle, which has user interface elements that display data detected by the vehicle's sensors (such as the vehicle's current gas level) and user interface elements that display data detected by the user device's sensors (such as the user device's signal level).

[0071] like Figure 4B As depicted, main area 408 includes speedometer 404 and fuel gauge 406. Figure 4A Compared to frame 400a in FIG. 400b, the speedometer 404 in frame 400b is in a digital format (as opposed to an analog format) and in the same location, and the fuel gauge 406 is in the same format and in a different location. Figure 4B Depicted with Figure 4A It should be understood that the format of either the speedometer 404 or the fuel gauge 406 may be the same as that of the fuel gauge 406 in FIG. Figure 4Bdepicted differently (e.g., speedometer 404 may still be in analog format), and the size, opacity, or position of any Figure 4B Different than what is depicted.

[0072] Figure 4B A side area 410 including the current time 402 is depicted. The current time 402 in the frame 400b is depicted as Figure 4A 402 in a smaller font size. It should be appreciated that there may be other differences, such as different font types (e.g., Times New Roman or Arial) or different formats (e.g., a 24-hour clock instead of a 12-hour clock). Additionally, the current time 402 may be rendered by a vehicle or user device.

[0073] In some examples, the content in the frame (e.g., current time 402) is in a language specified by the user device, such as a language for content displayed via a display of the user device. In other examples, the content in the frame is in a language specified for such content and may be different from a language used by the user device to display the content on the display of the user device. For example, the difference in current time 402 may be based on a preference associated with an application executing on the user device, such as a preference selected by a user of the user device. The preference may be provided to the carrier together with or separately from the second layout.

[0074] Side region 410 of frame 400b also includes a plurality of user interface elements, including a signal affordance 412, a plurality of application affordances corresponding to different applications of the user device (i.e., a map affordance 414, a music affordance 416, a phone affordance 418), and a dashboard affordance 420. It should be appreciated that more or fewer user interface elements may be included in side region 410.

[0075] Signal indication 412 indicates the communication technology used by the user equipment (i.e., LTE) and the signal strength of the user equipment for that communication technology (i.e., 2 of 3 signal bars). It should be understood that different ways of representing such information may be used, and as an alternative or in addition to signal indication 412, side area 410 may include an indication of the connection between the vehicle and the user equipment (e.g., wired, WiFi, or BLTE).

[0076] As mentioned above, side area 410 includes multiple application affordances corresponding to different applications of the user device. In some examples, the application affordance is configured to cause a user interface associated with the corresponding application to be displayed when selected. In such examples, the selection may cause the application to be executed by the user device when the application has not yet been executed. For example, map affordance 414 may correspond to a map application of the user device. The map application may draw physical locations in a representation of at least a portion of the world for identification and navigation. In such an example, selection of the map affordance may cause a map to be displayed. Similarly, music affordance 416 may correspond to a music application of the user device for searching and playing audio files, and phone affordance 418 may correspond to a phone application of the user device for searching contacts of the user device, initiating communication sessions with other devices, checking messages from contacts, or any combination thereof. It should be understood that such functionality of these applications may be different, and other applications may be represented in side area 410.

[0077] In some examples, side area 410 may include one or more application affordances (not shown) corresponding to different applications of the vehicle. Such application affordances may operate similarly to application affordances associated with a user device, except that the application is being executed by the vehicle rather than the user device. In some examples, side area 410 may be configured by a user to include specific application affordances corresponding to specific applications. In such examples, a specific affordance may be selected by a user using the vehicle or the user device.

[0078] As mentioned above, side region 410 also includes dashboard affordance 420. Dashboard affordance 420 may be configured to cause a different user interface, such as a dashboard associated with a user device or vehicle, to be displayed when selected. The dashboard may include affordances for other applications not included in side region 410. In some examples, dashboard affordance 420 is configured to exit the user interface corresponding to a particular application and allow navigation to a different application when selected.

[0079] In some examples, the content of main area 408 and side area 410 is rendered by the user device and transmitted to the vehicle for display by the vehicle. In such examples, some user interface elements of frame 400b may not be included in the content transmitted from the user device to the vehicle, but are rendered by the vehicle and combined with (e.g., rendered on top of) the content received from the user device. For example, speedometer 404 and fuel gauge 406 may have been rendered by the vehicle, and the application affordances may have been rendered by the user device.

[0080] Figure 4BUser input 415 is depicted at a location corresponding to map-graphic capable representation 414 (e.g., user input 415 corresponding to a selection of map-graphic capable representation 414). User input 415 may include any type of user input, including a tap on a touch-sensitive display at a location corresponding to map-graphic capable representation 414, a press of a physical button of the vehicle while focused on map-graphic capable representation 414, a voice request by a user, or any other type of user input that signals a selection of map-graphic capable representation 414. In response to user input 415, a signal indicative of user input 415 may be transmitted to a user device. Based on the signal, the user device may determine that an animation to be displayed by the vehicle transitions from content displayed in frame 400b to a user interface for a map application (e.g., Figure 4D The animation may include a modification to the current layout, thereby causing a change in the content being displayed, such as Figure 4C In some examples, animation may not be used, and in response to user input 415, such as after transmitting a signal and receiving a frame corresponding to frame 400d from a user device, the display Figure 4D Frame 400d.

[0081] Figure 4C Frame 400c is depicted and is Figure 4B Frame 400b is displayed after receiving the user selection of the map display capability representation 414. Frame 400b maintains the Figure 4B 4. The layout of the frame 400b of FIG. 400c (ie, the second layout) is changed and still includes the main area 408 and the side area 410. Specifically, the main area 408 and the side area 410 in the frame 400c are located at the same position as those in the frame 400b.

[0082] like Figure 4C As depicted, the only change to the side region 410 is that the current time 402 has been updated based on the passage of time (i.e., from 10:01 to 10:02). The side region 410 in frame 400c still includes Figure 4B It should be appreciated that further changes may occur to side region 410 in response to a user selection of map representation 414. Such changes may be determined by the vehicle and / or the user device.

[0083] like Figure 4C As depicted, main area 408 still includes speedometer 404 and fuel gauge 406, but speedometer 404 has been modified. Figure 4BCompared to frame 400b in FIG. 1 , speedometer 404 is at a different location and has a different size. In particular, speedometer 404 has moved downward and to the left and has become smaller. In some examples, the different location is a change in the second layout because the second layout identifies the previous location as the location where speedometer 404 is located. In other examples, the second layout defines that speedometer 404 may be located in a different location such as Figure 4B The location depicted or Figure 4C The depicted location, and the content received from the user device identifies a location at a third time (ie, Figure 4C ). It should be appreciated that other types of movement, size, and / or other changes may have been applied to the speedometer 404 between the frames 400b and 400c.

[0084] In some examples, parameters of the speedometer 404 (e.g., size, font, format, and position) are determined by the user device, and at least these changes are transmitted to the vehicle for rendering by the vehicle. In such examples, the changes may be transmitted to the vehicle before, at the same time, or after a frame (e.g., a frame corresponding to frame 400c, a frame without the speedometer 404 and the fuel gauge 406) is transmitted from the user device to the vehicle. Based on these changes, the vehicle may render the speedometer 404 and place the speedometer 404 at the location depicted in frame 400c. In other examples, the vehicle may not receive a new frame to be displayed at a third time. Instead, the vehicle receives instructions on how to modify the previously received frame and performs such modifications locally without receiving a frame from the vehicle.

[0085] Figure 4D Frame 400d is depicted and is Figure 4B The second time and optional Figure 4C Frame 400d is displayed after receiving the user selection of the map representation 414 and optionally after one or more frames included in the animation between frame 400b and frame 400d (for example, when frame 400c is not displayed, frame 400d may be displayed after frame 400b).

[0086] Frame 400d is based on a layout different from the second layout (ie, a third layout). In some examples, similar to the second layout, the third layout is selected by the user device. Figure 4D As depicted, similar to the second layout, the third layout includes two regions: main region 408 and side region 410. In the third layout, side region 410 includes the same user interface elements as side region 410 in the second layout. The only change to side region 410 is that current time 402 has been updated based on the passage of time (i.e., from 10:02 to 10:03). Side region 410 in frame 400d still includes Figure 4B and Figure 4COther user interface elements described above. It should be understood that more changes may occur to the side area 410 in the frame 400d. Such changes may be determined by the vehicle and / or the user device.

[0087] like Figure 4D As depicted, main area 408 includes a map with a current location indicator 422. In some examples, the map and current location indicator 422 are determined and rendered by the user device and then transmitted to the vehicle. In such examples, the map and current location indicator 422 may be determined by a map application executed on the user device.

[0088] In addition to the map, the main area 408 of the frame 400d still includes the speedometer 404 and the fuel gauge 406, but the speedometer 404 has been modified again. Figure 4C Compared to frame 400c in frame 400d, speedometer 404 in frame 400d is at a different location and has a different size. In particular, speedometer 404 has moved downward and to the left and has become smaller. In some examples, the different location is defined in the third layout because the third layout identifies the location of speedometer 404 in frame 400d as the location where speedometer 404 is located. It should be recognized that other types of movement, size, and / or other changes may have been applied to speedometer 404 between frame 400c and frame 400d. Similar to the above, in some examples, parameters of speedometer 404 (e.g., size, font, format, and location) are determined by the user device, and at least these changes are transmitted to the vehicle for rendering by the vehicle. Figure 4D As depicted, speedometer 404 and fuel gauge 406 are both overlaid with the map in primary area 408 and, in some examples, rendered on top of the map by the vehicle.

[0089] Figure 4E Frame 400e is depicted and is Figure 4D Frame 400e is displayed after the fourth time (ie, the fifth time) of . Frame 400e is displayed when the vehicle detects an error in the connection between the vehicle and the user equipment. Frame 400e maintains the Figure 4D 400e (i.e., the third layout) and still includes a main area 408 with a speedometer 404 and a fuel gauge 406 and a side area 410 with a current time 402, application affordances, and a dashboard affordance 420. In particular, such user interface elements in frame 400e are located in the same positions as in frame 400d.

[0090] like Figure 4EAs depicted, some changes to main area 408 include no longer displaying the map and current location indicator 422. In some examples, the removal of the map and current location indicator 422 indicates that the connection between the vehicle and the user device is not working. In other examples (not illustrated), the map and current location indicator 422 are maintained in frame 400e because frame 400d is reused at a fifth time when the connection is not working.

[0091] In some examples (not illustrated), when there is an error in the connection between the vehicle and the user device, all content rendered by the user device will not be displayed. In such examples, such content will not be displayed because there will be no content from the user device that is designated to be displayed at the current time (i.e., the fifth time). For example (not illustrated), the application enable representation and dashboard enable representation 420 in the side area 410 may no longer be displayed. As another example, user interface elements that are updated only at a certain rate (e.g., a predefined rate or a predefined user interface element type) will no longer be displayed. In this example, the map and current location indicator 422 may no longer be displayed, but the application enable representation and dashboard enable representation 420 in the side area 410 may still be displayed.

[0092] like Figure 4E As depicted, current time 402 has been updated based on the time elapsed (i.e., from 10:03 to 10:04), speedometer 404 has been updated based on the speed of the vehicle (i.e., from 10 MPH to 15 MPH), and signal affordance 412 has been replaced with error affordance 424 (e.g., error affordance 424 is displayed). Error affordance 424 indicates that there is a connection error. Such updates may have been performed by the vehicle, causing the vehicle to re-render current time 402 and speedometer 404 to reflect current data determined by the vehicle.

[0093] The side region 410 in the frame 400e still includes Figure 4D Other user interface elements described. It should be understood that in response to a lost connection, more changes may occur to side area 410. Such changes may be determined by the vehicle after detecting that the connection is not working or before detection by the vehicle or user device.

[0094] Figure 4F Frame 400f is depicted and is Figure 4E Frame 400f is displayed after the vehicle is reconnected to the user device. Frame 400f maintains the Figure 4D400e (i.e., the third layout) and still includes main area 408 with map, current location indicator 422, speedometer 404, and fuel gauge 406 and side area 410 with current time 402, application affordances, and instrument panel affordances 420. In particular, such user interface elements in frame 400e are located in the same positions as in frame 400d.

[0095] like Figure 4F Depicted and Figure 4E In contrast, the map and current position indicator 422 are displayed again, indicating that the map and current position indicator 422 have been received from the user device at the current time. In addition, the map has been updated to reflect the current position of the vehicle.

[0096] like Figure 4F As depicted, current time 402 has been updated based on the time that has passed (i.e., from 10:04 to 10:05), speedometer 404 has been updated based on the speed of the vehicle (i.e., from 15 MPH to 20 MPH), and error indication 424 has been replaced with signal indication 412 (e.g., signal indication 412 is displayed again). In some examples, even if current time 402 is re-rendered by the vehicle at a fifth time (i.e., Figure 4E ), user interface elements (such as current time 402 and / or signal indication 412, etc.) may also be rendered by the user device at a sixth time (ie, Figure 4F ).

[0097] The side region 410 in the frame 400e still includes Figure 4D Other user interface elements described. It should be understood that in response to re-establishing the connection, more changes may occur to side area 410. Such changes may be determined by the vehicle and / or user device after detecting that the connection is working or before detection by the vehicle or user device.

[0098] Figure 4G Depicts frame 400g and Figure 4D Frame 400g is displayed after the vehicle is shifted into reverse gear and when there is an error in the connection between the vehicle and the user device. Frame 400g does not maintain the Figure 4FIn some examples, the fourth layout is selected by the vehicle due to an error in the connection between the vehicle and the user device. In other examples, the new layout is selected by the user device when the connection between the vehicle and the user device is working. In such examples, the new layout can be the same layout selected by the vehicle or a different layout. The same layout can be selected because both the vehicle and the user device are selected from the same set of layouts, and the input that causes the change in layout (e.g., the vehicle shifts into reverse gear) produces a specific layout.

[0099] The fourth layout still includes the main area 408 with the speedometer 404 and the fuel gauge 406 (in the same position, size and font) and the side area 410 with the current time 402, the application enable indications and the instrument panel enable indications 420.

[0100] like Figure 4G Depicted and Figure 4F In contrast, one change to the main area 408 is that the map and current position indicator 422 are no longer displayed, but a feed 426 (e.g., a frame of an image or video) from a rear-view camera is displayed. In some examples, the rear-view camera generates feed 426 and transmits it to the vehicle's progress for display, so that no attempt is made to transmit feed 426 to the user device. In such examples, even when the connection with the user device is working, feed 426 may never be transmitted to the user device, but is maintained locally on the user device. It should be recognized that due to an error in the connection between the vehicle and the user device, feed 426 is not displayed. In some examples, feed 426 is displayed because the vehicle is shifted into reverse gear. In such examples, feed 426 is displayed whenever the vehicle is shifted into reverse gear, regardless of the connection status between the vehicle and the user device. In other words, feed 426 is displayed (1) when the vehicle is connected to a user device and is receiving content from the user device for display and (2) when the vehicle is not connected to a user device and is displaying content but not receiving content from the user device.

[0101] Similar to the above, the current time 402 has again been updated based on the time elapsed (i.e., from 10:05 to 10:05), the speedometer 404 has been updated based on the speed of the vehicle (i.e., from 20 MPH to 0 MPH), and the signal enable indication 412 has been replaced with the error enable indication 424 (e.g., the error enable indication 424 is displayed).

[0102] The side region 410 in the frame 400e still includes Figure 4FOther user interface elements described. It should be understood that in response to shifting into reverse gear and / or losing a connection, more changes may occur to the side area 410. Such changes may be determined by the vehicle after detecting that the connection is not working or before detection by the vehicle or user device.

[0103] Figure 4H Depicts frame 400h and Figure 4G Frame 400h is displayed after the vehicle is reconnected to the user device. Frame 400h maintains the Figure 4G The layout of frame 400g (i.e., the fourth layout) still includes the main area 408 having the feed 426, the speedometer 404 and the fuel gauge 406 and the side area 410 having the current time 402, the application enable indication and the dashboard enable indication 420.

[0104] like Figure 4H As depicted, the current time 402 has been updated based on the time elapsed (i.e., from 10:06 to 10:07), the speedometer 404 has been updated based on the speed of the vehicle (i.e., from 0 MPH to 2 MPH), the feed 426 has been updated based on content received from the rearview camera, and the error enable indication 424 has been replaced with the signal enable indication 412 (e.g., the signal enable indication 412 is displayed).

[0105] The side region 410 in the frame 400e still includes Figure 4G Other user interface elements described above. It should be understood that in response to reconnection, side area 410 may undergo more changes. Such changes may be determined by the vehicle and / or the user device.

[0106] FIG. 5A to FIG. 5H is a flowchart illustrating different operations performed by a vehicle 500 (eg, vehicle 302) and a user device 502 (eg, user device 320). These operations illustrate the operations that occur on various devices to generate 4A to 4H The content of the example described.

[0107] FIG. 5A to FIG. 5H The flowchart in includes operations performed by vehicle 500 on the left side of the vertical line and operations performed by user device 502 on the right side of the vertical line. The operations on both sides of the vertical line can be performed by either or both devices. The operations in the dashed boxes represent optional operations. In some examples, such optional operations are not performed. It should be recognized that other operations may be optional and some operations may be performed by different devices than those depicted.

[0108] In some examples, vehicle 500 is any means in which or through which a person travels or an object is carried or transported in or through it. Examples of vehicle 500 include motor vehicles (e.g., motorcycles, cars, trucks, buses, airplanes, ships, etc.) and rail vehicles (e.g., trains or trams). In some examples, user device 502 is an electronic device owned and / or operated by a user. Examples of user device 502 include mobile or other handheld devices (e.g., smart phones, tablets, laptops, or smart accessories (e.g., smart watches)).

[0109] Figure 5A is a flow chart illustrating a method 504 for establishing a layout package on both a vehicle 500 and a user device 502. Some operations in the method 504 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0110] At 504a, method 504 includes the vehicle 500 determining that first content is to be displayed. In some examples, 504a occurs after the vehicle 500 is powered on and before the vehicle 500 is connected to the user device 502. In some examples, 504a occurs before any communication (e.g., pairing and / or discovery) between the vehicle 500 and the user device 502. The vehicle 500 determines that the first content to be displayed is determined based on information accessible to the vehicle 500 (e.g., not information received from the user device 502).

[0111] Figure 4A 404, and a fuel gauge 406. For example, the vehicle 500 may determine what time to display in the current time 402, what speed to display in the speedometer 404, and what fuel level to display in the fuel gauge 406. In some examples, the determined first content also includes a layout for displaying the first content, the layout identifying which user interface elements to display and how to display those user interface elements (e.g., the position of the user interface elements within the frame). In some examples, the layout is stored by the vehicle 500, such as at the time of manufacture or by an update at some time after manufacture.

[0112] At 504b, method 504 includes vehicle 500 rendering (e.g., a process of generating an image from a 2D or 3D model via a software process) a first content. The rendering is performed by a renderer (e.g., vehicle renderer 306 or integrated renderer 310) executing on a computer system of vehicle 500. An example of the rendered first content is frame 400a, such as Figure 4A Depicted.

[0113] At 504c, method 504 includes vehicle 500 displaying the first content. The display is on a display (such as a touch-sensitive display, a head-up display, a surface or a screen formed by a projector) of vehicle 500. In some examples, vehicle 500 displays different content on different displays of vehicle 500.

[0114] At 504d, method 504 includes establishing a connection between carrier 500 and user device 502. In some examples, the connection is initiated by carrier 500 or user device 502. The connection can be hardwired (e.g., through one or more wirings) or wireless (e.g., through a wireless communication channel, such as Bluetooth or WiFi). If carrier 500 supports hardwired connections using a port of carrier 500, a connection can be established by inserting one side of a cable into a port of carrier 500 and the other side of the cable into a port of user device 502. If carrier 500 supports wireless connections, a connection can be established by turning on a wireless network on both carrier 500 and user device 502 and navigating to a user interface on either carrier 500 or user device 502 to select another device for connection. The connection may include pairing two devices together to establish one or more secure connections for transmitting data between carrier 500 and user device 502. This pairing will be performed the first time the device is connected and is not required at a subsequent time.

[0115] At 504e, method 504 includes the vehicle 500 transmitting an identification of the vehicle 500 to the user device 502. In some examples, the identification is a unique identifier that is specific to the vehicle 500 (e.g., a vehicle identification number (VIN)) or specific to a component of the vehicle 500 (e.g., an identifier of a display of the vehicle 500) or a non-unique identifier that is specific to the vehicle 500 (e.g., a brand and / or model of the vehicle 500) or specific to a component of the vehicle 500 (e.g., a brand or model of a component). The identification of the vehicle 500 may be transmitted when the connection is established at 504d or via the connection established at 504d (i.e., after the connection is established using the established connection). In some examples, the identification of the vehicle 500 is transmitted via a first connection (e.g., using a first communication technology, such as Bluetooth), and the subsequent communication of data (e.g., receiving the layout package at 504j) is transmitted via a second connection established using the first connection (e.g., using a second communication technology, such as WiFi). At 504f , method 504 includes the user device 502 receiving an identification of the vehicle 500 .

[0116] At 504g, method 504 includes user device 502 obtaining a layout package for vehicle 500. The layout package includes definitions of one or more layouts for vehicle 500. In some examples, the layout defines the initial position of one or more user interface elements within a frame. In such examples, the layout can be used to identify the position at which a specific user interface element is rendered within a frame. In some examples, the initial position of the user interface element can be modified, but the initial position provides the starting point and / or expected position of the user interface element. The layout package may also include one or more rendered user interface elements and / or scripts for rendering user interface elements. In some examples, some of the rendered user interface elements in the layout package are rendered by user device 502 and added to the layout package, so that those rendered user interface elements are not included in the layout package received by user device 502.

[0117] In some examples, the layout package is obtained using the identification of the vehicle 500. For example, the user device 502 may transmit a request for the current package of the vehicle 500 to the remote device, the request including the identification of the vehicle 500. The remote device may then transmit the layout package to the user device 502. In this example, at 504h, the method 504 includes the user device 502 storing the layout package received from the remote device. In some examples, the storage location of the layout package is local to the user device 502, so that the user device 502 can access the layout package when it cannot communicate with the remote device. For another example, the user device 502 may have stored one or more layout packages, and use the identification of the vehicle 500 to identify which layout package to use with respect to the vehicle 500.

[0118] At 504i, method 504 includes user device 502 transmitting a layout package to vehicle 500. In some examples, the layout package is transmitted via the connection established at 504d. Method 504 includes vehicle 500 receiving the layout package at 504j and storing the layout package at 504k. In some examples, the storage location of the layout package is local to vehicle 500, so that vehicle 500 can access the layout package when it is not connected to user device 502. In some examples, by storing the layout package on both vehicle 500 and user device 502, both devices are able to identify the location in the frame where the user interface element is to be placed and identify the location where another device can place the user interface element. In addition, when trying to display content, less data needs to be communicated between devices; and vehicle 500 can continue to operate and display content even when the connection with user device 502 is not working.

[0119] Figure 5B5 is a flow chart illustrating a method 506 for a vehicle 500 to display a frame including content rendered by both the vehicle 500 and the user device 502. Some operations in the method 506 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, the method 506 Figure 5A The method 504 occurs after that.

[0120] At 506a, method 506 includes user device 502 determining a layout to use for vehicle 500. In some examples, the layout is from the layout package obtained by user device 502 at 504g of method 504. The layout can be determined based on the context of vehicle 500 and / or user device 502, such as what vehicle 500 is currently displaying and / or what vehicle 500 will display in the future.

[0121] At 506b, method 506 includes the user device 502 transmitting an identification of the layout to the vehicle 500. The identification of the layout may be transmitted via the connection established at 504d of method 504 or a subsequent connection. In some examples, the identification of the layout may be transmitted via a connection configured for transmitting metadata and control information, while a different connection is configured to stream content between devices (e.g., the first frame of rendering transmitted at 506g). Method 506 includes the vehicle 500 receiving the identification of the layout at 506c, and storing the identification of the layout at 506d. In some examples, the storage location of the identification of the layout is local to the vehicle 500, so that the vehicle 500 can access the identification of the layout when it is not connected to the user device 502. By storing the identification of the layout, the vehicle 500 is able to determine how to combine the frame received from the user device 502 with the user interface element rendered by the vehicle 500. In other examples, the identification of the layout may be transmitted together with any content transmitted as metadata to the content.

[0122] At 506e, method 506 includes user device 502 determining a first frame to be displayed by vehicle 500. In some examples, the determination is based on the context of vehicle 500 and / or user device 502, such as what vehicle 500 is currently displaying and / or what vehicle 500 will display in the future. In such examples, the determination may be performed by an application executing on user device 502 that is pushing content to be displayed by vehicle 500.

[0123] Method 506 includes user device 502 rendering a first frame at 506f, and transmitting the rendered first frame and first rendering information to vehicle 500 at 506g. In some examples, the rendered first frame and the first rendering information are transmitted to vehicle 500 separately (such as through different communication channels). For example, the rendered first frame can be transmitted over a streaming connection used to transmit frames, and the first rendering information can be transmitted in a message addressed to vehicle 500 outside of the streaming connection. The first rendering information can include data that assists vehicle 500 in combining user interface elements with the first frame and / or displaying the first frame (e.g., the time to display the first frame). In some examples, the first rendering information includes instructions for modifying the appearance of user interface elements rendered by vehicle 500 and / or the location within the first frame where the user interface elements are to be included (i.e., different from the layout being used for the first frame).

[0124] The method includes the vehicle 500 receiving the rendered first frame and the first rendering information at 506h, and rendering a first combined frame at 506i. In some examples, the first combined frame is rendered by combining the rendered first frame with one or more user interface elements stored and / or rendered by the vehicle 500 (e.g., previously rendered prior to receiving the rendered first frame or rendered over the rendered first frame). In such examples, the combining may be based on the first rendering information, such as modifying how the combining is performed according to one or more instructions included with the first rendering information.

[0125] At 506j, the method includes the vehicle 500 displaying a first combined frame. An example of a first combined frame is Figure 4B Depicted frame 400b. In this example, the rendered first frame includes all user interface elements in frame 400b except speedometer 404 and fuel gauge 406, both of which may be rendered by vehicle 500 based on data detected by sensors of vehicle 500.

[0126] Figure 5C 5 is a flow chart illustrating a method 508 for user device 502 to coordinate the display of an animation by vehicle 500. Some operations in method 508 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, method 508 is Figure 5B The method 506 occurs after that.

[0127] At 508a, method 508 includes user device 502 determining that an animation is to be displayed on vehicle 500. In some examples, the animation is determined based on the context of vehicle 500 and / or user device 502, such as what vehicle 500 is currently displaying and / or what vehicle 500 will display in the future. In such examples, the determination may be performed by an application executing on user device 502 that is pushing content to be displayed by vehicle 500. The animation may define what vehicle 500 will display in multiple frames, including, for example, modifications to a layout over time.

[0128] At 508b, method 508 includes user device 502 determining a frame based on the animation. In some examples, the frame is further based on the context of vehicle 500 and / or user device 502 (such as what vehicle 500 is currently displaying and / or what vehicle 500 will display in the future).

[0129] Method 508 includes user device 502 rendering a frame at 508c and transmitting the frame and rendering information to vehicle 500 at 508 (similar to Figure 5B The method 506 may continue to execute 508b, 508c, and 508d until the animation is complete (e.g., a plurality of frames may be determined based on the animation, rendered, and transmitted to the vehicle 500).

[0130] Method 508 includes the vehicle 500 receiving the frame and rendering information at 508e and rendering the combined frame (similar to Figure 5B 506h and 506i). The method 508 then includes the vehicle 500 displaying the combined frame (similar to Figure 5B Method 508 may continue to execute 508e, 508f, and 508g for each frame received from user device 502. An example of different frames of an animation displayed by vehicle 500 is frames 400b to 400c having a speedometer 404 moving from the center of main area 408 to the lower left, as shown in FIG. FIG. 4B to FIG. 4C Depicted.

[0131] Figure 5D 5 is a flowchart illustrating a method 510 for disconnecting a vehicle 500 from a user device 502. Some operations in the method 510 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, the method 510 Figure 5C Method 508 Figure 5B Method 506 and / or Figure 5A The method 504 occurs after that.

[0132] At 510a, method 510 includes disconnecting vehicle 500 from user device 502. In some examples, the disconnection occurs due to a cable being unplugged from vehicle 500 and / or user device 502. In other examples, the disconnection occurs due to an intentional or unintentional loss of a wireless connection between vehicle 500 and user device 502.

[0133] Method 510 includes the vehicle 500 determining at 510b that a second content is to be displayed (similar to Figure 5A 504a), and rendering the second content at 510c (similar to Figure 5A 504c), and at 510d, the rendered second content is displayed (similar to Figure 5A 504d). An example of the second content is Figure 4E The depicted frame 400e. When the vehicle 500 and the user device 502 are connected (e.g., Figure 5B Method 506 and Figure 5C 508) and when the vehicle 500 and the user equipment 502 are disconnected (for example, Figure 5D One difference between the method 510 of the vehicle 500 and the user device 502 is which device is setting the content to be displayed. For example, when the vehicle 500 and the user device 502 are connected, the user device 502 is determining the frame to be displayed (e.g., Figure 5B 506e and Figure 5C 506b), such as identifying multiple user interface elements and where to place those user interface elements. On the other hand, when the vehicle 500 and the user device 502 are disconnected, the vehicle 500 determines the frame to be displayed (e.g., Figure 5D The reason for this difference is that the vehicle 500 no longer has input from the user device 502, causing the vehicle 500 to make at least one decision that the user device 502 would have made if the connection was working (e.g., where, what, or how to display a particular user interface element within the frame).

[0134] In some examples, the second content is based on a layout used by vehicle 500 before (e.g., immediately before) disconnecting from user device 502. In other examples, the second content is based on a layout determined by vehicle 500 in response to detecting a loss of connection with user device 502. In some examples, the layout may be based on the context of vehicle 500, such as what vehicle 500 is about to display.

[0135] Figure 5E 5 is a flowchart illustrating a method 512 for reconnecting the vehicle 500 with the user equipment 502. Some operations in the method 512 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, the method 512 Figure 5DThe method 510 occurs afterwards.

[0136] At 512a, method 512 includes vehicle 500 reconnecting with user device 502. The reconnection may be initiated by vehicle 500 or user device 502. The reconnection may be hardwired (e.g., via one or more wires) or wireless (e.g., via a wireless communication channel such as Bluetooth or WiFi), as described above in Figure 5A 504d as described in Section 504d.

[0137] At 512b, method 512 includes the vehicle 500 transmitting an identification of the vehicle 500 to the user device 502 (similar to Figure 5A The stored layout package may be a layout package in the form of a file or a directory. Figure 5A The layout package stored at 504h in the method 512 may be information included with the stored layout package. In some examples, the identification of the vehicle 500 and the identification of the version of the stored layout package are transmitted via the connection established at 512a. At 512c, the method 512 includes the user device 502 receiving the identification of the vehicle 500 and the identification of the version.

[0138] After receiving both identifications, the user device 502 may determine whether the version is the current version of the vehicle 500. If the version is outdated (i.e., not the current version of the vehicle 500), the method 512 proceeds to 512d. If the version is up to date (i.e., the current version of the vehicle 500), the method 512 proceeds to 512i.

[0139] At 512d, method 504 includes user device 502 obtaining a layout package for vehicle 500. As discussed further below, obtaining a new layout package may include transmitting a request for a new layout package or accessing a new layout package already stored by user device 502. The new layout package may include at least one difference from a layout package stored by vehicle 500. In some examples, the new layout package includes multiple differences from a layout package stored by vehicle 500, such that only the differences are sent to vehicle 500 rather than the entire layout package.

[0140] In some examples, the new layout package is obtained using an identification and / or an identification of a version of the vehicle 500. For example, the user device 502 may transmit a request for a current package of the vehicle 500 to the remote device, the request including an identification and / or an identification of a version of the vehicle 500. The remote device may then transmit the new layout package to the user device 502. In such an example, at 512e, the method 512 includes the user device 502 storing the new layout package received from the remote device. In some examples, the storage location of the new layout package is local to the user device 502, so that the user device 502 can access the new layout package when it is unable to communicate to the remote device.

[0141] At 512f, method 512 includes the user device 502 transmitting the new layout package to the vehicle 500. In some examples, the new layout package is transmitted via the connection established at 512a. Method 512 includes the vehicle 500 receiving the new layout package at 512g and storing the new layout package at 512h. In some examples, the storage location of the new layout package is local to the vehicle 500, so that the vehicle 500 can access the new layout package when it is not connected to the user device 502.

[0142] In some examples, after the user device 502 obtains or stores the new layout package (e.g., in response to the user device 502 obtaining or storing the new layout package), the method 512 proceeds to 512i. In other examples, after the user device 502 transmits the new layout package (e.g., in response to the user device 502 transmitting the new layout package), the method 512 proceeds to 512i.

[0143] At 512i, method 512 includes user device 502 determining to use a second layout. The second layout is optionally different from the layout used before reconnecting at 512a or used immediately before the most recent disconnection. The second layout can be determined based on the context of vehicle 500 and / or user device 502 (such as what vehicle 500 is currently displaying and / or what vehicle 500 will display in the future).

[0144] At 512j, method 512 includes user device 502 transmitting an identification of a second layout to vehicle 500. In some examples, the identification of the second layout is transmitted via the connection established at 512a (or a subsequent connection). In such examples, the second layout can be transmitted together with or separately from the new layout package.

[0145] At 512k and 512l, method 512 includes the vehicle 500 receiving and storing an identification of the second layout. In some examples, the storage location of the identification of the second layout is local to the vehicle 500, so that the vehicle 500 can access the second identification of the layout when it is not connected to the user device 502. By storing the identification of the second layout, the vehicle 500 can determine how to combine the frame received from the user device 502 with the user interface element rendered by the vehicle 500.

[0146] After determining to use the second layout or transmit an indication of the second layout (e.g., in response to determining to use the second layout or transmit an indication of the second layout), method 512 may proceed to 512m. At 512m, method 512 includes user device 502 determining a second frame to be displayed by vehicle 500. In some examples, the determination is based on the context of vehicle 500 and / or user device 502, such as what vehicle 500 is currently displaying and / or what vehicle 500 is to display in the future. In such examples, the determination may be performed by an application executing on user device 502 that is pushing content to be displayed by vehicle 500.

[0147] The method 506 includes the user device 502 rendering the second frame at 512n, and transmitting the rendered second frame and second rendering information to the vehicle 500 at 512o. The method includes the vehicle 500 receiving the rendered second frame and the second rendering information at 512p, and rendering a second combined frame at 512q. In some examples, the second combined frame is rendered by combining the rendered second frame with one or more user interface elements stored by the vehicle 500. In such examples, the combining can be based on the second rendering information, such as modifying how the combining is performed according to one or more instructions included with the second rendering information.

[0148] At 512r, the method includes the vehicle 500 displaying a second combined frame. An example of a second combined frame is Figure 4F Depicted frame 400f.

[0149] Fig. 5F 5 is a flow chart illustrating a method 514 for responding to user input detected by vehicle 500 or user device 502. Some operations in method 514 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, method 512 Figure 5E Method 512 Figure 5C Method 508 Figure 5B Method 506 or Figure 5A The method 504 occurs after that.

[0150] At 514a, method 512 includes vehicle 500 detecting a first user input. In some examples, the first user input is detected by a component of vehicle 500, such as a sensor of vehicle 500. Examples of components include physical buttons, touch-sensitive surfaces, cameras, microphones, and any other components of vehicle 500 capable of detecting user input. In response to vehicle 500 detecting the first user input, method 514 proceeds to 514b. At 514b, vehicle 500 transmits an indication of the first user input to user device 502, and at 504c, user device 502 receives the indication of the first user input.

[0151] In other examples, instead of the vehicle 500 detecting the first user input, at 514d, the user device 502 may detect a second user input. In such examples, the second user input is used to determine a change to the third layout without any communication with the vehicle 500 (e.g., the vehicle 500 does not detect the user input and does not transmit an indication of the user input to the user device 502). In some examples, the second user input is detected by a component of the user device 502, such as a sensor of the user device 502. Examples of components include physical buttons, touch-sensitive surfaces, cameras, microphones, and any other component of the user device 502 that is capable of detecting a user input.

[0152] After determining to change to the third layout, the remainder of the operations of method 514 (ie, 514f to 514n) are similar to Figure 5B 506b to 506j of method 506. An example of a third combined frame of 514n is Figure 4D Depicted frame 400d.

[0153] Figure 5G 5 is a flow chart illustrating a method 516 for recovering from a problem in communications between a vehicle 500 and a user device 502. Some operations in the method 516 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, the method 516 Fig. 5F Method 514 Figure 5E Method 512 Figure 5C Method 508 Figure 5B Method 506 or Figure 5A The method 504 occurs after that.

[0154] The method 516 includes the vehicle 500 detecting a third user input at 516a and attempting to transmit an indication of the third user input to the user device 502 at 516b. The third user input may be similar to the one described above in Fig. 5F In some examples, the vehicle 500 attempts to Figure 5E Later, when transmitting, Figure 5E The connection established at 512a in the transmission indication, or at Figure 5A , Figure 5B or Figure 5C Later, when transmitting, Figure 5A The connection established at 504d in transmits an indication.

[0155] At 516c, method 516 includes vehicle 500 determining that user device 502 has failed to respond to the indication. In some examples, this determination is based on determining that a connection used to transmit the indication to user device 502 is not working. In other examples, this determination is based on determining that a predefined amount of time has expired after attempting to transmit or transmitting an indication of a third user input. In other examples, this determination is based on determining that the remaining time to display the content reaches a threshold at which vehicle 500 can no longer wait for user device 502.

[0156] At 516d, method 516 includes vehicle 500 determining to change to a fourth layout based on the third user input. In some examples, the determination occurs after determining that user device 502 has failed to respond to the instruction.

[0157] At 516e, method 516 includes vehicle 500 determining a fourth frame to be displayed on vehicle 500. In some examples, the fourth frame is determined without input from user device 502. In other words, vehicle 500 attempts to receive input from user device 502, and when input is not received in a timely manner, vehicle 500 determines what to display (similar to the above description of the fourth frame). Figure 5D If the user device 502 has responded, the vehicle 500 will use the frame received from the user device 502.

[0158] At 516f and 516g, method 516 includes vehicle 500 rendering a fourth frame and displaying the rendered fourth frame. An example of the fourth frame is Figure 4G Depicted frame 400g.

[0159] Figure 5H is a flow chart illustrating a method 518 for when the vehicle 500 reconnects to the user equipment 502 after recovering from a problem in communication. Some operations in the method 518 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted. In some examples, the method 518 Figure 5G The method 516 occurs after that.

[0160] At 518a, method 516 includes vehicle 500 reconnecting with user device 502. In some examples, the reconnection is initiated by vehicle 500 or user device 502. The reconnection can be hardwired (e.g., via one or more wires) or wireless (e.g., via a wireless communication channel such as Bluetooth or WiFi), as described above in Figure 5A 504d as described in Section 504d.

[0161] The method 518 includes, at 518b, the vehicle 500 transmitting an identification of a current state of the vehicle 500, and, at 518c, the user device 502 receiving the identification. The identification of the current state may include information that helps the user device 502 determine what to cause to be displayed by the vehicle 500. For example, the identification may include an identification of a layout being used by the vehicle 500, an indication of an input signal detected by the vehicle 500 (e.g., from a Figure 5G 516a in the third user input), an indication of one or more user interface elements included in the frame displayed by the vehicle 500, or any combination thereof. In some examples, the vehicle 500 also transmits an identification of the vehicle 500 and an identification of the version of the stored layout package to the user device 502, similar to the Figure 5E In such an example, if the stored layout package is outdated, method 518 may include operations 512c to 512h.

[0162] At 518d, method 518 includes user device 502 determining to use a fifth layout based on the current state of vehicle 500. The fifth layout can be the same as or different from the current layout being used by vehicle 500. In examples where the fifth layout is different, user device 502 transmits an identification of the fifth layout to vehicle 500 (at 518e), and vehicle 500 receives and stores the identification of the fifth layout (at 518f and 518g).

[0163] The remaining steps of method 518 are similar to Figure 5B An example of a fifth combined frame of 518m is Figure 4H Depicted frame 400h.

[0164] Figure 6 is a flow chart illustrating a method 600 for establishing a layout for synchronous rendering on multiple devices. Some operations in method 600 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0165] In some examples, method 600 is performed at a first device (e.g., computing system 100, device 200, user device 320, or user device 502) (in some examples, the first device is a user device, such as a portable electronic device; in some examples, the first device is logged into a user account that the second device is logged into or not logged into; in some examples, any combination of operating system modules and / or application modules may execute the steps of method 600).

[0166] At 610, method 600 includes connecting to a second device (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) that is different from (e.g., separate from) the first device via a first connection (e.g., 504d) (in some examples, the second device is a vehicle, such as a computer system configured to display content on a display of the vehicle) (in some examples, the connection is included in a pairing process between the first device and the second device; in some examples, the connection occurs after the pairing process; in some examples, the connection is via a wired or wireless connection).

[0167] At 620, method 600 includes receiving, via the first connection, an identifier (e.g., 504f) associated with a second device (in some examples, the identifier refers to a display of the second device or a type of display; in some examples, the identifier refers to a type of the second device; in some examples, the identifier refers to a set of one or more layouts that are compatible with the display of the second device).

[0168] At 630, method 600 includes, after receiving an identifier associated with the second device, using the identifier to obtain a set of one or more layouts for a display (e.g., a screen or other visual output device) of the second device (e.g., 504g) (in some examples, the obtaining is performed by a device other than the second device; in some examples, the layout is not displayed by the display, but is used to identify the location of specific content; in some examples, the layout includes one or more sizes of the display; in some examples, the layout includes a resolution of the display). In some examples, the set of one or more layouts includes multiple layouts (e.g., multiple different layouts).

[0169] At 640 , method 600 includes storing (eg, in a memory of the first device) the set of one or more layouts (eg, 504 h ).

[0170] At 650 , method 600 includes transmitting to the second device (in some examples, the transmitting is via the first connection) a set of one or more layouts (eg, 504 i ) for use with a display of the second device.

[0171] At 660, method 600 includes, after transmitting the set, determining content (e.g., 506e, 508b, 512m, 514i, or 518h) for display via a display of the second device based on a layout in the set of one or more layouts stored at the first device (in some examples, the layout is determined by the first device) (in some examples, the determination includes rendering (e.g., locally rendering) the content (e.g., 506f, 508c, 512n, 514j, or 518i) on the first device; in some examples, the determination includes obtaining the rendered content from a remote device). In some examples, the layout includes a definition of an initial position of at least one user interface element (in some examples, the at least one user interface element is rendered by the first device; in some examples, the at least one user interface element is rendered by the second device).

[0172] At 670, method 600 includes transmitting to the second device (in some examples, the transmission is via the first connection) a message (e.g., 506g, 508d, 512o, 514k, or 518j) corresponding to the content (in some examples, the message includes the content; in some examples, the content includes a portion (e.g., a placeholder) intended for the second device to render a user interface element and added to the portion; in some examples, the message includes instructions for the second device to obtain the content, such as stored locally on the second device or on a device remote from the second device; in some examples, the message includes data used to generate the content on the second device).

[0173] In some examples, method 600 also includes, when the first device is connected to the second device (in some examples, when the first device is connected to the second device via the first connection or a different (e.g., subsequent) connection): receiving an indication of user input (e.g., 514c, 514d, or 518c) (in some examples, the indication of user input is an indication to a virtual assistant (e.g., an indication provided by the virtual assistant in response to the virtual assistant receiving an indication from the user; in some examples, the virtual assistant is hosted by the first device or the second device); in response to receiving the indication of user input, determining to change the layout being used by the second device to a new layout (e.g., 514e or 518d) (in some examples, the method also includes determining at the first device that the user input corresponds to a request to change the layout (e.g., the current layout)); and transmitting a message indicating the new layout to the second device (e.g., 514f or 518e) (in some examples, the message identifies the new layout; in some examples, the message includes the new layout; in some examples, the new layout is included in a set of one or more layouts; in some examples, the message includes an indication of a modification to the layout). In some examples, the user input corresponds to Activation of a physical button of the second device (in some examples, the physical button is embedded in the second device). In some examples, the user input corresponds to touch input detected via a touch-sensitive display of the second device (in some examples, the touch input corresponds to selection of an affordance (e.g., a user interface element, such as a button) displayed by the touch-sensitive display). In some examples, the user input corresponds to user input detected via a sensor of the first device (in some examples, the sensor includes a microphone (e.g., by a virtual assistant), a camera (e.g., by a virtual assistant), a touch-sensitive display, or a sensor that detects activation of a physical button of the first device). In some examples, the user input corresponds to voice input detected via a microphone (in some examples, the voice input corresponds to an audible request to change the layout; in some examples, the voice input involves a virtual assistant; in some examples, the voice input causes another application (e.g., a virtual assistant application) to execute and return control for changing the layout after the other application determines an output (e.g., user input); in some examples, the user input corresponds to a gesture detected via a camera; in some examples, the microphone is the first device or the second device).

[0174] In some examples, the set of one or more layouts is a first version (in some examples, the identifier of the first version is transmitted to the second device together with (or separately from) the set of one or more layouts). In such examples, method 600 also includes, after the first connection is disconnected: connecting to the second device via a second connection (e.g., 512a or 518a) (in some examples, the second connection is the same as or different from the first connection); receiving a second identifier associated with the second device via the second connection (in some examples, the second identifier is the same as the identifier) ​​(e.g., 512c) (in some examples, the second identifier refers to the display of the second device or the type of display; in some examples, the second identifier refers to the type of the second device; in some examples, the second identifier refers to a set of one or more layouts compatible with the display of the second device); receiving an identifier of a current version associated with the set of one or more layouts via the second connection (e.g., 512c) (in some examples, the identifier of the current version and the second identifier are included in a single message; in some examples, the identifier of the current version is transmitted separately from the second identifier (e.g., in different messages)); and based on determining that the current version is the same as the first version: based on Determining, based on a particular layout in a set of one or more layouts (in some examples, the particular layout is determined by the first device), second content (e.g., 512m) for display via a display of a second device (in some examples, the determination includes rendering (e.g., locally rendering) the second content (e.g., 512n) on the first device; in some examples, the determination includes obtaining the rendered content from a remote device); and transmitting to the second device (in some examples, the transmission is via a second connection) a second message (e.g., 512o) corresponding to the second content (in some examples, the second message includes the second content; in some examples, the second content includes a portion (e.g., a placeholder) intended for the second device to render a user interface element and added to the portion; in some examples, the second message includes instructions for the second device to obtain the second content, such as stored locally on the second device or on a device remote from the second device; in some examples, the second message includes data used to generate the content on the second device). In some examples, method 600 also includes transmitting a new set of one or more layouts to the second device (e.g., 512f) based on determining that the current version is different from the first version, wherein the new set is the current version, and wherein the new set is different from the set of one or more layouts (e.g., at least one layout from the new set is different from the set of one or more layouts). In some examples, the particular layout is a layout last used by the first device for the second device (e.g., the last used layout was used during a previous connection between the first device and the second device).

[0175] In some examples, method 600 also includes transmitting a script (e.g., a rendering script) for rendering a user interface element to the second device in addition to transmitting the set of one or more layouts (e.g., 504i or 512f) (in some examples, the script includes one or more instructions that, when executed, render a user interface element (e.g., an image, including one or more pixel values); in some examples, the script is transmitted in a message with the set of one or more layouts; in some examples, the script is transmitted in a message different from the message with the set of one or more layouts).

[0176] In some examples, method 600 also includes transmitting rendered content (e.g., 504i or 512f) (e.g., a bitmap or image, sometimes referred to as a rendering) to the second device in addition to transmitting the set of one or more layouts (in some examples, the rendered content is transmitted in a message with the set of one or more layouts; in some examples, the rendered content is transmitted in a message different from the message with the set of one or more layouts).

[0177] Note that the following description of method 700 (ie, Figure 7 ), 800 (i.e. Figure 8 ), 900 (i.e., Fig. 9 ), 1000 (i.e., Fig.10 ) and 1100 (i.e., Fig.11 The details of the process described in ) can also be applied in a similar manner to Figure 6 Method 600. For example, method 600 optionally includes one or more of the characteristics of the various methods described below with reference to method 700, such as the message transmitted at 670 of method 600 can be the rendered frame received at 710 of method 700. For another example, method 600 optionally includes one or more of the characteristics of the various methods described below with reference to method 800, such as the message transmitted at 670 of method 600 can be the rendered frame received at 810 of method 800. For another example, method 600 optionally includes one or more of the characteristics of the various methods described below with reference to method 900, such as the message transmitted at 670 of method 600 can be the first frame transmitted at 940 of method 900. For another example, method 600 optionally includes one or more of the characteristics of the various methods described below with reference to method 1000, such as the connection at 610 of method 600 can be the connection at 1020 of method 1000. For another example, method 600 optionally includes one or more of the characteristics of the various methods described below with reference to method 1100 , such as the message transmitted at 670 of method 600 may be the second message received from the device at 1140 of method 1100 .

[0178] Figure 7is a flow chart illustrating a method 700 for time-based rendering synchronization. Some operations in method 700 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0179] In some examples, method 700 is performed at a first device (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) (in some examples, the first device is a vehicle such as a computer system configured to display content on a display of the vehicle; in some examples, any combination of operating system modules and / or application modules may execute the steps of method 700).

[0180] At 710, method 700 includes receiving a rendered frame (e.g., a portion of a photo, image, video, or pixel information) (e.g., 506h, 508e, 512p, 514l, or 518k) from a second device different from the first device (in some examples, the second device is a user device, such as a portable electronic device; in some examples, the second device is logged into a user account that the first device is logged into or not logged into) (in some examples, the first device is wired or wirelessly connected to the second device, such as via Bluetooth or WiFi, and the rendered frame is transmitted through In some examples, the first device and the second device establish a streaming connection (e.g., 504d, 512a, 518a) before receiving the rendered frame, wherein the streaming connection is configured to transmit data from at least one device (e.g., the second device) to another device (e.g., the first device); in some examples, the rendered frame is rendered by the second device (e.g., 506f, 508c, 512n, 514j, 518i); in some examples, the rendered frame is rendered by a device other than the second device and provided to the second device for transmission to the first device). In some examples, the rendered frame is rendered by the second device (e.g., rendered locally). In some examples, a rendered frame received from a second device includes a placeholder portion (in some examples, the placeholder portion does not include content rendered by the second device; in some examples, the placeholder portion includes (in some examples, includes only) background content; in some examples, the placeholder portion does not include user interface elements rendered by the second device; in some examples, the placeholder portion does not include user interface elements other than background content rendered by the second device; in some examples, the placeholder portion does not include user interface elements that are unique to the placeholder portion as compared to the rest of the rendered frame (e.g., other than other placeholder portions); in some examples, the placeholder portion is used only by the second device to generate messages transmitted to the first device, and no indication of the placeholder portion is transmitted to the first device separately from the layout), and wherein the combined frame includes the user interface element at a location corresponding to the placeholder portion (e.g., the location is the placeholder portion).

[0181] At 720, method 700 also includes receiving, at the first time, a message (e.g., 506h, 508e, 512p, 514l, 518k) from a second device including a second time (e.g., a future time, a time after the current time, or a time in the future) (in some examples, the message includes an indication in addition to the time when the combined frame should be displayed (e.g., the next possible time)), where the second time is after the first time (in some examples, the message is received via the same channel as the rendered frame; in some examples, the message is received via a different channel than the rendered frame, such as a channel configured to transmit a smaller amount of data (e.g., Bluetooth compared to WiFi); in some examples, the rendered frame is received after the first time but before the second time, such that the second device transmits the message before transmitting the rendered frame). In some examples, the rendered frame is received at the first device after (e.g., separately from) the message is received at the first device (e.g., the rendered frame is included in a message different from the message). In some examples, the message includes an identification of a version of the user interface element (in some examples, multiple versions of the user interface element are stored on the first device, such as a light version and a dark version of the user interface element). In some examples, the message includes a modification to the user interface element other than a position within the rendered frame (e.g., other than a position within the user interface element where the user interface element is to be placed) (in some examples, the modification includes a change in font, size, color, or opacity, such as to make it more readable to the user; in some examples, the modification is determined by the user device based on settings of the user device (e.g., accessibility settings), settings of the application, etc.; in some examples, text and font size are specified by the layout rather than in the message). In some examples, the message includes a position of the user interface element within the rendered frame (in some examples, the position refers to the position where the user interface element is to be placed relative to the rendered frame).

[0182] At 730, method 700 also includes rendering (e.g., locally rendering) a user interface element (e.g., 506i, 508f, 512q, 514m, or 518l) (in some examples, the user interface element is rendered before or after receiving the message; in some examples, rendering includes executing a computer program to generate an image from a 2D or 3D model; in some examples, the rendering includes modifying content stored by the first device (e.g., pre-rendered content); in some examples, the rendered content is a bitmap; in some examples, the content is stored by the first device prior to receiving the message and / or the rendered frame; in some examples, the rendering includes accessing a script for the user interface element and executing the script; in some examples, the rendering is performed by a graphics processing unit (GPU) of the first device).

[0183] At 740, method 700 also includes, before the second time, generating a combined frame (e.g., 506i, 508f, 512q, 514m, or 518l) by combining the user interface element with the rendered frame (in some examples, the combined frame is generated in response to receiving the message (in some examples, in response means occurring without any additional user input); in some examples, generating the combined frame includes rendering (e.g., rendering locally) the user interface element so that the rendering is not separate from the combination). In some examples, combining the user interface element with the rendered frame includes placing (e.g., underlaying or overlaying) the user interface element on the rendered frame (e.g., on the bottom (e.g., below) or on the top of the rendered frame) (in some examples, the user interface element is placed to appear in front of content included in the rendered frame; in some examples, the rendered frame includes an area without content for placing the user interface element; in some examples, the rendered frame includes a portion having a higher opacity than another portion of the rendered frame, so that the user interface element is placed behind the rendered frame in unison with the portion so as to be visible together with the rendered frame).

[0184] At 750, method 700 also includes outputting (eg, communicating to another component or device or displaying) the combined frame for display at a second time (eg, 506j, 508g, 512r, 514n, or 518m).

[0185] In some examples, method 700 also includes receiving rendered content (e.g., a bitmap or image, sometimes referred to as a rendering) corresponding to the user interface element (e.g., the rendered content is rendered by a device other than the first device, such as a second device) (e.g., 504j or 512g) (in some examples, the rendered content is a version of the user interface element that is different from the user interface element rendered at the first device), wherein the rendered content is received at the first device before the message is received at the first device (in some examples, the rendered content is received in response to the first device connecting to the second device; in some examples, the rendered content is received at the first device via a previous connection between the first device and the second device, which occurs before a connection used to receive the rendered frame; in some examples, the rendered content is received via a connection different from the connection used to receive the rendered frame (e.g., a different connection to the second device or a different connection to a device other than the second device); in some examples, rendering the user interface element includes modifying the rendered content.

[0186] In some examples, method 700 also includes receiving a script (e.g., a rendering script) for rendering a user interface element (e.g., 504j or 512g) (in some examples, the script includes one or more instructions that, when executed, render a user interface element (e.g., an image, including one or more pixel values)), wherein the script is received at the first device before the message is received at the first device (in some examples, the script is received in response to the first device connecting to the second device; in some examples, the script is received at the first device via a previous connection between the first device and the second device, which occurs before a connection used to receive the rendered frame; in some examples, the script is received via a connection different from the connection used to receive the rendered frame (e.g., a different connection to the second device or a different connection to a device other than the second device, such as one configured on the first device during manufacturing, received from a server via a firmware update or an over-the-air update, etc.); in some examples, rendering the user interface element includes executing the script at the first device).

[0187] In some examples, method 700 also includes displaying the combined frame (eg, 506j, 508g, 512r, 514n, or 518m) at a second time.

[0188] In some examples, method 700 also includes detecting first data (in some examples, the first device) via a sensor of the first device (in some examples, the sensor is a speedometer, a tachometer, an odometer, a trip meter, an oil pressure gauge, a coolant temperature gauge, a battery / charging system sensor, a low oil pressure sensor, an air bag sensor, a coolant overheat sensor, a hand brake sensor, a door ajar sensor, a high beam sensor, an onboard diagnostic indicator (e.g., a check engine sensor), a fuel gauge, a low fuel sensor, a hand brake indicator, a turn signal, an engine service indicator, a seat belt indicator, or a camera). The data includes position, speed, distance, oil pressure, coolant temperature, oil pressure amount, whether the air bag is functioning, whether the coolant is overheating, whether the sensor is on or off, whether the sensor is functioning, fuel amount, whether a particular turn signal is functioning, whether the seat belt is engaged, or an image), wherein rendering the user interface element is based on the first data (in some examples, the data is derived (e.g., determined) based on the first data, such as the first data is an input to a model, table, heuristic, rule-based system, etc.; in such examples, the data derived based on the first data is used when rendering the user interface element). In some examples, the user interface element is rendered in response to detecting the first data (in some examples, the user interface element is rendered in response to deriving (e.g., determining) data based on the first data).

[0189] In some examples, method 700 also includes, after receiving the rendered frame, receiving a second rendered frame (e.g., 508e) from the second device (in some examples, the second rendered frame is the same as or different from the rendered frame; in some examples, the second rendered frame is not received, but an identification of a previous frame (e.g., the rendered frame) received from the second device is received repeatedly); at a third time, receiving a second message from the second device including a fourth time (e.g., a future time, a time after the current time, or a future time) (in some examples, the message includes an indication other than the time when the combined frame should be displayed (e.g., the next possible time); in some examples, the third time is after the first time and / or the second time; in some examples, the fourth time is after the first time and / or the second time), wherein the fourth time is after the third time; and outputting (e.g., transmitting to another component or device or displaying) a frame corresponding to the second rendered frame (in some examples, the frame is the second rendered frame; in some examples, the frame is a combination of the second rendered frame and a user interface element; in some examples, the frame is the same as or different from the combined frame) for display at the fourth time.

[0190] Note that the above and below descriptions of method 600 (i.e., Figure 6 ), 800 (i.e. Figure 8 ), 900 (i.e., Fig. 9 ), 1000 (i.e., Fig.10 ) and 1100 (i.e., Fig.11 The details of the process described in ) can also be applied in a similar manner to Figure 7 Method 700. For example, method 700 optionally includes one or more of the characteristics of the various methods described above with reference to method 600, such as the combined frame generated at 740 of method 700 may be based on the layout from 660 of method 600. For another example, method 700 optionally includes one or more of the characteristics of the various methods described below with reference to method 800, such as the second time received at 720 of method 700 may be the time at which the combined frame is output at 860 of method 800. For another example, method 700 optionally includes one or more of the characteristics of the various methods described below with reference to method 900, such as the message received at 720 of method 700 may include the first frame transmitted at 940 of method 900 and / or an indication of the position. For another example, method 700 optionally includes one or more of the characteristics of the various methods described below with reference to method 1000, such as the message received at 720 of method 700 may be a user-defined preference received at 1020 of method 1000. For another example, method 700 optionally includes one or more of the characteristics of the various methods described below with reference to method 1100 , such as the combined frame output at 750 of method 700 may be the content displayed at 1140 of method 1100 .

[0191] Figure 8 8 is a flowchart illustrating a method 800 for controlling rendering by another device (eg, controlling rendering by another device). Some operations in method 800 are optionally combined, the order of some operations are optionally changed, and some operations are optionally omitted.

[0192] In some examples, method 800 is performed at a first device (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) (in some examples, the first device is a vehicle such as a computer system configured to display content on a display of the vehicle; in some examples, any combination of operating system modules and / or application modules may execute the steps of method 800).

[0193] At 810, method 800 includes receiving a rendered frame (e.g., a portion of a photo, image, video, or pixel information) (e.g., 506h, 508e, 512p, 514l, or 518k) from a second device different from the first device (in some examples, the second device is a user device, such as a portable electronic device; in some examples, the second device is logged into a user account that the first device is logged into or not logged into) (in some examples, the first device is wired or wirelessly connected to the second device, such as via Bluetooth or WiFi, and the rendered frame is transmitted through In some examples, the first device and the second device establish a streaming connection (e.g., 504d, 512a, or 518a) prior to receiving the rendered frame, wherein the streaming connection is configured to transmit data from at least one device (e.g., the second device) to another device (e.g., the first device); in some examples, the rendered frame is rendered by the second device (e.g., 506f, 508c, 512n, 514j, or 518i); in some examples, the rendered frame is rendered by a device other than the second device and provided to the second device for transmission to the first device). In some examples, the rendered frame is rendered by the second device (e.g., rendered locally). In some examples, a rendered frame received from a second device includes a placeholder portion (in some examples, the placeholder portion does not include content rendered by the second device; in some examples, the placeholder portion includes (in some examples, only includes) background content; in some examples, the placeholder portion does not include user interface elements rendered by the second device; in some examples, the placeholder portion does not include user interface elements other than the background content rendered by the second device; in some examples, the placeholder portion does not include user interface elements that are unique to the placeholder portion compared to the rest of the rendered frame (e.g., other than other placeholder portions), and wherein the combined frame includes the user interface element at a location corresponding to the placeholder portion (e.g., the location is the placeholder portion).

[0194] At 820, method 800 also includes receiving a message (e.g., 506h, 508e, 512p, 514l, or 518k) including data (e.g., instructions) from a second device (in some examples, the message is received via the same channel as the rendered frame; in some examples, the message is received via a different channel than the rendered frame, such as a channel configured to transmit a smaller amount of data (e.g., Bluetooth compared to WiFi); in some examples, the rendered frame is received after the first time but before the second time, such that the second device transmits the message before transmitting the rendered frame; in some examples, the data indicates a location). In some examples, the message is received at the first device before the rendered frame is received at the first device. In some examples, the data includes an indication (e.g., an identifier) ​​of a size (e.g., text size) of a user interface element (in some examples, the indication is a change in size). In some examples, the data includes an indication (e.g., an identifier) ​​of a location within the rendered frame (in some examples, the location corresponds to a user interface element, such that the location is a location on the rendered frame where the first device is to place the user interface element). In some examples, the data includes an indication (e.g., an identification) of opacity (in some examples, the indication is a change in opacity; in some examples, the opacity is about a user interface element). In some examples, the data includes an indication (e.g., an identification) of color (in some examples, the indication is a change in color; in some examples, the color is about a user interface element).

[0195] At 830, method 800 also includes determining a modification regarding a user interface element (e.g., 506i, 508f, 512q, 514m, or 518l) based on the data (in some examples, the determination does not involve when to render the user interface element; in some examples, the determination includes determining a size, color, opacity, or any combination thereof for the user interface element; in some examples, the determination includes determining that there will be no change in how the user interface element will be rendered but rather a change in the position of the user interface element within a rendered frame will be made based on the data; in some examples, the modification includes a change in text size or font).

[0196] At 840, method 800 also includes rendering (e.g., locally rendering) a user interface element (e.g., 506i, 508f, 512q, 514m, or 518l) based on the determination (in some examples, the rendering is performed in response to receiving a message; in some examples, the rendering includes executing a computer program to generate an image from a 2D or 3D model; in some examples, the rendering includes modifying content stored by the first device (e.g., pre-rendered content); in some examples, the rendered content is a bitmap; in some examples, the content is stored by the first device prior to receiving the message and / or the rendered frame; in some examples, the rendering includes accessing a script for the user interface element and executing the script; in some examples, the rendering is performed by a graphics processing unit (GPU) of the first device).

[0197] At 850, method 800 also includes generating a combined frame (e.g., 506i, 508f, 512q, 514m, or 518l) by combining the user interface element with the rendered frame (in some examples, the combined frame is generated at the second time but not before the second time; in some examples, generating the combined frame includes rendering (e.g., locally rendering) the user interface element such that the rendering is not separate from the combination).

[0198] At 860, method 800 also includes outputting (eg, communicating to another component or device or displaying) the combined frame for display (eg, 506j, 508g, 512r, 514n, or 518m).

[0199] In some examples, method 800 also includes receiving rendered content (e.g., a bitmap or image, sometimes referred to as a rendering) corresponding to the user interface element (e.g., the rendered content is rendered by a device other than the first device, such as a second device) (e.g., 504j or 512g) (in some examples, the rendered content is a version of the user interface element that is different from the user interface element rendered at the first device), wherein the rendered content is received at the first device before the message is received at the first device (in some examples, the rendered content is received in response to the first device connecting to the second device; in some examples, the rendered content is received at the first device via a previous connection between the first device and the second device, which occurs before a connection used to receive the rendered frame; in some examples, the rendered content is received via a connection different from the connection used to receive the rendered frame (e.g., a different connection to the second device or a different connection to a device other than the second device); in some examples, rendering the user interface element includes modifying the rendered content.

[0200] In some examples, method 800 also includes receiving a script (e.g., a rendering script) for rendering a user interface element (e.g., 504j or 512g) (in some examples, the script includes one or more instructions that, when executed, render a user interface element (e.g., an image, including one or more pixel values)), wherein the script is received at the first device before the message is received at the first device (in some examples, the script is received in response to the first device connecting to the second device; in some examples, the script is received at the first device via a previous connection between the first device and the second device, which occurs before a connection used to receive the rendered frame; in some examples, the script is received via a connection different from the connection used to receive the rendered frame (e.g., a different connection to the second device or a different connection to a device other than the second device); in some examples, rendering the user interface element includes executing the script at the first device).

[0201] In some examples, method 800 also includes, after receiving the rendered frame, receiving a second rendered frame (e.g., 508e) from the second device (in some examples, the second rendered frame is the same as or different from the rendered frame; in some examples, the second rendered frame is not received, but an identification of a previous frame (e.g., the rendered frame) received from the second device is received repeatedly); at a third time, receiving a second message (e.g., 508e) from the second device including a fourth time (e.g., a future time, a time after the current time, or a future time) (in some examples, the message includes a time when the combined frame should be displayed); in some examples, the third time is after the first time and / or the second time; in some examples, the fourth time is after the first time and / or the second time), where the fourth time is after the third time; and outputting (e.g., transmitting to another component or device or displaying) a frame (e.g., 508g) corresponding to the second rendered frame (in some examples, the frame is the second rendered frame; in some examples, the frame is a combination of the second rendered frame and a user interface element; in some examples, the frame is the same as or different from the combined frame) for display at the fourth time.

[0202] Note that the above and below descriptions of method 600 (i.e., Figure 6 ), 700 (i.e. Figure 7 ), 900 (i.e., Fig. 9 ), 1000 (i.e., Fig.10 ) and 1100 (i.e., Fig.11 The details of the process described in ) can also be applied in a similar manner to Figure 8Method 800. For example, method 800 optionally includes one or more of the characteristics of the various methods described above with reference to method 600, such as the modification determined at 830 of method 800 can be to the layout from 660 of method 600. For another example, method 800 optionally includes one or more of the characteristics of the various methods described above with reference to method 700, such as the data received at 820 of method 800 also includes the second time mentioned at 720 of method 700. For another example, method 800 optionally includes one or more of the characteristics of the various methods described below with reference to method 900, such as the modification determined at 830 of method 800 can be to the location corresponding to the indication of the location transmitted at 940 of method 900. For another example, method 800 optionally includes one or more of the characteristics of the various methods described below with reference to method 1000, such as the combined frame output at 860 of method 800 can be the second frame displayed at 1030 of method 1000. For another example, method 800 optionally includes one or more of the characteristics of the various methods described below with reference to method 1100 , such as the combined frame output at 860 of method 800 may be the content displayed at 1140 of method 1100 .

[0203] Fig. 9 is a flow chart illustrating a method 900 for rendering animation across multiple devices. Some operations in method 900 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0204] In some examples, method 900 is performed at a first device (e.g., computing system 100, device 200, user device 320, or user device 502) (in some examples, the first device is a user device, such as a portable electronic device; in some examples, the first device is logged into a user account that the second device is logged into or not logged into; in some examples, any combination of an operating system module, an application module, a remote device or system (e.g., via an application programming interface (API) call), etc.) may execute the steps of method 900).

[0205] At 910, method 900 includes determining an animation (e.g., 508a) to be displayed on a second device (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) that is different from the first device (in some examples, the animation spans at least three frames) (in some examples, the animation is determined after one or more frames (e.g., 506g) are transmitted to the second device; in some examples, the animation is determined after a connection is established between the first device and the second device (e.g., 504d); in some examples, the animation is determined based on content determined by the first device to be displayed on the second device (e.g., 508a). In some examples, the first device is a user device and the second device is a vehicle.

[0206] At 920, method 900 also includes rendering (e.g., locally rendering) a first frame (e.g., 508c) according to the animation (in some examples, the rendering is performed in response to determining the animation; in some examples, the rendering includes executing a computer program to generate an image from a 2D or 3D model; in some examples, the rendering includes modifying content stored by the first device (e.g., pre-rendered content); in some examples, the rendered content is a bitmap; in some examples, the rendering includes accessing a script for a user interface element and executing the script; in some examples, the rendering is performed by a graphics processing unit (GPU) of the first device). In some examples, the first frame includes a placeholder image at the location.

[0207] At 930, method 900 also includes determining, based on an animation, a position within the first frame to be updated with a user interface element rendered by a second device (e.g., 508b) (in some examples, the user interface element is a vehicle instrument) (in some examples, the animation is determined after transmitting a frame to be displayed by the second device; in some examples, based on the animation: a placeholder in (1) a first position in the first frame and (2) a second position in the second frame, wherein the second position is different from the first position, and wherein the second frame is configured to be displayed after the first frame (e.g., subsequent to the first frame, such as immediately after the first frame); in some examples, the position is determined before rendering the first frame; in some examples, the position is used to render the first frame; in some examples, the position is metadata of the first frame).

[0208] At 940, method 900 also includes transmitting the first frame and an indication of the position (e.g., 508d) to the second device (in some examples, the indication is metadata of the first frame; in some examples, the indication is separate from the first frame; in some examples, the method is performed by an operating system of the first device; in some examples, the method is performed by an application executing on the first device (e.g., an application downloaded to the first device) rather than the operating system; in such examples, the operating system or the application of the first device may determine the animation; in some examples, some of the steps of the method are performed by an application executing on the first device that calls one or more operating system APIs (e.g., the application may call a single API to perform the determination and rendering steps) (e.g., the application may call a first API for determination and call a second API for rendering); in some examples, the application executing on the first device calls different applications to determine the position; in some examples, the application itself determines the position).

[0209] In some examples, method 900 also includes determining a time to display the first frame based on characteristics associated with the second device; and transmitting an indication of the time to the second device (e.g., 508d).

[0210] In some examples, method 900 also includes determining a current layout of a display of the second device, wherein determining the animation is based on the current layout (eg, 508a).

[0211] In some examples, method 900 also includes determining a modification to a user interface to be rendered by the second device at the location (e.g., 508a or 508b); and transmitting an indication of the modification to the second device (e.g., 508d). In some examples, the modification includes a change in a characteristic of a user interface element selected from the group consisting of location, opacity, color, font, size, and shape.

[0212] In some examples, method 900 also includes, before determining the animation, establishing a streaming connection with a second device to be used to transmit a plurality of frames corresponding to the animation (eg, 504d).

[0213] In some examples, method 900 also includes rendering (e.g., locally rendering) a second frame (e.g., 508c) that is different from the first frame based on the animation (in some examples, the rendering is performed in response to determining the animation; in some examples, the rendering includes executing a computer program to generate an image from a 2D or 3D model; in some examples, the rendering includes modifying content stored by the first device (e.g., pre-rendered content); in some examples, the rendered content is a bitmap; in some examples, the rendering includes accessing a script for a user interface element and executing the script; in some examples, the rendering is performed by a graphics processing unit (GPU) of the first device); and transmitting the second frame to the second device.

[0214] In some examples, method 900 also includes determining a second location (e.g., 508a or 508b) within the first frame to be updated with a user interface element rendered by the second device based on an animation (in some examples, the user interface element is a vehicle instrument) (in some examples, the animation is determined after transmitting a frame to be displayed by the second device; in some examples, based on the animation: a placeholder in (1) a first location in the first frame and (2) a second location in the second frame, wherein the second location is different from the first location, and wherein the second frame is configured to be displayed after the first frame (e.g., subsequent to the first frame, such as immediately after the first frame); in some examples, the location is determined prior to rendering the first frame; in some examples, the location is used to render the first frame; in some examples, the location is metadata of the first frame), wherein the second location is different from the first location, and wherein the second user interface element is different from the user interface element; and transmitting an indication of the second location to the second device (e.g., 508d).

[0215] Note that the above and below descriptions of method 600 (i.e., Figure 6 ), 700 (i.e. Figure 7 ), 800 (i.e. Figure 8 ), 1000 (i.e., Fig.10 ) and 1100 (i.e., Fig.11 The details of the process described in ) can also be applied in a similar manner to Fig. 9 Method 900. For example, method 900 optionally includes one or more of the characteristics of the various methods described above with reference to method 600, such as the first frame transmitted at 940 of method 900 can be the content transmitted at 670 of method 600. For another example, method 900 optionally includes one or more of the characteristics of the various methods described above with reference to method 700, such as the animation determined at 920 of method 900 can be used to determine the second time received at 720 of method 700. For another example, method 900 optionally includes one or more of the characteristics of the various methods described above with reference to method 800, such as the animation determined at 920 of method 900 can be used to generate the data received at 820 of method 800. For another example, method 900 optionally includes one or more of the characteristics of the various methods described below with reference to method 1000, such as the animation determined at 920 of method 900 can be used to determine the user-defined preference received at 1020 of method 1000. For another example, method 900 optionally includes one or more of the characteristics of the various methods described below with reference to method 1100 , such as the frame rendered according to the animation at 920 of method 900 may be the content displayed in the first layout at 1110 of method 1100 .

[0216] Fig.101 is a flow chart illustrating a method 1000 for customizing vehicle controls when connected to a user device (e.g., in response to a connection, sometimes referred to as a connection being made). Some operations in method 1000 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0217] In some examples, method 1000 is performed at a computer system of a vehicle (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) that communicates with a display component (in some examples, any combination of operating system modules and / or application modules may execute the steps of method 1000).

[0218] At 1010, method 1000 includes displaying a first frame (e.g., a display frame) (e.g., 504c) via a display component (in some examples, the first frame is an image; in some examples, the first frame is a frame in a frame train animation frame), including a first version of a vehicle instrument (in some examples, the vehicle instrument is a speedometer, a tachometer, an odometer, a trip meter, an oil pressure gauge, a coolant temperature gauge, a battery / charging system sensor, a low oil pressure sensor, an air bag sensor, a coolant overheat sensor, a handbrake sensor, a door ajar sensor, a high beam sensor, an onboard diagnostic indicator (e.g., a check engine sensor), a fuel gauge, a low fuel sensor, a handbrake indicator, a turn signal, an engine service indicator, or a seat belt indicator; in some examples, the vehicle instrument is a user interface element that indicates a status of a component of the vehicle; in some examples, the vehicle instrument is a user interface element that indicates data detected by a sensor of the vehicle), wherein the first version has a first appearance in the first frame (in some examples, the first appearance relates to a color, shape, or position of the first version within the first frame). In some examples, the first version is digital or analog, and wherein the second version is not the same version (e.g., digital or analog version) as the first version (in some examples, the first version is digital and the second version is analog; in some examples, the first version is analog and the second version is digital; in some examples, the second version is defined in user-defined preferences).

[0219] At 1020, method 1000 also includes, while displaying the first version: connecting to a user device (e.g., establishing a first connection between the user device and the vehicle) (e.g., 504d) (in some examples, the connection is via a wired (e.g., a cable connected to a USB port of the vehicle and a Lightning port of the user device) or a wireless (e.g., Bluetooth or WiFi) channel); and, without further user input after connecting to the user device, receiving user-defined preferences for display of vehicle instruments from the user device (e.g., 504j, 506c, 506h, 512g, 512k, 514g, or 518f) (in some examples, the user device transmits the user-defined preferences to the vehicle in response to connecting to the vehicle; in some examples, the user-defined preferences include text size or font).

[0220] At 1030, method 1000 also includes: rendering a second version of the vehicle instrument (e.g., 506i, 508f, 512q, 514m, or 518l) based on user-defined preferences (in some examples, the rendering is based on the user-defined preferences; in some examples, the second version is the same as the first version; in some examples, the second version is different from the first version; in some examples, the rendering is not based on the user-defined preferences; in some examples, the second version is rendered based on data received from sensors of the vehicle, such as sensors for detecting the speed of the vehicle); and displaying a second frame including the second version via a display component, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance (in some examples, the second frame is rendered based on the user-defined preferences; in some examples, rendering the second version and displaying the second frame are performed after connecting to the user device without any additional user input; in some examples, the second appearance relates to a color, shape, or position of the second version within the second frame; in some examples, displaying the second frame includes changing from the first frame to the second frame). In some examples, the second version includes a different color than the first version (in some examples, the different color is defined in user-defined preferences). In some examples, the second version is located at a different position within the frame than the first version (in some examples, the position of the second version is defined in user-defined preferences).

[0221] In some examples, method 1000 also includes, before rendering the second version, transmitting to the user device an identifier of a set of one or more layouts stored by the vehicle (e.g., 512b) (in some examples, the set of one or more layouts was received by the vehicle from the user device when the vehicle and the user device were previously connected (e.g., 504j); in some examples, the set of one or more layouts is stored separately by both the vehicle and the user device); based on determining that the set of one or more layouts is outdated (in some examples, determining that the set of one or more layouts is outdated is performed by the user device), receiving a new set of one or more layouts (e.g., 512g) from the user device (in some examples, the new set includes at least one different layout from the set), wherein after receiving the new set, rendering the second version (e.g., 512q, 514m, 516f, or 518l) and displaying the second frame (e.g., 512r, 514n, 516g, or 518m) are based on a layout in the new set of one or more layouts (in some examples, the layout is selected by the user device from the new set) selection; in some examples, the second version is modified based on the layout of the new set; in some examples, the second frame is arranged based on the layout of the new set (for example, the position of one or more user interface elements in the second frame is defined in the layout of the new set); and based on determining that the set of one or more layouts is up to date (in some examples, determining that the set of one or more layouts is up to date is performed by the user device), receiving an indication that the set is up to date (for example, 512k or 512p); wherein after receiving the indication that the set is up to date, rendering the second version (for example, 512q, 514m, 516f, or 518l) and displaying the second frame (for example, 512r, 514n, 516g, or 518m) are based on the layout of the set of one or more layouts (in some examples, the layout is selected by the user device from the set; in some examples, the second version is modified based on the layout of the set; in some examples, the second frame is arranged based on the layout of the set (for example, the position of one or more user interface elements in the second frame is defined in the layout of the set).

[0222] In some examples, method 1000 also includes detecting a disconnection of a user device (e.g., 510a or 516b) (e.g., disconnecting a connection between the user device and the vehicle), wherein a third version of the vehicle instrument is displayed via the display component immediately before detecting the disconnection of the user device (in some examples, the third version is the second version); and after detecting the disconnection of the user device, displaying a fourth version of the vehicle instrument (e.g., 510d or 516g) (in some examples, the fourth version is displayed in response to detecting the disconnection of the user device), wherein the fourth version is different from the third version (in some examples, the fourth version is the first version).

[0223] In some examples, method 1000 also includes detecting user input by a sensor of the vehicle when connected to the user device (e.g., when the vehicle is connected to the user device) (e.g., 514a); in response to detecting the user input, transmitting an indication of the user input to the user device (e.g., 514b); after transmitting the indication of the user input, receiving an indication of a modification corresponding to the vehicle instrument from the user device (e.g., 514g or 514l), wherein the modification is determined based on the indication of the user input (in some examples, the modification is determined by the user device; in some examples, the modification causes a modification to the vehicle instrument; in some examples, the modification causes a different placement of the vehicle instrument within the displayed frame); rendering a third frame including the vehicle instrument based on the indication of the modification (e.g., 514m); and displaying the third frame via the display component (e.g., 514n). In some examples, the sensor is a physical button (in some examples, the physical button is embedded in the vehicle), and wherein the user input corresponds to activation of the physical button. In some examples, the sensor is a touch-sensitive display, and wherein the user input corresponds to touch input detected via the touch-sensitive display (in some examples, the touch input corresponds to selection of an affordance (e.g., a user interface element, such as a button) displayed by the touch-sensitive display). In some examples, the sensor is a microphone, and wherein the user input corresponds to voice input detected via the microphone (in some examples, the voice input corresponds to an audible request to change the layout; in some examples, the voice input involves a virtual assistant; in some examples, the voice input causes another application (e.g., a virtual assistant application) to execute and return control for changing the layout after the other application determines an output (e.g., the user input); in some examples, the user input corresponds to a gesture detected via a camera.

[0224] Note that the above and below descriptions of method 600 (i.e., Figure 6 ), 700 (i.e. Figure 7 ), 800 (i.e. Figure 8 ), 900 (i.e., Fig. 9 ) and 1100 (i.e., Fig.11 The details of the process described in ) can also be applied in a similar manner to Fig.10Method 1000. For example, method 1000 optionally includes one or more of the characteristics of the various methods described above with reference to method 600, such as the second version rendered at 1030 of method 1000 may be placed at a specific position in the frame based on the layout transmitted at 650 of method 600. For another example, method 1000 optionally includes one or more of the characteristics of the various methods described above with reference to method 800, such as the second version rendered at 1030 of method 1000 may correspond to the user interface element rendered at 730 of method 700. For another example, method 1000 optionally includes one or more of the characteristics of the various methods described above with reference to method 900, such as the second version rendered at 1030 of method 1000 may correspond to the user interface element rendered at 840 of method 800. For another example, method 1000 optionally includes one or more of the characteristics of the various methods described below with reference to method 1000, such as the user-defined preference received at 1020 of method 1000 may be the indication of the location transmitted at 940 of method 900. For another example, method 900 optionally includes one or more of the characteristics of the various methods described below with reference to method 1100, such as the second frame displayed at 1030 of method 1000 may be the content displayed at 1110 of method 1100 in the first layout.

[0225] Fig.11 is a flow chart illustrating a method 1100 for changing a layout used during synchronous rendering in the event of a connection loss. Some operations in method 1100 are optionally combined, the order of some operations is optionally changed, and some operations are optionally omitted.

[0226] In some examples, method 1100 is performed at a first device (e.g., computing system 100, device 200, vehicle 302, or vehicle 500) (in some examples, the first device is a vehicle such as a computer system configured to display content on a display of the vehicle; in some examples, any combination of operating system modules and / or application modules may execute the steps of method 1100).

[0227] At 1110, method 1100 includes receiving an input signal (e.g., 516a) (in some examples, the input signal is a message transmitted by a component of the first device; in some examples, the input signal represents an indication of user input to the component of the first device; in some examples, the input signal is received from a different device, such as a server, remote from the first device) while displaying content (e.g., 514n) in a first layout (in some examples, the layout represents positions where one or more elements are placed in a frame displayed by a first device; in some examples, the display is on a display component of the first device), wherein the first layout is displayed by a second device (e.g., computing system 10 ... Device 200, user device 320, or user device 502) (in some examples, the second device is a user device, such as a portable electronic device; in some examples, the second device is logged into a user account that the first device is logged into or not logged into) selects (in some examples, the connection is included in a pairing process between the first device and the second device (e.g., 504d); in some examples, the connection occurs after the pairing process; in some examples, the connection is via a wired or wireless connection; in some examples, the first layout is from a set of layouts; in some examples, the content is a combination of content rendered by the first device and content rendered by the second device; in some examples, the content includes one or more images).

[0228] At 1120, method 1100 also includes, in response to receiving the input signal, attempting to transmit a first message (e.g., 516b) indicating the input signal to the second device (in some examples, the first message is attempted to be transmitted via a first channel (e.g., 504d or 512a) established prior to displaying the content in the first layout; in some examples, the first message includes an indication of the input signal; in some examples, the first message includes an identification of the component that detected the input signal).

[0229] At 1130, method 1100 also includes, after attempting to transmit the first message: determining, based on a determination that the second device has failed to respond to the first message (e.g., 516c) (in some examples, determining that the second device has failed to respond to the first message includes the first device not receiving a confirmation message from the second device, the confirmation message indicating that the second device has received the first message; in some examples, determining that the second device has failed to respond to the first message includes determining that a predefined amount of time has elapsed since attempting to transmit the first message without receiving a response from the second device; in some examples, determining that the second device has failed to respond to the first message includes determining that a channel used to transmit messages between the first device and the second device is no longer connected): determining, based on an input signal, to change from the first layout to the second layout (e.g., 516d) (in some examples, determining to change to the second layout is not based on the first message; in some examples, the second layout The layout is from a set of layouts; in some examples, determining to change to the second layout is not based on a message received from the second device after the first message is transmitted; in some examples, the second layout is stored on the first device); and displaying content in the second layout (e.g., 516g) (in some examples, the display is on a display component of the first device such as a display that displays content in the first layout; in some examples, the content in the second layout includes content different from the content in the first layout; in some examples, the content in the second layout is a combination of content rendered by the first device and content rendered by the second device; in some examples, the content in the second layout includes only content rendered by the second device; in some examples, the content in the second layout includes only content rendered by the second device or stored by the second device before the first message is transmitted; in some examples, the content includes one or more images).

[0230] At 1140, method 1100 also includes, after attempting to transmit the first message: based on determining that the transmission of the first message is successful (e.g., 518a or 518f): determining to change from the first layout to a third layout based on a second message received from the second device (e.g., 518f) (in some examples, the second message includes an indication of the third layout; in some examples, determining to change to the third layout is not based on an input signal; in some examples, the third layout is from a set of layouts); and displaying content in the third layout (e.g., 518m) (in some examples, the content in the third layout includes content different from the content in the first layout; in some examples, the content in the third layout is different from the content in the first layout). The content in the third layout includes the same content as the content in the second layout; in some examples, the content in the third layout is a combination of content rendered by the first device and content rendered by the second device (e.g., 518l); in some examples, the content in the third layout includes content rendered by the first device, and the content rendered by the first device is also included in the content in the second layout, so that one difference between the content in the second layout and the content in the third layout is that the content in the third layout includes content rendered by the second device, and the content rendered by the second device is not included in the content in the second layout; in some examples, the content in the third layout includes one or more images).

[0231] In some examples, method 1100 also includes, based on determining that the second device failed to respond to the first message, the content in the second layout includes placeholder content and does not include content received by the first device after the first device transmits the first message (in some examples, the content in the second layout includes content received from the second device before attempting to transmit the first message, such as content received when the second layout is received or when connected to the second device (e.g., 504j, 512g, or 514l); in some examples, the placeholder content is included at the first position), based on determining that transmission of the first message was successful, the content in the third layout includes content received by the first device after the first device transmits the first message (in some examples, the content received by the first device after the first device transmits the first message is included at the first position, and when transmission of the first message fails, the placeholder content is located at the first position; in some examples, the content in the third layout does not include placeholder content), and the third layout is the same as the second layout. In some examples, the content in the second layout includes media (e.g., an image or video) captured by a camera of the first device, the content in the third layout includes the media, the content in the third layout also includes additional content (which, in some examples, is received and / or rendered by the second device), and the content in the second layout does not include the additional content.

[0232] In some examples, method 1100 also includes, at a first time (in some examples, the first time is after displaying the content in a third layout): based on determining that the first device has received the first content from the second device for display at the first time, displaying the first content (in some examples, the first content is included in a message including an indication of the first time); and based on determining that the first device has not received the content from the second device for display at the first time, displaying second content (e.g., old or previous content) received from the second device for display at a second time, wherein the second time is before the first time (in some examples, the second content is different from the first content; in some examples, the second content is included in a message including a first indication of when to display the second content, wherein the first indication includes an indication of the second time without including an indication corresponding to the first time (e.g., the first indication does not relate to a time frame including the first time).

[0233] In some examples, method 1100 also includes, at a third time (in some examples, the first time is after displaying the content in the third layout; in some examples, the third time is after the first time): based on determining that the first device has received third content from the second device for display at the third time, displaying third content (in some examples, the third content is included in a message including an indication of the third time); and based on determining that the first device has not received content from the second device for display at the third time, displaying fourth content (e.g., placeholder content) configured to be displayed when the connection between the first device and the second device is not working (in some examples, the fourth content is rendered by the first device or the second device).

[0234] In some examples, method 1100 also includes, after receiving the input signal: displaying a user interface element (in some examples, the user interface element is a vehicle instrument); after displaying the user interface element, receiving fifth content from a second device (in some examples, the fifth content is in a specific layout); generating combined content by combining the fifth content with the user interface element (in some examples, the combined content is generated using a specific layout); and displaying the combined content (in some examples, the combined content replaces display of the user interface element).

[0235] In some examples, method 1100 also includes initiating rendering of the user interface element (e.g., rendering locally) (in some examples, the rendering includes executing a computer program to generate an image from a 2D or 3D model; in some examples, the rendering includes modifying content stored by the first device (e.g., pre-rendered content); in some examples, the rendered content is a bitmap; in some examples, the rendering includes accessing a script for the user interface element and executing the script; in some examples, the rendering is performed by a graphics processing unit (GPU) of the first device); after initiating rendering of the user interface element (in some examples, after rendering the user interface element; in some examples, before ending rendering of the user interface element), receiving sixth content from the second device; generating a combined frame by combining the user interface element with the sixth content; and outputting (e.g., transmitting to another component or device or displaying) the combined frame for display.

[0236] Note that the above and below descriptions of method 600 (i.e., Figure 6 ), 700 (i.e. Figure 7 ), 800 (i.e. Figure 8 ), 900 (i.e., Fig. 9 ) and 1000 (i.e., Fig.10 The details of the process described in ) can also be applied in a similar manner to Fig.11 Method 1100. For example, method 1100 optionally includes one or more of the characteristics of the various methods described above with reference to method 600, such as the first layout and the second layout at 1130 of method 600 may be included in the set of one or more layouts transmitted at 640 of method 1100. For another example, method 1100 optionally includes one or more of the characteristics of the various methods described above with reference to method 700, such as the content in the first layout at 1110 of method 700 may correspond to the combined frame output at 750 of method 1100. For another example, method 1100 optionally includes one or more of the characteristics of the various methods described above with reference to method 800, such as the content in the first layout at 1110 of method 800 may correspond to the combined frame output at 860 of method 1100. For another example, method 1100 optionally includes one or more of the characteristics of the various methods described above with reference to method 900, such as the content in the first layout at 1110 of method 900 may be the first frame transmitted at 940 of method 1100. For another example, method 1100 optionally includes one or more of the characteristics of the various methods described above with reference to method 1000 , such as the content in the first layout at 1110 of method 1100 may be the second frame displayed at 1030 of method 1000 .

[0237] In some examples, frames transmitted from one device to another do not include placeholder portions, but rather the placeholder locations are used locally by the transmitting device to render the frames. In some examples, one or more layouts or assets having a layout are (1) configured on a device during manufacturing (e.g., at a factory), (2) configured as part of a firmware update, (3) configured with an over-the-air (OTA) update, or (4) configured by another device. In such examples, one or more layouts or assets having a layout may be generated by the manufacturer of either device (e.g., according to a standard). In some examples, the voice input that causes the change in layout includes a specific question, such as "set to sport mode," "I cannot read the instruments," "show me my fuel gage," or "let me know when I need to exit the freeway to charge the car." In some examples, the change in layout is caused when the device's sensors detect a specific state (e.g., the fuel level is low and the vehicle changes to navigate to a charging station). In some examples, the virtual assistant is running on the user device and / or vehicle such that if the user device is disconnected, some virtual assistant intelligence / functionality is still operable (e.g., navigate me to a charging station).

[0238] For the purpose of explanation, the foregoing description is described by reference to specific examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible based on the above teachings. These examples are selected and described in order to explain the principles of these technologies and their practical applications. Others skilled in the art can thus utilize these technologies and various examples with various modifications suitable for the specific purposes contemplated.

[0239] Although the disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the disclosure and examples defined by the claims.

[0240] As described above, one aspect of the present technology is to collect and use data from various sources to improve the rendering of content. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to identify a specific person and / or a specific location. Such personal information data may include preferences of a person, data stored on a personal device, an image of a person, an image of a location, a reference to a person's current location, or any other identification or personal information.

[0241] The disclosure contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other uses of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside of these legal purposes. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. Therefore, different privacy practices may be maintained for different types of personal data in each country.

[0242] Furthermore, the intention of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting data collection and data deletion.

Claims

1. A method for establishing a collection of one or more layouts on a plurality of devices for use with displaying content on one of the devices, the method comprising: At the first device: connecting via a first connection to a second device different from the first device; receiving, via the first connection, an identification associated with the second device; upon receiving the identification associated with the second device, obtaining a set of one or more layouts for a display of the second device using the identification; storing said collection of one or more layouts; sending, to the second device, the set of one or more layouts for use with the display of the second device; after sending the set, determining content for display via the display of the second device based on a layout in the set of one or more layouts stored at the first device; as well as A message corresponding to the content is sent to the second device.

2. The method according to claim 1, further comprising: At the first device: When the first device is connected to the second device: receiving an indication of a user input; in response to receiving the indication of the user input, determining to change a layout currently being used by the second device to a new layout; as well as A message indicating the new layout is sent to the second device. 3 . The method of claim 2 , wherein the user input corresponds to activation of a physical button of the second device. The method of claim 2 , wherein the user input corresponds to a touch input detected via a touch-sensitive display of the second device. The method of claim 2 , wherein the user input corresponds to a user input detected via a sensor of the first device. The method of claim 2 , wherein the user input corresponds to voice input detected via a microphone.

7. The method according to any one of claims 1 to 6, wherein the set of one or more layouts is a first version, and wherein the method further comprises: At the first device: After the first connection is disconnected: connecting to the second device via a second connection; receiving, via the second connection, a second identifier associated with the second device; receiving, via the second connection, an identification of a current version associated with the set of one or more layouts; as well as According to determining that the current version is the same as the first version: determining, based on a particular layout in the set of one or more layouts, second content for display via the display of the second device; as well as A second message corresponding to the second content is sent to the second device.

8. The method according to claim 7, further comprising: At the first device: Based on determining that the current version is different from the first version, a new set of one or more layouts is sent to the second device, wherein the new set is the current version, and wherein the new set is different from the set of one or more layouts.

9. The method of claim 7, wherein the specific layout is a layout last used by the first device for the second device.

10. The method according to any one of claims 1 to 9, wherein the layout comprises a definition of an initial position of at least one user interface element.

11. The method according to any one of claims 1 to 10, further comprising: At the first device: In addition to sending the set of one or more layouts, a script for rendering a user interface element is sent to the second device.

12. The method according to any one of claims 1 to 11, further comprising: At the first device: In addition to sending the set of one or more layouts, rendered content is sent to the second device.

13. The method of any one of claims 1 to 12, wherein the set of one or more layouts comprises a plurality of layouts.

14. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

15. A first device, comprising: one or more processors; and A memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

16. A first device, comprising: Device for carrying out the method according to any one of claims 1 to 13.

17. A computer program product, comprising one or more programs, the one or more programs being configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

18. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: connecting via a first connection to a second device different from the first device; receiving, via the first connection, an identification associated with the second device; upon receiving the identification associated with the second device, obtaining a set of one or more layouts for a display of the second device using the identification; storing said collection of one or more layouts; sending, to the second device, the set of one or more layouts for use with the display of the second device; after sending the set, determining content for display via the display of the second device based on a layout in the set of one or more layouts stored at the first device; as well as A message corresponding to the content is sent to the second device.

19. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: connecting via a first connection to a second device different from the first device; receiving, via the first connection, an identification associated with the second device; upon receiving the identification associated with the second device, obtaining a set of one or more layouts for a display of the second device using the identification; storing said collection of one or more layouts; sending, to the second device, the set of one or more layouts for use with the display of the second device; after sending the set, determining content for display via the display of the second device based on a layout in the set of one or more layouts stored at the first device; as well as A message corresponding to the content is sent to the second device.

20. A first device, comprising: means for connecting via a first connection to a second device different from said first device; means for receiving, via the first connection, an identification associated with the second device; upon receiving the identification associated with the second device, means for using the identification to obtain a set of one or more layouts for a display of the second device; means for storing said collection of one or more layouts; means for sending to said second device said set of one or more layouts for use with said display of said second device; means for determining, after sending the set, content for display via the display of the second device based on a layout in the set of one or more layouts stored at the first device; and Means for sending a message corresponding to the content to the second device.

21. A computer program product, the computer program product comprising one or more programs, the one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: connecting via a first connection to a second device different from the first device; receiving, via the first connection, an identification associated with the second device; upon receiving the identification associated with the second device, obtaining a set of one or more layouts for a display of the second device using the identification; storing said collection of one or more layouts; sending, to the second device, the set of one or more layouts for use with the display of the second device; after sending the set, determining content for display via the display of the second device based on a layout in the set of one or more layouts stored at the first device; as well as A message corresponding to the content is sent to the second device.

22. A method for combining user interface elements rendered on different devices, the method comprising: At the first device: receiving a rendered frame from a second device different from the first device; receiving, at a first time, a message including a second time from the second device, wherein the second time is after the first time; Rendering user interface elements; before the second time, generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display at the second time.

23. The method of claim 22, wherein the rendered frame is rendered by the second device.

24. The method of any one of claims 22 to 23, wherein the rendered frame is received at the first device after the message is received at the first device.

25. The method according to any one of claims 22 to 24, further comprising: At the first device: Rendered content corresponding to the user interface element is received, wherein the rendered content is received at the first device before the message is received at the first device.

26. The method according to any one of claims 22 to 25, further comprising: At the first device: A script is received for rendering the user interface element, wherein the script is received at the first device before the message is received at the first device.

27. The method of any one of claims 22 to 26, wherein combining the user interface element with the rendered frame comprises placing the user interface element on the rendered frame.

28. The method of any one of claims 22 to 27, wherein the rendered frame received from the second device includes a placeholder portion, and wherein the combined frame includes the user interface element at a position corresponding to the placeholder portion.

29. The method according to any one of claims 22 to 28, further comprising: At the first device, the combined frame is displayed at the second time.

30. The method according to any one of claims 22 to 29, further comprising: At the first device: First data is detected via a sensor of the first device, wherein rendering the user interface element is based on the first data.

31. The method of claim 30, wherein the user interface element is rendered in response to detecting the first data.

32. A method according to any one of claims 22 to 31, wherein the message includes an identification of a version of the user interface element.

33. A method according to any one of claims 22 to 32, wherein the message comprises a modification to the user interface element other than to a position within the rendered frame.

34. A method according to any one of claims 22 to 33, wherein the message comprises a position of the user interface element within the rendered frame.

35. The method according to any one of claims 22 to 34, further comprising: At the first device: After receiving the rendered frame, receiving a second rendered frame from the second device; receiving, at a third time, from the second device, a second message including a fourth time, wherein the fourth time is after the third time; and A frame corresponding to the second rendered frame is output for display at the fourth time.

36. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for executing the method according to any one of claims 22 to 35.

37. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 22 to 35.

38. A first device, comprising: Device for carrying out the method according to any one of claims 22 to 35.

39. A computer program product comprising one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for performing the method according to any one of claims 22 to 35.

40. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving a rendered frame from a second device different from the first device; receiving, at a first time, from the second device, a message including a second time, wherein the second time is after the first time; Rendering user interface elements; before the second time, generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display at the second time.

41. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a rendered frame from a second device different from the first device; receiving, at a first time, from the second device, a message including a second time, wherein the second time is after the first time; Rendering user interface elements; before the second time, generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display at the second time.

42. A first device, comprising: means for receiving a rendered frame from a second device different from the first device; means for receiving, at a first time, from the second device, a message including a second time, wherein the second time is after the first time; means for rendering user interface elements; before said second time, means for generating a combined frame by combining said user interface element with said rendered frame; and means for outputting the combined frame for display at the second time.

43. A computer program product, comprising one or more programs, the one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving a rendered frame from a second device different from the first device; receiving, at a first time, from the second device, a message including a second time, wherein the second time is after the first time; Rendering user interface elements; before the second time, generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display at the second time.

44. A method for combining user interface elements rendered on different devices, the method comprising: At the first device: receiving a rendered frame from a second device different from the first device; receiving a message including data from the second device; determining a modification to a user interface element based on the data; Based on the determination, rendering the user interface element; generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display.

45. The method of claim 44, wherein the rendered frame is rendered by the second device.

46. ​​A method according to any one of claims 44 to 45, wherein the message is received at the first device before the rendered frame is received at the first device.

47. The method according to any one of claims 44 to 46, further comprising: At the first device: Rendered content corresponding to the user interface element is received, wherein the rendered content is received at the first device before the message is received at the first device.

48. The method according to any one of claims 44 to 47, further comprising: At the first device: A script is received for rendering the user interface element, wherein the script is received at the first device before the message is received at the first device.

49. A method according to any one of claims 44 to 48, wherein the rendered frame received from the second device includes a placeholder portion, and wherein the combined frame includes the user interface element at a position corresponding to the placeholder portion.

50. A method according to any one of claims 44 to 49, wherein the data comprises an indication of a size of the user interface element.

51. A method according to any one of claims 44 to 50, wherein the data comprises an indication of a position within the rendered frame.

52. A method according to any one of claims 44 to 51 , wherein the data comprises an indication of opacity.

53. A method according to any one of claims 44 to 52, wherein the data comprises an indication of a colour.

54. The method according to any one of claims 44 to 53, further comprising: At the first device: After receiving the rendered frame, receiving a second rendered frame from the second device; receiving, at a third time, from the second device, a second message including a fourth time, wherein the fourth time is after the third time; and A frame corresponding to the second rendered frame is output for display at the fourth time.

55. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for executing the method according to any one of claims 44 to 54.

56. An apparatus comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 44 to 54.

57. An apparatus comprising: Apparatus for carrying out the method according to any one of claims 44 to 54.

58. A computer program product comprising one or more programs configured to be executed by one or more processors of a device, the one or more programs comprising instructions for performing the method according to any one of claims 44 to 54.

59. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving a rendered frame from a second device different from the first device; receiving a message including data from the second device; determining a modification to a user interface element based on the data; Based on the determination, rendering the user interface element; generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display.

60. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving a rendered frame from a second device different from the first device; receiving a message including data from the second device; determining a modification to a user interface element based on the data; Based on the determination, rendering the user interface element; generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display.

61. A first device, comprising: means for receiving a rendered frame from a second device different from the first device; means for receiving a message comprising data from the second device; means for determining a modification to a user interface element based on the data; means for rendering said user interface element based on said determination; means for generating a combined frame by combining the user interface element with the rendered frame; and means for outputting said combined frame for display.

62. A computer program product, comprising one or more programs, the one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving a rendered frame from a second device different from the first device; receiving a message including data from the second device; determining a modification to a user interface element based on the data; Based on the determination, rendering the user interface element; generating a combined frame by combining the user interface element with the rendered frame; as well as The combined frame is output for display.

63. A method for changing a layout used when displaying content, the method comprising: At the first device: receiving an input signal while displaying content in a first layout, wherein the first layout is selected by a second device different from the first device; In response to receiving the input signal, attempting to send a first message indicative of the input signal to the second device; and After trying to send the first message: Based on determining that the second device fails to respond to the first message: determining to change from the first layout to a second layout based on the input signal; and displaying the content in the second layout; and According to determining that sending the first message is successful: determining to change from the first layout to a third layout based on a second message received from the second device; and The content is displayed in the third layout.

64. The method of claim 63, wherein: based on determining that the second device failed to respond to the first message, the content in the second layout includes placeholder content and does not include content received by the first device after the first device sent the first message, Based on determining that the first message is sent successfully, the content in the third layout includes content received by the first device after the first device sends the first message, and The third layout is the same as the second layout.

65. A method according to any one of claims 63 to 64, wherein the first device is a vehicle, and wherein the input signal corresponds to a request to change the vehicle from a first driving mode to a second driving mode.

66. The method of claim 65, wherein: The content in the second layout includes media captured by a camera of the first device, said content in said third layout comprises said media, The content in the third layout also includes additional content, and The content in the second layout does not include the additional content.

67. The method according to any one of claims 63 to 66, further comprising: At the first device: At the first time: Based on determining that the first device receives the first content from the second device for display at the first time, displaying the first content; and Based on determining that the first device has not received content from the second device for display at the first time, second content received from the second device for display at a second time is displayed, wherein the second time is before the first time.

68. The method according to any one of claims 63 to 67, further comprising: At the first device: In the third time: Based on determining that the first device receives third content from the second device for display at the third time, displaying the third content; and Based on determining that the first device has not received content from the second device for display at the third time, fourth content configured to be displayed when the connection between the first device and the second device is not working is displayed.

69. The method according to any one of claims 63 to 68, further comprising: At the first device: After receiving the input signal: Display user interface elements; After displaying the user interface element, receiving fifth content from the second device; generating combined content by combining the fifth content with the user interface element; as well as The contents of the combination are displayed.

70. The method according to any one of claims 63 to 69, further comprising: At the first device: Initiating rendering of the user interface element; After initiating rendering of the user interface element, receiving sixth content from the second device; generating a combined frame by combining the user interface element with the sixth content; as well as The combined frame is output for display.

71. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for executing a method according to any one of claims 63 to 70.

72. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 63 to 70.

73. A first device, comprising: Apparatus for carrying out the method according to any one of claims 63 to 70.

74. A computer program product comprising one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for performing a method according to any one of claims 63 to 70.

75. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving an input signal while displaying content in a first layout, wherein the first layout is selected by a second device different from the first device; In response to receiving the input signal, attempting to send a first message indicative of the input signal to the second device; and After trying to send the first message: Based on determining that the second device fails to respond to the first message: determining to change from the first layout to a second layout based on the input signal; and displaying the content in the second layout; and According to determining that sending the first message is successful: determining to change from the first layout to a third layout based on a second message received from the second device; and The content is displayed in the third layout.

76. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: receiving an input signal while displaying content in a first layout, wherein the first layout is selected by a second device different from the first device; In response to receiving the input signal, attempting to send a first message indicative of the input signal to the second device; and After trying to send the first message: Based on determining that the second device fails to respond to the first message: determining to change from the first layout to a second layout based on the input signal; and displaying the content in the second layout; and According to determining that sending the first message is successful: determining to change from the first layout to a third layout based on a second message received from the second device; and The content is displayed in the third layout.

77. A first device, comprising: means for receiving an input signal while displaying content in a first layout, wherein the first layout is selected by a second device different from the first device; means for attempting to send a first message indicative of the input signal to the second device in response to receiving the input signal; as well as After trying to send the first message: Based on determining that the second device fails to respond to the first message: means for determining to change from the first layout to a second layout based on the input signal; and means for displaying content in said second layout; as well as According to determining that sending the first message is successful: means for determining to change from the first layout to a third layout based on a second message received from the second device; and Means for displaying content in the third layout.

78. A computer program product, comprising one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: receiving an input signal while displaying content in a first layout, wherein the first layout is selected by a second device different from the first device; In response to receiving the input signal, attempting to send a first message indicative of the input signal to the second device; and After trying to send the first message: Based on determining that the second device fails to respond to the first message: determining to change from the first layout to a second layout based on the input signal; and displaying the content in the second layout; and According to determining that sending the first message is successful: determining to change from the first layout to a third layout based on a second message received from the second device; and The content is displayed in the third layout.

79. A method for orchestrating rendering of an animation across multiple devices, the method comprising: At the first device: determining an animation to be displayed on a second device different from the first device; According to the animation, rendering the first frame; determining, based on the animation, a location within the first frame to be updated with a user interface element rendered by the second device; as well as The first frame and an indication of the location are sent to the second device.

80. The method of claim 79, wherein the first frame includes a placeholder image at the location.

81. The method of any one of claims 79 to 80, further comprising: At the first device: determining a time to display the first frame based on a characteristic associated with the second device; as well as An indication of the time is sent to the second device.

82. The method of any one of claims 79 to 81, further comprising: At the first device: A current layout of a display of the second device is determined, wherein determining the animation is based on the current layout.

83. The method of any one of claims 79 to 82, further comprising: At the first device: determining a modification to a user interface to be rendered by the second device at the location; as well as An indication of the modification is sent to the second device.

84. The method of claim 83, wherein the modification comprises a change in a property of the user interface element, the property selected from the group consisting of position, opacity, color, font, size, and shape.

85. The method of any one of claims 79 to 84, further comprising: At the first device: Before determining the animation, a streaming connection is established with the second device to be used to transmit a plurality of frames corresponding to the animation.

86. The method of any one of claims 79 to 85, further comprising: At the first device: Rendering a second frame different from the first frame according to the animation; as well as The second frame is sent to the second device.

87. The method of any one of claims 79 to 86, further comprising: At the first device: determining, based on the animation, a second location within the first frame to be updated with a second user interface element rendered by the second device, wherein the second location is different from the first location, and wherein the second user interface element is different from the first user interface element; and An indication of the second location is sent to the second device.

88. A method according to any one of claims 79 to 87, wherein the first device is a user device, and wherein the second device is a vehicle.

89. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for executing a method according to any one of claims 79 to 88.

90. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 79 to 88.

91. A first device, comprising: Apparatus for carrying out the method according to any one of claims 79 to 88.

92. A computer program product comprising one or more programs configured to be executed by one or more processors of a first device, the one or more programs including instructions for executing a method according to any one of claims 79 to 88.

93. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: determining an animation to be displayed on a second device different from the first device; According to the animation, rendering the first frame; determining, based on the animation, a location within the first frame to be updated with a user interface element rendered by the second device; as well as The first frame and an indication of the location are sent to the second device.

94. A first device, comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: determining an animation to be displayed on a second device different from the first device; According to the animation, rendering the first frame; determining, based on the animation, a location within the first frame to be updated with a user interface element rendered by the second device; as well as The first frame and an indication of the location are sent to the second device.

95. A first device, comprising: means for determining an animation to be displayed on a second device different from the first device; means for rendering a first frame according to the animation; means for determining, based on the animation, a location within the first frame to be updated with a user interface element rendered by the second device; and Means for sending the first frame and an indication of the location to the second device.

96. A computer program product, comprising one or more programs configured to be executed by one or more processors of a first device, the one or more programs comprising instructions for: determining an animation to be displayed on a second device different from the first device; According to the animation, rendering the first frame; determining, based on the animation, a location within the first frame to be updated with a user interface element rendered by the second device; as well as The first frame and an indication of the location are sent to the second device.

97. A method for enabling connection vehicle customization, the method comprising: At a computer system of the vehicle, the computer system in communication with the display assembly: displaying, via the display component, a first frame including a first version of a vehicle instrument, wherein the first version has a first appearance in the first frame; When displaying the first version: Connect to user device; as well as receiving, without further user input after connecting to the user device, user-defined preferences for display of the vehicle instrumentation from the user device; as well as According to the user defined preferences: rendering a second version of the vehicle instrument; as well as A second frame including the second version is displayed via the display component, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance.

98. The method of claim 97, further comprising: At the computer system of the vehicle: Prior to rendering the second version, sending to the user device an identification of a set of one or more layouts stored by the vehicle; receiving, from the user device, a new set of one or more layouts based on determining that the set of one or more layouts is out of date, wherein after receiving the new set, rendering the second version and displaying the second frame are based on a layout in the new set of one or more layouts; as well as Based on determining that the set of one or more layouts is up-to-date, receiving an indication that the set is up-to-date; wherein after receiving the indication that the set is up-to-date, rendering the second version and displaying the second frame are based on a layout in the set of one or more layouts.

99. A method according to any one of claims 97 to 98, wherein the first version is digital or analog, and wherein the second version is not the same version as the first version.

100. The method of any one of claims 97 to 99, wherein the second version comprises a different color than the first version.

101. A method according to any one of claims 97 to 100, wherein the second version is located at a different position within the frame than the first version.

102. The method of any one of claims 97 to 101, further comprising: At the computer system of the vehicle: detecting a disconnection of the user device, wherein a third version of the vehicle instrument is being displayed via the display assembly immediately before the disconnection of the user device is detected; as well as After detecting a disconnection of the user device, a fourth version of the vehicle instrument is displayed, wherein the fourth version is different from the third version.

103. The method of any one of claims 97 to 102, further comprising: At the computer system of the vehicle: detecting user input by a sensor of the vehicle while connected to the user device; In response to detecting the user input, sending an indication of the user input to the user device; receiving, after sending the indication of the user input, an indication of a modification corresponding to the vehicle instrument from the user device, wherein the modification is determined based on the indication of the user input; rendering a third frame including said vehicle instrumentation based on said indication of said modification; as well as The third frame is displayed via the display component.

104. The method of claim 103, wherein the sensor is a physical button, and wherein the user input corresponds to activation of the physical button.

105. The method of claim 103, wherein the sensor is a touch-sensitive display, and wherein the user input corresponds to a touch input detected via the touch-sensitive display.

106. The method of claim 103, wherein the sensor is a microphone, and wherein the user input corresponds to voice input detected via the microphone.

107. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a vehicle, the one or more programs comprising instructions for executing a method according to any one of claims 97 to 106.

108. A vehicle comprising: Display components; one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 97 to 106.

109. A vehicle comprising: Apparatus for carrying out the method according to any one of claims 97 to 106.

110. A computer program product comprising one or more programs configured to be executed by one or more processors of a vehicle, the one or more programs including instructions for executing a method according to any one of claims 97 to 106.

111. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a vehicle, the one or more programs comprising instructions for: displaying a first frame including a first version of a vehicle instrument, wherein the first version has a first appearance in the first frame; When displaying the first version: Connect to user device; as well as receiving, without further user input after connecting to the user device, user-defined preferences for display of the vehicle instrumentation from the user device; as well as According to the user defined preferences: rendering a second version of the vehicle instrument; as well as A second frame including the second version is displayed, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance.

112. A vehicle comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying a first frame including a first version of a vehicle instrument, wherein the first version has a first appearance in the first frame; When displaying the first version: Connect to user device; as well as receiving, without further user input after connecting to the user device, user-defined preferences for display of the vehicle instrumentation from the user device; as well as According to the user defined preferences: rendering a second version of the vehicle instrument; as well as A second frame including the second version is displayed, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance.

113. A vehicle comprising: means for displaying a first frame including a first version of a vehicle instrument, wherein the first version has a first appearance in the first frame; When displaying the first version: means for connecting to user equipment; as well as means for receiving, from the user device, user-defined preferences for display of the vehicle instrumentation in the absence of further user input after connecting to the user device; as well as According to the user defined preferences: means for rendering a second version of said vehicle instrument; as well as Means for displaying a second frame including the second version, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance.

114. A computer program product, comprising one or more programs configured to be executed by one or more processors of a vehicle, the one or more programs comprising instructions for: displaying a first frame including a first version of a vehicle instrument, wherein the first version has a first appearance in the first frame; When displaying the first version: Connect to user device; as well as receiving, without further user input after connecting to the user device, user-defined preferences for display of the vehicle instrumentation from the user device; as well as According to the user defined preferences: rendering a second version of the vehicle instrument; as well as A second frame including the second version is displayed, wherein the second version has a second appearance in the second frame, and wherein the second appearance is different from the first appearance.

Images