Systems and methods for cloud-based digital audio and video processing

Abstract

Embodiments are provided for on-demand processing of audiovisual data. In one example, an on-demand audiovisual (AV) data processing method for playback in a vehicle includes preprocessing, by a local digital signal processor of the vehicle, an AV data file, where the preprocessing includes forming a plurality of digital AV signals from the AV data file, transmitting at least some of the plurality of digital AV signals to a cloud-based server configured to apply one or more processing features to at least some of the plurality of digital AV signals, receiving a processed version of at least some of the plurality of digital AV signals from the cloud-based server, and outputting the processed version of at least some of the plurality of digital AV signals to one or more AV sinks of the vehicle.

Description

[Technical Field]

[0001] The present disclosure relates to the field of in-vehicle audio and video systems. [Background technology]

[0002] Some automobiles may include in-vehicle entertainment systems, such as media players and radios. These vehicle entertainment systems may be used to deliver media, such as audio and video content, to the vehicle's driver and / or passengers. The media may be sourced from a radio signal (e.g., frequency modulation, FM), an external device (e.g., a cell phone), or numerous other sources. To enhance the listening and / or viewing experience, digital signal processing may be used to adjust the quality of the audio and / or video. Digital signal processing may add desirable audio and video effects to suit the end user's preferences. The digitally processed (or unprocessed) signals may be played using the vehicle's internal audio system, for example, using speakers and / or a video screen. Summary of the Invention [Means for solving the problem]

[0003] In various embodiments, the aforementioned problems may be addressed by a method for on-demand audiovisual (AV) data processing for playback in a vehicle, the method including: preprocessing, by a local digital signal processor of the vehicle, AV data files, the preprocessing including forming a plurality of digital AV signals from the AV data files; transmitting at least some of the plurality of digital AV signals to a cloud-based server configured to apply one or more processing features to at least some of the plurality of digital AV signals in real time or non-real time; receiving processed versions of at least some of the plurality of digital AV signals from the cloud-based server; and outputting the processed versions of at least some of the plurality of digital AV signals to one or more AV sinks of the vehicle.

[0004] It should be understood that the foregoing summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description. It is not intended to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Moreover, the claimed subject matter is not limited to implementations that solve any disadvantages discussed above or noted anywhere in this disclosure. The present specification also provides, for example, the following: (Item 1) 1. A method for processing on-demand audiovisual (AV) data for playback in a vehicle, comprising: pre-processing, by a local digital signal processor of the vehicle, AV data files, the pre-processing including forming a plurality of digital AV signals from the AV data files; transmitting at least a portion of the plurality of digital AV signals to a cloud-based server configured to apply one or more processing features to the at least a portion of the plurality of digital AV signals; receiving a processed version of at least the portion of the plurality of digital AV signals from the cloud-based server; outputting the processed versions of at least the portions of the plurality of digital AV signals to one or more AV sinks in the vehicle. (Item 2) 2. The method of claim 1, wherein transmitting at least the portion of the plurality of digital AV signals to the cloud-based server includes receiving a request from the cloud-based server indicating which of the plurality of digital AV signals to transmit to the cloud-based server, and transmitting at least the portion of the plurality of digital AV signals to the cloud-based server in response to the request. (Item 3) 3. The method of claim 2, further comprising sending to the cloud-based server an indication of the one or more processing features to apply to at least the portion of the plurality of digital AV signals, wherein the cloud-based server is configured to determine which of the plurality of digital AV signals to request based on the one or more processing features to apply. (Item 4) 2. The method of claim 1, wherein the plurality of digital AV signals is stored in a first buffer, and further comprising saving the processed versions of the at least some of the plurality of digital AV signals in a second buffer. (Item 5) 5. The method of claim 4, further comprising: determining that a network connection to the cloud-based server has been lost; and, in response, cross-morphing the processed versions of at least the portion of the plurality of digital AV signals stored in the second buffer with locally processed versions of at least the portion of the plurality of digital AV signals, wherein the locally processed versions of at least the portion of the plurality of digital AV signals are processed by the local digital signal processor. (Item 6) 6. The method of claim 5, further comprising: determining that the network connection to the cloud-based server has been restored; and responsively cross-morphing the processed versions of at least the portion of the plurality of digital AV signals received from the cloud-based server with locally processed versions of at least the portion of the plurality of digital AV signals after the network connection is restored. (Item 7) 10. The method of claim 1, further comprising transmitting only the portion of the plurality of digital AV signals to the cloud-based server, receiving processed versions of only the portion of the plurality of digital AV signals from the cloud-based server, and outputting the remaining portion of the plurality of digital AV signals, along with the processed versions of only the portion of the plurality of digital AV signals, to the one or more AV sinks. (Item 8) Item 10. The method of item 1, wherein the AV data file is obtained from a local media source located within the vehicle. (Item 9) 2. The method of claim 1, wherein the AV data files include audio files, the one or more AV sinks include one or more speakers of the vehicle, and the one or more processing features applied to at least the portions of the plurality of digital AV signals include one or more of surround sound, immersion effects, reverberation, virtual speakers, virtual bass, and adaptive equalization. (Item 10) A multimedia system for a vehicle, comprising: a digital signal processor including: an input buffer configured to receive audiovisual (AV) data files; an upmixer configured to pre-process the AV data files to form a plurality of digital AV signals; a main signal processor configured to apply a first, smaller set of processing features to the plurality of digital AV signals; and an output buffer; a transmitter configured to transmit the AV data file in the input buffer or one or more of the plurality of digital AV signals from the upmixer to a cloud server, the cloud server configured to apply a second, larger set of processing features to the AV data file or the plurality of digital AV signals to form a set of cloud-processed digital AV signals; a receiver configured to receive the set of cloud-processed digital AV signals; one or more AV sinks configured to output the set of cloud-processed digital AV signals for playback. (Item 11) Item 11. The multimedia system of item 10, wherein the AV data file includes an audio file, the one or more AV sinks include one or more speakers of the vehicle, and the second, larger set of processing features includes one or more of surround sound, immersion effects, reverberation, virtual speakers, virtual bass, and adaptive equalization. (Item 12) Item 11. The multimedia system of item 10, wherein the AV data file is obtained from a local media source located within the vehicle. (Item 13) Item 13. The multimedia system of item 12, wherein the local media source includes a mobile phone, and the transmitter is configured to transmit the AV data file in the input buffer or one or more of the plurality of digital AV signals from the upmixer to the cloud server via the mobile phone. (Item 14) Item 11. The multimedia system of item 10, further comprising a crossmorpher configured to crossmorph the set of cloud-processed digital AV signals with locally processed versions of the digital AV signals in response to determining that network connectivity to the cloud server has been lost, the locally processed versions of the digital AV signals being processed by the digital signal processor. (Item 15) Item 11. The multimedia system of item 10, wherein the receiver is configured to convert a requested abstract target buffer received together with the set of cloud-processed digital AV signals into one or more physical buffers corresponding to the requested abstract target buffer, and transmit the set of cloud-processed digital AV signals to the one or more AV sinks according to the one or more physical buffers.

[0005] The present disclosure may be better understood by reading the following description of non-limiting embodiments with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0006] [Figure 1] 1 is an example partial view of a vehicle cabin, in accordance with one or more embodiments of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an example in-vehicle computing system for a vehicle with a digital media player, in accordance with one or more embodiments of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an example car audio / video system. [Figure 4] FIG. 1 is a block diagram of an example of a car multimedia system. [Figure 5] FIG. 1 is a block diagram of cross-morphing a multi-channel locally processed digital signal processing signal and a multi-channel cloud-processed signal. [Figure 6] FIG. 1 is a block diagram of cloud-based signal processing in accordance with one or more embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates a method for performing digital signal processing using components from a car multimedia system and a cloud server from the perspective of a car multimedia system. [Figure 8] FIG. 1 illustrates a method for performing digital signal processing using components from a car multimedia system and a cloud server from the perspective of a cloud server. DETAILED DESCRIPTION OF THE INVENTION

[0007] Many modern automobiles are equipped with in-car multimedia entertainment systems, such as radios and / or video players. The multimedia systems may output multiple signals accessible to the end user. Processing these signals may enable a wide range of audio and / or video quality adjustments depending on the user's preferences. Digital signal processing algorithms may include, for example, bass boost, noise filtering, and other techniques. Vehicle multimedia systems may or may not include an extensive library of digital signal processing features. In either case, the availability of digital processing features is generally fixed for a given car multimedia system; for example, the set of digital signal processing features may not be easily updatable by the user.

[0008] Some car multimedia systems may include numerous digital storage options, digital signal processing options, and / or video subsystems. Such systems may be capable of providing a high degree of customization and fine-tuning, allowing end users to fine-tune audiovisual (AV) data of varying quality. Digital signal processing may be implemented using algorithms, which may be calculated on a digital signal processing unit within the multimedia system itself. However, the options available to users of such car multimedia systems are generally a fixed set of algorithms, often not updatable via software. Digital signal processing algorithms may also be licensed. As new or updated digital signal processing algorithms are created, many multimedia systems may be unable to implement them due to memory and processing power limitations. Other car multimedia systems may include a relatively small selection of basic digital signal processing features and may have only a small computational budget, meaning that complex digital processing features are unavailable due to processing power and memory limitations.

[0009] Thus, according to embodiments described herein, one or more digital audio and / or video signals may be upstreamed to one or more remote computers (e.g., cloud-based servers / systems), which may perform digital signal processing using one or more cloud-based digital processing algorithms in real time. The processed signals may be sent back to the multimedia entertainment system, where they may be played, for example, through multiple audio and / or video sinks (e.g., loudspeakers, screens, etc.) in the automobile. Digital signal processing services may be offered as part of a subscription service, allowing users to select their preferred digital effects and optimizations. Furthermore, as new algorithms are developed, the available catalog of effects may be updated, effectively allowing users to upgrade and customize their listening experience. In some embodiments, a network connection to the cloud-based digital signal processor may be routed through an external device (e.g., a mobile phone). The external device acts as a transmitter and receiver, and the digitally processed audio and / or video may then be transmitted to the automobile's multimedia system. This allows virtually any car's multimedia system to be (indirectly) connected to a cloud server, giving even the most basic car multimedia system access to cloud-based features on request.

[0010] By executing signal processing algorithms on a cloud-based service, advanced digital signal processing tasks can be outsourced to a cloud server. The cloud-based service may perform extensive or complex computational tasks that are not performed on the car multimedia system itself. However, processing AV data via a cloud-based service may require a stable network connection to the server performing the cloud-based digital signal processing. The network connection may be established directly between the cloud server and the car multimedia system, or it may be established through an intermediate device, such as a mobile phone, that has sufficient mobile connectivity through, for example, 5G broadband mobile networking.

[0011] Mobile connectivity may be limited in some areas and may depend on factors such as the user's cellular phone coverage. If a network connection to a cloud-based digital signal processing server is not feasible, the car multimedia entertainment system may switch to one or more of its available on-board digital signal processing effects. Abruptly switching from one signal processing system to another may introduce undesirable audiovisual side effects, such as pops, crackles, sudden volume changes, and other artifacts. To mitigate the undesirable side effects, a cross-morphing effect may be used to facilitate the transition. As an example, cloud-processed audio data may be cached for a certain period of time. If network coverage is suddenly lost, the decrease in network coverage may be detected, and then on-board digital signal processing may be initiated, initiating a fade-in. At the same time, the digital signal received from the cloud may fade out. This technology, implemented in a crossfader, may reduce or eliminate the aforementioned side effects (e.g., pops, crackles, sudden volume changes, etc.).

[0012] Cross-morphing (as described above) may be insufficient to completely eliminate the aforementioned side effects after activating one or more cloud-based features on-demand. For example, if a particular cloud-based feature on-demand includes a delay line (e.g., a finite impulse response (FIR) filter) or feedback (e.g., an infinite impulse response (IIR) filter), a sudden change in the signal after a period of silence may create a step response in the cloud signal that may persist longer than the cross-morphing time used when the network connection is lost. In some embodiments, a different cross-morphing time may be used when the connection drops compared to when the network connection is re-established (e.g., a relatively short cross-morphing time may be applied when the connection is lost). However, the variable cross-morphing time may require an undesirably long time (e.g., on the order of several seconds) to complete before the transient effect (e.g., the step signal) dissipates. In another embodiment, the crossmorpher may be inserted at the input of the signal processing feature (e.g., on the cloud), so that crossmorphing can begin when a connection is established (e.g., processed AV data processed on the cloud can be crossmorphed with unprocessed AV data in the cloud). In this approach, selecting the appropriate time to fade in becomes part of the feature, allowing the car multimedia system to always use the same crossfading time without having to worry about whether the selected time is sufficient.

[0013] FIG. 1 illustrates the interior of a cabin 100 of a vehicle 102 in which a driver and / or one or more passengers may be seated. The vehicle 102 may be a road vehicle, among other types of vehicles. In particular, the vehicle 102 of FIG. 1 may be an automobile including drive wheels (not shown) and an engine 104. In some examples, the engine 104 may be an internal combustion engine. In other examples, the engine 104 may be an electric engine or may include hybrid components. For example, the vehicle 102 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy into an energy form suitable for storage by an energy storage device. The vehicle 102 may also include a fully electric vehicle incorporating a fuel cell, a solar energy capture element, and / or other energy storage system to power the vehicle.

[0014] The vehicle 102 may include multiple vehicle systems, including a brake system for providing braking, an engine system for providing power to the wheels of the vehicle, a steering system for adjusting the direction of the vehicle, a transmission system for controlling gear selection for the engine, an exhaust system for treating exhaust gases, etc. Additionally, the vehicle 102 includes an onboard computing system 109.

[0015] As shown, instrument panel 106 may include various displays and controls accessible to a human user (also referred to as a passenger) of vehicle 102. For example, instrument panel 106 may include a user input device, such as a touchscreen 108 of onboard computing system 109, an audio system control panel, a display 111, and an instrument cluster 110. Touchscreen 108 may receive user input to onboard computing system 109 for controlling audio output, visual display output, user preferences, control parameter selections, etc.

[0016] Cabin 100 may also include one or more user objects, such as a mobile device 128, stored within the vehicle before, during, and / or after travel. Mobile device 128 may include a smartphone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. Mobile device 128 may connect to an on-board computing system via a communication link 130. Communication link 130 may be wired (e.g., via Universal Serial Bus [USB], Mobile High-Definition Link [MHL], High-Definition Multimedia Interface [HDMI], Ethernet, etc.) or wireless (e.g., via BLUETOOTH, Wi-Fi, Wi-Fi Direct Near Field Communication [NFC], cellular connectivity, etc.) and may be configured to provide bidirectional communication between the mobile device and the on-board computing system. Mobile device 128 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices within the mobile device. For example, communication link 130 may provide sensor and / or control signals from various vehicle systems (e.g., vehicle audio system, sensor subsystem, etc.) and touchscreen 108 to mobile device 128, and may also provide control and / or display signals from mobile device 128 to on-board systems and touchscreen 108. Communication link 130 may also provide power to mobile device 128 from an on-board power source for charging the mobile device's internal battery.

[0017] The on-board computing system 109 may also be communicatively coupled to additional devices (e.g., one or more external devices 150) that are operated and / or accessed by a user but are located outside the vehicle 102. In the illustrated embodiment, the external devices are located outside the vehicle 102, but it is understood that in alternative embodiments, the external devices may be located inside the cabin 100. The external devices may include server computing systems, personal computing systems, portable electronic devices, electronic wristbands, electronic headbands, portable music players, electronic activity tracking devices, pedometers, smart watches, GPS systems, etc. The external devices 150 may connect to the on-board computing system via a communication link 136, which may be wired or wireless, as described with reference to the communication link 130, and may be configured to provide bidirectional communication between the external devices and the on-board computing system. For example, the external devices 150 may include one or more sensors, and the communication link 136 may transmit sensor output from the external devices 150 to the on-board computing system 109 and the touchscreen 108. The external device 150 may also store and / or receive information regarding navigation map data, image feature mapping data, etc., and may transmit such information from the external device 150 to the in-vehicle computing system 109 and / or touchscreen 108.

[0018] The on-board computing system 109 may analyze input received from the external device 150, the mobile device 128, and / or other input sources, provide output via the touchscreen 108 and / or the speaker 112, communicate with the mobile device 128 and / or the external device 150, and / or perform other actions based on the evaluation. In some embodiments, all or part of the evaluation may be performed by the mobile device 128 and / or the external device 150. In some embodiments, the external device 150 may include an on-board computing device of another vehicle.

[0019] In some embodiments, one or more of the external devices 150 may be indirectly communicatively coupled to the in-vehicle computing system 109 via the mobile device 128 and / or another external device 150. For example, the communication link 136 may communicatively couple the external device 150 to the mobile device 128 such that output from the external device 150 is relayed to the mobile device 128. The data received from the external device 150 may then be aggregated at the mobile device 128 with the data collected by the mobile device 128, and the aggregated data may then be transmitted to the in-vehicle computing system 109 and the touchscreen 108 via the communication link 130. Similar data aggregation may also be performed at a server system and then transmitted to the in-vehicle computing system 109 and the touchscreen 108 via the communication links 136 / 130.

[0020] 2 illustrates a block diagram of an in-vehicle computing system 109 configured and / or integrated within a vehicle 102. The in-vehicle computing system 109, in some embodiments, may perform one or more of the methods described herein. The in-vehicle computing system may include or be coupled to various vehicle systems, subsystems, hardware components, and software applications and systems that are integrated or integratable within the vehicle 102 to enhance the in-vehicle experience for the driver and / or passengers. Additionally, the in-vehicle computing system may be coupled to a system for providing autonomous vehicle control.

[0021] The in-vehicle computing system 109 may include one or more processors, including an operating system processor 214 and an interface processor 220. The operating system processor 214 may execute an operating system on the in-vehicle computing system and control the input / output, display, playback, and other operations of the in-vehicle computing system. The interface processor 220 may interface with and function with a vehicle control system 230 via an in-vehicle system communication module 224.

[0022] The intra-vehicle system communication module 224 may receive data input from other vehicle components and systems, for example, via the vehicle control system 230, while simultaneously outputting data to the vehicle control system 230. When outputting data, the intra-vehicle system communication module 224 may provide signals over a bus corresponding to any state of the vehicle, the vehicle's surroundings, or the output of any other information source connected to the vehicle. The vehicle data outputs may include, for example, analog signals (e.g., flow rate), digital signals provided by individual information sources (e.g., clocks, thermometers, position sensors such as global positioning system (GPS) sensors, inertial measurement systems (IMS), etc.), and digital signals propagated through vehicle data networks (e.g., an engine controller area network (CAN) bus through which engine-related information may be communicated, a climate control CAN bus through which climate control-related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components within the vehicle). For example, the vehicle data outputs may be output to the vehicle control system 230, and the vehicle control system 230 may adjust vehicle control based on the vehicle data outputs. For example, the on-board computing system 109 may retrieve from the engine CAN bus the vehicle's current speed as estimated by wheel sensors, the vehicle's power status via the vehicle's battery and / or power distribution system, the vehicle's ignition status, etc. Additionally, other interfacing means, such as Ethernet, may be used as well without departing from the scope of this disclosure.

[0023] A storage device 208 may be included within the in-vehicle computing system 109 to store data, such as instructions executable by the processors 214 and 220, in a non-volatile form. The storage device 208 may store application data (including pre-recorded sounds) to enable the in-vehicle computing system 109 to execute applications to connect to and / or collect information for transmission to a cloud-based server. The applications may retrieve information collected by vehicle systems / sensors, input devices (e.g., user interface 218), data stored in volatile memory 219A or non-volatile storage device (e.g., memory) 219B, devices communicating with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth link), etc. The in-vehicle computing system 109 may further include volatile memory 219A. The volatile memory 219A may be random access memory (RAM). The non-transitory storage device (e.g., non-volatile storage device 208 and / or non-volatile memory 219B) may store instructions and / or code that, when executed by a processor (e.g., operating system processor 214 and / or interface processor 220), control in-vehicle computing system 109 to perform one or more of the actions described in this disclosure. Storage device 208 may also store audio and / or visual data to be used as a source for car multimedia system 226.

[0024] A microphone 202 may be included within the on-board computing system 200 to measure ambient noise within the vehicle, measure ambient noise outside the vehicle, review user input, etc. A navigation subsystem 211 of the on-board computing system 109 may generate and / or receive navigation information, such as location information (e.g., via the GPS / IMS sensor 204 and / or other sensors from the sensor subsystem), route guidance, traffic information, point of interest (POI) identification, and / or provide other navigation services to the user. The navigation subsystem 211 may include input / output 280 (e.g., analog-to-digital converters, digital inputs, digital outputs, network outputs, radio frequency transmitting devices, etc.). In some examples, the navigation subsystem 211 may function in conjunction with a vehicle control system 230.

[0025] The external device interface 212 of the in-vehicle computing system 109 may be coupled to and / or communicate with one or more external devices 150 located outside the vehicle 102. While the external devices are shown as being located outside the vehicle 102, it should be understood that they may be temporarily housed within the vehicle 102, such as when a user is operating the external device while operating the vehicle 102. In other words, the external device 150 is not integrated into the vehicle 102. The external device 150 may include a mobile device 128 (e.g., connected via Bluetooth, NFC, WIFI Direct, or other wireless network connection) or an alternative Bluetooth-enabled device 252. The mobile device 128 may be a mobile phone, a smartphone, a wearable device / sensor that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). The external device 150 also includes a cloud server 246 for performing remote processing of digital audio and / or video. In some embodiments, a network connection to the cloud server may be established through an intermediary, such as, for example, a mobile device 128. For example, an external device may include an off-vehicle device located away from and outside the vehicle. Still other external devices include external storage devices 254 (e.g., solid-state drives, pen drives, USB drives, etc.). The external device 150 may communicate with the on-vehicle computing system 109 either wirelessly or through a connector without departing from the scope of this disclosure. For example, the external device 150 may communicate with the on-vehicle computing system 109 through the external device interface 212, via a network 260, a universal serial bus (USB) connection, a direct wired connection, a direct wireless connection, and / or other communication links.

[0026] The external device interface 212 may provide a communications interface to enable the in-vehicle computing system to communicate with a mobile device associated with a user contact. For example, the external device interface 212 may enable establishing a voice call and / or sending a text message (e.g., SMS, MMS, etc.) to the mobile device 128 associated with the user contact (e.g., over a cellular communications network). Furthermore, in some examples, a vehicle user may coordinate autonomous vehicle operation via an application on the mobile device 128 associated with the user. The external device interface 212 may additionally or alternatively provide a wireless communications interface to enable the in-vehicle computing system to synchronize data with one or more devices in the vehicle (e.g., the user's mobile device) via Wi-Fi Direct.

[0027] The on-board computing system 109 may further include an antenna 206. While the antenna 206 is illustrated as a single antenna, in some embodiments, it may include one or more antennas. The on-board computing system may obtain broadband wireless Internet access via the antenna 206 and may also receive broadcast signals such as radio, television, weather, traffic, etc. The on-board computing system may receive positioning signals such as GPS signals via one or more antennas 206. The on-board computing system may receive wireless commands via FR, for example, via antenna(s) 206 or via infrared or other means with an appropriate receiving device. For example, the antenna 206 may receive voice calls (e.g., telephone calls, etc.). Additionally, the antenna 206 may provide AM / FM radio signals to an external device 150 (e.g., a mobile device 128) via an external device interface 212.

[0028] The in-vehicle system communication module 224 may be communicatively coupled to a vehicle control system 230. The vehicle control system 230 may include, for example, acceleration control, braking control, and / or steering control, in addition to many other controls, such as climate control. Signals from the vehicle control system 230 may be used as inputs for digital signal processing. For example, the control signals may adjust audio output characteristics, such as volume, equalization, sound imaging (e.g., the configuration of audio signals to produce audio output that appears to the user to originate from one or more defined locations), audio distribution among multiple speakers, etc. In another example, control signals from an engine may be used for digital signal processing to synthesize engine noise, e.g., to enhance the sound of the engine through the use of loudspeakers.

[0029] The in-vehicle computing system 109 may include a car multimedia system 226 configured to play locally available media and / or cloud-based media. The car multimedia system includes a digital signal processor for improving the quality of audio and / or visual data. In addition, it may perform some or all of the digital signal processing using external sources. For certain digital processing effects, the car multimedia system 226 may be communicatively coupled to other sensors and input sources (e.g., microphone 202) within the in-vehicle computing system. The car multimedia system 226 and its interaction with the cloud server are described in more detail with respect to FIGS. 3-7. While the term "car multimedia system" is used herein, it should be understood that the multimedia system 226 may be implemented in virtually any type of vehicle (e.g., without limitation, an automobile, a truck, a locomotive, heavy equipment (e.g., a mining vehicle), etc.).

[0030] FIG. 3 shows an example block diagram 300 of a typical vehicle AV system including local media sources, sinks, and a car multimedia system 226. The car multimedia system may receive one or more local media sources 302 as input. The local media sources may be, for example, music stored on one or more hard drives, a mobile phone connected via Bluetooth, a CD, or virtually any other audio source (including audio stored in the storage device 208), and may or may not include video data, e.g., audiovisual (AV) data. The car multimedia system 226 may be coupled to one or more AV sinks (herein, one or more audio sinks 310 and one or more video sinks 312), which may include multiple loudspeakers, a radio antenna, and other input / output devices. The car multimedia system 226 may also include a head unit. The head unit may serve as a user interface for a user to control parameters of audio playback (e.g., volume, FM radio frequency, and media selection).

[0031] The car multimedia system 226 may also include a digital signal processor 308. The digital signal processor 308 may be able to apply a set of effects available to the car multimedia system 226 itself, independent of cloud computing or internet connectivity. For a basic multimedia system, the digital signal processor 308 may only have access to basic effects (e.g., simple equalization, bass boost, etc.). Digital signal processing may be performed at several steps (e.g., reception, upmixing, main signal processing, and output). This is described in more detail with respect to FIG. 4 below.

[0032] The digital signal processor 308 processes AV data from local media sources using various methods. Each digital signal processing method used may process AV data from one or more of the sources. For example, audio from two sources may be used to cross-morph between two subsequently played songs. As another example, audio from a single source may be processed with a reverberation effect. The processed audio may be output to one or more audiovisual sinks (e.g., audio sink 310 and / or video sink 312). The audio sinks may be, for example, loudspeakers connected to the car multimedia system 226. The audio sinks may include speakers designed for various frequency ranges (e.g., subwoofers or mid-range speakers). The digital signal processor 308 may perform digital-to-analog conversion before the output is passed to the audio sink. Amplification may also be used to adjust the volume of the audio sink's playback, if necessary.

[0033] 4 shows a block diagram 400 of a digital signal processor 402. The digital signal processor 402 is an example of the digital signal processor 308 of the car multimedia system 226. The digital signal processor 402 includes a receiver 404, which in some embodiments may be a digital buffer that loads some or all of a given piece of digital media provided by one or more local media sources 302. The digital signal processor 402 includes memory and one or more processors, which may vary in memory size and processing power depending on the desired functionality of the car multimedia system.

[0034] The digital signal processor 402 includes resources (e.g., memory and processing power) for performing upmixing 406. Upmixing, as described herein, refers to the process of mixing audio to increase the number of available audio channels. For example, given a digital audio source represented in stereo sound, upmixing may create an intermediate number of channels (e.g., front left, front right, rear left, rear right, and a subwoofer). The digital audio channels produced in the intermediate upmixing process may be stored in volatile and / or non-volatile memory in a format usable for main signal processing or transmittable to a cloud server for further processing.

[0035] The digital signal processor 402 includes resources for performing main signal processing 408. Main signal processing (performed using the resources herein) may include applying one or more digital audio effects (e.g., frequency-dependent filtering, noise filtering, bass boost, etc.) to AV data obtained from local media sources. Main signal processing may perform multiple adjustments, including adjustments for surround sound. During main signal processing 408, the intermediate channels generated by upmixing 406 may be further divided into specific speaker channels, the number of which depends on the number of speakers in the system. The output generated by main signal processing 408 may be stored in a digital buffer and available for digital-to-analog conversion and / or cross-morphing.

[0036] The conditioned audio produced from main signal processing 408 may be stored, for example in a buffer, as output 412. To be usable as an audio output, the digital signal produced by main signal processing 408 may be converted to an analog signal, which may also be amplified or conditioned before being sent at output 412 to one or more audio sinks 310 and / or video sinks 312.

[0037] It should be noted that the digital signal processor 402 may include more, fewer, or different components than those shown in Figure 4. In some embodiments, for example, upmixing 406 may be performed as part of main signal processing 408. The internal components of the digital signal processor 402 may generally include executable instructions stored in volatile and / or non-volatile memory and executable using one or more digital processor(s) within the digital signal processor.

[0038] FIG. 5 shows a block diagram 500 for cross-morphing audio and / or video from two sources (first source 502 and second source 504). For example, the first source may be a local digital signal processor, and the second source may be a cloud-based digital signal processor. Cross-morphing may be performed by digital signal processor 402 using memory and processing resources on the memory. Cross-morphing, as used herein, is the process of fading audio and / or video from first source 502 to second source 504 (or vice versa). The two AV data sources may be buffered audio and / or video stored in memory, such as the buffer of main signal processing 408. The sources are input into cross-morpher 506 and cross-morphed. Cross-morpher 506 performs three tasks simultaneously: That is, gradually adjusting (e.g., lowering) the volume of the first source 502, gradually adjusting (e.g., raising) the volume of the second source 504, and adding the adjusted signals (e.g., the raised and lowered volumes) together to form the resulting signal 508 as its output. Cross-morphing is advantageous over simply switching between sources of AV data because cross-morphing minimizes audio effects associated with abrupt switching, such as pops and clicks. In the present disclosure, cross-morphing can be used to fade between audio processed on a cloud-based digital signal processor and a local digital signal processor. If the cloud-based digital signal processor is unreachable, the output from the local digital signal processor may be smoothly faded in, allowing for a less noticeable transition in the event of a network connection interruption. To enable a transition from the AV data processed by the cloud-based digital signal processor to the locally processed version of the AV data, the AV data processed by the cloud-based digital signal processor may be buffered for a sufficient period of time using memory available to the local digital signal processor.Depending on the digital signal processing features you are using on the cloud server, this buffer may contain, by way of non-limiting example, 6-20 milliseconds of audio.

[0039] Additionally, a delay line 505 may be introduced on either the first source 502 or the second source 504 so that the first source 502 and the second source 504 can be properly synchronized in time. The delay line 505 operates to delay the signal of one of the sources (relative to the other) and is adjustable so that the two source signals are aligned in time. The delay line 505 is an optional component of the Crossmorpher and may not be included if computational resources are unavailable.

[0040] The cross-morphing performed by the cross-morpher 506 is user-configurable through a variety of different parameters. For example, the cross-morphing time (e.g., the time it takes to transition from the first source 502 and the second source 504) may be adjusted. As one example, a cross-morphing may have a default time of 10 milliseconds. Other adjustments may include adjusting the manner in which the volume of the AV data source is increased and decreased. The volume of the input source may, for example, be increased and decreased linearly over time. In another example, the cross-morphing may be performed approximately exponentially over time. Settings related to cross-morphing may be adjustable by the user and / or manufacturer of the car multimedia system.

[0041] FIG. 6 shows a block diagram 600 including a car multimedia system 606 and a cloud-based digital signal processor 620. The car multimedia system 606 includes an interface for transmitting and receiving data to and from the cloud. The car multimedia system 606 is a non-limiting example of (e.g., an extended version of) the car multimedia system 226 of FIG. 3, which may be configured to play digital audio and / or video from several sources (e.g., a local media source 602). As described with respect to FIGS. 3-5, the car multimedia system can be configured to generate output on one or more audiovisual sinks (e.g., an audio sink 630 and / or a video sink 632) by using its local digital signal processor 608, which is a non-limiting example of (e.g., an extension of) the digital signal processor 402 (see FIG. 4). The local digital signal processor 608 processes the local media source 602 and receives AV data as input from the cloud-based digital signal processor 620. The cloud-based digital signal processor 620 uses its own resources to generate the digitally processed result, as described below. The car multimedia system 606 further includes resources that can be configured to perform cross-morphing, as described above with respect to Figure 5. When used by the car multimedia system 606, cross-morphing can be used to cross-morph from a local version of the digitally processed AV data and a cloud-based version of the digitally processed AV data.

[0042] The local media source 602 may be input directly into the car multimedia system 606 or may be passed into the local digital signal processor 608 (e.g., in its receiving section, which may include a buffer, as described with respect to the receiving section 404 of the digital signal processor 402 of FIG. 4). The local digital signal processor 608 may be configured to perform several adjustments (e.g., upmixing and main digital signal processing) on ​​the local media source 602. The output of the main digital signal processing may be buffered in memory to enable cross-morphing to another audio source (e.g., audio provided by a cloud digital signal processor).

[0043] The block diagram 600 includes a cloud-based digital signal processor 620 that is configurable to receive at least a portion of a digital audiovisual file from the car multimedia system 606 and / or from one or more cloud media sources 604. The cloud-based digital signal processor 620 is configurable to perform similar operations as the local digital signal processor (e.g., upmixing, primary digital signal processing, and output). The cloud-based digital signal processor may include a larger set of computing resources (e.g., processing power and memory) compared to the car multimedia system. If a network connection to the cloud-based digital signal processor 620 can be successfully established, the cloud-based digital signal processor 620 may process audiovisual data in parallel with the local digital signal processor 608. Compared to the car multimedia system, the cloud-based digital signal processor 620 includes a second, larger set of AV digital signal processing features (referred to as cloud-based features on request 622) that can be applied to execute computationally intensive digital signal processing algorithms. Computationally intensive digital signal processing algorithms include, but are not limited to, complex reverberation to simulate large spaces (e.g., cathedrals), adapting audio playback based on external weather conditions and noise levels, changing filtering options depending on the genre of the source music, simulating more acoustic space inside a vehicle, various surround and immersion effects, virtual speakers, virtual bass (for vehicles without subwoofers), engine sound synthesis, add-on standard acoustic vehicle alert systems (AVAS) for hybrid and electric vehicles, automatic vehicle equalization and metering, e.g., acoustic balancing depending on the number and / or position of passengers, speed-dependent volume control, synchronization of audio and video streams, audio / video flow improvement, etc.When combinable, several on-demand audio / video features (e.g., advanced AVAS sound and surround effects) can be requested in parallel by the same user. To perform the processing of the local media sources 602, the local digital signal processor 608 may transmit partially processed audiovisual data. The partially processed data may be provided directly from the receiver of the local digital signal processor 608. In another example, the local digital signal processor may perform upmixing on the local media sources 602 before transmitting them to the cloud-based digital signal processor.

[0044] Once the cloud-based digital signal processor 620 processes the digital signal received from the local digital signal processor 608 (e.g., through its own primary digital signal processing), the cloud-based digital signal processor 620 may transmit the processed digital signal back to the local digital signal processor 608. Usable output for an audio and / or video sink may be generated, for example, through digital-to-analog conversion. In this manner, primary digital signal processing may be outsourced to the cloud-based digital signal processor 620. Output from the cloud-based digital signal processor may be stored (e.g., buffered) on the local digital signal processor 608, allowing for cross-morphing in the event of a network connection interruption and / or re-establishment. For example, if the connection between the cloud-based digital signal processor 620 and the car multimedia system 606 is interrupted, the local digital signal processor 608 may cross-morph to buffered audio and / or video generated through its own processing. At least a short-term AV buffer (e.g., containing 20-30 milliseconds of AV data) may be included as part of the local digital signal processor 608 to store data received from the cloud-based digital signal processor. In some examples, for example, when the media source is a broadcast media source (e.g., an Internet radio station), the AV data processed by the cloud-based digital signal processor may be processed in ultra-real-time (e.g., faster than the playback time of the AV data) and transmitted to the car multimedia system. This may allow a small amount of processed AV data to be stored in the buffer for cross-morphing.

[0045] In addition to the local media source 602, AV data from one or more cloud media sources 604 may be input into the cloud-based digital signal processor 620. The cloud media sources 604 may be stored on a cloud server accessible to the cloud-based digital signal processor 620. If one or more cloud media sources 604 are used, all digital signal processing of the cloud media sources 604 may be performed by the cloud-based digital signal processor 620. The processed audio and / or video data may be sent to the local digital signal processor 608 and output (e.g., using digital-to-analog conversion) to an audio sink 630 and / or a video sink 632.

[0046] To facilitate communication with the cloud-based digital signal processor 620, the car multimedia system may include a transmitter 610 and a receiver 624. As described in more detail below, the local digital signal processor 608 may transmit and / or receive AV data to and from the cloud-based digital signal processor 620. Data transmitted to the cloud-based digital signal processor 620 may include AV data from one or more local media sources 602. The AV data from the local media sources 602 may include audio and / or visual components and may or may not be partially processed through methods implemented on the local digital signal processor 608. Data received by the local digital signal processor 608 via the receiver may include audio and / or visual sources that have been processed using the cloud-based digital signal processor 620. Both the transmitter 610 and the receiver 624 are communicatively coupled to the cloud-based digital signal processor 620 via a sufficiently fast communication protocol (e.g., 5G broadband mobile networking). The communication coupling may be achieved via three channels. That is, an AV upstream channel 616 (for transmitting audiovisual data from the transmitter to the cloud server), an AV downstream channel 618 (for transmitting audiovisual data from the cloud server to the receiver), and a control channel 614 (for transmitting commands, requests, and other information (e.g., sensor data) between the transmitter, the cloud server, and the receiver).

[0047] The transmitter 610 is configured to transmit user requests via a control channel 614 and audiovisual data via an AV upstream channel 616. The transmitter may transmit received AV data through one or more configurable input ports 612 connected to the local digital signal processor 608. The configurable input ports 612 may be configured to receive one or more sources, some of which may be partially or fully processed audio and / or video from the local media source 602, and stored in one or more dedicated buffers in memory. To accomplish this, the configurable input ports may receive data from the local digital signal processor 608 from multiple different intermediate stages of processing. For example, the configurable input ports 612 may receive data directly from a buffer in the receiver section 404 of the local digital signal processor 608. In another example, upmixing 406 may be performed on the local digital signal processor 608 before being sent to the transmitter via the configurable input ports 612. In both of the foregoing examples, the main signal processing 408 may be outsourced to a cloud-based digital signal processor 620, performed using one or more compatible cloud-based features on request 622. To select which cloud-based features to use, the transmitter 610 may request a list of available features and transmit the user's selection of compatible features.

[0048] Cloud-based features on request can be configured to be either dependent or independent of the configuration of the vehicle (e.g., automobile) to which the processed signal will ultimately be output for playback. For example, adding a reverberation effect to AV data does not take into account the vehicle's configuration, e.g., how many output channels the vehicle has, the type, placement, and / or number of loudspeakers, or other parameters. Other features may depend on the vehicle and / or its loudspeaker configuration. For example, a surround sound effect depends on the number of speakers, their type, location, their location and / or proximity to the driver and / or passengers, etc. Thus, the same digital signal processing features may be used in a variety of different cases, but the fine-tuning may depend on the vehicle's configuration and / or its multimedia system.

[0049] The configurable input port 612 may also include data from sensors within the vehicle. For example, a microphone (e.g., microphone 202) may be configured to pick up engine noise for input into one or more digital signal processing algorithms. In another example, digital and / or analog signals from the vehicle control system 230 may be used for signal enhancement. For example, the transmitter 610 may receive a noise profile of the vehicle's engine as input and then broadcast the engine noise data over the control channel 614. One or more cloud-based features on request 622 may be used to adjust audio from the local media source 602 and / or the cloud media source 604 based on the engine noise sample, for example, by selectively boosting frequency components of the digital audio associated with the frequency components of the engine noise. In this example, the configurable input port 612 may be adjusted depending on, for example, the positioning of the microphone within the vehicle and / or the type of engine. In a similar manner, other configurable input ports 612 may apply cloud-based digital signal processing based on input from sensors.

[0050] The receiver 624 can be configured to receive information from the AV downstream channel 618 and data from the control channel 614. The AV downstream channel 618 may provide audio and / or video data processed via cloud-based features on request 622, which may originate from either local media sources 602 and / or cloud media sources 604.

[0051] The receiver 624 includes a configurable output port 626 that allows data to be sent to the local digital signal processor 608. The processed data collected by the receiver 624 may be output using one or more of the configurable output ports 626 into the local digital signal processor 608, which may perform some further processing, such as, for example, a compressor, limiter, cross-morphing, etc. In the event of a network connection interruption and / or re-establishment, cross-morphing may be performed to shift from audiovisual data generated by the cloud-based digital signal processor 620 to audiovisual data generated by the local digital signal processor 608, as will be described in more detail with respect to FIG. 7.

[0052] The resulting output (which may be provided from a combination of the output of the local digital signal processor and the data collected from the receiver 624) may be passed into one or more audio sinks 630 and / or video sinks 632 via the output signal generated by the local digital signal processor 608 (e.g., via digital-to-analog conversion). For example, analog-to-digital conversion may be used to play the audio on one or more loudspeakers. The output may also be sent to another audio sink 630 or video sink 632, such as a cell phone connected to the car multimedia system 606 (e.g., via Bluetooth, USB, etc.).

[0053] The data received from the control channel 614 may include a list of available cloud features on request 622. The cloud features listed for a given user may depend on several factors, such as the user's subscription plan or feature compatibility. For example, some filters may not be usable together (e.g., if they are inverses of each other). The cloud-based features on request 622 may include various filters that require high computational complexity and memory (e.g., a complex reverberation effect designed to mimic the acoustics of a cathedral).

[0054] As mentioned above, some cloud-based feature-on-request may be vehicle-independent, while others depend on the configuration of the vehicle and the car multimedia system. Vehicle-independent features may not receive, process, and / or output digital signals on a physical channel when a network connection is established, but may utilize a logical channel (e.g., 7.1) as a target buffer (e.g., writing data to a 7.1 buffer when the feature is a reverberation or surround algorithm). Upon receiving a request for an abstract source / target buffer, the car multimedia system may locally convert the received, processed digital signal to a physical audio / video buffer corresponding to the requested logical audio / video buffer. Support for this mechanism presupposes agreement among feature-on-demand manufacturers or even standardization in referencing logical channels. One advantage of vehicle-independent features is easier maintenance and a wider range of potential users, since they are suitable for a wide family of audio systems, not just a given audio / video system in a specific car model. However, not all features may be vehicle-independent, and therefore, for some processing features applied by a cloud-based digital signal processor, the processed signal may be received by the car multimedia system using a designated physical channel (e.g., a physical AV buffer).

[0055] 7 illustrates an example method 700 for processing audiovisual data using a system including a vehicle multimedia system in communication with a cloud-based digital signal processor, such as the aforementioned system illustrated by block diagram 600. Method 700 may be performed using instructions stored on a memory of the vehicle multimedia system (e.g., in-vehicle computing system 109 and / or car multimedia system 606) in combination with instructions stored in a memory of a remote service (e.g., cloud-based digital signal processor 620). Method 700 generates output usable at one or more AV sinks (e.g., audio sink 630 and / or video sink 632, as non-limiting examples). The AV sinks may further include one or more other audio channels to be processed, for example, using locally available digital signal processing features, as described in the context of method 700. At 702, method 700 includes obtaining one or more AV data files (also referred to as digital media files) from one or more local and / or cloud media sources. Local media sources may include, for example, digital media files stored within the car multimedia system and / or one or more external devices communicatively coupled to the car multimedia system (e.g., without limitation, a hard drive, a navigation system, and / or a mobile phone). Cloud media sources may be accessible from or stored on a cloud server communicatively coupled to the cloud-based digital signal processor (e.g., cloud media source 604). Retrieval of AV data file(s) from media source(s) may occur in response to a user request to listen to and / or view digital media files and / or automatically in response to the car multimedia system being activated (e.g., following vehicle start), a navigation prompt or automobile alert being issued, or another event (e.g., AV data files may be pushed to the car multimedia system via a mobile phone).

[0056] If the AV data file is locally available (e.g., obtained from a local media source), at 704, method 700 may begin processing the audiovisual data in the AV data file via methods available to a local digital signal processor (e.g., local digital signal processor 608) of the car multimedia system. The local digital signal processor may receive the audiovisual data of the AV data file from the local media source(s), perform upmixing, and perform main signal processing to create a locally processed version of the AV data using one or more locally available processing features (which may include a first, smaller set of processing features). As described above with respect to FIG. 4, the AV data from the local media source(s) may be upmixed to form multiple channels of AV data (e.g., configured to be output to different speakers). The main signal processing may include processing the (upmixed) AV data to enable surround sound or other lower processing load audio features. The results of the main signal processing may be stored in a buffer of the car multimedia system. The contents of the buffer (e.g., the buffered signal) may be used in the event of a network connection interruption with the cloud-based digital signal processor. This will be described in more detail below. The local digital signal processor may include limited features compared to the cloud server. Audiovisual data processed in an intermediate step (e.g., AV data after upmixing) may also be available for transmission to the cloud-based digital signal processor (e.g., via an AV upstream channel). In this way, when an AV data file is provided from a local media source, the car multimedia system may preprocess the AV data of the AV data file by the vehicle's local digital signal processor.The pre-processing may include forming multiple digital AV signals (e.g., corresponding to the multiple channels described above) from the AV data file and storing the multiple digital AV signals in a buffer (e.g., in an input buffer such as the receiving unit 404 or in another buffer of a local digital signal processor).

[0057] At 706, method 700 includes checking network connectivity to the cloud-based digital signal processor and attempting to connect to the cloud-based digital signal processor. As described with respect to block diagram 600 above, the network connectivity may be established via an AV upstream channel, an AV downstream channel, and a control channel.

[0058] If a network connection with the cloud-based digital signal processor cannot be established, method 700 proceeds to 710, which includes modifying / using the processed AV data on the local digital signal processor. If the first attempted network connection is not established, the processed audiovisual data of the local digital signal processor may be used, and method 700 proceeds to 722, which outputs the audiovisual data (as processed by the car multimedia system's local digital signal processor) to one or more AV sinks (e.g., one or more loudspeakers and / or display screens). The processing performed by the local digital signal processor may include further processing beyond that described above (e.g., further processing beyond forming different channels, such as applying local processing features). Output to AV sinks may include performing digital-to-analog conversion as required for some types of output, such as loudspeakers. Thus, when a network connection with the cloud-based digital signal processor cannot be established, AV data processed by the local digital signal processor may be output to one or more AV sinks (e.g., speakers) using only the digital signal processing features available to the local digital signal processor, e.g., without using the digital signal processing features available to the cloud-based digital signal processor.

[0059] Alternatively, if a network connection with the cloud-based digital signal processor is established at 708, method 700 proceeds to 712. 712 includes enabling the user to select one or more cloud-based features on request. Enabling the user to select one or more cloud-based features on request may include receiving a list of available cloud-based features on request from the cloud (such as cloud-based features on request 622 described above) using a control channel (such as control channel 614). Through interaction with the user interface, the user may, for example, select one or more features of the cloud-based features on request displayed via the user interface. An indication of the requested features (e.g., user-selected features) may be sent to the cloud-based digital signal processor via the control channel, and the requested features may be used to process the digital audiovisual data. If features have already been selected by the user, method 700 may instead skip to 714. Additionally, in some examples, vehicle-specific system / playback features may also be sent to the cloud-based digital signal processor (e.g., loudspeaker configuration (e.g., type, size, position, etc.) and locally available processing features).

[0060] The features selected by the user at 712 may be checked against each other for compatibility. For example, features may be considered incompatible with each other if they utilize overlapping resources. In one example, two digital signal processing features may use the same buffer for audio processing, potentially creating undesirable effects by allowing one processing feature to overwrite the AV data of another processing feature. Feature compatibility may be checked by the car multimedia system and / or the cloud server. If one or more features are incompatible, the user may be notified at 712 and prompted to select a compatible feature.

[0061] At 714, the AV data is transmitted to the cloud server. For example, the AV data may be transmitted to the cloud server using an AV upstream channel. If the media source for the AV data is a cloud media source, the AV data does not need to be transmitted from the local digital signal processor because the source is already accessible to the cloud-based digital signal processor. However, if the media source for the AV data is a local media source, the AV data is transmitted to the cloud-based digital signal processor (714). Transmitting the AV data may include transmitting partially processed data (e.g., multiple digital AV signals generated from the preprocessing described above, which may correspond to different channels as described above) or unprocessed data via the AV upstream channel. In some examples, as described above, the local digital signal processor may perform upmixing, such that the AV data transmitted to the cloud server via the AV upstream channel is preprocessed (e.g., upmixed) AV data. In other examples, upmixing is not performed using the local digital signal processor, and the AV data transmitted to the cloud server may not be preprocessed.

[0062] In some examples, the AV data to be sent to the cloud server may be selected based on the processing features selected by the user. For example, the cloud server may send a request to the car multimedia system indicating which AV data (e.g., which of multiple digital AV signals) to send to the cloud server, and the car multimedia system may send the requested AV data. In some examples, the cloud server may request that all AV data be sent. In other examples, the cloud server may request that only a portion of the AV data be sent, such as only a portion of the digital AV signals (e.g., only the channels requested for the selected processing feature(s)).

[0063] At 718, method 700 includes receiving processed AV data from the cloud server and storing the received processed AV data in a buffer (e.g., an output buffer such as output 412). The processed AV data may be received via an AV downstream channel. The processed AV data stored in the buffer may be used, for example, to cross-morph from or cross-morph to the cloud-processed audiovisual data. At 720, method 700 includes modifying or using the AV data processed by the cloud server. If a network connection to the cloud server is successfully established, the AV data processed by the cloud-based digital signal processor may be used as an output source. If the network connection remains stable, method 700 may continue playing the cloud-processed AV data without any cross-morphing or modification in the source of the processed AV data. If playback of AV data is currently ongoing via output from the local digital signal processor, cross-morphing may be performed (720) to cross-morph into the audiovisual data processed in the cloud-based digital signal processor and received at 718. Method 700 then proceeds to 722, using the AV data processed by the cloud server as an output, to output the processed AV data to one or more AV sinks. In some examples, the cloud-based digital signal processor may transmit the AV data to a local digital signal processor for further digital signal processing, for example through one or more features available to the local digital signal processor, before outputting the processed AV data to one or more AV sinks.

[0064] In some examples, as described above, only a portion of the channels (e.g., digital signals) of AV data may be processed by a cloud-based digital signal processor. In such examples, the channels / digital signals received from the cloud server may be output to the appropriate AV sink(s), and the remaining locally processed channels / digital signals (e.g., not sent to or received from the cloud server) may be output to the appropriate AV sink(s).

[0065] The method 700 continuously checks for an active network connection to the cloud-based digital signal processor. Once the AV output is generated and sent to the AV sink (722), the method 700 loops back to 706 to check that the network connection to the cloud server is maintained (e.g., the connection continues to allow for the sending and receiving of AV data and control information).

[0066] An interruption or corruption of the network connection may be detected through various means. If AV data is transmitted via packets, the multimedia system's internal system clock may determine the time between the two most recently received packets. If the time between the two most recently received packets exceeds some threshold, method 700 may determine that the network connection has been lost. Additionally, an interruption or corruption of the network connection may be detected by using a checksum. The checksum may be any of a variety of encryption methods that can be used to determine the integrity of a given packet of data. The checksum method used by the car multimedia system may depend on the communication protocol used to establish the network connection to the cloud-based digital signal processor. If the checksum determines that a sufficient number of packets are complete, the car multimedia system determines that the network connection is stable and uncorrupted.

[0067] If the network connection is maintained, method 700 proceeds to 712, where it continues transmitting AV data to the cloud server and receiving processed AV data from the cloud server for playback. Otherwise, for example, if a previously established network connection is interrupted, method 700 proceeds to 710. At 710, data processed on the local digital signal processor may be used. Using the data processed on the local digital signal processor may be achieved, for example, by cross-morphing it to buffered AV data stored by the local digital signal processor and proceeding to 722. Because the processed AV data received from the cloud server is stored in a buffer (e.g., 10 milliseconds or more worth of data) before being output to the AV sink, the processed AV data in the buffer may be cross-morphed briefly with the AV data locally processed by the car multimedia system before only the locally processed AV data is output. Thus, method 700 switches between using a cloud-based AV output and using the output of the local digital signal processor based on the availability of the cloud-based AV output. Optionally, method 700 may end instead of looping back to 706, for example, if the user turns off the vehicle or otherwise stops playback.

[0068] 8 illustrates an example method 800 for processing audiovisual data using a system including a vehicle multimedia system in communication with a cloud-based digital signal processor, such as the previously described system illustrated by block diagram 600. Method 800 may be performed using instructions stored in a memory of a remote service, such as cloud-based digital signal processor 620 residing on a cloud-based server (also referred to as a cloud server).

[0069] At 802, method 800 includes receiving a request to process AV data. The request may be received, for example, via a control channel established between the car multimedia system and the cloud-based digital signal processor. Receiving the request to process the AV data further includes, at 804, receiving the AV data via an AV upstream channel, such as AV upstream channel 616. Receiving the AV data via the AV upstream channel may include receiving the AV data from the car multimedia system. The car multimedia system may be configured to obtain the AV data from one or more local media sources. Additionally or alternatively, receiving the request to process the AV data further includes obtaining data from one or more cloud media sources, such as cloud media source 604. The cloud media sources may include, for example, music available from a streaming service communicatively coupled to the cloud-based digital signal processor.

[0070] At 808, method 800 receives requested features on demand from a user. The request may be transmitted over a control channel. The features requested from the user over the control channel may include several digital signal processing features, such as cloud-based features on request 622. The user's request is stored in the memory of the cloud-based digital signal processor. Before receiving the requested features on demand, the cloud-based digital signal processor may transmit a list of available features over the control channel. Further, in some examples, based on the requested processing features, the cloud server may transmit a request to the car multimedia system indicating which aspects of the AV data to send to the cloud server for processing (e.g., which pre-processed digital AV signals to send). Compatibility of the digital signal processing features may be checked by the car multimedia system at 808, as further described at 712 of FIG. 7. If the requested features are incompatible, the cloud-based digital signal processor may transmit a notification to the car multimedia system indicating that the features are incompatible.

[0071] At 810, method 800 includes processing AV data on a cloud-based digital signal processor. The AV data processed at 810 is the AV data obtained at 802 and, therefore, may be received from one or more cloud media sources and / or local media sources. Data received via an AV upstream channel (e.g., data obtained at 806) may be partially processed (e.g., upmixed) or partially unprocessed by the car multimedia system's local digital signal processor. If the received audio is not upmixed, upmixing may be performed by the cloud-based digital signal processor at 810. Regardless of whether the received data is upmixed, primary digital signal processing may be performed by the cloud-based digital signal processor at 810 using one or more processing features requested by the user at 808. These features may include, for example, effects not available to the local digital signal processor itself, such as reverberation and frequency filtering effects that are too computationally intensive for the local digital signal processor. Computationally intensive digital processing effects include, but are not limited to, virtual bass, virtual speakers, advanced surround sound processing, adaptive equalization, and the like.

[0072] At 812, the method 800 includes transmitting the processed AV data to the car multimedia system. The processed AV data may be transmitted via an AV downstream channel, such as, for example, AV downstream channel 618. The method 800 then ends.

[0073] A technical effect of outsourcing the main signal processing to a cloud-based digital signal processor is that premium signal processing features that would not otherwise be available to users become available via a networking connection. Another technical effect is that new signal processing features can be added to the cloud-based digital signal processor, or existing signal processing features of the cloud-based digital signal processor can be adjusted without having to update the local digital signal processor in each vehicle. This can reduce network traffic and memory and processing loads on each vehicle's multimedia system.

[0074] Alternatively expressed, a method for on-demand processing of audiovisual (AV) data includes receiving a request from a vehicle's multimedia system to apply one or more selected processing features to selected AV data files; acquiring AV data corresponding to the selected AV data files; applying the one or more selected processing features to the AV data by a cloud-based digital signal processor to form a set of cloud-processed digital AV signals; and transmitting the set of cloud-processed digital AV signals to the vehicle's multimedia system. In a first example of the method, acquiring the AV data corresponding to the selected AV data files includes receiving the AV data from the vehicle's multimedia system. In a second example of the method, which optionally includes the first example, acquiring the AV data corresponding to the selected AV data files includes receiving the AV data from a cloud-based AV source. In a third example of the method, which optionally includes one or both of the first and second examples, the AV data corresponding to the selected AV data files includes one or more digital AV signals preprocessed by a local digital signal processor of the multimedia system. In a fourth example of the method optionally including one or more or each of the first through third examples, the one or more selected processing features include one or more of surround sound, immersion effects, reverberation, virtual speakers, virtual bass, and adaptive equalization. In a fifth example of the method optionally including one or more or each of the first through fourth examples, the method further includes determining a multimedia configuration of the vehicle and applying the one or more selected processing features to the AV data based on the multimedia configuration. In a sixth example of the method optionally including one or more or each of the first through fifth examples, the multimedia configuration of the vehicle includes one or more of a number of speakers in the vehicle, a position of each speaker in the vehicle, and a type of each speaker in the vehicle.In a seventh example of the method optionally including one or more or each of the first through sixth examples, the method further includes determining that the one or more selected processing features are independent of the vehicle's multimedia configuration and applying the one or more selected processing features to the AV data without determining the vehicle's multimedia configuration. In an eighth example of the method optionally including one or more or each of the first through seventh examples, obtaining AV data corresponding to the selected AV data file includes receiving the AV data from a broadcast media source, and applying the one or more selected processing features to the AV data to form the set of cloud-processed digital AV signals includes performing hyper-processing to apply the one or more selected processed features to the AV data at a rate faster than a playback rate of the AV data. In a ninth example of the method optionally including one or more or each of the first through eighth examples, applying the one or more selected processing features to the AV data to form the set of cloud-processed digital AV signals includes cross-morphing the set of cloud-processed digital AV signals with the AV data for a time threshold based on the one or more selected processing features.

[0075] The description of the embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be made in light of the foregoing description or may result from practicing the methods. For example, unless otherwise indicated, one or more of the described methods may be performed by any suitable device and / or combination of devices, such as the car multimedia system 226 described with reference to FIG. 2 . The methods may be performed by executing stored instructions by one or more logic devices (e.g., processors) in combination with one or more additional hardware elements (e.g., storage devices, memories, hardware network interfaces / antennas, switches, actuators, clock circuits, etc.). The described methods and associated operations may also be performed in various orders, in parallel, and / or simultaneously in addition to the order described in this application. The described systems are exemplary in nature and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and subcombinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0076] As used in this application, elements or steps listed in the singular and preceded by the terms "a" or "an" should be understood as not excluding a plurality of such elements or steps, unless such exclusion is expressly stated. Furthermore, references to "one embodiment" or "one example" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms "first," "second," "third," etc. are used merely as labels and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the foregoing disclosure that is believed to be novel and non-obvious.

Claims

1. 1. A method for on-demand audiovisual (AV) data processing for playback in a vehicle, comprising: pre-processing an AV data file by a local digital signal processor of the vehicle, the pre-processing including forming a plurality of digital AV signals from the AV data file; transmitting at least a portion of the plurality of digital AV signals to a cloud-based server configured to apply one or more processing features to the at least a portion of the plurality of digital AV signals; receiving a processed version of at least the portion of the plurality of digital AV signals from the cloud-based server; outputting the processed versions of at least the portions of the plurality of digital AV signals to one or more AV sinks in the vehicle.

2. 2. The method of claim 1, wherein transmitting at least the portion of the plurality of digital AV signals to the cloud-based server comprises: receiving a request from the cloud-based server indicating which of the plurality of digital AV signals to transmit to the cloud-based server; and transmitting the at least the portion of the plurality of digital AV signals to the cloud-based server in response to the request.

3. 3. The method of claim 2, further comprising sending to the cloud-based server an indication of the one or more processing features to apply to at least the portion of the plurality of digital AV signals, the cloud-based server being configured to determine which of the plurality of digital AV signals to request based on the one or more processing features to apply.

4. 10. The method of claim 1, wherein the plurality of digital AV signals are stored in a first buffer, and further comprising saving the processed version of the at least the portion of the plurality of digital AV signals in a second buffer.

5. 5. The method of claim 4, further comprising: determining that a network connection to the cloud-based server has been lost; and, in response, cross-morphing the processed versions of at least the portion of the plurality of digital AV signals stored in the second buffer with local processed versions of at least the portion of the plurality of digital AV signals, wherein the local processed versions of at least the portion of the plurality of digital AV signals are processed by the local digital signal processor.

6. 6. The method of claim 5, further comprising: determining that the network connection to the cloud-based server has been restored; and responsively cross-morphing the processed versions of at least the portion of the plurality of digital AV signals received from the cloud-based server with locally processed versions of at least the portion of the plurality of digital AV signals after the network connection is restored.

7. 10. The method of claim 1, further comprising transmitting only the portion of the plurality of digital AV signals to the cloud-based server, receiving processed versions of only the portion of the plurality of digital AV signals from the cloud-based server, and outputting the remaining portion of the plurality of digital AV signals, along with the processed versions of only the portion of the plurality of digital AV signals, to the one or more AV sinks.

8. The method of claim 1 , wherein the AV data files are obtained from a local media source located within the vehicle.

9. 2. The method of claim 1 , wherein the AV data files include audio files, the one or more AV sinks include one or more speakers of the vehicle, and the one or more processing features applied to at least the portions of the plurality of digital AV signals include one or more of surround sound, immersion effects, reverberation, virtual speakers, virtual bass, and adaptive equalization.

10. A multimedia system for a vehicle, comprising: a digital signal processor including: an input buffer configured to receive an AV data file; an upmixer configured to pre-process the AV data file to form a plurality of digital AV signals; a main signal processor configured to apply a first, smaller set of processing features to the plurality of digital AV signals; and an output buffer; a transmitter configured to transmit the AV data file in the input buffer or one or more of the plurality of digital AV signals from the upmixer to a cloud server, the cloud server configured to apply a second, larger set of processing features to the AV data file or the plurality of digital AV signals to form a set of cloud-processed digital AV signals; a receiver configured to receive the set of cloud-processed digital AV signals; one or more AV sinks configured to output the set of cloud-processed digital AV signals for playback.

11. 11. The multimedia system of claim 10, wherein the AV data file includes an audio file, the one or more AV sinks include one or more speakers of the vehicle, and the second, larger set of processing features includes one or more of surround sound, immersion effects, reverberation, virtual speakers, virtual bass, and adaptive equalization.

12. The multimedia system of claim 10 , wherein the AV data files are obtained from a local media source located within the vehicle.

13. 13. The multimedia system of claim 12, wherein the local media source includes a mobile phone, and the transmitter is configured to transmit the AV data file in the input buffer or one or more of the plurality of digital AV signals from the upmixer to the cloud server via the mobile phone.

14. 11. The multimedia system of claim 10, further comprising a crossmorpher configured to crossmorph the set of cloud-processed digital AV signals with locally processed versions of the digital AV signals in response to determining that network connectivity to the cloud server has been lost, the locally processed versions of the digital AV signals being processed by the digital signal processor.

15. 11. The multimedia system of claim 10, wherein the receiver is configured to convert a requested abstract target buffer received together with the set of cloud-processed digital AV signals into one or more physical buffers corresponding to the requested abstract target buffer, and transmit the set of cloud-processed digital AV signals to the one or more AV sinks according to the one or more physical buffers.

Images