Adaptive audio system for occupant-aware vehicles

Abstract

Generally, the present disclosure describes various aspects of techniques relating to adaptive audio systems for occupant-aware vehicles. A device configured to render a sound field based on audio data within a vehicle may implement these techniques. The apparatus may include: a memory configured to store the audio data; and a processing circuit coupled to the memory. The processing circuit may be configured to: obtain a state of an occupant residing within a cabin of the vehicle from an occupant monitoring system; modifying playback of the audio data within at least a portion of the cabin of the vehicle based on the status of the occupant residing within the cabin of the vehicle to obtain modified playback data for the audio data; and reproducing the sound field based on the modified playback data.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 470,800, filed on September 20, 2023, and U.S. Provisional Application No. 63 / 380,168, filed on October 19, 2022, the entire contents of which are hereby incorporated by reference. U.S. Patent Application No. 18 / 470,800, filed on September 20, 2023, claims the benefit of U.S. Provisional Application Serial No. 63 / 380,168, filed on October 19, 2022. Technical Field

[0002] The present disclosure relates to processing media data, such as audio data. Background Art

[0003] Many vehicles are equipped with an Occupant Monitoring System (OMS) that provides information about the vehicle occupants, such as the operator of the vehicle, to determine the awareness of the operating environment in which the operator is operating the vehicle (e.g., the operator's state - such as impaired, asleep, distracted, etc.), provides information about the vehicle's passengers to monitor presence (e.g., for airbag deployment), safety (e.g., notifying the status of seat belts in a dashboard notification system), etc. Such vehicles may also be equipped with entertainment or infotainment systems that reproduce a sound field via speakers based on audio data (or, in other words, an audio signal). Summary of the Invention

[0004] The present disclosure generally relates to an adaptive audio system for occupant awareness in a vehicle. A vehicle head unit or other device may implement the adaptive audio system to interface with an Occupant Monitoring System (OMS) to receive status updates about the occupants residing in the vehicle's cabin. The adaptive audio system may receive status updates that indicate, as an example, that a rear passenger residing in the rear passenger area of the vehicle's cabin is sleeping (e.g., an infant is sleeping). Instead of reproducing the sound field throughout the vehicle, the adaptive audio system may modify the playback of the audio data based on the status updates provided by the OMS (where, in this example, the adaptive audio system may mute the playback of the audio data in the rear passenger area of the vehicle's cabin).

[0005] In this regard, various aspects of the technology can improve the operation of the infotainment system (which otherwise refers to the vehicle head unit) itself. For example, rather than reproducing the sound field indiscriminately in all parts of the vehicle cockpit, an adaptive audio system can modify (or in other words, adapt) the audio playback based on the status updates provided by the OMS to mute the reproduction of the sound field in one or more parts of the vehicle cockpit (or in other words, the vehicle cabin). Muting the volume of the playback in a particular part of the vehicle cabin can maintain, facilitate, or support the continuation of the status of the occupants (e.g., a sleeping child) monitored by the OMS. By facilitating the continuation of the occupants' status, other occupants of the vehicle can continue to enjoy the content (e.g., the playback of audio data) without being distracted or otherwise disturbing the status of other occupants within the vehicle cabin, thereby enhancing the enjoyment of the content reproduction provided by the infotainment system itself.

[0006] In one example, various aspects of the technology relate to an apparatus configured to reproduce a sound field based on audio data within a vehicle, the apparatus including: a memory configured to store the audio data; and a processing circuit coupled to the memory and configured to: obtain the status of an occupant residing within the vehicle cabin from an occupant monitoring system; modify the playback of the audio data within at least a portion of the vehicle cabin based on the status of the occupant residing within the vehicle cabin to obtain modified playback data for the audio data; and reproduce the sound field based on the modified playback data.

[0007] In another example, various aspects of the technology relate to a method of reproducing a sound field based on audio data within a vehicle, the method including: obtaining the status of an occupant residing within the vehicle cabin from an occupant monitoring system; modifying the playback of the audio data within at least a portion of the vehicle cabin based on the status of the occupant residing within the vehicle cabin to obtain modified playback data for the audio data; and reproducing the sound field based on the modified playback data.

[0008] In another example, various aspects of the technology relate to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors of a vehicle head unit to: obtain the status of an occupant residing within the vehicle cabin of a vehicle including the vehicle head unit from an occupant monitoring system; modify the playback of audio data representing the sound field within at least a portion of the vehicle cabin based on the status of the occupant residing within the vehicle cabin to obtain modified playback data for the audio data; and reproduce the sound field based on the modified playback data and the audio data.

[0009] Details of one or more examples of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the various aspects of the technology will be apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an example vehicle configured to perform various aspects of the adaptive audio system technology described in the present disclosure.

[0011] Figures 2A to 2D is an illustration of Figure 1 example operations of the AAS shown in an example of the AAS when performing various aspects of the adaptive audio playback technology described in the present disclosure.

[0012] Figure 3 is an illustration of Figure 1 a flowchart of example operations of the vehicle shown in an example of the vehicle when performing various aspects of the adaptive audio playback technology described in the present disclosure.

[0013] Figure 4 is a conceptual diagram illustrating an example of a wireless communication system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0014] Many vehicles are equipped with entertainment or infotainment systems (which are otherwise referred to as vehicle head units) that reproduce a sound field via speakers based on audio data (or, in other words, audio signals). Additionally, many vehicles such as vehicle 100 include an occupant monitoring system (OMS) that can monitor the status of occupants within the vehicle's cabin.

[0015] The infotainment system can be configured to adapt operating conditions (e.g., current safety notifications such as the status of seat belts for passengers) based on status updates provided by the OMS. That is, the infotainment system can use the status updates provided by the OMS to adapt the user interface, activate one or more cameras, activate one or more microphones, etc., to enable occupant awareness functionality regarding safety, awareness, activities, etc. that occur within the vehicle's cabin. However, the infotainment system may not adapt the reproduction of the sound field based on OMS status updates regarding the various occupants within the vehicle.

[0016] In accordance with various aspects of the technology described in this disclosure, a vehicle or other device (e.g., a vehicle host unit) may implement an Adaptive Audio System (AAS) that interfaces with an OMS to receive status updates regarding the occupants residing within the vehicle's cabin. The Adaptive Audio System may receive status updates that indicate (as an example) that a rear passenger residing within the rear passenger area of the vehicle's cabin is sleeping (e.g., a child - an infant is sleeping). Instead of reproducing the sound field identically throughout the vehicle, the Adaptive Audio System may modify the playback of audio data (AD) based on the status updates provided by the OMS (where, in this example, the Adaptive Audio System may mute the playback of the audio data within the rear passenger area of the vehicle's cabin).

[0017] In this regard, various aspects of the technology may improve the operation of the infotainment system (which otherwise refers to the vehicle host unit) itself. For example, instead of reproducing the sound field indiscriminately throughout all parts of the vehicle's cabin, the Adaptive Audio System may modify (or in other words, adapt) the audio playback based on the status updates provided by the OMS to mute the reproduction of the sound field within one or more parts of the vehicle's cabin (or in other words, the vehicle cockpit). Muting the volume of the playback within a particular part of the vehicle cockpit may maintain, facilitate, or support the continuance of the status of the occupant (e.g., a sleeping child) being monitored by the OMS. By facilitating the continuance of the occupant's status, other occupants of the vehicle may continue to enjoy the content (e.g., the playback of audio data) without being distracted or otherwise disturbing the status of other occupants within the vehicle cabin, thereby enhancing the enjoyment of the content reproduction provided by the infotainment system itself.

[0018] Figure 1 FIG. is a block diagram of an example vehicle configured to perform various aspects of the transparent audio mode technology described in this disclosure. In the following description, vehicle 100 is assumed to be an automobile. However, the technology described in this disclosure may be applied to any type of vehicle capable of transporting occupants within a cabin, such as a bus, a recreational vehicle (RV), a semi - tractor truck, a tractor or other types of farm equipment, a train, an airplane, a personal transportation vehicle, etc.

[0019] In Figure 1 the example of, vehicle 100 includes a processing circuit 112, an audio circuit 114, and a memory device 116. In some examples, processing circuit 112 and audio circuit 114 may be formed as an integrated circuit (IC). For example, the IC may be regarded as a processing chip within a chip package and may be a system - on - a - chip (SoC).

[0020] Examples of the processing circuit 112 and the audio circuit 114 include, but are not limited to, one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), fixed function circuits, programmable processing circuits, any combination of fixed function and programmable processing circuits, or other equivalent integrated circuits or discrete logic circuits. The processing circuit 112 may be a central processing unit (CPU) of the vehicle 100. In some examples, the audio circuit 114 may be dedicated hardware including integrated and / or discrete logic circuits that provide parallel processing capabilities for the audio circuit 114.

[0021] The processing circuit 112 may execute various types of applications, such as various occupant experience related applications, including climate control interface applications, entertainment and / or infotainment applications, cellular phone interfaces (e.g., as implemented using a link), navigation applications, vehicle functionality interface applications, web or directory browsers, or other applications that enhance the occupant experience within the bounds of the vehicle 100. The memory device 16 may store instructions for executing one or more applications.

[0022] The memory device 116 may include the total memory for the vehicle 100, may be the total memory for the vehicle 100, or may be a part of the total memory for the vehicle 100. The memory device 116 may include one or more computer-readable storage media. Examples of the memory device 116 include, but are not limited to, random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other media that can be used to carry or store desired program code in the form of instructions and / or data structures and that can be accessed by a computer or one or more processors (e.g., the processing circuit 112 and / or the audio circuit 114).

[0023] In some aspects, the memory device 116 may include instructions that cause the processing circuit 112 and / or the audio circuit 114 to perform the functions ascribed to the processing circuit 112 and / or the audio circuit 114 in this disclosure. Thus, the memory device 16 may be a computer-readable storage medium (including a non-transitory computer-readable storage medium) having instructions stored thereon that, when executed, cause one or more processors (e.g., the processing circuit 112 and / or the audio circuit 114) to perform various functions ascribed to the processing circuit 112 and / or the audio circuit 114.

[0024] The memory device 116 is a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be construed to mean that the memory device 116 is immovable or that its contents are static. As an example, the memory device 116 can be removed from the vehicle 100 and moved to another device. As another example, a memory substantially similar to the memory device 116 can be inserted into one or more receiving ports of the vehicle 100. In some examples, the non-transitory storage medium can store data that changes over time (e.g., in RAM).

[0025] As Figure 1 further shown in the example of, the vehicle 100 can include an interface device 122, a camera 124, a plurality of microphones 128, and one or more speakers 126. In some examples, the interface device 122 can include one or more microphones configured to capture audio data within the vehicle 100. In some examples, the interface device 122 can include an interactive input / output display device, such as a touch screen or other presence-sensitive display. For example, the display device that can form part of the interface device 122 can represent any type of passive screen on which an image can be projected, or an active screen capable of projecting an image (such as a light-emitting diode (LED) display, an organic LED (OLED) display, a liquid crystal display (LCD), or any other type of active display) with built-in input reception capabilities.

[0026] Although shown as a single device for ease of illustration in Figure 1 the interface device 122 can include a plurality of user-facing devices configured to receive input and / or provide output. In various examples, the interface device 122 can include a display in wired or wireless communication with the vehicle 100, such as a head-up display, a head-mounted display, an augmented reality computing device (such as "smart glasses"), a virtual reality computing device or display, a laptop computer or netbook, a mobile phone (including so-called "smartphones"), a tablet computer, a gaming system, or another type of computing device capable of serving as an extension of or replacing the display integrated into the vehicle 100.

[0027] The interface device 122 can represent any type of physical or virtual interface with which a user can interface to control various functions of the vehicle 100. The interface device 122 can include physical buttons, knobs, sliders, or other physical control implements. The interface device 122 can also include a virtual interface whereby an occupant of the vehicle 100 interacts with virtual buttons, knobs, sliders, or other virtual interface elements via, as an example, a touch-sensitive screen. The occupant can interface with the interface device 122 to control one or more of the climate within the vehicle 100, the audio playback of the vehicle 100, the video playback of the vehicle 100, transmissions via the vehicle 100 (such as cellular phone calls), or any other operation that can be performed by the vehicle 100.

[0028] When acting as an extension of or in place of a display integrated into the vehicle 100, the interface device 122 can also represent an interface extending from the vehicle 100. That is, the interface device 122 can include a virtual interface presented via the above-described HUD, augmented reality computing device, virtual reality computing device, or display, tablet computer, or any other extended display of the different types listed above. The vehicle 100 can include a steering wheel for controlling the direction of travel of the vehicle 100, one or more pedals for controlling the rate of travel of the vehicle 100, one or more manual brakes, etc. In some examples, the steering wheel and pedals can be included in a specific in-vehicle area of the vehicle 100, such as in a driver area or a pilot area.

[0029] For illustrative purposes, the processing circuitry 112, the audio circuitry 114, and the interface device 122 can form or otherwise support the operation of a so-called host unit (which can also be referred to as a vehicle host unit). Thus, a reference to the host unit can refer to a computing device integrated within the vehicle 100 that includes the processing circuitry 112, the audio circuitry 114, and the interface device 122. The processing circuitry 112 can execute an operating system (OS) having a kernel (the kernel being an OS layer that facilitates interaction with the underlying hardware of the host unit and other connected hardware components and that executes within a protected OS space) that supports the execution of applications within an application space provided by the OS.

[0030] The camera 124 of the vehicle 100 may represent one or more image and / or video capture devices configured to capture image data (where a sequence of the image data may form video data). The vehicle 100 may include a single camera capable of capturing 360-degree image / video data, or multiple cameras configured to capture a portion of the surrounding environment of the vehicle 100 (where each portion may be stitched together to form 360-degree image / video data). In some examples, the camera 124 may only capture discrete portions of the 360-degree image / video data (and not all of the portions required to form 360-degree image / video data). In other examples, the camera 124 may enable the capture of three-dimensional image / video data representing the entire visual scene around the vehicle 100.

[0031] The camera 124 may be disposed in a single location on the body of the vehicle 100 (e.g., the roof of the vehicle 100) or in multiple locations around the body of the vehicle 100 and pointing outward from the vehicle to capture image / video data representing the external visual scene in which the vehicle 100 operates. The camera 124 may assist with various levels of autonomous driving, safety systems (e.g., lane assist, dynamic cruise control, etc.), vehicle operations (e.g., a rearview camera for assisting with backing up the vehicle 100), and the like.

[0032] The camera 124 may also be disposed inside the cockpit of the vehicle 100. The camera 124 may capture images depicting the interior of the cockpit of the vehicle 100 in order to evaluate the state of the occupants within the vehicle 100. For example, the camera 124 may capture an image of the operator (or, in other words, the driver) of the vehicle 100 to evaluate the level of awareness of the operating state in which the vehicle 100 is being operated. The camera 124 may also identify one or more occupants who are passengers of the vehicle 100 and identify various states of the passengers (e.g., sleeping, consuming media, talking, resting, etc.).

[0033] The microphone 128 of the vehicle 100 may represent a microphone array, which represents multiple different microphones 128 placed outside the vehicle 100 to capture the sound scene of the environment in which the vehicle 100 is operating. Each microphone 128 may represent a transducer that converts sound waves into electrical signals (which may be referred to as audio signals and, when processed into digital signals, may be referred to as audio data). One or more of the microphones 128 may represent a reference microphone and / or an error microphone for performing audio signal processing (e.g., wind noise cancellation, active noise cancellation, etc.).

[0034] The microphone 128 can also be disposed inside the cockpit of the vehicle 100. As pointed out above, the microphone 128 can include a reference microphone and / or an error microphone for performing audio signal processing (e.g., active noise cancellation, etc.). The microphone 128 can be internally disposed inside the cockpit of the vehicle 100 in a specific area (e.g., the driver area or, in other words, the driver area, the front passenger area, the rear passenger area (including both the driver-side rear passenger area and the passenger-side rear passenger area), etc.). The microphone 128 can capture audio data representing the sound field in the corresponding driver / passenger / rear passenger area.

[0035] The speaker 126 represents a component of the vehicle 100 that reproduces a sound field based on an audio signal directly or indirectly provided by the processing circuit 112 and / or the audio circuit 114. For example, the speaker 126 can generate a pressure wave based on one or more electrical signals received from the processing circuit 112 and / or the audio circuit 114. The speaker 126 can include various types of speaker hardware, including speakers based on full-range drivers, individual speakers including multiple range-specific dynamic drivers, or speakers including a single dynamic driver such as a tweeter or a woofer.

[0036] The audio circuit 114 can be configured to perform audio processing on the audio signal / audio data captured via the microphone 128 in order to drive the speaker 126. The audio circuit 114 can also receive audio signal / audio data from the processing circuit 112, and the audio circuit 114 can process the audio signal / audio data in order to drive the speaker 126. As used herein, the term "drive" can refer to the process of providing an audio signal to the speaker 126, which includes a driver through which the audio signal is converted into a pressure wave (which is otherwise referred to as a sound wave). The term "drive" refers to providing such an audio signal to the driver of the speaker 126 in order to reproduce the sound field represented by the audio signal (which is otherwise referred to as a sound scene).

[0037] Many vehicles such as the vehicle 100 are equipped with an entertainment or infotainment system (which is otherwise referred to as a vehicle head unit) that reproduces a sound field based on an audio signal (or, in other words, an audio signal) via a speaker such as the speaker 126. Additionally, many vehicles such as the vehicle 100 include an Occupant Monitoring System (OMS) 115, which in this example is software executed by the processing circuit 112, the audio circuit 114, etc. and interacts with the interface device 122, the camera 124, the microphone 128, etc. That is, the OMS 115 can monitor the state of the occupants inside the cockpit of the vehicle 100.

[0038] The infotainment system can be configured to adapt to operating conditions (e.g., current safety notifications such as the status of seat belts for passengers) based on the status updates provided by the OMS 115. That is, the infotainment system can use the status updates provided by the OMS 115 to adapt the user interface, activate one or more of the cameras 124, activate one or more of the microphones 128, etc., to enable occupant awareness functionality regarding safety, awareness, activities, etc. occurring within the cabin of the vehicle 100. However, the infotainment system may not adapt the reproduction of the sound field based on the OMS status updates regarding the various occupants within the vehicle 100.

[0039] In accordance with various aspects of the techniques described in the present disclosure, the vehicle 100 or other device (e.g., the vehicle host unit) can implement an Adaptive Audio System (AAS) 117 that interfaces with the OMS 115 to receive status updates regarding the occupants residing within the cabin of the vehicle 100. The Adaptive Audio System 117 can receive status updates that indicate (by way of example) that a rear passenger residing in the rear passenger area of the cabin of the vehicle 100 is sleeping (e.g., a child - an infant is sleeping). Instead of reproducing the sound field fully throughout the vehicle 100, the Adaptive Audio System 117 can modify the playback of the Audio Data (AD) 127 based on the status updates provided by the OMS 115 (wherein, in this example, the Adaptive Audio System 117 can mute the playback of the Audio Data 127 in the rear passenger area of the cabin of the vehicle 100).

[0040] In operation, the AAS 117 can obtain the status of the occupants residing within the cabin of the vehicle 100 from the OMS 115. The OMS 115 (executed by the processing circuit 112) can invoke one or more of the cameras 124 to capture occupant visual data (not shown in the example for ease of illustration) that may include images, videos, etc. of the occupants residing within the vehicle cabin. The OMS 115 can also invoke one or more of the microphones 128 to further capture occupant audio data (which may be similar to the AD 127) in one or more areas of the vehicle cabin. Figure 1 In the example of

[0041] OMS115 can determine the current state of each occupant residing in the cockpit of vehicle 100 based on occupant visual data and occupant audio data. OMS115 can represent one or more trained machine learning modules that are applied to the occupant visual data and / or the occupant audio data to identify the state of each occupant residing in the vehicle cockpit (which may also be referred to in another way as the cockpit of vehicle 100). OMS115 can identify the current state of each occupant residing in the vehicle cockpit, where such state includes a level of awareness (such as attentive, awake, active, distracted, asleep, resting, etc.) based on the occupant visual data and / or the occupant audio data.

[0042] OMS115 can push a status update to AAS117, where "push" refers to an application programming interface (API), where the corresponding application (i.e., AAS117 in this example) registers for occupant status updates and receives various software notifications (such as exceptions or interruptions) that signal that a new occupant status is available for processing. Alternatively or in combination, OMS115 can operate in a pull status update for AAS117, where AAS117 issues a request to obtain an occupant status update in response to an indication from OMS115 that the occupant status of a particular occupant among the occupants residing in the vehicle cockpit has changed.

[0043] In any case, AAS117 (as performed by the processing circuit and / or the audio circuit 114) can obtain an occupant status update that defines the state of the occupants residing in the cockpit of vehicle 100. Next, AAS117 can modify the playback of the audio data 127 in at least a portion of the cockpit of vehicle 100 based on the state of the occupants residing in the cockpit of vehicle 100 to obtain modified playback data for the audio data 127.

[0044] For example, OMS115 can determine that a rear passenger in the cockpit of vehicle 100 is asleep (e.g., a child is sleeping in the driver-side rear passenger area of the vehicle cockpit). In this example, OMS115 can interface with AAS117 (via the API exposed by AAS117) to convey a status update indicating that the passenger in the driver-side rear passenger area of the cockpit of vehicle 100 is sleeping. AAS117 can be configured to generate modified audio playback data in response to being called and receiving the status update, where the modified audio playback data indicates that the audio playback of AD 127 is to be muted in the driver-side rear passenger area of the cockpit of vehicle 100.

[0045] Then, the AAS 117 can render the AD 127 based on the modified audio playback data such that the gain associated with the channels of the AD 127 associated with the rear passenger area on the driver's side of the vehicle cockpit is muted. In this regard, the AAS 117 can respond to an indication of a sleeping state that an occupant residing in the rear passenger area (in this instance, on the driver's side) of the cockpit of the vehicle 100 is sleeping to mute the audio playback in the rear passenger area (in this instance, on the driver's side) of the cockpit of the vehicle 100. Then, the AAS 117 can output the rendered channels to the speaker 127 to reproduce the sound field (represented by the AD 127) based on the modified playback data.

[0046] In this regard, various aspects of the technology can improve the operation of the infotainment system (which is otherwise referred to as the vehicle head unit) itself. For example, rather than reproducing the sound field represented by the AD 127 indiscriminately in all parts of the cockpit of the vehicle 100, the AAS 117 can modify (or in other words, adapt) the audio playback based on the status update provided by the OMS 115 to mute the reproduction of the sound field in one or more parts of the cockpit of the vehicle 100 (or in other words, the vehicle cockpit). Muting the volume of the playback in a particular part of the vehicle cockpit can maintain, facilitate, or support the continuation of the status of the occupant (such as a sleeping child) monitored by the OMS 115. By facilitating the continuation of the occupant's status, other occupants of the vehicle 100 can continue to enjoy the content (such as the playback of audio data) without being distracted or otherwise disturbing the status of other occupants within the vehicle cockpit, thereby enhancing the enjoyment of the content reproduction provided by the infotainment system itself by the occupants.

[0047] Figures 2A to 2D is an illustration Figure 1 of an example operation of the AAS when performing various aspects of the adaptive audio playback technology described in the present disclosure. First referring to Figure 2A the example of, a vehicle 200A is shown, which includes a vehicle head unit (VHU) 202 (the "VHU 202") representing an example of the processing circuit 112, the audio circuit 114, and / or the interface device 122, etc. shown in the example of Figure 1 The VHU 202 can represent an example of a device configured to perform (such as execute the instructions of the processing circuit 112 and / or the audio circuit 114 - 112 / 114 represented by the OMS 115 and / or the AAS 117) various aspects of the adaptive audio technology described in the present disclosure.

[0048] As Figure 2AAs shown in the example of, VHU 202 can dock with speakers 226A - 226E ("speakers 226"), and these speakers can represent Figure 1 the example of speaker 126 shown in the example of. VHU 202 can also dock with microphones 228A - 228F ("microphones 228"), and these microphones can similarly represent the example of microphone 128. Although described with respect to five (5) speakers (i.e., speakers 226) and six (6) microphones 228, VHU 202 can dock with more or fewer speakers 226 and more or fewer microphones 228.

[0049] In any case, VHU 202 can execute OMS115 (which in other examples can be executed by other processing circuits located in other positions within vehicle 200A, including different devices not associated with vehicle 200A, such as smart phones, smart watches, smart glasses, tablet computers, laptop computers, gaming systems, portable computing devices, etc.). In any case, OMS115 can dock with camera 124 (not shown in the Figure 2A example of for ease of illustration) to obtain occupant visual data representing the interior (or in other words, the cabin) of vehicle 200A. OMS115 can also dock with microphones 228 to obtain occupant audio data representing the interior of vehicle 200A.

[0050] In Figure 2A the example of, vehicle 200A includes a cabin (or in other words, the interior of vehicle 200A) that is divided into separate regions 230B - 230E ("regions 230"), where region 230B represents the driver - side front - passenger (or in other words, occupant) region, region 230C represents the passenger - side front - passenger region, region 230D represents the driver - side rear - passenger region, and region 230E represents the passenger - side rear - passenger region. OMS115 can dock with camera 124 and / or microphone 228A to capture occupant visual data and / or occupant audio data to identify whether the driver (or in other words, the operator) of vehicle 200A is present. When the driver / operator is present, OMS115 can then dock with AAS117 to provide the status of the driver / operator of vehicle 200A.

[0051] Similarly, OMS115 can dock with various cameras 124 and / or microphones 228 for monitoring each of the regions in regions 230 in order to determine the presence of occupants at each of the regions in regions 230. Next, when an occupant is present in any of the regions in regions 230, OMS115 can analyze the visual data and / or audio data to determine the current status of the occupants residing at each of the regions in regions 230 of vehicle 200A.

[0052] That is, in addition to docking with the camera 124 and the microphone 228A for the driver-side front row area 230B, the OMS 115 can also dock with the camera 124 and the microphone 228B to obtain occupant visual data and / or occupant audio data for the front passenger-side area 230C. The OMS 115 can dock with the camera 124 and / or the microphone 228C and / or 228E to obtain occupant visual data and / or occupant audio data for the driver-side rear passenger area 230D. The OMS 115 can dock with the camera 124 and the microphone 228C and / or 228F to obtain occupant visual data and / or occupant audio data for the passenger-side rear passenger area 230E.

[0053] In Figure 2A the example, the OMS 115 can determine the presence of an occupant in the driver-side rear passenger area 230D based on the occupant visual data and / or occupant audio data captured for the driver-side rear passenger area 230D. The OMS 115 can analyze the occupant visual data and / or occupant audio data to determine the current state of the occupant residing in the driver-side rear passenger area 230D. In this example, the OMS 115 can determine that the occupant residing in the driver-side rear passenger area 230D is sleeping (or, in other words, is asleep). The OMS 115 can dock with the AAS 117 (via the API pointed out above) to transmit the updated state of the occupant residing in the driver-side rear passenger area 230D.

[0054] In response to receiving the updated state of the occupant residing in the driver-side rear passenger area 230D, the AAS 117 can reduce the gain (or, in other words, mute) associated with the speaker channel rendered for the speaker 226D located near (or, in other words, closest to) the driver-side rear passenger area 230D. The AAS 117 can use the modified playback data indicating the channel-specific volume (e.g., reduced volume or no volume) of a specific speaker (i.e., the speaker 226D in this example) to mute the speaker channel rendered for the speaker 226D.

[0055] Then, the AAS 117 can render the AD 127 based on the modified playback data to obtain one or more speaker feeds representing the electrical signals for driving the speaker 226. The AAS 117 can output the speaker feeds to the speaker 226, and these speakers can reproduce the sound field based on the speaker feeds. As Figure 2A shown in the example, the speaker 226D does not reproduce any sound field, as represented by the circle with a straight deletion.

[0056] Next, referring to Figure 2B the example, the vehicle 200B can represent Figure 1Another example of the vehicle 100 shown in the example of

[0057] In response to receiving an updated status of an occupant residing in the driver-side rear passenger area 230D, the AAS 117 may perform active noise cancellation on the driver-side rear passenger area 230D of the vehicle 200B's cockpit in response to a sleeping status indicating that the occupant residing in the rear passenger area (e.g., area 230D) of the vehicle's cockpit is sleeping, including modifying the audio data before playback within the driver-side rear passenger area 230D to limit the sound field reproduction in the driver-side rear passenger area 230D.

[0058] Active noise cancellation may involve generating a reverse wave that interprets the audio sound fields captured by a reference microphone and / or an error microphone (which may be represented as one or more of the microphones 128 / 228) (meaning a wave that is 180 degrees out of phase with these audio sound fields). When the sound field exists in the physical domain, the anti-sound wave may cancel out the sound waves output by other speakers in the speaker 226, which may reduce or eliminate the unwanted noise in any of the areas 230 (including the reproduced sound field based on the AD 127). Active noise cancellation is represented in Figure 2B the example as a circle with wavy strike-through lines.

[0059] In Figure 2C the example of Figure 1 Another example of the vehicle 100 shown in the example of Figures 2A to 2D The vehicle 200C may be similar to (if not substantially similar to) the vehicle 200A and / or 200B, except that the AAS 117 may either alternatively or in addition to the examples described herein with respect to

[0060] The VHU 202 that executes AAS117 can retrieve soothing audio data from a streaming audio source, from a dedicated in-vehicle memory for soothing audio data, and / or from the microphone 228 (e.g., the recording of another occupant residing within the vehicle 200C—or from the microphone of a device associated with the vehicle 200C such as a smart phone, smart watch, smart glasses, laptop computer, tablet computer, etc. of a passenger associated with the vehicle 200C). AAS117 can retrieve or otherwise obtain soothing audio data from almost any source, including sources configured via the VHA 202 to provide soothing audio data. Based on the reproduction of the sound field of the soothing audio data, in Figure 2C the example is shown as curved sound field lines.

[0061] In Figure 2D the example, the vehicle 200D can represent Figure 1 another example of the vehicle 100 shown in the example of Figures 2A to 2D . The vehicle 200D can be similar (if not substantially similar) to the vehicles 200A - 200C, except that the VHU 2 - 2 can, as an alternative or in addition to the examples described herein with respect to Figures 2A to 2D the example, alert the operator of the vehicle that the occupant in the rear child area of the cockpit of the vehicle 200D is a child in response to detecting the state of the occupant in the area 230. In some cases, the VHU 202 can adjust the heating, ventilation, and air conditioning (HVAC) settings for the rear passenger area of the cockpit of the vehicle 200D (i.e., the rear passenger areas 230D and / or 230E in this example) in response to detecting the state of the occupant in the rear passenger area (i.e., the rear passenger areas 230D and / or 230E in this example).

[0062] Additionally, the VHU 202 can determine the operating state of the vehicle 200D, which can refer to the state of the vehicle 200D such as driving, waiting, stalling, parked, internal temperature, external temperature, etc. The VHU 202 can perform safety actions based on the state of the occupants residing within the vehicle 200D and the operating state of the vehicle 200D to promote the safety of the occupants residing within the vehicle.

[0063] That is, an operator of the vehicle 200D may leave the vehicle 200D while there is another occupant in the cabin of the vehicle 200D. The interior of the vehicle 200D may exceed a safe temperature (above or below a habitable temperature—e.g., 50 degrees Fahrenheit to 80 degrees Fahrenheit), and these safe temperatures define habitable conditions when the vehicle 200D is parked without the ability to use the HVAC system (e.g., when parked and locked). In some examples, when an occupant of the vehicle 200D is detected and when external (or, in other words, outside) conditions exceed habitable conditions, the VHU 202 may automatically turn on the HVAC system to adjust the interior temperature of the vehicle 200D to maintain habitable conditions within the vehicle 200D. Automatically adjusting (meaning adjusting without input or other interaction with the operator / owner / driver of the vehicle 200D) the temperature provided by the HVAC system of the vehicle 200D to maintain habitable conditions may represent an example of a safety action.

[0064] In other examples, the VHU 202 may perform a safety action by initiating a phone call (e.g., via a cellular network using cellular phone service) to one or more of the owner of the vehicle 200D, the (possibly temporary) operator of the vehicle 200D, the preferred contact of the vehicle 200D (possibly specified such as via settings exposed by an operating system—OS executed by the VHU 202), and emergency services (e.g., 911 service in the United States) in response to an operating state of the vehicle 200D indicating that an occupant remains in the vehicle 200D and a status of the occupant indicating that the occupant remains in the vehicle 200D (even when locked).

[0065] Although described with respect to a cellular phone call, the VHU 202 may initiate a text message in addition to or in place of a cellular phone call, where such cellular services may utilize one or more cellular services including a data cellular service. In Figure 2D the example of, the VHU 202 may initiate a cellular security service 240 with the network 250 (which may represent a cellular phone call, a cellular text message, cellular data messaging, etc.), and the network may represent a wireless network connection to a public network (such as the Internet) and / or a private network via a cellular standard or other wireless standard (such as WiFi TM ).

[0066] In addition to or as an alternative to the above safety operations, the VHU 202 can initiate a safety alert to warn people in the vicinity of the occupants within the vehicle 200D in response to an indication of the operating state of the vehicle 200D that the occupants still remain within the vehicle 200D and an indication of the state of the occupants that the occupants still remain within the vehicle 200D. The safety alert can include making the horn of the vehicle 200D sound, triggering an alarm via a network (e.g., via an application that monitors nearby vehicles and is connected to the wireless network of the vehicle 200D), making the headlights of the vehicle 200D flash, accelerating the engine, etc.

[0067] In some instances, the VHU 202 can perform safety actions according to a configurable escalation policy. The escalation policy can provide a prioritized list of actions, where one or more of the above safety actions can be performed according to metrics based on time, based on temperature (e.g., internal temperature, external temperature, or some combination of both internal and external temperatures), based on an action (e.g., with respect to the vehicle 200D, such as an attempt to open and / or unlock the door of the vehicle 200D), or other criteria-based metrics.

[0068] For example, when a child may be residing in the driver-side rear passenger area 230D while the vehicle 200D is locked, inoperable, and parked, the VHU 202 can perform a safety action including initiating a cellular phone call according to a configurable escalation policy. When it is determined that the cellular phone call is not answered, the VHU 202 can continue to escalate the safety action according to the escalation policy by sending a text message requesting a response. When it is determined that the text message is not answered, the VHU 20s can continue to escalate the safety action according to the escalation policy by issuing a safety alert. The owner or other authorized operator of the vehicle 200D can configure the escalation policy according to which communication mode (meaning, for example, cellular phone call, text message, safety alert, etc.) is preferred and the order in which each communication mode is to be executed.

[0069] Figure 3 is illustrative Figure 1 is a flowchart of an example operation of the vehicle shown in the example of performing various aspects of the adaptive audio playback technology described in the present disclosure. The AAS 117 can obtain the state of the occupants residing within the cockpit of the vehicle 100 from the OMS 115 (300). The OMS 115 (executed by the processing circuit 112) can call one or more of the cameras 124 to capture occupant visual data that can include images, videos, etc. of the occupants residing within the vehicle cockpit (not shown in the example for ease of illustration). The OMS 115 can also call one or more of the microphones 128 to further capture occupant audio data in one or more areas of the vehicle cockpit (which can be similar to the AD 127). Figure 1 In the example of

[0070] Next, AAS 117 may modify the playback of audio data 127 within at least a portion of the cabin of vehicle 100 based on the state of the occupants residing within the cabin of vehicle 100 to obtain modified playback data for the audio data 127 (302). For example, OMS 115 may determine that a rear passenger in the cabin of vehicle 100 is asleep (e.g., a child is sleeping in the rear passenger area on the driver side of the vehicle cabin). In this example, OMS 115 may interface with AAS 117 (via the API exposed by AAS 117) to convey a status update indicating that the passenger in the rear passenger area on the driver side of the cabin of vehicle 100 is sleeping. AAS 117 may be configured to generate modified audio playback data in response to being invoked and receiving the status update, the modified audio playback data indicating that the audio playback of AD 127 is to be muted in the rear passenger area on the driver side of the cabin of vehicle 100.

[0071] Then, AAS 117 may render AD 127 based on the modified audio playback data (304) such that, in this example, the gain associated with the channel associated with the rear passenger area on the driver side of the vehicle cabin for AD 127 is muted. In this regard, AAS 117 may mute the audio playback in the rear passenger area (in this instance, on the driver side) of the cabin of vehicle 100 in response to the sleeping state indicating that the occupant within the rear passenger area (in this instance, on the driver side) of the cabin of vehicle 100 is asleep. Then, AAS 117 may output the rendered channels to speaker 127 in order to reproduce the sound field (represented by AD 127) based on the modified playback data (306).

[0072] Figure 4 An example of a wireless communication system 400 in accordance with aspects of the present disclosure is illustrated. Wireless communication system 400 includes a base station 405, a UE 415, and a core network 430. In some examples, wireless communication system 400 may be a Long Term Evolution (LTE) network, an Advanced LTE (LTE-A) network, an LTE-A Pro network, a 5th generation cellular network, or a New Radio (NR) network. In some cases, wireless communication system 100 may support enhanced broadband communication, ultra-reliable (e.g., mission-critical) communication, low-latency communication, or communication with low-cost and low-complexity devices. Wireless communication system 400 may represent Figure 2D an example of network 250 shown in the example of.

[0073] Base station 405 may communicate wirelessly with UE 415 via one or more base station antennas. The base station 405 described herein may include or may be referred to by those skilled in the art as a base station transceiver, radio base station, access point, radio transceiver, NodeB, eNodeB (eNB), next-generation NodeB, or gigabit NodeB (any of which may be referred to as a gNB), home NodeB, home eNodeB, or some other suitable term. The wireless communication system 400 may include different types of base stations 105 (e.g., macro cell base stations or small cell base stations). The UE 415 described herein may be capable of communicating with various types of base stations 405 and network equipment including macro eNBs, small cell eNBs, gNBs, relay base stations, and the like.

[0074] Each base station 405 may be associated with a particular geographic coverage area 410 in which communication with various UEs 415 is supported. Each base station 405 may provide communication coverage for the corresponding geographic coverage area 410 via a communication link 425, and the communication link 425 between the base station 405 and the UE 415 may utilize one or more carriers. The communication link 425 shown in the wireless communication system 400 may include: an uplink transmission from the UE 415 to the base station 405, or a downlink transmission from the base station 405 to the UE 415. The downlink transmission may also be referred to as a forward link transmission, and the uplink transmission may also be referred to as a reverse link transmission.

[0075] The geographic coverage area 410 for the base station 405 may be divided into sectors, which form part of the geographic coverage area 410, and each sector may be associated with a cell. For example, each base station 405 may provide communication coverage for a macro cell, small cell, hotspot, or other type of cell, or various combinations thereof. In some examples, the base station 405 may be movable and thus provide communication coverage for a mobile geographic coverage area 410. In some examples, different geographic coverage areas 410 associated with different technologies may overlap, and the overlapping geographic coverage areas 410 associated with different technologies may be supported by the same base station 405 or different base stations 405. The wireless communication system 400 may include, for example, a heterogeneous LTE / LTE-A / LTE-A Pro, fifth-generation, or NR network, where different types of base stations 405 provide coverage for various geographic coverage areas 410.

[0076] UE 415 can be dispersed throughout the wireless communication system 400, and each UE 415 can be stationary or mobile. UE 415 can also be referred to as a mobile device, wireless device, remote device, handheld device, or subscriber device, or some other appropriate term, where "device" can also be referred to as a unit, station, terminal, or client. UE 415 can also be a personal electronic device, such as a cellular phone, personal digital assistant (PDA), tablet computer, laptop computer, or personal computer. In an example of the present disclosure, UE 415 can be any audio source among the audio sources described in the present disclosure, including VR headsets, XR headsets, AR headsets, vehicles, smartphones, microphones, microphone arrays, or any other device including a microphone, or capable of transmitting a captured and / or synthesized audio stream. In some examples, the synthesized audio stream can be an audio stream stored in a memory or previously created or synthesized. In some examples, UE 415 can also refer to a wireless local loop (WLL) station, Internet of Things (IoT) device, Internet of Everything (IoE) device, or machine type communication (MTC) device, etc., which can be implemented in various items such as appliances, vehicles, meters, etc.

[0077] Some UEs 415 such as MTC or IoT devices can be low-cost or low-complexity devices, and can provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC can refer to a data communication technology that allows devices to communicate with each other or with the base station 405 without human intervention. In some examples, M2M communication or MTC can include communication from devices that exchange and / or use audio metadata, which can include timing metadata for affecting an audio stream and / or an audio source.

[0078] In some cases, UE 415 may also be capable of communicating directly with other UEs 415 (e.g., using peer-to-peer (P2P) or device-to-device (D2D) protocols). One or more UEs in a group of UEs 415 utilizing D2D communication can be within the geographical coverage area 410 of the base station 405. Other UEs 415 in such a group can be outside the geographical coverage area 410 of the base station 405, or otherwise unable to receive transmissions from the base station 405. In some cases, multiple groups of UEs 415 communicating via D2D communication can utilize a one-to-many (1:M) system, where each UE 415 transmits to each other UE 415 in the group. In some cases, the base station 405 facilitates the scheduling of resources for D2D communication. In other cases, D2D communication is performed between UEs 415 without involving the base station 405.

[0079] Base station 405 can communicate with the core network 430 and with each other. For example, base station 405 can interface with the core network 430 via a backhaul link 432 (e.g., via S1, N2, N3, or other interfaces). Base station 405 can communicate with each other directly (e.g., directly between base stations 405) or indirectly (e.g., via the core network 430) on a backhaul link 434 (e.g., via X2, Xn, or other interfaces).

[0080] In some cases, wireless communication system 400 can utilize both licensed and unlicensed radio frequency spectrum bands. For example, wireless communication system 400 can employ licensed-assisted access (LAA), LTE-unlicensed (LTE-U) radio access technology, or NR technology in an unlicensed band such as the 5 GHz industrial, scientific, and medical (ISM) band. When operating in an unlicensed radio spectrum band, wireless devices such as base station 405 and UE 415 can employ a listen-before-talk (LBT) procedure to ensure that the frequency channel is clear before transmitting data. In some cases, operation in the unlicensed band can be based on a carrier aggregation configuration that combines component carriers operating in a licensed band (e.g., LAA). Operation in the unlicensed spectrum can include downlink transmissions, uplink transmissions, peer-to-peer transmissions, or a combination of these. Duplexing in the unlicensed spectrum can be based on frequency-division duplexing (FDD), time-division duplexing (TDD), or a combination of both.

[0081] There are a variety of different ways to represent a sound field. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to 5.1 surround sound formats, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that locates audio channels to specific positions around a listener in order to recreate a sound field.

[0082] Object-based audio formats can refer to a format in which audio objects (usually encoded using pulse code modulation (PCM) and referred to as PCM audio objects) are specified to represent a sound field. Such audio objects can include position information such as metadata that identifies the position of the audio object relative to the listener or other reference points in the sound field, such that the audio object can be rendered to one or more speaker channels for playback in an effort to recreate the sound field. The techniques described in this disclosure can be applicable to any of the following formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

[0083] Scene-based audio formats can include a hierarchical set of elements that define a sound field in three dimensions. An example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression shows a description or representation of a sound field using SHC:

[0084]

[0085] This expression indicates that the pressure p at any point in the sound field at time t is i uniquely represented by SHC Here, c is the speed of sound (about 343 m / s), is the reference point (or observation point), j n (·) is the spherical Bessel function of order n, and is the spherical harmonic basis function of order n and sub-order m (which can also be referred to as the spherical basis function). It can be recognized that the term in the square brackets is the frequency-domain representation of the signal (e.g., ), and this frequency-domain representation can be approximated by various time-frequency transforms such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or the wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other coefficient sets of multi-resolution basis functions.

[0086] SHC can be physically obtained (e.g., recorded) through various microphone array configurations, or alternatively, they can be derived from channel-based or object-based descriptions of the sound field. SHC (which can also be referred to as the ambisonic coefficient) represents scene-based audio, where SHC can be input into an audio encoder to obtain the encoded SHC that can facilitate more efficient transmission or storage. For example, a fourth-order representation involving (1 + 4) 2 (25, and thus fourth-order) coefficients can be used.

[0087] As described above, SHC can be derived from microphone recordings using a microphone array. Various examples of how SHC can be physically obtained from a microphone array are described in the following document: Poletti, M., "Three-Dimensional Surround Sound Systems Based on Spherical Harmonics", Journal of the Audio Engineering Society, Vol. 53, No. 11, November 2005, pp. 1004 - 1025.

[0088] The following equation can illustrate how SHC can be derived from an object-based description. The coefficients for the sound field corresponding to individual audio objects can be expressed as:

[0089]

[0090] where i is is a spherical Hankel function of order n (of the second kind), and is the position of the object. Knowing the object source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques such as performing a fast Fourier transform on a pulse-code modulated (PCM) stream) enables the conversion of each PCM object and corresponding position into SHC In addition, it can be shown (since the above is a linear and orthogonal decomposition) that the coefficients for each object are additive. Thus, multiple PCM objects can be represented by coefficients (e.g., as the sum of coefficient vectors for individual objects). These coefficients can contain information about the sound field (pressure as a function of three-dimensional (3D) coordinates), and the above represents the transformation of the representation from individual objects to the entire sound field near the observation point According to the techniques of the present disclosure, individual audio streams can be restricted from being reproduced or can be reproduced on a temporary basis based on timing information such as time or duration. Certain individual audio streams or clusters of audio streams can be enabled or disabled for a fixed duration for better audio interpolation. Thus, the techniques of the present disclosure provide a flexible way to control access to audio streams based on time.

[0091] It should be noted that the methods described herein describe possible specific implementations, and the operations and steps can be rearranged or otherwise modified and other specific implementations are possible. In addition, aspects from two or more methods can be combined.

[0092] It should be recognized that, according to examples, certain actions or events of any of the techniques described herein can be performed in a different order, can be added, combined, or completely omitted (e.g., not all actions or events described are necessary for implementing the technique). In addition, in certain examples, actions or events can be performed concurrently (e.g., via multi-threading, interrupt handling, or multiple processors) rather than sequentially.

[0093] In some examples, a VR device (or streaming device) can use a network interface coupled to the memory of the VR / streaming device to exchange messages with an external device, where the exchanged messages are associated with multiple available representations of the sound field. In some examples, a VR device can use an antenna coupled to the network interface to receive wireless signals including data packets, audio packets, video packets, or transport protocol data associated with multiple available representations of the sound field. In some examples, one or more microphone arrays can capture the sound field.

[0094]

[0095] ​In some examples, multiple available representations of an acoustic field stored in a memory device may include multiple object-based representations of the acoustic field, a high-order immersive audio representation of the acoustic field, a hybrid-order immersive audio representation of the acoustic field, a combination of an object-based representation of the acoustic field and a high-order immersive audio representation of the acoustic field, a combination of an object-based representation of the acoustic field and a hybrid-order immersive audio representation of the acoustic field, or a combination of a hybrid-order representation of the acoustic field and a high-order immersive audio representation of the acoustic field.

[0096] In some examples, one or more acoustic field representations among the multiple available representations of the acoustic field may include at least one high-resolution region and at least one low-resolution region, and wherein a rendering selected based on a steering angle provides higher spatial accuracy with respect to the at least one high-resolution region and lower spatial accuracy with respect to the low-resolution region.

[0097] As used herein, "A and / or B" means "A or B" or "both A and B".

[0098] In this way, various aspects of these techniques can implement the following clauses.

[0099] Clause 1A. A device configured to reproduce an acoustic field based on audio data within a vehicle, the device comprising: a memory configured to store the audio data; and a processing circuit coupled to the memory and configured to: obtain a state of an occupant residing within a cabin of the vehicle from an occupant monitoring system; modify playback of the audio data within at least a portion of the cabin of the vehicle based on the state of the occupant residing within the cabin of the vehicle to obtain modified playback data for the audio data; and reproduce the acoustic field based on the modified playback data.

[0100] Clause 2A. The device according to Clause 1A, wherein the processing circuit, when configured to obtain the state of the occupant residing within the vehicle, is configured to obtain a state of a rear occupant residing within a rear passenger area of the cabin of the vehicle.

[0101] Clause 3A. The device according to any combination of Clauses 1A and 2A, wherein the state of the occupant residing within the vehicle includes a sleeping state of an occupant residing within a rear passenger area of the cabin of the vehicle.

[0102] Clause 4A. The device according to Clause 3A, wherein the processing circuit, when configured to modify the playback of the audio data, is configured to mute audio playback within the rear passenger area of the cabin of the vehicle in response to the sleeping state indicating that the occupant residing within the rear passenger area of the cabin of the vehicle is sleeping.

[0103] Clause 5A. The apparatus according to any combination of Clauses 3A and 4A, wherein the processing circuitry, when configured to modify the playback of the audio data, is configured to perform active noise cancellation for the rear passenger area of the vehicle cabin in response to the sleep state indicating that the occupant residing in the rear passenger area of the vehicle cabin is sleeping, the active noise cancellation including modifying the audio data prior to playback in the rear passenger area to limit the sound field reproduction in the rear passenger area.

[0104] Clause 6A. The apparatus according to any combination of Clauses 3A to 5A, wherein the processing circuitry, when configured to modify the playback of the audio data, is configured to replace a portion of the audio data with soothing audio data in response to the sleep state indicating that the occupant residing in the rear passenger area of the vehicle cabin is sleeping, to facilitate the sleep state of the occupant residing in the rear passenger area of the vehicle cabin.

[0105] Clause 7A. The apparatus according to any combination of Clauses 2A to 6A, wherein the processing circuitry is further configured to alert an operator of the vehicle that the occupant in the rear passenger area of the vehicle cabin is a child in response to detecting the state of the occupant in the rear passenger area of the vehicle cabin.

[0106] Clause 8A. The apparatus according to any combination of Clauses 2A to 7A, wherein the processing circuitry is further configured to adjust the HVAC settings for the rear passenger area of the vehicle cabin in response to detecting the state of the occupant in the rear passenger area of the vehicle cabin.

[0107] Clause 9A. The apparatus according to any combination of Clauses 1A to 8A, wherein the processing circuitry is further configured to: obtain an operating state of the vehicle; and perform a safety action based on the state of the occupant residing in the vehicle and the operating state of the vehicle to promote the safety of the occupant residing in the vehicle.

[0108] Clause 10A. The device according to Clause 9A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a phone call to one or more of the owner of the vehicle, the operator of the vehicle, the preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

[0109] Clause 11A. The device according to any combination of Clauses 9A and 10A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a text message to one or more of the owner of the vehicle, the operator of the vehicle, the preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

[0110] Clause 12A. The device according to any combination of Clauses 9A to 11A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a safety alarm to alert people nearby of the occupant in the vehicle in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

[0111] Clause 13A. The device according to any combination of Clauses 1A to 12A, wherein the processing circuit is coupled to one or more speakers, wherein the processing circuit is configured to output the modified playback data to the one or more speakers when configured to reproduce the sound field, and wherein the one or more speakers are configured to reproduce the sound field based on the modified playback data and the audio data.

[0112] Clause 14A. The device according to any combination of Clauses 1A to 13A, wherein the device includes a vehicle host unit.

[0113] Clause 15A. A method for reproducing a sound field based on audio data in a vehicle, the method comprising: obtaining a state of an occupant residing in a cockpit of the vehicle from an occupant monitoring system; modifying playback of the audio data in at least a portion of the cockpit of the vehicle based on the state of the occupant residing in the cockpit of the vehicle to obtain modified playback data for the audio data; and reproducing the sound field based on the modified playback data.

[0114] Clause 16A. The method according to Clause 15A, wherein obtaining the state of the occupant residing in the vehicle includes: obtaining a state of a rear occupant residing in a rear passenger area of the cockpit of the vehicle.

[0115] Clause 17A. The method according to any combination of Clauses 15A and 16A, wherein the state of the occupant residing in the vehicle includes a sleeping state of the occupant residing in the rear passenger area of the cockpit of the vehicle.

[0116] Clause 18A. The method according to Clause 17A, wherein modifying the playback of the audio data includes: muting the audio playback in the rear passenger area of the cockpit of the vehicle in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping.

[0117] Clause 19A. The method according to any combination of Clauses 17A and 18A, wherein modifying the playback of the audio data includes: performing active noise cancellation on the rear passenger area of the cockpit of the vehicle in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping, the active noise cancellation including modifying the audio data before playback in the rear passenger area to limit the reproduction of the sound field in the rear passenger area.

[0118] Clause 20A. The method according to any combination of Clauses 17A to 19A, wherein modifying the playback of the audio data includes: replacing a portion of the audio data with soothing audio data in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping, to facilitate the sleeping state of the occupant residing in the rear passenger area of the cockpit of the vehicle.

[0119] Clause 21A. The method according to any combination of Clauses 16A to 20A further includes: warning an operator of the vehicle that an occupant in the rear passenger area of the cockpit of the vehicle is a child in response to detecting the state of the occupant in the rear passenger area of the cockpit of the vehicle.

[0120] Clause 22A. The method according to any combination of Clauses 16A to 21A further includes: adjusting a heating, ventilation, and air conditioning (HVAC) setting for the rear passenger area of the cockpit of the vehicle in response to detecting the state of the occupant in the rear passenger area of the cockpit of the vehicle.

[0121] Clause 23A. The method according to any combination of Clauses 15A to 22A further includes: obtaining an operating state of the vehicle; and performing a safety action based on the state of the occupant residing in the vehicle and the operating state of the vehicle to promote the safety of the occupant residing in the vehicle.

[0122] Clause 24A. The method according to Clause 23A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action includes: initiating a phone call to one or more of an owner of the vehicle, an operator of the vehicle, a preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

[0123] Clause 25A. The method according to any combination of Clauses 23A and 24A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action includes: sending a text message to one or more of an owner of the vehicle, an operator of the vehicle, a preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

[0124] Clause 26A. A method according to any combination of Clauses 23A to 25A, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action includes: initiating a safety alert in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle to alert a person near the occupant in the vehicle.

[0125] Clause 27A. A method according to any combination of Clauses 15A to 26A, wherein reproducing the sound field includes: outputting the modified playback data to one or more speakers.

[0126] Clause 28A. A method according to any combination of Clauses 15A to 27A, wherein the device includes a vehicle host unit.

[0127] Clause 29A. A non-transitory computer-readable storage medium having instructions stored thereon, which when executed cause one or more processors of a vehicle host unit to: obtain the state of an occupant residing in a cockpit of a vehicle including the vehicle host unit from an occupant monitoring system; modify playback of audio data representing a sound field in at least a portion of the cockpit of the vehicle based on the state of the occupant residing in the cockpit of the vehicle to obtain modified playback data for the audio data; and reproduce the sound field based on the modified playback data and the audio data.

[0128] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, which includes, for example, any medium that facilitates transfer of a computer program from one place to another according to a communication protocol. Thus, the computer-readable medium generally may correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or a carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0129] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead are directed to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0130] The instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Further, the techniques can be fully implemented in one or more circuits or logic elements.

[0131] The techniques of the present disclosure can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or a group of ICs (e.g., a chipset). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but are not necessarily implemented by different hardware units. Rather, as described above, the various units can be combined in a codec hardware unit, or can be provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with appropriate software and / or firmware.

[0132] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A device configured to reproduce an acoustic field based on audio data within a vehicle, the device comprising: a memory configured to store the audio data; and a processing circuit coupled to the memory and configured to: obtain a state of an occupant residing within a cabin of the vehicle from an occupant monitoring system; modify playback of the audio data within at least a portion of the cabin of the vehicle based on the state of the occupant residing within the cabin of the vehicle to obtain modified playback data for the audio data; and reproduce the acoustic field based on the modified playback data.

2. The device according to claim 1, wherein the processing circuit, when configured to obtain the state of the occupant residing within the vehicle, is configured to obtain a state of a rear occupant residing within a rear passenger area of the cabin of the vehicle.

3. The device according to claim 1, wherein the state of the occupant residing within the cabin of the vehicle includes a sleeping state of the occupant residing within a rear passenger area of the cabin of the vehicle.

4. The device according to claim 3, wherein the processing circuit, when configured to modify the playback of the audio data, is configured to mute audio playback within the rear passenger area of the cabin of the vehicle in response to the sleeping state indicating that the occupant residing within the rear passenger area of the cabin of the vehicle is sleeping.

5. The device according to claim 3, wherein the processing circuit, when configured to modify the playback of the audio data, is configured to perform active noise cancellation for the rear passenger area of the cabin of the vehicle in response to the sleeping state indicating that the occupant residing within the rear passenger area of the cabin of the vehicle is sleeping, the active noise cancellation including modifying the audio data prior to playback within the rear passenger area to limit reproduction of the acoustic field within the rear passenger area.

6. The device according to claim 3, wherein the processing circuit, when configured to modify the playback of the audio data, is configured to replace a portion of the audio data with soothing audio data in response to the sleeping state indicating that the occupant residing within the rear passenger area of the cabin of the vehicle is sleeping, to facilitate the sleeping state of the occupant residing within the rear passenger area of the cabin of the vehicle.

7. The device according to claim 2, wherein the processing circuit is further configured to alert an operator of the vehicle that the occupant within the rear passenger area of the cabin of the vehicle is a child in response to detecting the state of the occupant within the rear passenger area of the cabin of the vehicle.

8. The apparatus according to claim 2, wherein the processing circuit is further configured to adjust the HVAC settings for the rear passenger area of the vehicle cabin in response to detecting the status of the occupant in the rear passenger area of the vehicle cabin.

9. The apparatus according to claim 1, wherein the processing circuit is further configured to: obtain an operating state of the vehicle; and perform a safety action based on the status of the occupant residing in the vehicle and the operating state of the vehicle to promote the safety of the occupant residing in the vehicle.

10. The apparatus according to claim 9, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the status of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a telephone call to one or more of the owner of the vehicle, the operator of the vehicle, a preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the status of the occupant indicating that the occupant still resides in the vehicle.

11. The apparatus according to claim 9, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the status of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a text message to one or more of the owner of the vehicle, the operator of the vehicle, a preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the status of the occupant indicating that the occupant still resides in the vehicle.

12. The apparatus according to claim 9, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the status of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein the processing circuit, when configured to perform the safety action, initiates a safety alert to warn people in the vicinity of the occupant in the vehicle in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the status of the occupant indicating that the occupant still resides in the vehicle.

13. The apparatus according to claim 1, wherein the processing circuit is coupled to one or more speakers, wherein the processing circuit is configured to output the modified playback data to the one or more speakers when configured to reproduce the sound field, and wherein the one or more speakers are configured to reproduce the sound field based on the modified playback data and the audio data.

14. The apparatus according to claim 1, wherein the apparatus includes a vehicle host unit.

15. A method for reproducing a sound field based on audio data within a vehicle, the method comprising: obtaining a state of an occupant residing in a cockpit of the vehicle from an occupant monitoring system; modifying playback of the audio data within at least a portion of the cockpit of the vehicle based on the state of the occupant residing in the cockpit of the vehicle to obtain modified playback data for the audio data; and reproducing the sound field based on the modified playback data.

16. The method according to claim 15, wherein obtaining the state of the occupant residing in the cockpit of the vehicle comprises: obtaining a state of a rear occupant residing in a rear passenger area of the cockpit of the vehicle.

17. The method according to claim 15, wherein the state of the occupant residing in the vehicle includes a sleeping state of the occupant residing in a rear passenger area of the cockpit of the vehicle.

18. The method according to claim 17, wherein modifying the playback of the audio data comprises: muting audio playback in the rear passenger area of the cockpit of the vehicle in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping.

19. The method according to claim 17, wherein modifying the playback of the audio data comprises: performing active noise cancellation for the rear passenger area of the cockpit of the vehicle in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping, the active noise cancellation including modifying the audio data before playback in the rear passenger area to limit reproduction of the sound field in the rear passenger area.

20. The method according to claim 17, wherein modifying the playback of the audio data comprises: replacing a portion of the audio data with soothing audio data in response to the sleeping state indicating that the occupant residing in the rear passenger area of the cockpit of the vehicle is sleeping to facilitate the sleeping state of the occupant residing in the rear passenger area of the cockpit of the vehicle.

21. The method according to claim 16, further comprising: alerting an operator of the vehicle that an occupant in the rear passenger area of the cockpit of the vehicle is a child in response to detecting the state of the occupant in the rear passenger area of the cockpit of the vehicle.

22. The method according to claim 16, further comprising: adjusting heating, ventilation, and air conditioning (HVAC) settings for the rear passenger area of the cockpit of the vehicle in response to detecting the state of the occupant in the rear passenger area of the cockpit of the vehicle.

23. The method according to claim 15, further comprising: obtaining an operating state of the vehicle; and performing a safety action based on the state of the occupant residing in the vehicle and the operating state of the vehicle to promote the safety of the occupant residing in the vehicle.

24. The method according to claim 23, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action comprises: initiating a phone call to one or more of the owner of the vehicle, the operator of the vehicle, the preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

25. The method according to claim 23, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action comprises: initiating a text message to one or more of the owner of the vehicle, the operator of the vehicle, the preferred contact of the vehicle, and emergency services in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

26. The method according to claim 23, wherein the operating state of the vehicle indicates that the vehicle is locked in a situation where the HVAC cannot be used, wherein the state of the occupant residing in the vehicle indicates that the occupant still resides in the vehicle, and wherein performing the safety action comprises: initiating a safety alarm to alert people in the vicinity of the occupant in the vehicle in response to the operating state of the vehicle indicating that the occupant still resides in the vehicle and the state of the occupant indicating that the occupant still resides in the vehicle.

27. The method according to claim 15, wherein reproducing the sound field comprises: outputting the modified playback data to one or more speakers.

28. The method according to claim 15, wherein obtaining comprises obtaining by a vehicle host unit, modifying comprises modifying by the vehicle host unit, and reproducing comprises reproducing by the vehicle host unit.

29. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors of a vehicle host unit to: obtain the state of an occupant residing in the cockpit of a vehicle including the vehicle host unit from an occupant monitoring system; Modify playback of audio data representing an acoustic field in at least a portion of a cabin of a vehicle based on a state of an occupant residing in the cabin of the vehicle to obtain modified playback data for the audio data; and Reproduce the acoustic field based on the modified playback data and the audio data.

Images