Synchronous Multi-Channel Loopback in Embedded Architectures

Abstract

To provide synchronized multichannel loopback within embedded architectures.SOLUTION: In at least one embodiment, an embedded Linux(R) system is provided. The Linux(R) system includes a memory, a system on a chip (SoC) device, and a first circuit. The SoC device includes the memory and is programmed to process at least a reference signal indicative of undesired audio content and a measured signal indicative of measured audio data in a listening environment. The first circuit is programmed to receive the reference signal and the measured signal. The first circuit is further programmed to merge the reference signal with the measured signal to provide a combined system input to the SoC device to prevent temporal misalignment between the reference signal and the measured signal caused by one or more software layers of the Linux(R) system.SELECTED DRAWING: None

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 799,338, filed Jan. 31, 2019, the disclosure of which is incorporated by reference herein in its entirety.

[0002] Aspects disclosed herein relate generally to synchronous multi-channel loopback in an embedded architecture. These aspects and others are described in more detail below. [Background technology]

[0003] Often, processing in embedded systems (e.g., in Linux operating systems) is not real-time because buffers applied to data systems are not immutable. Such buffers may change over time due to memory usage and system load. This behavior makes those general-purpose, general-purpose System-on-Chip (SoC) systems widely used in mobile phones almost unusable for advanced algorithms such as Acoustic Echo Cancellation (AEC) algorithms. This situation also increases the need for expensive companion chips. Summary of the Invention [Means for solving the problem]

[0004] In at least one embodiment, an embedded Linux system is provided. The Linux system includes a memory, a system-on-chip (SoC) device, and a first circuit. The SoC device includes a memory and is programmed to at least process a reference signal indicative of unwanted audio content and a measurement signal indicative of audio data measured in a listening environment. The first circuit is programmed to receive the reference signal and the measurement signal. The first circuit is further programmed to merge the reference signal with the measurement signal to provide a combined system input to the SoC device and prevent temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux system.

[0005] In at least one other embodiment, a computer program product embodied on a non-transitory computer readable medium programmed to prevent temporary misalignments between a reference signal and a measurement signal for an embedded Linux® system is provided. The computer program product includes instructions for receiving a reference signal indicative of unwanted audio content and receiving a measurement signal indicative of audio data measured in a listening environment. The computer program product includes instructions for processing the reference signal and the measurement signal at a system-on-chip (SoC) device, merging the reference signal with the measurement signal to provide a combined system data stream to the SoC device, and preventing temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux® system.

[0006] In at least one other embodiment, a computer program product embodied on a non-transitory computer readable medium is provided that is programmed to prevent temporary misalignments between a reference signal and a measurement signal for an embedded Linux® system. The computer program product includes instructions for receiving a reference signal indicative of output data for an adaptive control system. The computer program product includes instructions for processing the reference signal and the measurement signal at a system-on-chip (SoC) device, merging the reference signal with the measurement signal to provide a combined system data stream to the SoC device, and preventing temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux® system. For example, the present application provides the following: (Item 1) An embedded Linux system for an audio processing device, comprising: Memory, a system-on-chip (SoC) device including the memory and programmed to at least process a reference signal indicative of unwanted audio content and a measurement signal indicative of audio data measured in the listening environment; A first circuit, receiving said reference signal; receiving said measurement signal; merging the reference signal with the measurement signal to provide a combined system input to the SoC device to prevent temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux system; The first circuit is programmed to perform the following: The above system. (Item 2) The system described in the preceding item, further comprising a mixer programmed to transmit the reference signal to the first circuit. (Item 3) 2. The system of claim 1, further comprising an input sensor programmed to transmit the reference signal to the first circuit. (Item 4) 2. The system of claim 1, further comprising an input sensor programmed to transmit the measurement signal to the first circuit. (Item 5) 2. The system of claim 1, wherein the input sensor is one of a microphone or an acceleration sensor. (Item 6) The system of any one of the preceding claims, wherein the first circuit is further programmed to merge the reference signal with the measurement signal before the combined system input is sent to the one or more software layers of the Linux® system. (Item 7) a first software layer of the one or more software layers including at least one audio-based kernel; the at least one audio-based kernel includes a sound driver; a second software layer of the one or more software layers includes a speech-based library; a third software layer of the one or more software layers including a sound server for managing a sound card; A system according to any one of the preceding claims. (Item 8) 2. The system of claim 1, wherein the first circuit is further programmed to receive the reference signal as a plurality of N audio-based channels. (Item 9) 2. The system of claim 1, wherein the first circuit is further programmed to receive the measurement signal as a plurality of M audio-based channels. (Item 10) 2. The system of claim 1, wherein the first circuit is further programmed to merge the reference signal with the measurement signal to generate M+N voice-based channels to form a time division multiplexed (TDM) data stream. (Item 11) 1. A computer program product embodied in a non-transitory computer readable medium programmed to prevent temporary misalignment between a reference signal and a measurement signal for an embedded Linux® system, the computer program product comprising: receiving said reference signal indicative of unwanted audio content; receiving said measurement signal indicative of audio data measured in a listening environment; processing the reference signal and the measurement signal in a system-on-chip (SoC) device; merging the reference signal with the measurement signal to provide a combined system input to the SoC device to prevent temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux® system; The computer program product as described above, comprising instructions. (Item 12) 20. The computer program product of claim 19, further comprising instructions for transmitting the reference signal via a mixer to a first circuit. (Item 13) 20. The computer program product of claim 19, further comprising instructions for transmitting the reference signal to a first circuit via an input sensor. (Item 14) 20. The computer program product of claim 19, further comprising instructions for transmitting the measurement signal to a first circuit. (Item 15) 2. The computer program product of claim 1, further comprising instructions for merging the reference signal with the measurement signal before the combined system input is sent to the one or more software layers of the Linux system. (Item 16) a first software layer of the one or more software layers including at least one audio-based kernel; the at least one audio-based kernel includes a sound driver; a second software layer of the one or more software layers includes a speech-based library; a third software layer of the one or more software layers including a sound server for managing a sound card; 2. A computer program product according to any one of the preceding items. (Item 17) 2. The computer program product of claim 1, further comprising instructions for receiving the reference signal as a plurality of N audio-based channels. (Item 18) 20. The computer program product of claim 19, further comprising instructions for receiving the measurement signal as a plurality of M audio-based channels. (Item 19) 2. The computer program product of claim 1, further comprising instructions for packing or adding / merging the reference signal with the measurement signal to generate M+N voice-based channels to form a time division multiplexed (TDM) data stream. (Item 20) 1. A computer program product embodied in a non-transitory computer readable medium programmed to prevent temporary misalignment between a reference signal and a measurement signal for an embedded Linux® system, the computer program product comprising: receiving said reference signal indicative of output data for an adaptive control system; receiving the measurement signal; processing the reference signal and the measurement signal in a system-on-chip (SoC) device; merging the reference signal with the measurement signal to provide a combined system input to the SoC device to prevent temporary misalignments between the reference signal and the measurement signal caused by one or more software layers of the Linux® system; The above computer program product further comprising instructions. (Summary) In at least one embodiment, an embedded Linux system is provided. The Linux system includes a memory, a system-on-chip (SoC) device, and a first circuit. The SoC device includes a memory and is programmed to process at least one reference signal indicative of unwanted audio content and a measurement signal indicative of measured audio data in a listening environment. The first circuit is programmed to receive the reference signal and the measurement signal. The first circuit is further programmed to merge the reference signal with the measurement signal to provide a combined system input to the SoC device and prevent a temporary misalignment between the reference signal and the measurement signal from being caused by one or more software layers of the Linux system. [Brief description of the drawings]

[0007] While embodiments of the present disclosure are set forth with particularity in the appended claims, other features of various embodiments will become more apparent and will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which:

[0008] [Figure 1] 1 shows an example of an embedded adaptive control system. [Diagram 2] 1 illustrates an advanced implementation of a closed loop control implementation according to one embodiment. [Diagram 3] 1 illustrates an embedded adaptive control system according to one embodiment. [Figure 4] 1 illustrates a detailed example implementation of an adaptive control system used in conjunction with an acoustic echo canceller / cancellation according to one embodiment. [Diagram 5] 1 illustrates a detailed example implementation of an adaptive control system for use in connection with active noise cancellation according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The embodiments for carrying out the invention of the present invention are disclosed herein as necessary. However, it should be understood that the disclosed embodiments are merely illustrative of the invention, which may be embodied in various and alternative forms. The figures are not necessarily drawn to scale. Some features may be exaggerated or minimized to show the details of specific components. Therefore, specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for guiding those skilled in the art to variously utilize the present invention.

[0010] It will be appreciated that the various electrical devices, such as servers, controllers, and clients disclosed herein, may include a variety of microprocessors, integrated circuits, memory devices (e.g., FLASH, random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable PROM (EEPROM), or other suitable variations thereof), and software that cooperate with one another to perform the operation(s) disclosed herein. Additionally, these electrical devices utilize one or more microprocessors to execute computer programs embodied in non-transitory computer readable media that are programmed to perform any number of the functions disclosed herein. Additionally, the various electrical devices provided herein include a housing and a variety of microprocessors, integrated circuits, and memory devices (e.g., FLASH, random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable PROM (EEPROM)) disposed therein. The electrical devices also include hardware-based inputs and outputs for receiving data from and transmitting data to other hardware-based devices, respectively, as described herein.

[0011] Generally, the embodiments disclosed herein provide, among other things, a synchronized reference measurement embedded input signal architecture and a measurement embedded input signal architecture that may be used in conjunction with, for example, a Linux operating system that provides the feasibility of many advanced algorithms. The architecture is reliable and may be implemented for any number of adaptive control strategies. The Linux operating system may be used for any number of audio processing devices. For example, the architecture may utilize synchronized reference and measurement signals in conjunction with, but not limited to, acoustic echo canceller or cancellation (AEC) applications, active noise cancellation (ANC) systems, any other suitable audio-based application, or generally any control application that requires synchronized relative reference and measurement signals. The illustrated enhancements (or architectures) may utilize signal conditioning circuits that may be simple and provide a low-cost way to provide a relative synchronization design between the reference and measurement signals, while the end-to-end latency may vary dynamically. It is recognized that the embodiments described herein may be applied to any system that does not include an audio system, whereby it is desired to achieve a control target by utilizing relative synchronized reference and measurement signals to minimize errors, for example.

[0012] 1 illustrates an example of an embedded adaptive control system 100. The adaptive control system 100 may be implemented, for example, in an embedded Linux operating system (OS) 101. It is recognized that the system 100 may include a Linux kernel (e.g., the core of the OS) and related support tools and libraries. The system 100 generally includes an application layer 102, a server (or server layer) 104, a sound subsystem layer 106, or, for example, an Extended Linux Sound Architecture (ALSA) library layer 106 or an Open Sound System (OSS) library layer 106, a kernel (or driver) layer (e.g., an ALSA kernel / driver or an OSS kernel / driver) 108, and a hardware layer 110. The application layer 102, the server 104, the sound subsystem layer 106, the kernel layer 108, and the hardware layer 110 may form at least one system-on-chip (SoC) device 111. The SoC device 111 may be an integrated circuit that integrates all associated components therein. It is recognized that the SoC device 111 may include a central processing unit (CPU), memory 113, input / output ports, and / or secondary storage that may be packaged on a single substrate or microchip. The SoC device 111 may also include digital signal processing, analog signal processing, and / or mixed signal processing. The SoC device 111 may be part of a Linux® embedded system.

[0013] It will be appreciated that the system 100 further includes at least one controller 112 (hereinafter controller 112) for executing instructions to perform any and all tasks performed by the application layer 102, the server layer 104, the sound subsystem layer 106, and the kernel (or driver) layer 108. Additionally, the controller 112 may interface with the hardware layer 110 to process data as received from the hardware layer 110 or to transmit data to the hardware layer 110.

[0014] It is recognized that the control system 100 may be utilized in any adaptive control implementation (or closed-loop control strategy). For example, FIG. 2 illustrates a control means 200 utilizing a closed-loop strategy. The control means 200 includes a signal conditioning circuit 201 (i.e., first circuit 201), a controller 202, a system or plant 204 (hereafter "plant 204") under the control of the controller 202, and a sensor 206. The signal conditioning circuit 201 may include a mixer 203 for isolating a reference signal from a system / plant input signal, and a pack circuit 209 for combining the reference signal with a measurement signal to provide a controller input signal. It is recognized that all references to any signal herein may be singular or plural (e.g., any number of signals for each respective input and output). The reference signal may be provided to the signal conditioning circuit 201 via the mixer 203 and may be separated by the system / plant input signal or by a separate input to the signal conditioning circuit 201. The reference signal may directly correspond to audio output data previously output from the hardware layer 110 shown in connection with FIG. 1. This may be referred to as a loopback mechanism. The reference signal may include data provided on any number of audio channels M. The reference signal may correspond to audio data that is determined to be unwanted by the user. The measurement signal may also be provided to the signal conditioning circuit 201. The measurement signal may correspond to audio data transmitted from, for example, a sensor 206 (e.g., a microphone). The sensor 206 generates a measurement signal that corresponds to a measurement output from the plant 204, also defined as a plant output signal from the plant 204. In this case, the measurement signal may correspond to audio on a plant input signal or a controller output signal. The measurement signal may include the actual audio that the user hears in the listening environment. The measurement signal may include any number of audio channels, N. The controller 202 receives a controller input signal that is a synchronous combination of the reference signal and the measurement signal from, for example, the sensor 206 (i.e., a microphone). It is recognized that an analog-to-digital converter may be disposed between the sensor 206 and the signal conditioning circuit 201. In general, the pack circuit 209 may merge data between the measurement signal and the reference signal.In one example, the merging operation may correspond to the packing circuit 209 adding or subtracting the measurement signal from the reference signal to provide a combined controller input signal to the controller 202. Since the reference signal is superimposed on the measurement signal, the controller input signal may be in the form of M channels. Alternatively, the merging operation may correspond to the packing circuit 209 combining (or packing) the measurement signal with the reference signal to provide the controller input signal. However, since the reference signal and the measurement signal are packed into individual slots of the controller input signal, as a result of the packing operation, the controller input signal may be in the form of M+N channels. For example, the controller input signal may form M+N voice-based channels that provide a time division multiplexed (TDM) data stream, packed with M voice-based channels from the measurement signal and N voice-based channels from the reference signal. In general, the signal conditioning circuit 201 is configured to merge data on the reference signal with data on the measurement signal to prevent temporal misalignment between the data on the reference signal and the data on the measurement signal to provide a combined system data stream to the SoC device 111. Such temporary inconsistencies may be caused by one or more of the layers 102, 104, 106, and 108 during operation of the control means 200. This aspect is described in more detail below.

[0015] As described above, it is desirable to reduce the temporal mismatch between two signals propagating through one or more software layers of a Linux® system by applying the embodiments herein to any system, not including an audio system. In an adaptive control system, the reference signal may simply correspond to input data to the plant utilizing a loopback mechanism. The measurement signal may correspond to data such as provided by a sensor in the adaptive control system, which provides some form of feedback information. Thus, in this case, the signal conditioning circuit 201 may merge the data on the reference signal with the data on the measurement signal. After merging the data between these signals, the signal conditioning circuit 201 provides a combined system data stream to the SoC device 111, preventing the temporal mismatch between the data on the reference signal and the data on the measurement signal. The temporal mismatch may generally be caused by one or more of the software layers 102, 104, 106, and 108. This aspect may achieve a control objective, for example, by minimizing errors utilizing relatively synchronized reference and measurement signals. It is recognized that implementations as described herein may be used not only for audio-related purposes, but for any system that seeks to resolve or reduce temporal mismatch between two signals propagating through the software layers of a Linux® system.

[0016] The controller 202 executes this process iteratively based on the measurement and reference signals using an adaptive control strategy to adapt the control means 200 towards the plant 204 in a desired manner. However, it is recognized that due to inherent system errors or due to inconsistencies, the controller 202 may not be able to fully control the plant 204 to achieve the desired results. The adaptive control means 200 described above can be executed any number of times to provide control actions in an optimal manner that may require at least relatively stable latency and timing between the reference and measurement signals to ensure control stability and optimal performance for the plant 204 under the control of the controller 202. Any dynamic time / latency mismatch issues between the measurement and reference signals, for example as delivered in a Linux system 101 such as the embodiment shown in FIG. 1, may cause the controller 202 (or controller 112 as shown in FIG. 1) to perform poorly or even become unstable. However, it is recognized that due to inherent system (or plant) errors or inconsistencies, the controller 202 may not be perfectly able to control the plant 204 to achieve the desired results.

[0017] Referring back to FIG. 1, the application layer 102 typically includes audio clients and associated user programs that can, for example, play music. These audio clients can connect to a sound server via audio streams. The server layer 104 (or sound server) can be software that manages the use of and access to audio devices, for example, sound cards. The server layer 104 typically runs as a background process for sink and source handling. The server layer 104 can be implemented as PulseAudio or an audio connection kit, for example, JACK. PulseAudio is generally a sound system for various operating systems. PulseAudio typically acts as a proxy for sound applications. When implementing the server layer 104 for PulseAudio, it is recognized that PulseAudio can be, for example, a sound server utilized in the UBUNTU distribution that connects to a lower software layer, such as the kernel layer 108 (or ASLA layer). PulseAudio provides server / client connectivity within the application layer 102. JACK can be included in any number of Linux applications.

[0018] The sound system layer 106 typically includes software that provides libraries for ALSA and OSS based on whether ALSA or OSS is used for implementation. The kernel or driver layer 108 may include sound drivers and sound devices. If an ALSA-related kernel layer 108 is used, the ALSA kernel layer 108 may provide application program interface (API) access for supporting multi-channel audio interfaces and managing hardware control. ALSA is a standard software layer in multiprocessor Linux-based embedded systems and may manage all audio streams and stream mixing with buffer handling between audio-hardware peripherals and upper application layers. If an OSS-related kernel layer 108 is used, the OSS kernel layer 108 includes Linux kernel sound drivers and sound devices to connect the application layer 102 with the actual audio-hardware peripherals.

[0019] Furthermore, the system 100 includes hardware-based input device(s) 120 that provide input signals to the hardware layer 110. The input device 120 may include any number of sensors, such as a microphone, an acceleration sensor, and the like. Furthermore, the system 100 includes a hardware-based output device 122 that receives output signals from the hardware layer 110. In one embodiment, the hardware-based output device 122 may include at least one controller for an audio system, such as an acoustic echo canceller (AEC), an active noise canceller (ANC) system, and the like. In general, when the system 100 receives an input signal from the input device 120 and the input signal propagates directly through the hardware layer 110, the kernel layer 108, the sound-subsystem layer 106, and the server layer 104, the processing performed by the controller 112 that executes such layers 102, 104, 106, and 108, the sound-subsystem layer 106, and the server layer 104 may cause latency issues for different and separate input streams of data processed by the system 100.

[0020] In general, the application layer 102 includes a controller unit 124 that can utilize the reference signal and the measurement signal independently of each other. For example, the reference signal and the measurement signal each include individual and separate data from each other (e.g., data between each signal is packed together or added (i.e., merged together)). For example, the input to the controller unit 124 generally corresponds to the measurement signal as provided by the sensor 206 and the reference signal as described in FIG. 2. However, in this case, the system 100 does not use the signal conditioning circuit 201 as described in FIG. 2. In this case, the system 100 buffers the reference signal and the measurement signal as separate data streams. Thus, the individual buffering (i.e., dynamic buffer resizing) of the measurement signal and the reference signal in FIG. 1 may cause these individual signals to be misaligned or asynchronous with each other in the various layers 102, 104, 106, and 108. Although the layer 110 can be hardware-based, this layer 110 may not participate in the latency problem between the reference signal and the measurement signal.

[0021] However, it should be noted that the data stream for the signal controller output or plant input is processed in a downstream manner by layers 102, 104, 106, 108, and 110, and such signals can be fully or partially looped back as reference signals for upstream processing. Furthermore, it should be noted that layers 102, 104, 106, and 108 process measurement signals and reference signals in an upstream manner. Given that the reference signal can be based on the signal controller output or plant input, this may be applied in an inconsistent or asynchronous manner as described above. In general, the hardware layer 110 may include programmable subunits and is generally defined as hardware in a Linux system. The latency described above may cause the hardware layer 110 to be programmable, but deterministic and static. All layers above layer 106, as well as kernel layer 108, may be software-based layers, and the latency attributed to such layers 102, 104, 106, 108 may be assumed to be dynamic and not static (or deterministic due to the operating system (OS) setup within such layers 102, 104, 106, 108).

[0022] Generally, the data provided by the hardware-based input device 120 can be digital. The hardware-based input device 120 can include any number of analog-to-digital converters to provide digital data onto the measurement signal and into some controller applications, for example, when the ANC also provides a reference signal, and as herein, the reference signal does not deviate from the signal controller output or plant input, but still requires the signal conditioning circuit 201 to add or pack the reference and measurement signals together into a combined signal controller input and forward the signal to the various layers 110, 108, 106, and 104 upstream for processing (see, for example, FIG. 5). Similarly, the data transmitted from the hardware-based output device 122 can be digital. The hardware-based output device 122 can include any number of digital-to-analog converters (DACs) to convert the digital data back to analog data.

[0023] FIG. 3 illustrates an embedded adaptive control system 100′ according to one embodiment. It is recognized that the system 100′ can be implemented in a mobile device (not shown), such as a mobile phone, laptop, tablet, etc. (or any other device that enables cellular communication). The system 100′ can be implemented in an audio processing device of a mobile device. This system 100 generally includes a signal conditioning circuit 201 that is intermediate to the hardware layer 110 and to the hardware-based input device(s) 120 and the hardware-based output device(s) 122. As described above, the input device(s) 120 can correspond to a micro or acceleration sensor that provides measured audio data (e.g., see the measured signal from FIG. 2) from the listening environment. Also as described above, the hardware-based output device 122 can include any number of D / A converters as well as a power amplifier to convert the digital data on the signal controller output or plant input to analog data and provide the same to any loudspeaker in the room or at the listening environment. As described above, the reference signal can fully or partially correspond to the signal controller output or plant input. In addition, the hardware-based output device 122 may also include at least one controller for an audio system such as an acoustic echo canceller (AEC), an active noise canceller (ANC) system, etc. The signal conditioning circuit 201 may be a field programmable gate array (FGPA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), etc. Similar to the signal conditioning circuit 201 of FIG. 2, the signal conditioning circuit 201 as shown in FIG. 3 may also include a mixer 203 and a pack circuit 209 to combine the reference signal and the measurement signal and provide a combined controller input data stream. As shown in FIG. 3, the combined system data stream may be referred to as a combined reference and measurement signal SIGNAL(s).As mentioned above, the reference signal is not shown in FIG. 2 but may be utilized entirely or may partially deviate from (or originate from) the controller output signal or the plant input signal.

[0024] As described above, layers 102, 104, 106, and 108 can be software-based layers and can be considered to be non-deterministic (or time-varying). Thus, layers 102, 104, 106, and 108 such as these may generally contribute to causing mismatches between reference signals and measurement signals due to dynamic buffer resizing and / or processing latency when executed by controller 112. However, assuming that signal conditioning circuit 201 (e.g., mixer 203 and / or pack circuit 209) is hardware-based and time-deterministic, signal conditioning circuit 201 is generally utilized or placed before all non-time-deterministic (or time-varying) layers 102, 104, 106, and 108, which can mitigate any mismatches between reference signals and measurement signals when both signals, the reference signal and the measurement signal, are combined with each other via signal conditioning circuit 201 to provide a combined reference signal and measurement signal, referred to as a controller input signal.

[0025] For example, typical dynamic buffer resizing between layers 102, 104, 106, and 108 can change the overall end-to-end latency. However, because the location of signal conditioning circuit 201 is located past potential dynamic buffer latency changes associated with SW layer processing, signal conditioning circuit 201 can synchronize the reference signal and measurement signal, and the latency of the combined reference signal and measurement signal can be considered constant. For example, any latency applied to the combined reference signal and measurement signal can affect the reference signal and measurement signal in the same manner, so the relative latency can be considered constant. The reference signal and measurement signal, when present in the combined reference signal and measurement signal, can correspond to the reference signal and measurement signal, as described in FIG. 2. The same can be applied to the combined reference signal and measurement signal, which can correspond to the controller input signal, as described in FIG. 2. The signal conditioning circuit 201 merges or combines the reference and measurement signals with each other (via pack circuits or mixers) to provide a combined system input (or combined system data stream) to the SoC device 111 and prevents any temporary inconsistencies within one or more of the non-deterministic software layers 102, 104, 106, and 108 during device operation.

[0026] FIG. 4 illustrates an example of an adaptive control system 100'' when used in connection with an AEC system, according to one embodiment. The server layer 104 (or PulseAudio daemon) generally includes a sink layer 300 and a source layer 302. The adaptive control system 100'' generally illustrates an example of independent sink and source processing that is built into a typical PulseAudio daemon framework. The PulseAudio I / O buffers of the server layer 104 cannot be assumed to be synchronous.

[0027] The hardware-based input device(s) 120 can provide audio data on the measurement signal, such as captured from a microphone or acceleration sensor (not shown) when used in conjunction with an AEC system. The captured audio data corresponds to the desired audio data, but may include unwanted echo data (e.g., data on the measurement signal) captured in the room or listening environment. The signal conditioning circuitry 201 can forward the audio data from the system input signal and provide the same to the hardware-based output device(s) 122 and broadcast to an acoustic path, representing the plant 204 as shown in FIG. 2. As described above, the reference signal can correspond to the system input signal. In the case of an AEC, the audio data on the system input signal can correspond to music data that is played independently of the controller operation, but is considered an unwanted echo when forwarded to the system or plant during playback. The signal conditioning circuitry 201 can fully or partially mix the system input signal with the reference signal and combine it with the measurement signal to enable an echo canceller in the controller. It will be appreciated that the hardware-based output device(s) 122 may include any number of D / A converters as well as power amplifiers to convert the digital data on the system input signal to analog data and provide the same to any loudspeakers in the room or listening environment.

[0028] The signal conditioning circuit 201 can pack or add data on the reference signal with the measured microphone data (i.e., desired audio with unwanted echo) on the measurement signal in real time to provide a combined reference and measurement signal, or controller input signal. If the latency between the microphone data and the reference data can remain relatively constant on the combined reference and measurement signal, or controller input signal, the combined reference and measurement signal, or controller input signal can be considered robust against any dynamic system behavior. Although the input / output (I / O) end-to-end latency can change dynamically, AEC convergence can be guaranteed by the stable / constant relative latency between the microphone data and the reference data on the combined reference and measurement signal.

[0029] FIG. 5 shows an example of an adaptive control system 100''' as used in connection with an ANC system, according to one embodiment. The server layer 104 (or PulseAudio daemon) generally includes a sink layer 300 and a source layer 302. The adaptive control system 100''' generally illustrates an example of independent sink and source processing built into a typical PulseAudio daemon framework. The PulseAudio I / O buffers of the server layer 104 cannot be assumed to be synchronous.

[0030] The hardware-based input device(s) 120 can provide audio data on the measurement signal, such as captured from a microphone or acceleration sensor (not shown) when used in conjunction with an ANC system. The captured audio data can correspond to unwanted noise in addition to the desired music or audio data (e.g., data on the measurement signal) captured in the room or listening environment. The signal conditioning circuit 201 can forward the audio data from the controller output signal and provide the same to the hardware-based output device(s) 122 and broadcast to an acoustic path for the noise canceller, representing the plant as shown in FIG. 2. As described above, the reference signal may not correspond to the controller output signal, but can correspond only to unwanted noise, such as provided by another sensor, not shown in FIG. 2, that provides only the unwanted noise. In the case of ANC, the audio data on the controller output signal can correspond to anti-noise data that serves as a loopback towards the audio source via the controller operation in the application layer 102 to regenerate an anti-noise signal for noise cancellation via the plant 204. The signal conditioning circuit 201 merges the reference signal with the measurement signal to enable noise cancellation by the controller via the plant 204. It will be appreciated that the hardware-based output device(s) 122 may include any number of D / A converters, as well as power amplifiers, to convert the digital data on the controller output signal to analog data and provide the same to any loudspeakers in the room or listening environment.

[0031] The signal conditioning circuit 201 can pack or add (i.e., merge data) the measured microphone data (i.e., desired music / voice with unwanted noise) on the measurement signal and the data on the reference signal in real time to provide a combined reference and measurement signal, or controller input signal. If the latency between the microphone data and the reference data can remain relatively constant on the combined reference and measurement signal, or controller input signal, the combined reference and measurement signal, or controller input signal can be considered robust against any dynamic system behavior. Although the input / output (I / O) end-to-end latency may change dynamically, ANC convergence can be guaranteed by the stable / constant relative latency between the microphone data and the reference data on the combined reference and measurement signal.

[0032] The data streams corresponding to the measurement signal, the reference signal, the combined reference and measurement signal, the controller input signal, the plant input signal, and the controller output signal can be implemented as time division multiplexed (TDM) data streams between the software layers 102, 104, 106, or as internal linear or ring buffers. For the embodiment described in connection with FIG. 4 above, the TDM data stream is partially filled with input data (e.g., audio data from the measured microphone), output data (e.g., data from the controller output signal), and data from the reference signal. In the embodiment as described in connection with the system 100'', the TDM-based data stream can be packed with 1 to N+M channels (or bits of data), where M corresponds to audio data from the measured microphone (or from the hardware-based input device(s) 120) and N corresponds to the reference data. In the case of AEC, after the AEC controller process, which ideally cancels the echo completely and processes the audio as a sink for further application services, the microphone content may include the desired audio, and the unwanted echo. Generally, music can act as a source (device transceiver music). Audio can act as a sink, as the device receives audio for communication purposes, e.g., phone calls. In the case of ANC, ideally after the ANC controller process to completely cancel noise and process the anti-noise signal for audio source applications as a sink, the microphone content may contain the desired music / audio, as well as unwanted noise, so the controller's anti-noise signal can be looped back to the device output, allowing noise cancellation to be performed in the acoustic path.

[0033] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms of the present invention. Rather, the words used herein are words of description rather than limitation, and it is understood that various modifications can be made without departing from the spirit and scope of the present invention. In addition, features of various implementing embodiments can be combined to form further embodiments of the present invention.

Claims

1. 1. An embedded Linux system for an audio processing device, the system comprising: Memory, a system-on-chip (SoC) device including the memory, the SoC device being programmed to execute one or more software layers of the Linux system and to process at least a reference signal indicative of unwanted audio content and a measurement signal indicative of audio data measured in a listening environment; and A first circuit, the first circuit comprising: receiving the reference signal; Receiving a measurement signal; merging the reference signal with the measurement signal to provide a combined system input to the one or more software layers of the Linux system to mitigate temporal mismatch between the reference signal and the measurement signal; a first circuit programmed to A system comprising:

2. The system of claim 1 , further comprising a mixer programmed to send the reference signal to the first circuit.

3. The system of claim 1 , further comprising an input sensor programmed to transmit the reference signal to the first circuit.

4. The system of claim 1 , further comprising an input sensor programmed to transmit the measurement signal to the first circuit.

5. The system of claim 4 , wherein the input sensor is one of a microphone or an acceleration sensor.

6. 2. The system of claim 1, wherein the first circuit is further programmed to merge the reference signal with the measurement signal before the combined system input is sent to the one or more software layers of the Linux system.

7. a first software layer of the one or more software layers including at least one audio-based kernel; the at least one audio-based kernel includes a sound driver; a second software layer of the one or more software layers includes a voice-based library; a third software layer of the one or more software layers including a sound server that manages a sound card; The system of claim 1 .

8. The system of claim 1 , wherein the first circuit is further programmed to receive the reference signal as a plurality of N audio-based channels.

9. The system of claim 8 , wherein the first circuit is further programmed to receive the measurement signal as a plurality of M audio-based channels.

10. 10. The system of claim 9, wherein the first circuit is further programmed to merge the reference signal with the measurement signal to generate M+N voice-based channels to form a time division multiplexed (TDM) data stream.

11. A storage medium having a program stored thereon, the program, when executed by a processor, causing the processor to perform a method of mitigating temporary mismatch between a reference signal and a measurement signal for an embedded Linux system, the method comprising: receiving the reference signal indicative of unwanted audio content; receiving the measurement signal indicative of audio data measured in a listening environment; processing the reference signal and the measurement signal in a system-on-chip (SoC) device, the SoC device executing one or more software layers of the Linux system; merging the reference signal with the measurement signal to provide a combined system input to the one or more software layers of the Linux system to mitigate temporal mismatch between the reference signal and the measurement signal; A storage medium including

12. The storage medium of claim 11, wherein the method further includes transmitting the reference signal to a first circuit via a mixer.

13. The storage medium of claim 11, wherein the method further includes transmitting the reference signal to a first circuit via an input sensor.

14. The storage medium of claim 11, wherein the method further includes transmitting the measurement signal to a first circuit.

15. The method according to claim 1, 12. The storage medium of claim 11, further comprising merging the reference signal with the measurement signal before the combined system input is sent to the one or more software layers of the Linux system.

16. a first software layer of the one or more software layers including at least one audio-based kernel; the at least one audio-based kernel includes a sound driver; a second software layer of the one or more software layers includes a voice-based library; a third software layer of the one or more software layers including a sound server that manages a sound card; The storage medium according to claim 11.

17. The storage medium of claim 11, wherein the method further comprises receiving the reference signal as a plurality of N audio-based channels.

18. The storage medium of claim 17, wherein the method further comprises receiving the measurement signal as a plurality of M voice-based channels.

19. The method according to claim 1, 20. The storage medium of claim 18, further comprising packing or adding / merging the reference signal with the measurement signal to generate M+N voice-based channels to form a time division multiplexed (TDM) data stream.

20. A storage medium having a program stored thereon, the program, when executed by a processor, causing the processor to perform a method of mitigating temporary mismatch between a reference signal and a measurement signal for an embedded Linux system, the method comprising: receiving the reference signal indicative of output data for an adaptive control system; receiving the measurement signal; processing the reference signal and the measurement signal in a system-on-chip (SoC) device, the SoC device executing one or more software layers of the Linux system; merging the reference signal with the measurement signal to provide a combined system input to the one or more software layers of the Linux system to mitigate temporal mismatch between the reference signal and the measurement signal; A storage medium including

Images