Methods and apparatus for processing audio data, electronic devices

Abstract

This invention discloses a method, apparatus, and electronic device for processing audio data. The method includes: in response to an audio input command, receiving at least two audio data points through at least two digital audio interfaces, and storing the at least two audio data points in at least two input buffer queues corresponding to the at least two digital audio interfaces; transferring the at least two audio data points from the at least two input buffer queues to addresses in a memory corresponding to the at least two input buffer queues, such that the at least two audio data points corresponding to the at least two digital audio interfaces are alternately stored in the memory; and reading the alternately stored at least two audio data points from the memory as concatenated data from the at least two digital audio interfaces. This invention, while meeting multi-channel requirements, reduces system CPU load and latency, and improves synchronization stability.

Description

Technical Field

[0001] This invention relates to the field of audio data processing technology, and in particular to methods, apparatus, and electronic devices for processing audio data. Background Technology

[0002] In applications such as voice assistants, automatic speech recognition (ASR) systems, and voice control devices, a single digital audio interface (DAI) is often insufficient to meet the required number of channels. However, in a System-on-Chip (SoC), multiple DAIs can be combined to meet multi-channel requirements. Currently, multi-channel audio systems are typically implemented by combining multiple DAIs on the CPU (Central Processing Unit) using software (such as the multi plugin in alsa-lib).

[0003] However, this splicing method requires CPU processing, making it susceptible to the influence of the operating system (OS) and prone to instability and significant jitter. For synchronous applications requiring strict stability across multiple channels, debugging is extremely difficult. Furthermore, CPU processing of audio data incurs high overhead and latency, necessitating additional software porting and resulting in a substantial workload. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method, apparatus, and electronic device for processing audio data, which can reduce the CPU load and latency of the system and improve synchronization stability while meeting the requirements of multi-channel operation.

[0005] In one aspect of the invention, a method for processing audio data is provided. The method includes: receiving at least two audio data points via at least two digital audio interfaces in response to an audio input command, and storing the at least two audio data points in at least two input buffer queues corresponding to the at least two digital audio interfaces; transferring the at least two audio data points in the at least two input buffer queues to addresses in a memory corresponding to the at least two input buffer queues, such that the at least two audio data points corresponding to the at least two digital audio interfaces are alternately stored in the memory; and reading the alternately stored at least two audio data points from the memory as concatenated data from the at least two digital audio interfaces.

[0006] In another aspect of the invention, an apparatus for processing audio data is provided. The apparatus includes: an application layer module configured to acquire audio output data in response to an audio output command, the audio output data including at least two distribution data, each of the at least two distribution data corresponding to at least two digital audio interfaces; a DMA module configured to store the at least two distribution data of the audio output data into addresses in a memory corresponding to the at least two digital audio interfaces, such that the at least two distribution data are alternately stored in the memory, and configured to transfer the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces; and an interface module configured to output the at least two distribution data from the at least two output buffer queues through the at least two digital audio interfaces for audio playback based on the at least two distribution data.

[0007] In another aspect of the invention, an electronic device is provided. The electronic device includes a memory configured to store a computer program; and a processor configured to execute the computer program to perform the described method of processing audio data.

[0008] According to the technical solution of this invention, audio data received by at least two digital audio interfaces are first stored in their respective corresponding input buffer queues, and then the audio data in each input buffer queue is transferred to the corresponding address in the memory, so that the audio data received by different digital audio interfaces are alternately stored in the memory. When the audio data received by different digital audio interfaces are alternately stored in the memory, the audio data at adjacent addresses can be merged. Finally, the merged audio data from different digital audio interfaces can be directly read from the memory. This invention, based on storing different audio data in input buffer queues, can avoid the delay caused by inconsistent data transmission times between different digital audio interfaces, thus improving synchronization stability. Furthermore, transferring different audio data to the memory in an alternate storage manner to complete the merging can avoid the introduction of related software for data merging, effectively reducing the merging workload and improving merging efficiency. In addition, when reading the merged data, only one memory needs to be read to obtain the merged data, effectively reducing the CPU load and avoiding the delay caused by reading data from different memories.

[0009] According to another aspect of the technical solution of the present invention, at least two distribution data included in the audio output data are respectively transmitted to addresses in the memory corresponding to at least two digital audio interfaces, such that the at least two distribution data are alternately stored in the memory, and the at least two distribution data correspond to the at least two digital audio interfaces respectively; then the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output buffer queues corresponding to the at least two digital audio interfaces; and the at least two distribution data in the at least two output buffer queues are respectively output through the at least two digital audio interfaces for audio playback based on the at least two distribution data. In this way, the audio output data can be divided into multiple different distribution data, thereby playing the distribution data based on digital audio interfaces with different numbers of channels, satisfying the requirements of multi-channel audio playback while reducing data processing latency and improving processing efficiency. The present invention, by storing different audio data based on output buffer queues, can avoid latency caused by inconsistent data transmission times between different digital audio interfaces and improve synchronization stability. In addition, transmitting different audio data to the memory in an alternating manner to complete data distribution can avoid the introduction of related software for data distribution, effectively reduce the workload of distribution, improve splicing efficiency, effectively reduce CPU load, and avoid the latency caused by outputting data from different memories. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of an audio recording method in the prior art;

[0011] Figure 2 This is a schematic diagram of an audio playback method in the prior art;

[0012] Figure 3 This is a flowchart of a method for processing audio data according to an embodiment of the present invention;

[0013] Figure 4 A flowchart illustrating another method for processing audio data according to an embodiment of the present invention;

[0014] Figure 5 This is a flowchart of an audio recording method according to an embodiment of the present invention;

[0015] Figure 6 This is a schematic diagram of data transmission in an audio recording method according to an embodiment of the present invention;

[0016] Figure 7 This is a hardware block diagram of an audio recording method according to an embodiment of the present invention;

[0017] Figure 8 This is a flowchart of an audio playback method according to an embodiment of the present invention;

[0018] Figure 9 This is a data transmission diagram of an audio playback method according to an embodiment of the present invention;

[0019] Figure 10 This is a hardware block diagram of an audio playback method according to an embodiment of the present invention;

[0020] Figure 11 This is a schematic diagram of the structure of an audio data processing device according to an embodiment of the present invention;

[0021] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] In existing technologies, multi-channel audio systems are typically implemented by splicing multiple DAIs on the CPU using software. For example... Figure 1 As shown, when recording data by concatenating multiple DAIs, the DMA controller needs to first read the DMA buffers of DAI0 and DAI1 separately via software during data transfer. Then, the audio data is merged using the alsa-lib multi plugin before being transmitted to the speech recognition application algorithm for processing. The application layer requires two Read operations to obtain the audio data of all DAIs. Similarly, as... Figure 2 As shown, when playing audio by splicing multiple DAIs, the playback application first needs to output audio data to the alsa-lib multi plugin for processing on the CPU. Then, the multi plugin writes the audio data to the corresponding DMA buffers of DAI0 and DAI1 respectively, and finally transmits it to each DAI for playback. The application layer also needs to perform two write operations to output the audio data. During recording and audio playback, the time between the two read operations and the two write operations is easily affected by OS scheduling, and the latency of the PCM data buffer between the two DAI times is unstable, with a significant increase in latency. Therefore, in existing technologies, multi-channel splicing systems require CPU processing for audio data processing, which is easily affected by the OS, resulting in significant instability and jitter. For synchronous applications requiring strict stability across multiple channels, debugging is extremely difficult. Furthermore, the CPU has high overhead and latency in processing audio data, and additional software porting is required, resulting in a large workload.

[0024] To address at least the aforementioned technical problems, this disclosure provides a method for processing audio data. According to this disclosure, audio data received by at least two digital audio interfaces is first stored in their respective corresponding input buffer queues. Then, the audio data in each input buffer queue is transferred to the corresponding address in the memory, so that the audio data received by different digital audio interfaces is alternately stored in the memory. When the audio data received by different digital audio interfaces is alternately stored in the memory, the audio data at adjacent addresses can be concatenated and merged. Finally, the concatenated and merged audio data from different digital audio interfaces can be directly read from the memory. In this way, according to the embodiments of this disclosure, storing different audio data based on input buffer queues can avoid latency caused by inconsistent data transmission times between different digital audio interfaces, improving synchronization stability. Furthermore, transferring different audio data to the memory through alternate storage to complete concatenation and merging avoids the introduction of related software for data concatenation, effectively reducing the concatenation workload and improving concatenation efficiency. In addition, when reading the concatenated and merged data, only one memory needs to be read to obtain the concatenated and merged data, effectively reducing CPU load and avoiding latency caused by reading data from different memories separately.

[0025] According to another aspect of this disclosure, at least two distribution data included in the audio output data are respectively transmitted to addresses in the memory corresponding to at least two digital audio interfaces, such that the at least two distribution data are alternately stored in the memory, and the at least two distribution data correspond to the at least two digital audio interfaces respectively; then, the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output buffer queues corresponding to the at least two digital audio interfaces; and the at least two distribution data in the at least two output buffer queues are respectively output through the at least two digital audio interfaces for audio playback based on the at least two distribution data. In this way, the audio output data can be divided into multiple different distribution data, thereby playing the distribution data based on digital audio interfaces with different numbers of channels, meeting the requirements of multi-channel audio playback while reducing data processing latency and improving processing efficiency. In this way, according to the embodiments of this disclosure, storing different audio data based on output buffer queues can avoid latency caused by inconsistent data transmission times between different digital audio interfaces, improving synchronization stability. In addition, transmitting different audio data to the memory in an alternating manner to complete data distribution can avoid the introduction of related software for data distribution, effectively reduce the workload of distribution, improve splicing efficiency, effectively reduce CPU load, and avoid the latency caused by outputting data from different memories.

[0026] In the following, the technical solutions according to this disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0027] Figure 3 This is a flowchart illustrating a method 100 for processing audio data according to an embodiment of the present disclosure. (Refer to...) Figure 3 The method 100 includes the following steps 102 to 106.

[0028] In step 102, in response to an audio input command, at least two audio data are received through at least two digital audio interfaces, and the at least two audio data are stored in at least two input buffer queues corresponding to the at least two digital audio interfaces.

[0029] In some embodiments, receiving at least two audio data points through at least two digital audio interfaces includes: receiving first audio data through a first digital audio interface and simultaneously receiving second audio data through a second digital audio interface, driven by the same clock. In this manner, audio data is received through multiple digital audio interfaces to meet the multi-channel requirements of the current system.

[0030] In some embodiments, storing the at least two audio data sets in at least two input buffer queues corresponding to the at least two digital audio interfaces includes: under the same clock drive, storing the first audio data in a first input buffer queue via a first interface controller corresponding to the first digital audio interface, and simultaneously storing the second audio data in a second input buffer queue via a second interface controller corresponding to the second digital audio interface. In this way, based on the same clock drive, the two digital audio interfaces are hardware-guaranteed to be synchronized, avoiding latency when the two digital audio interfaces receive audio data. Simultaneously, storing the corresponding audio data in the input buffer queues temporarily stores the data received by the digital audio interfaces, achieving smooth data stream processing and preventing data loss or overflow.

[0031] In some embodiments, receiving first audio data through a first digital audio interface includes: receiving first audio data comprising multiple sets of first audio sub-data, each set of first audio sub-data corresponding to a first number of channels.

[0032] Receiving second audio data via the second digital audio interface includes receiving second audio data comprising multiple sets of second audio sub-data, each set of second audio sub-data corresponding to a second number of channels. In this manner, the audio data received by each digital audio interface under its corresponding number of channels is accurately obtained, facilitating subsequent splicing based on the number of channels in the memory.

[0033] In step 104, the at least two audio data in the at least two input buffer queues are respectively transferred to the addresses in the memory corresponding to the at least two input buffer queues, so that the at least two audio data corresponding to the at least two digital audio interfaces are alternately stored in the memory.

[0034] In some embodiments, transferring the at least two audio data from the at least two input buffer queues to addresses in memory corresponding to the at least two input buffer queues includes: under the same clock drive, storing the first audio data from the first input buffer queue in the memory at a first range address in the memory via a first memory controller corresponding to the first digital audio interface, and simultaneously storing the second audio data from the second input buffer queue in the memory at a second range address in the memory via a second memory controller corresponding to the second digital audio interface, wherein the first range address and the second range address are alternately set in an interleaved manner. In this way, audio data from different digital audio interfaces are stored alternately in the memory, thereby completing the concatenation and merging of audio data based on memory addresses, avoiding the introduction of third-party software, and reducing the CPU load.

[0035] In some embodiments, in response to the storage capacity of the first input buffer queue or the second input buffer queue reaching a preset threshold, the first audio data or the second audio data is transferred to the first range address or the second range address. In this way, when the storage capacity of the input buffer queue reaches the preset threshold, the audio data is synchronously transferred to the corresponding address, avoiding latency between the two digital audio interfaces and improving synchronization stability.

[0036] In some embodiments, transferring the at least two audio data from the at least two input buffer queues to addresses in the memory corresponding to the at least two input buffer queues includes: storing multiple sets of first audio sub-data at multiple sets of first sub-range addresses in the memory, and storing multiple sets of second audio sub-data at multiple sets of second sub-range addresses in the memory, such that each set of first audio sub-data and each set of second audio sub-data are alternately stored in the memory in an interleaved manner, wherein each set of first sub-range addresses and each set of second sub-range addresses are alternately set in an interleaved manner, such that a set of second sub-range addresses immediately follows a set of first sub-range addresses, the next set of first sub-range addresses immediately follows the set of second sub-range addresses, and the next set of second sub-range addresses immediately follows the next set of first sub-range addresses. In this manner, since the different sub-range addresses are alternately set in an interleaved manner, audio sub-data received by different digital audio interfaces can be directly spliced ​​based on the alternating addresses in the memory when stored at the corresponding sub-range addresses, without the need for third-party splicing software, effectively improving data processing efficiency.

[0037] In step 106, the at least two audio data stored alternately are read from the memory as concatenated data from the at least two digital audio interfaces.

[0038] In some embodiments, reading the at least two alternately stored audio data from the memory as concatenated and merged data from the at least two digital audio interfaces includes: reading each set of first audio sub-data and each set of second audio sub-data from the memory, and combining each set of first audio sub-data and each set of second audio sub-data read as a set of concatenated and merged sub-data, wherein the concatenated and merged data includes multiple sets of the concatenated and merged sub-data. In this way, only the corresponding data needs to be read from one memory, avoiding the need to read data from two different memories separately and then concatenate them, thereby avoiding data latency.

[0039] In some embodiments, reading the at least two alternately stored audio data from the memory as concatenated and merged data from the at least two digital audio interfaces includes: in response to each set of first audio sub-data and each set of second audio sub-data being stored in a set of joint sub-range addresses in the memory, reading a set of first audio sub-data and a set of second audio sub-data from the set of joint sub-range addresses in the memory as a set of concatenated and merged sub-data. In this manner, since the different digital audio interfaces are alternately stored in memory addresses, concatenation can be completed by reading only the data at the corresponding addresses, without having to read data separately from two different memories or introduce third-party software for concatenation, effectively improving data processing efficiency.

[0040] In some embodiments, an echo cancellation algorithm is used to process the concatenated and merged data to remove background noise. The audio data includes microphone recording data and audio re-sampling data. This improves the quality of data processing.

[0041] In some embodiments, the method may further include: in response to an audio output command, transmitting at least two distribution data included in the audio output data to addresses in a memory corresponding to at least two digital audio interfaces, such that the at least two distribution data are alternately stored in the memory, each corresponding to one of the at least two digital audio interfaces; transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces; and outputting the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces for audio playback based on the at least two distribution data. In this manner, audio output data can be divided into multiple different distribution data, thereby allowing the distribution data to be played based on digital audio interfaces with different numbers of channels, satisfying multi-channel audio playback requirements while reducing data processing latency and improving processing efficiency.

[0042] Figure 4 This is a flowchart illustrating another method 200 for processing audio data according to an embodiment of the present disclosure. (Refer to...) Figure 4 The method 200 includes the following steps 202 to 208.

[0043] In step 202, in response to an audio output command, audio output data is acquired, the audio output data including at least two distribution data, each of which corresponds to at least two digital audio interfaces.

[0044] In step 204, the at least two distribution data of the audio output data are stored in the memory at addresses corresponding to the at least two digital audio interfaces, such that the at least two distribution data are stored alternately in the memory.

[0045] In some embodiments, storing the at least two distributed data points of the audio output data into addresses in memory corresponding to the at least two digital audio interfaces includes: storing the first distributed data point of the audio output data into a first range address in memory corresponding to the first digital audio interface, and storing the second distributed data point of the audio output data into a second range address in memory corresponding to the second digital audio interface, wherein the first range address and the second range address are alternately set in an interleaved manner. In this way, the audio output data is split based on memory addresses, eliminating the need for third-party splitting software and reducing CPU load.

[0046] In some embodiments, storing the first distribution data of the audio output data in the memory at a first range address corresponding to the first digital audio interface includes: storing multiple sets of first distribution sub-data of the first distribution data in multiple sets of first sub-range addresses in the memory, each set of first distribution sub-data corresponding to a first number of channels;

[0047] The storage of the second portion of the audio output data in the memory at the second range address corresponding to the second digital audio interface includes: storing multiple sets of second portion sub-data of the second portion of the second portion of the data in multiple sets of second sub-range addresses in the memory, each set of second portion sub-data corresponding to a second number of channels, wherein each set of first sub-range addresses and each set of second sub-range addresses are alternately set in an interleaved manner, such that a set of second sub-range addresses immediately follows a set of first sub-range addresses, the next set of first sub-range addresses immediately follows the set of second sub-range addresses, and the next set of second sub-range addresses immediately follows the next set of first sub-range addresses. In this way, the audio output data is precisely divided into the number of channels corresponding to each digital audio interface, so as to facilitate subsequent playback output through the corresponding digital audio interface.

[0048] In step 206, the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output buffer queues corresponding to the at least two digital audio interfaces.

[0049] In some embodiments, transmitting the at least two distributed data points alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces includes: under the same clock drive, storing the first distributed data at the first range address in the memory to the first output buffer queue corresponding to the first digital audio interface via a first memory controller corresponding to the first digital audio interface, and simultaneously storing the second distributed data at the second range address in the memory to the second output buffer queue corresponding to the second digital audio interface via a second memory controller corresponding to the second digital audio interface. In this manner, storing different distributed data based on the output buffer queues temporarily stores the data played by the digital audio interfaces, achieving smooth data stream processing, avoiding data loss or overflow, and ensuring synchronous playback of audio data between the two digital audio interfaces, thus reducing data latency.

[0050] In some embodiments, transferring the at least two sets of distributed data, alternately stored in the memory, to at least two output buffer queues corresponding to the at least two digital audio interfaces includes: in response to each set of the first distributed sub-data and each set of the second distributed sub-data being stored in a set of joint sub-range addresses in the memory, transferring the set of the first distributed sub-data and the set of the second distributed sub-data stored in the set of joint sub-range addresses in the memory to a first output buffer queue corresponding to the first digital audio interface and a second output buffer queue corresponding to the second digital audio interface, respectively. In this manner, since different distributed data are alternately stored in memory addresses, data distribution can be completed simply by transferring data from the corresponding addresses to different output buffer queues, eliminating the need to read data from two different memories separately, thus effectively improving data processing efficiency.

[0051] In step 208, the at least two distribution data in the at least two output buffer queues are output through the at least two digital audio interfaces respectively, for audio playback based on the at least two distribution data.

[0052] In some embodiments, outputting the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces includes: under the same clock drive, outputting the first distribution data in the first output buffer queue to the outside via the first digital audio interface through a first interface controller corresponding to the first digital audio interface, and simultaneously outputting the second distribution data in the second output buffer queue to the outside via the second digital audio interface through a second interface controller corresponding to the second digital audio interface, such that a first audio playback device corresponding to the first digital audio interface and a second audio playback device corresponding to the second digital audio interface respectively play audio based on the first distribution data and the second distribution data for the audio output data. In this way, based on the same clock drive, it is ensured that the two output buffer queues can synchronously transmit different distribution data to the corresponding digital audio interfaces for playback, thereby ensuring data synchronization between the two digital audio interfaces and reducing latency.

[0053] In some embodiments, outputting the at least two distributed data items from the at least two output buffer queues through the at least two digital audio interfaces includes: in response to the storage capacity of the first output buffer queue or the second output buffer queue reaching a preset threshold, transmitting the first distributed data item or the second distributed data item to the first digital audio interface or the second digital audio interface for output. In this manner, when the storage capacity of the output buffer queue reaches the preset threshold, the distributed data is synchronously transmitted to the digital audio interface, avoiding latency between the two digital audio interfaces and improving synchronization stability.

[0054] Figure 5 This is a flowchart illustrating an audio recording method 300 according to an embodiment of the present invention. (Refer to...) Figure 5 The method 300 records two audio data points through two digital audio interfaces (DAI0 and DAI1), then alternately stores the two audio data points through memory addresses to obtain concatenated data, and finally transmits the concatenated data to a speech recognition application using a multi-channel microphone array for processing. The method 300 includes steps 302 to 310.

[0055] In step 302, in response to the audio input command, audio data D0 with 8 channels is recorded through the digital audio interface DAI0, and audio data D1 with 2 channels is recorded through the digital audio interface DAI1.

[0056] Figure 6 This is a schematic diagram illustrating data transmission of an audio recording method 300 according to an embodiment of the present invention.

[0057] In step 304, under the same clock drive, the interface controllers corresponding to the digital audio interfaces DAI0 and DAI1 are simultaneously enabled to control the digital audio interfaces DAI0 and DAI1 to simultaneously receive the audio data D0 and D1 into their respective input buffer queues FIFO for temporary storage.

[0058] In some embodiments, all digital audio interfaces use the same CLK (clock signal) derived from the same PLL (Phase-locked loop) to implement the same clock configuration in hardware, ensuring that different digital audio interfaces can be synchronized from the same source.

[0059] In step 306, when the storage capacity of the input buffer queue FIFO reaches the preset threshold, the digital audio interfaces DAI0 and DAI1 simultaneously trigger the DMA controller to start transmitting the audio data D0 and D1 in their input buffer queue FIFO.

[0060] In step 308, based on the DMA controller in Interleaved transfer mode, audio data D0 in the input buffer queue FIFO corresponding to digital audio interface DAI0 is transferred to a first range address in the memory, and audio data D1 in the input buffer queue FIFO corresponding to digital audio interface DAI1 is transferred to a second range address in the memory. The first and second range addresses are alternately set in an interleaved manner.

[0061] In some embodiments, the CPU configures the DMA controller's CH0 and CH1 to enable the Interleaved transfer mode.

[0062] In some embodiments, after the DMA controller's CH0 transfers audio data D0 with a first number of channels (8CH) to the first range address of the DMA buffer, it skips the second range address with a data length of 2CH, and then transfers the next audio data D0 with a first number of channels (8CH) to the first range address of the DMA buffer. Similarly, after the DMA controller's CH1 transfers audio data D1 with a second number of channels (2CH) to the second range address of the DMA buffer, it skips the first range address with a data length of 8CH, and then transfers the next audio data D1 with a second number of channels (2CH) to the second range address of the DMA buffer. This allows the addresses of the DMA controller's CH0 and CH1 to be interleaved. At this time, the number of channels of the concatenated and merged data D obtained in a set of joint sub-range addresses is 10CH.

[0063] In step 310, the application reads concatenated data D with 10 channels from the DMA buffer at memory address 10. That is, the application reads 10 channels of data from the DMA buffer each time.

[0064] Figure 7 This is a hardware block diagram illustrating an audio recording method 300 according to an embodiment of the present invention. (Refer to...) Figure 7 In some embodiments, the digital audio interface DAI0 connects to the microphone array, and the digital audio interface DAI1 connects to the speaker feedback signal. The ADC (analog-to-digital converter) converts the analog microphone signal into a PCM digital signal. The microphone array is used to acquire the microphone recording signal, and the speaker feedback signal is used to capture the electrical signal of the music played from the speaker. When performing speech recognition on the acquired recording, an echo cancellation algorithm (AEC) is needed to filter out the background noise in the recording and extract the speech signal, thereby improving the accuracy of subsequent speech recognition. The AEC algorithm requires both the microphone recording signal and the PCM data of the feedback signal to process the audio, and it requires that the microphone recording signal and the PCM data of the feedback signal be phase-synchronized. Based on existing audio processing methods, the upper-layer application needs to interact with the operating system twice to obtain the microphone recording signal and PCM data. During this process, the system scheduling may be affected, causing instability and jitter in the phase of the microphone recording signal and the PCM data of the feedback signal, affecting the AEC algorithm's processing and thus impacting accuracy. The audio processing method of the present invention can avoid the influence of OS scheduling through a single read operation, and both digital audio interfaces use the same clock source, which ensures the phase synchronization of the PCM data between the MIC recording signal and the retrieval signal from the hardware principle.

[0065] In some embodiments, the application that records audio using speech recognition issues a multi-channel audio recording request. At this time, the corresponding device is opened via PCM_OPEN, relevant parameters are set, and then a signal to start recording is sent to the OS system. The OS system then triggers the DMA channels corresponding to the digital audio interfaces DAI0 and DAI1, and the concatenated data is obtained through the above method 300. The application then reads the concatenated data via PCM_READ, and finally hands the concatenated data over to the speech recognition application algorithm for processing.

[0066] It should be noted that the MIC recording signal refers to the sound recorded by the microphone (including speech, background music played from speakers, etc.). The purpose of the audio backsampling signal is to provide a reference signal for the playback path for the AEC algorithm. Based on this reference signal, the AEC algorithm can filter out the background noise in the recording and extract the speech signal for subsequent speech recognition. AEC (Acoustic Echo Cancellation) is a technique used in communication systems to handle echo problems. Echoes typically occur in voice communication when the audio signal from the speaker is propagated back through the microphone, causing an echo heard by the other party. The AEC algorithm aims to reduce or eliminate this echo to improve communication quality. DMA (Direct Memory Access) is a technique in computer architecture used to improve data transfer efficiency. It allows external devices (such as hard drives, network adapters, audio cards, etc.) to directly access computer memory without intervention from the central processing unit (CPU). This reduces the CPU load and increases data transfer speed, especially when large amounts of data need to be transferred between memory and external devices. PCM (Pulse Code Modulation) data is the representation of an analog signal after it has been converted into a digital signal at a fixed sampling frequency.

[0067] Figure 8 This is a flowchart illustrating an audio playback method 400 according to an embodiment of the present invention. (Refer to...) Figure 8 The method 400 plays audio output data through two digital audio interfaces (DAI0 and DAI1). The method 400 includes steps 402 to 408.

[0068] In step 402, the audio output data D with 10 channels is stored alternately in the memory address DMA Buffer according to the first range address and the second range address, wherein the number of channels corresponding to the first range address is 8 channels and the number of channels corresponding to the second range address is 2 channels.

[0069] Figure 9 This is a schematic diagram illustrating data transmission of an audio playback method 400 according to an embodiment of the present invention.

[0070] In step 404, in response to the audio output command, the DMA controller based on the Interleaved transfer mode reads the first distribution data D0 with a first number of channels of 8CH from the first range address, and at the same time reads the second distribution data D1 with a second number of channels of 2CH from the second range address.

[0071] In some embodiments, the DMA controller's CH0 reads distribution data D0 with a first number of channels (8CH) from a first address range, skips a second address range with a second number of channels (2CH), and then reads distribution data D0 with a first number of channels (8CH) from the next first address range. Similarly, the DMA controller's CH1 reads distribution data D1 with a second number of channels (2CH) from a second address range, skips a first address range with a first number of channels (8CH), and then reads distribution data D1 with a second number of channels (2CH) from the next second address range. This allows the DMA controller's CH0 and CH1 addresses to be read alternately. In this case, the resulting distribution data D0 has 8 channels, and the distribution data D1 has 2 channels.

[0072] In step 406, based on the DMA controller in Interleaved transfer mode, the distributed data D0 is transferred to the output buffer queue FIFO of digital audio interface DAI0, and the distributed data D1 is transferred to the output buffer queue FIFO of digital audio interface DAI1. Since DAI0 and DAI1 are driven by the same clock source, the output buffer queue FIFO receives data for synchronization.

[0073] In some embodiments, when configuring the transfer mode of the DMA controller, the source address and destination address of the data transferred by the DMA controller are configured synchronously. For the digital audio interface DAI0, the source address configured for its DMACH0 is the first range address in the DMA buffer, and the destination address is the output buffer queue FIFO in the digital audio interface DAI0. This allows the digital audio interface DAI0 to transfer the 8CH data in the first range address to the output buffer queue FIFO of the digital audio interface DAI0 through the CH0 of the DMA controller, and finally send the data of the output buffer queue FIFO out through the hardware interface of the digital audio interface DAI0.

[0074] In step 408, when the storage capacity of the output buffer queue FIFO reaches the preset threshold, the DMA controller is triggered to start transmitting the distribution data D0 and D1 in its output buffer queue FIFO to the digital audio interfaces DAI0 and DAI1 for playback.

[0075] Figure 10 This is a hardware block diagram illustrating an audio playback method 400 according to an embodiment of the present invention. (Refer to...) Figure 10 In some embodiments, digital audio interfaces DAI0 and DAI1 are connected to a DAC array and then to a speaker array. The DAC (digital-to-analog converter) converts the PCM digital audio signal into an analog signal to drive the speaker array to produce sound.

[0076] In some embodiments, multi-channel playback refers to playing PCM data through multiple channels. For example, Dolby's 7.1 channel (8-channel audio playback) or 7.1.4 (12-channel audio playback). When an application needs to play a 7.1.4 channel audio source file, existing audio playback methods typically require decoding the 7.1.4 channel audio source into 7.1.4 channel PCM data, then transferring the user-space 7.1.4 channel PCM data buffer to the OS system through two write operations. Upon receiving the start playback command, the OS system simultaneously triggers DMA data transfer operations corresponding to the two digital audio interfaces (DAI). During this process, system scheduling may affect data transfer, leading to asynchronous audio playback. However, the audio processing method of this invention avoids the influence of OS scheduling with a single write operation, and since both digital audio interfaces use the same clock source, hardware principles guarantee synchronous audio playback between the two audio interfaces.

[0077] According to another aspect of the invention, Figure 11 This is a schematic diagram illustrating the structure of an apparatus 500 for processing audio data according to an embodiment of the present invention. (Refer to...) Figure 11 The audio data processing apparatus 500 includes an interface module 502, a DMA module 504, and an application layer module 506. The interface module 502 is configured to receive at least two audio data points through at least two digital audio interfaces in response to an audio input command, and to store the at least two audio data points in at least two input buffer queues corresponding to the at least two digital audio interfaces. The DMA module 504 is configured to transfer the at least two audio data points from the at least two input buffer queues to addresses in memory corresponding to the at least two input buffer queues, such that the at least two audio data points corresponding to the at least two digital audio interfaces are alternately stored in the memory. The application layer module 506 is configured to read the alternately stored at least two audio data points from the memory as concatenated data from the at least two digital audio interfaces.

[0078] In some embodiments, the application layer module 506 is further configured to acquire audio output data in response to an audio output command, the audio output data including at least two distribution data, each corresponding to at least two digital audio interfaces. The DMA module 504 is further configured to store the at least two distribution data of the audio output data into addresses in memory corresponding to the at least two digital audio interfaces, such that the at least two distribution data are alternately stored in the memory, and is configured to transfer the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces. The interface module 502 is further configured to output the at least two distribution data from the at least two output buffer queues through the at least two digital audio interfaces for audio playback based on the at least two distribution data.

[0079] It should be understood that the interface module 502, the DMA module 504, and the application layer module 506 can be further configured to perform the corresponding steps or actions in the methods described above, which will not be elaborated here.

[0080] According to another aspect of the invention, Figure 12 This is a schematic diagram illustrating an electronic device 600 according to an embodiment of the present invention. (Refer to...) Figure 12 The electronic device 600 includes a memory 602, a processor 604, and a computer program stored in the memory 602 and executable on the processor 604. When the processor 604 executes the computer program, it implements the various steps of the method for processing audio data as described above.

[0081] In summary, the method, apparatus, and electronic device for processing audio data provided by this invention, through the design of a hardware DMA controller, splices or distributes audio data from each channel of multiple Digital Audio Interfaces (DAIs), eliminating the need for porting related software, thus improving data processing efficiency and reducing CPU load. During data processing, the upper-layer application of this invention only needs one read operation or one write operation to splice or distribute the audio data input or output from the DAIs into multiple channels of audio data during the DMA operation within the SoC, and finally transmit it to the corresponding audio processing module or audio playback module. This processing method can significantly reduce the CPU load of the entire system, reduce latency, improve synchronization stability, and make the system easier to expand and integrate into various application scenarios.

[0082] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method of processing audio data, characterized by, comprises: in response to an audio input instruction, receiving at least two audio data respectively through at least two digital audio interfaces under the same clock driving, and storing the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces; transmitting the at least two audio data in the at least two input buffer queues respectively to addresses in a memory corresponding to the at least two input buffer queues under the same clock driving, so that the at least two audio data corresponding to the at least two digital audio interfaces are alternately stored in the memory respectively; and reading the alternately stored at least two audio data from the memory as spliced and merged data from the at least two digital audio interfaces, wherein receiving at least two audio data respectively through at least two digital audio interfaces comprises: receiving first audio data through a first digital audio interface and simultaneously receiving second audio data through a second digital audio interface under the same clock driving, wherein storing the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces comprises: storing the first audio data in a first input buffer queue through a first interface controller corresponding to the first digital audio interface and simultaneously storing the second audio data in a second input buffer queue through a second interface controller corresponding to the second digital audio interface under the same clock driving, wherein receiving first audio data through a first digital audio interface comprises: receiving the first audio data comprising a plurality of groups of first audio sub-data through the first digital audio interface, each group of the first audio sub-data corresponding to a first number of channels, wherein receiving second audio data through a second digital audio interface comprises: receiving the second audio data comprising a plurality of groups of second audio sub-data through the second digital audio interface, each group of the second audio sub-data corresponding to a second number of channels, wherein transmitting the at least two audio data in the at least two input buffer queues respectively to addresses in a memory corresponding to the at least two input buffer queues comprises: storing a plurality of groups of the first audio sub-data respectively at a plurality of groups of first sub-range addresses in the memory and storing a plurality of groups of the second audio sub-data respectively at a plurality of groups of second sub-range addresses in the memory, so that each group of the first audio sub-data and each group of the second audio sub-data are alternately stored in the memory in an interleaved manner, wherein each group of the first sub-range addresses and each group of the second sub-range addresses are alternately arranged in an interleaved manner, so that a group of second sub-range addresses is immediately followed by a group of first sub-range addresses, a next group of first sub-range addresses is immediately followed by the group of second sub-range addresses, and a next group of second sub-range addresses is immediately followed by the next group of first sub-range addresses. transmitting the at least two audio data in the at least two input buffer queues respectively to addresses in a memory corresponding to the at least two input buffer queues comprises:

2. The method of claim 1, wherein, ​ under the same clock driving, storing the first audio data in the first input cache queue at a first range of addresses in the memory through a first storage controller corresponding to the first digital audio interface, and simultaneously storing the second audio data in the second input cache queue at a second range of addresses in the memory through a second storage controller corresponding to the second digital audio interface, wherein the first range of addresses and the second range of addresses are alternately arranged in an interleaved manner.

3. The method of claim 2, wherein, reading the alternately stored at least two audio data from the memory as spliced and merged data from the at least two digital audio interfaces includes: reading the alternately stored first audio data and second audio data from the memory, and combining the read first audio data and second audio data as the spliced and merged data.

4. The method of claim 1, wherein, reading the alternately stored at least two audio data from the memory as spliced and merged data from the at least two digital audio interfaces includes: reading each group of the first audio sub-data and each group of the second audio sub-data from the memory, and combining each group of the read first audio sub-data and each group of the second audio sub-data as a group of spliced and merged sub-data, the spliced and merged data including a plurality of groups of the spliced and merged sub-data.

5. The method of claim 2, wherein, transmitting the at least two audio data in the at least two input cache queues to addresses in the memory corresponding to the at least two input cache queues, respectively, includes: in response to the storage amount of the first input cache queue or the second input cache queue reaching a preset waterline, transmitting the first audio data or the second audio data to the first range of addresses or the second range of addresses.

6. The method of claim 4, wherein, reading the alternately stored at least two audio data from the memory as spliced and merged data from the at least two digital audio interfaces includes: in response to each group of the first audio sub-data and each group of the second audio sub-data having been stored in a group of joint sub-range addresses in the memory, reading a group of first audio sub-data and a group of second audio sub-data from the group of joint sub-range addresses in the memory as a group of spliced and merged sub-data.

7. The method of claim 1, wherein, further comprising: processing the spliced and merged data using an echo cancellation algorithm to remove background sound, the audio data including microphone recording data and audio back sampling data.

8. The method of claim 1, wherein, further comprising: in response to an audio output instruction, transmitting at least two distribution data included in audio output data to addresses in the memory corresponding to at least two digital audio interfaces, respectively, so that the at least two distribution data are alternately stored in the memory, respectively, the at least two distribution data corresponding to the at least two digital audio interfaces, respectively; transmitting the alternately stored at least two distribution data in the memory to at least two output cache queues corresponding to the at least two digital audio interfaces, respectively; and further comprising: outputting the at least two pieces of distribution data in the at least two output buffer queues through the at least two digital audio interfaces respectively for audio playing based on the at least two pieces of distribution data.

9. A method of processing audio data, characterized by, The method comprises: in response to an audio output instruction, obtaining audio output data, the audio output data comprising at least two pieces of distribution data, the at least two pieces of distribution data corresponding to at least two digital audio interfaces respectively; storing the at least two pieces of distribution data of the audio output data in the memory at addresses corresponding to the at least two digital audio interfaces respectively, so that the at least two pieces of distribution data are alternately stored in the memory; transmitting the at least two pieces of distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively under the same clock driving; and outputting the at least two pieces of distribution data in the at least two output buffer queues through the at least two digital audio interfaces respectively for audio playing based on the at least two pieces of distribution data under the same clock driving, wherein the storing of the at least two pieces of distribution data of the audio output data in the memory at addresses corresponding to the at least two digital audio interfaces respectively comprises: storing first distribution data of the audio output data in the memory at a first range of addresses corresponding to a first digital audio interface, and storing second distribution data of the audio output data in the memory at a second range of addresses corresponding to a second digital audio interface, wherein the first range of addresses and the second range of addresses are alternately arranged in an interleaved manner, wherein the storing of the first distribution data of the audio output data in the memory at the first range of addresses corresponding to the first digital audio interface comprises: storing a plurality of groups of first distribution sub-data of the first distribution data in the memory at a plurality of groups of first sub-range of addresses respectively, each group of the first distribution sub-data corresponding to a first number of channels, wherein the storing of the second distribution data of the audio output data in the memory at the second range of addresses corresponding to the second digital audio interface comprises: storing a plurality of groups of second distribution sub-data of the second distribution data in the memory at a plurality of groups of second sub-range of addresses respectively, each group of the second distribution sub-data corresponding to a second number of channels, wherein each group of the first sub-range of addresses and each group of the second sub-range of addresses are alternately arranged in an interleaved manner, so that a group of second sub-range of addresses is immediately followed by a group of first sub-range of addresses, a next group of first sub-range of addresses is immediately followed by the group of second sub-range of addresses, and a next group of second sub-range of addresses is immediately followed by the next group of first sub-range of addresses. the transmitting of the at least two pieces of distribution data alternately stored in the memory to the at least two output buffer queues corresponding to the at least two digital audio interfaces respectively comprises:

10. The method of claim 9, wherein, ​ The first distribution data stored in the memory at the first range of addresses is stored into a first output cache queue corresponding to the first digital audio interface by a first storage controller corresponding to the first digital audio interface, and at the same time, the second distribution data stored in the memory at the second range of addresses is stored into a second output cache queue corresponding to the second digital audio interface by a second storage controller corresponding to the second digital audio interface.

11. The method of claim 10, wherein, The outputting of the at least two distribution data in the at least two output cache queues by the at least two digital audio interfaces comprises: The first distribution data in the first output cache queue is outputted to the outside via the first digital audio interface by a first interface controller corresponding to the first digital audio interface, and at the same time, the second distribution data in the second output cache queue is outputted to the outside via the second digital audio interface by a second interface controller corresponding to the second digital audio interface, under the same clock driving, so that a first audio playing device corresponding to the first digital audio interface and a second audio playing device corresponding to the second digital audio interface respectively play audio based on the first distribution data and the second distribution data for the audio output data.

12. The method of claim 9, wherein, The transmission of the at least two distribution data alternately stored in the memory to the at least two output cache queues corresponding to the at least two digital audio interfaces comprises: In response to each group of the first distribution sub-data and each group of the second distribution sub-data having been stored in a group of joint sub-range addresses of the memory, a group of first distribution sub-data and a group of second distribution sub-data stored in the group of joint sub-range addresses of the memory are respectively transmitted to a first output cache queue corresponding to the first digital audio interface and a second output cache queue corresponding to the second digital audio interface.

13. The method of claim 10, wherein, The outputting of the at least two distribution data in the at least two output cache queues by the at least two digital audio interfaces comprises: In response to the storage amount of the first output cache queue or the second output cache queue reaching a preset waterline, the first distribution data or the second distribution data is transmitted to the first digital audio interface or the second digital audio interface for output.

14. An apparatus for processing audio data, the apparatus comprising: Comprise: An interface module configured to, in response to an audio input instruction, receive at least two audio data by at least two digital audio interfaces respectively under the same clock driving, and store the at least two audio data in at least two input cache queues corresponding to the at least two digital audio interfaces respectively; A DMA module configured to, under the same clock driving, transmit the at least two audio data in the at least two input cache queues to addresses in a memory corresponding to the at least two input cache queues respectively, so that the at least two audio data corresponding to the at least two digital audio interfaces are alternately stored in the memory respectively; And an application layer module configured to read the at least two audio data alternately stored in the memory as spliced and merged data from the at least two digital audio interfaces, wherein the interface module is configured to receive the first audio data through the first digital audio interface and the second audio data through the second digital audio interface simultaneously under the same clock driving, and store the first audio data in the first input buffer queue through the first interface controller corresponding to the first digital audio interface and the second audio data in the second input buffer queue through the second interface controller corresponding to the second digital audio interface simultaneously under the same clock driving, wherein the interface module is configured to receive the first audio data including a plurality of groups of first audio sub-data through the first digital audio interface, each group of the first audio sub-data corresponding to a first number of channels, and receive the second audio data including a plurality of groups of second audio sub-data through the second digital audio interface, each group of the second audio sub-data corresponding to a second number of channels, wherein the DMA module is configured to store a plurality of groups of the first audio sub-data at a plurality of groups of first sub-range addresses in the memory respectively, and store a plurality of groups of the second audio sub-data at a plurality of groups of second sub-range addresses in the memory respectively, such that each group of the first audio sub-data and each group of the second audio sub-data are alternately stored in the memory in an interleaved manner, wherein each group of the first sub-range addresses and each group of the second sub-range addresses are alternately arranged in an interleaved manner, such that a group of second sub-range addresses is immediately followed by a group of first sub-range addresses, a next group of first sub-range addresses is immediately followed by the group of second sub-range addresses, and a next group of second sub-range addresses is immediately followed by the next group of first sub-range addresses.

15. An apparatus for processing audio data, the apparatus comprising: comprising: an application layer module configured to obtain audio output data in response to an audio output instruction, the audio output data including at least two distribution data corresponding to at least two digital audio interfaces respectively; a DMA module configured to store the at least two distribution data of the audio output data to addresses corresponding to the at least two digital audio interfaces in a memory respectively, such that the at least two distribution data are alternately stored in the memory respectively, and configured to transmit the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively under the same clock driving; and an interface module configured to output the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces respectively under the same clock driving, for audio playing based on the at least two distribution data, wherein the DMA module is configured to store first distribution data of the audio output data to a first range of addresses in the memory corresponding to a first digital audio interface, and to store second distribution data of the audio output data to a second range of addresses in the memory corresponding to a second digital audio interface, wherein the first range of addresses and the second range of addresses are arranged in an interleaved manner, wherein the DMA module is configured to store a plurality of groups of first distribution sub-data of the first distribution data to a plurality of groups of first sub-ranges of addresses in the memory respectively, each group of the first distribution sub-data corresponding to a first number of channels, and to store a plurality of groups of second distribution sub-data of the second distribution data to a plurality of groups of second sub-ranges of addresses in the memory respectively, each group of the second distribution sub-data corresponding to a second number of channels, wherein each group of the first sub-ranges of addresses and each group of the second sub-ranges of addresses are arranged in an interleaved manner, such that a group of second sub-ranges of addresses is immediately followed by a group of first sub-ranges of addresses, a next group of first sub-ranges of addresses is immediately followed by the group of second sub-ranges of addresses, and a next group of second sub-ranges of addresses is immediately followed by the next group of first sub-ranges of addresses.

16. An electronic device, comprising: comprising: a memory configured to store an executable program; and a processor configured to execute the program to perform the method according to any one of claims 1 to 8 or the method according to any one of claims 9 to 13.

Images