Method and device for processing audio data and electronic equipment

By alternately storing and reading audio data of multiple digital audio interfaces in memory, the problems of stability and delay of multi-channel audio systems in the prior art are solved, and lower CPU load and higher synchronous stability are achieved. sex.

CN119049477AActive Publication Date: 2024-11-29FUZHOU ROCKCHIP SEMICON
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410948725.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-29
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

In voice assistants, voice recognition systems and voice control devices, the splicing of multiple digital audio interfaces usually depends on CPU processing, resulting in poor system stability, high latency and large CPU load, and requires additional software migration, which has a large workload. .

Method used

By storing the audio data of multiple digital audio interfaces in the input cache queue, and alternately storing and reading in memory, the delay caused by inconsistent data transmission time is avoided, synchronization stability is improved, and directly through the DMA module Data stitching and distribution are completed in memory, reducing CPU load and workload.

Benefits of technology

It realizes that while meeting multi-channel needs, it reduces CPU load and delay, improves synchronization stability, reduces the workload of data processing and distribution, and simplifies system design.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049477A_ABST
    Figure CN119049477A_ABST
Patent Text Reader

Abstract

The invention discloses a method and device for processing audio data and electronic equipment. The method comprises the following steps: in response to an audio input instruction, respectively receiving at least two audio data through at least two digital audio interfaces, and respectively storing the at least two audio data in at least two input cache queues corresponding to the at least two digital audio interfaces; respectively transmitting the at least two pieces of audio data in the at least two input cache queues to addresses corresponding to the at least two input cache queues in a memory, so that the at least two pieces of audio data corresponding to the at least two digital audio interfaces are respectively and alternately stored in the memory; and reading the at least two alternately stored audio data from the memory to serve as spliced and merged data from the at least two digital audio interfaces. According to the method, the CPU load and the time delay of the system are reduced while the multi-channel requirement is met, and the synchronization stability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of audio data processing, and in particular to a method and device for processing audio data, and electronic equipment. Background Art

[0002] In applications such as voice assistants, speech recognition (ASR) systems, and voice control devices, one digital audio interface (DAI) is usually unable to meet the number of channels required. However, in SoC (System on Chip) systems, multiple digital audio interfaces (DAIs) can be spliced ​​together to meet the multi-channel requirements. Currently, multi-channel audio systems are usually implemented by splicing multiple DAIs on the CPU (Central Processing Unit) through software (such as alsa-lib's multi plug-in).

[0003] However, this splicing method is easily affected by the OS (Operating System) because it needs to be processed by the CPU, and the instability jitter is relatively large, which makes debugging quite difficult for applications that require strict and stable synchronization of multiple channels. In addition, the CPU has a high overhead and high latency for processing audio data, and additional software porting is required, which is a lot of work. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for processing audio data, and an electronic device, which can reduce the CPU load and delay of the system and improve synchronization stability while meeting multi-channel requirements.

[0005] In one aspect of the present invention, a method for processing audio data is provided. The method comprises: in response to an audio input instruction, receiving at least two audio data respectively through at least two digital audio interfaces, and storing the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces; transmitting the at least two audio data in the at least two input buffer queues to addresses corresponding to the at least two input buffer queues in a memory respectively, so that the at least two audio data corresponding to the at least two digital audio interfaces are respectively stored alternately in the memory; and reading the at least two audio data stored alternately from the memory as spliced ​​and merged data from the at least two digital audio interfaces.

[0006] In another aspect of the present invention, a device for processing audio data is provided. The device includes: an application layer module configured to obtain audio output data in response to an audio output instruction, wherein the audio output data includes at least two distribution data, and the at least two distribution data correspond to at least two digital audio interfaces respectively; a DMA module configured to store the at least two distribution data of the audio output data in a memory at addresses corresponding to the at least two digital audio interfaces respectively, so that the at least two distribution data are alternately stored in the memory respectively, and configured to transmit the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively; and an interface module configured to output the at least two distribution data in the at least two output buffer queues respectively through the at least two digital audio interfaces, so as to play audio based on the at least two distribution data.

[0007] In another aspect of the present invention, an electronic device is provided, comprising a memory configured to store a computer program and a processor configured to execute the computer program to perform the above method for processing audio data.

[0008] According to the technical solution of the present invention, the audio data received by at least two digital audio interfaces are first stored in the input buffer queues corresponding to each other, and then the audio data in the input buffer queues are respectively transmitted to the corresponding addresses in the memory, so that the audio data received by different digital audio interfaces are alternately stored in the memory. When the audio data received by different digital audio interfaces are alternately stored in the memory, the audio data at the adjacent addresses can complete the splicing and merging. Finally, the audio data after the splicing and merging of different digital audio interfaces can be directly read from the memory. The present invention is based on the input buffer queue to store different audio data, which can avoid the delay caused by inconsistent data transmission time between different digital audio interfaces and improve synchronization stability. And different audio data are transmitted to the memory in an alternate storage mode to complete the splicing and merging, which can avoid introducing related software to perform data splicing, effectively reduce the splicing workload, and improve splicing efficiency. In addition, when reading the splicing and merging data, only one memory needs to be read to obtain the splicing and merging data, which effectively reduces the CPU load and avoids the delay caused by reading data from different memories.

[0009] According to another aspect of the technical solution of the present invention, at least two distribution data included in the audio output data are respectively transmitted to the addresses corresponding to at least two digital audio interfaces in the memory, so that at least two distribution data are respectively stored alternately in the memory, and at least two distribution data correspond to the at least two digital audio interfaces; then the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output cache queues corresponding to the at least two digital audio interfaces; and the at least two distribution data in the at least two output cache queues are respectively output through the at least two digital audio interfaces, so as to play audio based on the at least two distribution data. In this way, the audio output data can be divided into a plurality of different distribution data, so that the distribution data can be played based on digital audio interfaces with different numbers of channels, while meeting the requirements of multi-channel audio playback, reducing data processing delay and processing efficiency. The present invention stores different audio data based on the output cache queue, which can avoid the delay caused by inconsistent data transmission time between different digital audio interfaces and improve synchronization stability. In addition, data distribution is completed by transferring different audio data to the memory through alternating storage, which can avoid the introduction of related software for data distribution, effectively reduce the distribution workload, improve splicing efficiency, effectively reduce CPU load, and avoid the delay caused by outputting data from different memories separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of an audio recording method in the prior art;

[0011] Figure 2 A schematic diagram of an audio playback method in the prior art;

[0012] Figure 3 A flowchart of a method for processing audio data according to an embodiment of the present invention;

[0013] Figure 4 A flowchart of another method for processing audio data according to an embodiment of the present invention;

[0014] Figure 5 is a flow chart of an audio recording method according to an embodiment of the present invention;

[0015] Figure 6 A data transmission schematic diagram of an audio recording method according to an embodiment of the present invention;

[0016] Figure 7 A hardware block diagram of an audio recording method according to an embodiment of the present invention;

[0017] Figure 8 A flowchart of an audio playback method according to an embodiment of the present invention;

[0018] Fig. 9 A data transmission schematic diagram of an audio playback method according to an embodiment of the present invention;

[0019] Fig.10 A hardware block diagram of an audio playback method according to an embodiment of the present invention;

[0020] Fig.11 A schematic diagram of the structure of a device for processing audio data according to an embodiment of the present invention;

[0021] Fig.12 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0023] In the prior art, a multi-channel audio system is usually implemented by splicing multiple DAIs on a CPU using software. Figure 1 As shown in the figure, when recording is processed by splicing multiple DAIs, the DMA controller needs to read the DMA Buffer of DAI0 and DAI1 respectively through software during the data transfer process, and then merge the audio data through the multi plug-in of alsa-lib, and finally transmit it to the speech recognition application algorithm for processing. The application layer needs to perform two Read operations to obtain the audio data of all DAIs. Similarly, Figure 2 As shown, when playing audio through multiple DAI splicing, the playback application first needs to output audio data to the multi plug-in of alsa-lib for processing on the CPU, and then the multi plug-in writes the audio data to the corresponding DMA Buffer of DAI0 and DAI1 respectively, and finally transmits it to each DAI for playback. The application layer also needs to perform two Write operations to output the audio data. During the recording process and audio playback, the two Read operations and the two Write operations are easily affected by OS scheduling, and the PCM data buffer delay of two DAI times is unstable, and the delay is also increased a lot. Therefore, in the existing technology, when processing audio data, the multi-channel splicing system needs to process it through the CPU, which is easily affected by the OS, and the instability jitter is relatively large. For synchronous applications that require strict stability of multiple channels, the debugging difficulty is quite large, and the CPU has a large overhead and high latency for processing audio data, and additional software porting is required, which is a large workload.

[0024] In order to solve at least the above technical problems, the present disclosure provides a method for processing audio data. According to the present disclosure, the audio data received by at least two digital audio interfaces are first stored in the input buffer queues corresponding to each other, and then the audio data in the input buffer queues are respectively transmitted to the corresponding addresses in the memory, so that the audio data received by different digital audio interfaces are alternately stored in the memory. When the audio data received by different digital audio interfaces are alternately stored in the memory, the audio data at the adjacent addresses can be spliced ​​and merged. Finally, the audio data after splicing and merging of different digital audio interfaces can be directly read from the memory. In this way, according to the embodiment of the present disclosure, different audio data are stored based on the input buffer queue, which can avoid the delay caused by inconsistent data transmission time between different digital audio interfaces and improve synchronization stability. And different audio data are transmitted to the memory in an alternate storage mode to complete the splicing and merging, which can avoid the introduction of related software for data splicing, effectively reduce the workload of splicing, and improve splicing efficiency. In addition, when reading the spliced ​​and merged data, only one memory needs to be read to obtain the spliced ​​and merged data, which effectively reduces the CPU load and avoids the delay caused by reading data from different memories.

[0025] According to another aspect of the present disclosure, at least two distribution data included in the audio output data are respectively transmitted to the addresses corresponding to at least two digital audio interfaces in the memory, so that at least two distribution data are respectively stored alternately in the memory, and at least two distribution data correspond to the at least two digital audio interfaces; then the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output cache queues corresponding to the at least two digital audio interfaces; and the at least two distribution data in the at least two output cache queues are respectively output through the at least two digital audio interfaces, so as to be used for audio playback based on the at least two distribution data. In this way, the audio output data can be divided into a plurality of different distribution data, so that the distribution data can be played based on digital audio interfaces with different numbers of channels, while meeting the multi-channel audio playback requirements, reducing data processing delay and processing efficiency. In this way, according to the embodiment of the present disclosure, different audio data are stored based on the output cache queue, which can avoid the delay caused by inconsistent data transmission time between different digital audio interfaces and improve synchronization stability. In addition, data distribution is completed by transferring different audio data to the memory through alternating storage, which can avoid the introduction of related software for data distribution, effectively reduce the distribution workload, improve splicing efficiency, effectively reduce CPU load, and avoid the delay caused by outputting data from different memories separately.

[0026] Hereinafter, the technical solution according to the present disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0027] Figure 3 1 is a flow chart showing a method 100 for processing audio data according to an embodiment of the present disclosure. Figure 3 The method 100 includes the following steps 102 to 106.

[0028] In step 102, in response to an audio input instruction, at least two audio data are received through at least two digital audio interfaces respectively, and the at least two audio data are stored in at least two input buffer queues corresponding to the at least two digital audio interfaces respectively.

[0029] In some embodiments, receiving at least two audio data respectively through at least two digital audio interfaces includes: receiving the first audio data through the first digital audio interface and receiving the second audio data through the second digital audio interface at the same time under the same clock drive. In this way, audio data are respectively received through multiple digital audio interfaces to meet the multi-channel requirements of the current system.

[0030] In some embodiments, storing the at least two audio data in at least two input cache queues corresponding to the at least two digital audio interfaces respectively includes: under the same clock drive, the first audio data is stored in the first input cache queue by the first interface controller corresponding to the first digital audio interface, and at the same time, the second audio data is stored in the second input cache queue by the second interface controller corresponding to the second digital audio interface. In this way, based on the same clock drive, the two digital audio interfaces are guaranteed to be synchronous in hardware, avoiding the delay caused by the two digital audio interfaces when receiving audio data. At the same time, the corresponding audio data is stored based on the input cache queue to temporarily store the data received by the digital audio interface, so as to realize smooth processing of the data stream and avoid data loss or overflow.

[0031] In some embodiments, receiving the first audio data through the first digital audio interface includes: receiving the first audio data including a plurality of groups of first audio sub-data through the first digital audio interface, each group of the first audio sub-data corresponding to a first number of channels.

[0032] Receiving the second audio data through the second digital audio interface includes: receiving the second audio data including a plurality of groups of second audio sub-data through the second digital audio interface, each group of the second audio sub-data corresponding to a second number of channels. In this way, the audio data received by each digital audio interface under the corresponding number of channels is accurately obtained, so as to facilitate subsequent splicing in the memory based on the number of channels.

[0033] In step 104, the at least two audio data in the at least two input buffer queues are respectively transmitted to addresses corresponding to the at least two input buffer queues in the memory, so that the at least two audio data corresponding to the at least two digital audio interfaces are respectively and alternately stored in the memory.

[0034] In some embodiments, the at least two audio data in the at least two input cache queues are respectively transmitted to the addresses corresponding to the at least two input cache queues in the memory, including: under the same clock drive, the first audio data in the first input cache queue is stored at the first range address in the memory through the first storage controller corresponding to the first digital audio interface, and at the same time, the second audio data in the second input cache queue is stored at the second range address in the memory through the second storage controller corresponding to the second digital audio interface, wherein the first range address and the second range address are alternately set in an interleaved manner. In this way, the audio data of different digital audio interfaces are alternately stored in the memory, thereby completing the splicing and merging of the audio data based on the memory address, avoiding the introduction of third-party software, and reducing the CPU load.

[0035] In some embodiments, in response to the storage capacity of the first input cache queue or the second input cache queue reaching a preset watermark, the first audio data or the second audio data is transferred to the first range address or the second range address. In this way, when the storage capacity of the input cache queue reaches the preset watermark, the audio data is synchronously transferred to the corresponding address, avoiding the time delay between the two digital audio interfaces and improving the synchronization stability.

[0036] In some embodiments, transmitting the at least two audio data in the at least two input buffer queues to the addresses corresponding to the at least two input buffer queues in the memory includes: storing multiple groups of the first audio sub-data at multiple groups of first sub-range addresses in the memory, and storing multiple groups of the second audio sub-data at multiple groups of second sub-range addresses in the memory, so that each group of the first audio sub-data and each group of the second audio sub-data are alternately stored in the memory in an interleaved manner, wherein each group of the first sub-range addresses and each group of the second sub-range addresses are alternately set in an interleaved manner, so that a group of second sub-range addresses is immediately after a group of first sub-range addresses, the next group of first sub-range addresses is immediately after the group of second sub-range addresses, and the next group of second sub-range addresses is immediately after the next group of first sub-range addresses. In this way, since different sub-range addresses are alternately set in an interleaved manner, the audio sub-data received by different digital audio interfaces can be directly spliced ​​based on the alternating addresses of the memory when stored at the corresponding sub-range addresses, without the need to introduce third-party splicing software, effectively improving data processing efficiency.

[0037] In step 106, the at least two audio data stored alternately are read from the memory as spliced ​​and merged data from the at least two digital audio interfaces.

[0038] In some embodiments, reading the at least two audio data stored alternately from the memory as the spliced ​​merged data from the at least two digital audio interfaces includes: reading each group of the first audio sub-data and each group of the second audio sub-data from the memory, and taking the combination of each group of the first audio sub-data and each group of the second audio sub-data read as a group of spliced ​​merged sub-data, the spliced ​​merged data including multiple groups of the spliced ​​merged sub-data. In this way, only corresponding data needs to be read from one memory, avoiding reading data from two different memories and then splicing, thereby avoiding data delay.

[0039] In some embodiments, reading the at least two audio data stored alternately from the memory as the spliced ​​merged data from the at least two digital audio interfaces includes: in response to each group of the first audio sub-data and each group of the second audio sub-data being stored in a set of joint sub-range addresses of the memory, reading a group of first audio sub-data and a group of second audio sub-data from the set of joint sub-range addresses of the memory as a group of spliced ​​merged sub-data. In this way, since different digital audio interfaces are alternately stored in the memory addresses, when reading data, only the data at the corresponding address needs to be read to complete the splicing, without the need to read the data from two different memories separately, and without the need to introduce third-party software for splicing, which effectively improves data processing efficiency.

[0040] In some embodiments, the spliced ​​and merged data is processed using an echo cancellation algorithm to remove background sound, and the audio data includes microphone recording data and audio echo data. In this way, the data processing quality is improved.

[0041] In some embodiments, the method may further include: in response to the audio output instruction, transmitting at least two distribution data included in the audio output data to addresses corresponding to at least two digital audio interfaces in the memory, so that the at least two distribution data are respectively stored alternately in the memory, and the at least two distribution data correspond to the at least two digital audio interfaces; transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces; and outputting the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces, so as to play audio based on the at least two distribution data. In this way, the audio output data can be divided into a plurality of different distribution data, so that the distribution data can be played based on digital audio interfaces with different numbers of channels, while meeting the multi-channel audio playback requirements, reducing data processing delay and processing efficiency.

[0042] Figure 4 2 is a flow chart showing another method 200 for processing audio data according to an embodiment of the present disclosure. Figure 4 The method 200 includes the following steps 202 to 208.

[0043] In step 202, in response to the audio output instruction, audio output data is acquired, where the audio output data includes at least two distribution data, and the at least two distribution data correspond to at least two digital audio interfaces respectively.

[0044] In step 204, the at least two distribution data of the audio output data are respectively stored in the memory at addresses corresponding to the at least two digital audio interfaces, so that the at least two distribution data are respectively stored alternately in the memory.

[0045] In some embodiments, storing the at least two distribution data of the audio output data in the memory at addresses corresponding to the at least two digital audio interfaces respectively includes: storing the first distribution data of the audio output data in the memory at a first range address corresponding to the first digital audio interface, and storing the second distribution data of the audio output data in the memory at a second range address corresponding to the second digital audio interface, wherein the first range address and the second range address are alternately set in an interleaved manner. In this way, the audio output data is split based on the address of the memory without introducing third-party splitting software, thereby reducing the CPU load.

[0046] In some embodiments, storing the first distribution data of the audio output data at a first range address corresponding to the first digital audio interface in the memory comprises: storing a plurality of groups of first distribution sub-data of the first distribution data at a plurality of groups of first sub-range addresses in the memory respectively, each group of the first distribution sub-data corresponding to a first number of channels;

[0047] Storing the second distribution data of the audio output data in the memory at the second range address corresponding to the second digital audio interface includes: storing multiple groups of second distribution sub-data of the second distribution data at multiple groups of second sub-range addresses in the memory, each group of the second distribution sub-data corresponds to a second number of channels, wherein each group of the first sub-range addresses and each group of the second sub-range addresses are alternately arranged in an interleaved manner, so that one group of second sub-range addresses is immediately after one group of first sub-range addresses, the next group of first sub-range addresses is immediately after the first group of second sub-range addresses, and the next group of second sub-range addresses is immediately after the next group of first sub-range addresses. In this way, the audio output data is accurately divided into the number of channels corresponding to each digital audio interface, so as to facilitate subsequent playback output through the corresponding digital audio interface.

[0048] In step 206, the at least two distribution data alternately stored in the memory are respectively transmitted to at least two output buffer queues corresponding to the at least two digital audio interfaces.

[0049] In some embodiments, the at least two distribution data alternately stored in the memory are transmitted to at least two output cache queues corresponding to the at least two digital audio interfaces respectively, including: under the same clock drive, the first distribution data stored at the first range address in the memory is stored in the first output cache queue corresponding to the first digital audio interface through the first storage controller corresponding to the first digital audio interface, and at the same time, the second distribution data stored at the second range address in the memory is stored in the second output cache queue corresponding to the second digital audio interface through the second storage controller corresponding to the second digital audio interface. In this way, different distribution data are stored based on the output cache queue to temporarily store the data played by the digital audio interface, so as to realize smooth processing of data stream, avoid data loss or overflow, and ensure that the audio data can be played synchronously between the two digital audio interfaces to reduce data delay.

[0050] In some embodiments, transmitting the at least two distribution data alternately stored in the memory to at least two output cache queues corresponding to the at least two digital audio interfaces includes: in response to each group of the first distribution sub-data and each group of the second distribution sub-data being stored in a group of joint sub-range addresses of the memory, transmitting a group of first distribution sub-data and a group of second distribution sub-data stored in the group of joint sub-range addresses of the memory to a first output cache queue corresponding to the first digital audio interface and a second output cache queue corresponding to the second digital audio interface. In this way, since different distribution data are alternately stored in the memory addresses, when transmitting data, data distribution can be completed by only transferring the data at the corresponding addresses to different output cache queues, without reading data from two different memories respectively, effectively improving data processing efficiency.

[0051] In step 208, the at least two distribution data in the at least two output buffer queues are outputted respectively through the at least two digital audio interfaces, so as to perform audio playback based on the at least two distribution data.

[0052] In some embodiments, outputting the at least two distribution data in the at least two output cache queues respectively through the at least two digital audio interfaces includes: under the same clock drive, outputting the first distribution data in the first output cache queue to the outside through the first digital audio interface through the first interface controller corresponding to the first digital audio interface, and simultaneously outputting the second distribution data in the second output cache queue to the outside through the second digital audio interface through the second interface controller corresponding to the second digital audio interface, so that the first audio playback device corresponding to the first digital audio interface and the second audio playback device corresponding to the second digital audio interface respectively play audio for the audio output data based on the first distribution data and the second distribution data. In this way, based on the same clock drive, it is ensured that the two output cache queues can synchronously transmit different distribution data to the corresponding digital audio interface for playback, so as to ensure data synchronization between the two digital audio interfaces and reduce latency.

[0053] In some embodiments, outputting the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces respectively includes: in response to the storage capacity of the first output buffer queue or the second output buffer queue reaching a preset waterline, transmitting the first distribution data or the second distribution data to the first digital audio interface or the second digital audio interface for output. In this way, when the storage capacity of the output buffer queue reaches the preset waterline, the distribution data is synchronously transmitted to the digital audio interface, avoiding the time delay between the two digital audio interfaces and improving the synchronization stability.

[0054] Figure 5 3 is a flow chart showing an audio recording method 300 according to an embodiment of the present invention. Figure 5 The method 300 records two audio data through two digital audio interfaces (DAI0 and DAI1), then stores the two audio data alternately through memory addresses to obtain spliced ​​and merged data, and finally transmits the spliced ​​and merged data to a speech recognition application of a multi-channel microphone array for processing. The method 300 includes the following steps 302 to 310.

[0055] In step 302, in response to an audio input instruction, audio data D0 of a first number of channels of 8CH is recorded through the digital audio interface DAI0, and audio data D1 of a second number of channels of 2CH is recorded through the digital audio interface DAI1.

[0056] Figure 6 FIG. 3 is a schematic diagram showing data transmission according to an audio recording method 300 according to an embodiment of the present invention.

[0057] In step 304, driven by the same clock, the interface controllers corresponding to the digital audio interfaces DAI0 and DAI1 are enabled simultaneously to control the digital audio interfaces DAI0 and DAI1 to simultaneously receive the audio data D0 and D1 into their respective input buffer queues FIFO for temporary storage.

[0058] In some embodiments, all digital audio interfaces use the CLK (clock signal) derived from the same PLL (Phase-locked loop) to implement the same clock configuration in hardware, thereby ensuring that different digital audio interfaces can be synchronized with the same source.

[0059] In step 306 , when the storage amount of the input buffer queue FIFO reaches a preset watermark, the digital audio interfaces DAI0 and DAI1 simultaneously trigger the DMA controller to start transmitting the audio data D0 and D1 in the input buffer queue FIFO.

[0060] In step 308, based on the DMA controller in the interleaved transmission mode, the audio data D0 in the input buffer queue FIFO corresponding to the digital audio interface DAI0 is transmitted to the first range address of the memory, and the audio data D1 in the input buffer queue FIFO corresponding to the digital audio interface DAI1 is transmitted to the second range address of the memory. The first range address and the second range address are alternately set in an interleaved manner.

[0061] In some embodiments, the Interleaved transfer mode is enabled by configuring CH0 and CH1 of the DMA controller by the CPU.

[0062] In some embodiments, after CH0 of the DMA controller transfers the audio data D0 with the first number of channels of 8CH to the first range address of the memory address DMA Buffer, it skips the second range address with the data length of 2CH, and then transfers the next audio data D0 with the first number of channels of 8CH to the first range address of the memory address DMA Buffer; similarly, after CH1 of the DMA controller transfers the audio data D1 with the second data channel of 2CH to the second range address of the memory address DMA Buffer, it skips the first range address with the data length of 8CH, and then transfers the next audio data D1 with the second data channel of 2CH to the second range address of the memory address DMA Buffer. So that the CH0 and CH1 addresses of the DMA controller are stored alternately. At this time, the number of channels of the spliced ​​and merged data D obtained in a group of joint sub-range addresses is 10CH.

[0063] In step 310, the application reads the spliced ​​and merged data D with a number of channels of 10CH in the memory address DMA Buffer. That is, the application reads 10CH of data from the DMA Buffer each time.

[0064] Figure 7 3 is a hardware block diagram showing an audio recording method 300 according to an embodiment of the present invention. Figure 7 In some embodiments, the digital audio interface DAI0 is connected to the microphone array, and the digital audio interface DAI1 is connected to the speaker recollection. Among them, the ADC (analog-to-digital converter) converts the analog MIC signal into a PCM digital signal. The microphone array is used to collect the MIC recording signal, and the speaker recollection is to collect the electrical signal of the music played by the speaker. When collecting the recording for speech recognition, it is necessary to filter out the background sound of the recording through the echo cancellation algorithm (AEC) to extract the speech signal, thereby improving the accuracy of subsequent speech recognition. When the echo cancellation algorithm (AEC) processes the audio, it is necessary to obtain the PCM data of the MIC recording signal and the audio recollection signal at the same time, and requires the MIC recording signal to be phase-synchronized with the PCM data of the recollection signal. If the audio processing method based on the prior art is used, the upper-layer application needs to interact with the OS operating system twice to obtain the MIC recording signal and PCM data. In this process, it may be affected by the system scheduling, resulting in unstable and jittery phases of the PCM data of the MIC recording signal and the recollection signal each time, affecting the AEC algorithm processing is not clean enough, thereby affecting the accuracy. The audio processing method of the present invention can avoid the influence of OS scheduling through a single read operation, and the two digital audio interfaces use the same clock source, which ensures the PCM data phase synchronization of the MIC recording signal and the recovered signal from the hardware principle.

[0065] In some embodiments, a multi-channel audio recording request is issued by a speech recognition recording application. At this time, the corresponding device is opened through PCM_OPEN and relevant parameters are set, and then a signal to start recording is sent to the OS system. At this time, the OS system triggers the DMA channel corresponding to the digital audio interface DAI0 and DAI1, and obtains the spliced ​​and merged data through the above method 300. The application then reads the spliced ​​and merged data through PCM_READ, and finally the spliced ​​and merged data is handed over to the speech recognition application algorithm for processing.

[0066] It should be noted that the MIC recording signal refers to the sound recorded by the microphone (including speaking voice, background music played by the speaker, etc.). The role of the audio recollection signal is to provide the AEC algorithm with a reference signal of the playback path. Based on the reference signal, the AEC algorithm can filter out the background sound in the recording and extract the voice signal for subsequent voice recognition. The AEC (Acoustic Echo Cancellation) algorithm is a technology used to deal with echo problems in communication systems. Echoes usually occur in voice communications. When the audio signal of the speaker is transmitted back again through the microphone, it will cause an echo heard by the other party. The AEC algorithm aims to reduce or eliminate this echo to improve the quality of communication. DMA (Direct Memory Access) is a technology used in computer architecture to improve data transmission efficiency. It allows external devices (such as hard drives, network adapters, audio cards, etc.) to directly access computer memory without the intervention of the central processing unit (CPU). This can reduce the burden on the CPU and increase the data transmission speed, especially when a large amount of data needs to be transferred between the memory and external devices. PCM (Pulse Code Modulation) data is the representation of an analog signal converted into a digital signal at a fixed sampling frequency.

[0067] Figure 8 4 is a flow chart showing an audio playback method 400 according to an embodiment of the present invention. Figure 8 The method 400 plays audio output data through two digital audio interfaces (DAI0 and DAI1). The method 400 includes the following steps 402 to 408.

[0068] In step 402, audio output data D with 10 channels is alternately stored in the memory address DMA Buffer according to the first range address and the second range address, wherein the first range address corresponds to 8 channels and the second range address corresponds to 2 channels.

[0069] Fig. 9 FIG. 4 is a schematic diagram showing data transmission according to an audio playing method 400 according to an embodiment of the present invention.

[0070] In step 404, in response to the audio output instruction, based on the DMA controller in the Interleaved transmission mode, the first distribution data D0 of the first number of channels 8CH is read from the first range address, and the second distribution data D1 of the second number of channels 2CH is read from the second range address.

[0071] In some embodiments, CH0 of the DMA controller reads the distribution data D0 of the first number of channels of 8CH from the first range address, skips the second range address of the second number of channels of 2CH, and then reads the distribution data D0 of the first number of channels of 8CH from the next first range address. Similarly, CH1 of the DMA controller reads the distribution data D1 of the second number of channels of 2CH from the second range address, skips the first range address of the first number of channels of 8CH, and then reads the distribution data D1 of the second number of channels of 2CH from the next second range address. So that the CH0 and CH1 addresses of the DMA controller are read alternately. At this time, the number of channels of the obtained distribution data D0 is 8CH, and the number of channels of the obtained distribution data D1 is 2CH.

[0072] In step 406, based on the DMA controller in the Interleaved transmission mode, the distribution data D0 is transmitted to the output buffer queue FIFO of the digital audio interface DAI0, and the distribution data D1 is transmitted to the output buffer queue FIFO of the digital audio interface DAI1. Since DAI0 and DAI1 are driven based on the same clock source, the output buffer queue FIFO receives data synchronously.

[0073] In some embodiments, when configuring the transmission mode of the DMA controller, the source address and destination address of the data carried by the DMA controller are synchronously configured. For the digital audio interface DAI0, the source address configured by its DMACH0 is the first range address in the DMA Buffer, and the destination address is the output buffer queue FIFO in the digital audio interface DAI0, so that the digital audio interface DAI0 transfers the 8CH data in the first range address to the output buffer queue FIFO of the digital audio interface DAI0 through the CH0 of the DMA controller, and finally sends the data of the output buffer queue FIFO out through the hardware interface of the digital audio interface DAI0.

[0074] In step 408, when the storage capacity of the output buffer queue FIFO reaches the preset waterline, the DMA controller is triggered to start transmitting the distribution data D0 and D1 in its output buffer queue FIFO to the digital audio interfaces DAI0 and DAI1 for playback.

[0075] Fig.10 4 is a hardware block diagram showing an audio playback method 400 according to an embodiment of the present invention. Fig.10 In some embodiments, the digital audio interfaces DAI0 and DAI1 are respectively connected to the DAC array and then to the speaker array. The DAC (digital-to-analog converter) is used to convert PCM digital audio signals into analog signals to drive the speaker array to produce sound.

[0076] In some embodiments, multi-channel playback refers to multi-channel playback of PCM data. For example, Dolby's 7.1 channel (8-channel audio playback) or 7.1.4 (12-channel audio playback). When an application needs to play a 7.1.4-channel audio source file, if the audio playback method based on the prior art, it is usually necessary to decode the 7.1.4-channel audio source into 7.1.4-channel PCM data, and then transmit the Buffer of the 7.1.4-channel PCM data in the user space to the OS system through two write operations. After the OS system receives the start playback command, it simultaneously triggers the data transfer operation of the DMA corresponding to the two digital audio interfaces DAI. In this process, it may be affected by the system scheduling, resulting in asynchronous data transfer, thereby causing asynchronous audio playback. The audio processing method of the present invention can avoid the influence of OS scheduling through a single write operation, and the two digital audio interfaces use the same clock source, which ensures that the two audio interfaces play audio synchronously from the hardware principle.

[0077] According to another aspect of the present invention, Fig.11 2 is a schematic diagram showing the structure of an apparatus 500 for processing audio data according to an embodiment of the present invention. Fig.11 , the device 500 for processing audio data includes an interface module 502, a DMA module 504 and an application layer module 506. The interface module 502 is configured to receive at least two audio data respectively through at least two digital audio interfaces in response to an audio input instruction, and store the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces. The DMA module 504 is configured to transfer the at least two audio data in the at least two input buffer queues to the addresses corresponding to the at least two input buffer queues in the memory, so that the at least two audio data corresponding to the at least two digital audio interfaces are alternately stored in the memory. The application layer module 506 is configured to read the at least two audio data alternately stored from the memory as the spliced ​​and merged data from the at least two digital audio interfaces.

[0078] In some embodiments, the application layer module 506 is further configured to obtain audio output data in response to an audio output instruction, wherein the audio output data includes at least two distribution data, and the at least two distribution data correspond to at least two digital audio interfaces respectively. The DMA module 504 is further configured to store the at least two distribution data of the audio output data in the memory at addresses corresponding to the at least two digital audio interfaces, respectively, so that the at least two distribution data are alternately stored in the memory, and is configured to transmit the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively. The interface module 502 is further configured to output the at least two distribution data in the at least two output buffer queues respectively through the at least two digital audio interfaces, so as to play audio based on the at least two distribution data.

[0079] It should be understood that the interface module 502, the DMA module 504 and the application layer module 506 may be further configured to execute the corresponding steps or actions in the above-described method, which will not be described in detail herein.

[0080] According to yet another aspect of the present invention, Fig.12 is a schematic diagram showing an electronic device 600 according to an embodiment of the present invention. Fig.12 The electronic device 600 includes a memory 602, a processor 604, and a computer program stored in the memory 602 and executable on the processor 604. When the processor 604 executes the computer program, the various steps of the method for processing audio data as described above are implemented.

[0081] In summary, the method and device for processing audio data, and the electronic device provided by the present invention, by designing a hardware DMA controller, the audio data of each channel of multiple digital audio interfaces (DAI) are spliced ​​or distributed, without the need to transplant related software, thereby improving data processing efficiency and reducing CPU load. During the data processing process, the upper-layer application of the present invention only needs to splice or distribute the audio data input or output by the digital audio interface (DAI) into audio data of multiple channels during the DMA operation inside the SoC through a read operation or a write operation, and finally transmit it to the corresponding audio processing module or the audio playback module. This processing method can significantly reduce the CPU load of the entire system, reduce latency, improve synchronization stability, and make the system easier to expand and integrate into various application scenarios.

[0082] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for processing audio data, characterized in that: include: In response to the audio input instruction, receiving at least two audio data respectively through at least two digital audio interfaces, and storing the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces; The at least two audio data in the at least two input buffer queues are respectively transmitted to addresses corresponding to the at least two input buffer queues in a memory, so that the at least two audio data corresponding to the at least two digital audio interfaces are respectively and alternately stored in the memory; as well as The at least two audio data alternately stored are read from the memory as spliced ​​combined data from the at least two digital audio interfaces.

2. The method according to claim 1, characterized in that Receiving at least two audio data respectively through at least two digital audio interfaces includes: receiving first audio data through a first digital audio interface and receiving second audio data through a second digital audio interface at the same time under the same clock drive, Storing the at least two audio data respectively in at least two input cache queues corresponding to the at least two digital audio interfaces includes: under the same clock drive, storing the first audio data in a first input cache queue through a first interface controller corresponding to the first digital audio interface, and storing the second audio data in a second input cache queue through a second interface controller corresponding to the second digital audio interface.

3. The method according to claim 2, characterized in that Receiving the first audio data through the first digital audio interface includes: receiving the first audio data including a plurality of groups of first audio sub-data through the first digital audio interface, each group of the first audio sub-data corresponding to a first number of channels, Receiving the second audio data through the second digital audio interface includes: receiving the second audio data including a plurality of groups of second audio sub-data through the second digital audio interface, each group of the second audio sub-data corresponding to a second number of channels.

4. The method according to claim 2, characterized in that: The at least two audio data in the at least two input buffer queues are transmitted to addresses corresponding to the at least two input buffer queues in the memory respectively, comprising: Under the same clock drive, the first audio data in the first input buffer queue is stored at a first range address in the memory through a first storage controller corresponding to the first digital audio interface, and the second audio data in the second input buffer queue is stored at a second range address in the memory through a second storage controller corresponding to the second digital audio interface. The first range address and the second range address are alternately arranged in an interleaved manner.

5. The method according to claim 3, characterized in that: The at least two audio data in the at least two input buffer queues are transmitted to addresses corresponding to the at least two input buffer queues in the memory respectively, comprising: storing a plurality of groups of the first audio sub-data at a plurality of groups of first sub-range addresses in the memory, respectively, and storing a plurality of groups of the second audio sub-data at a plurality of groups of second sub-range addresses in the memory, respectively, so that each group of the first audio sub-data and each group of the second audio sub-data are alternately stored in the memory in an interleaved manner, Each group of the first sub-range addresses and each group of the second sub-range addresses are alternately arranged in an interleaved manner, so that a group of second sub-range addresses is immediately followed by a group of first sub-range addresses, the next group of first sub-range addresses is immediately followed by the group of second sub-range addresses, and the next group of second sub-range addresses is immediately followed by the next group of first sub-range addresses.

6. The method according to claim 4, characterized in that Reading the at least two audio data alternately stored in the memory as the spliced ​​combined data from the at least two digital audio interfaces comprises: The first audio data and the second audio data alternately stored are read from the memory, and a combination of the read first audio data and the second audio data is used as the spliced ​​merged data.

7. The method according to claim 5, characterized in that Reading the at least two audio data alternately stored in the memory as the spliced ​​combined data from the at least two digital audio interfaces comprises: Each group of the first audio sub-data and each group of the second audio sub-data are read from the memory, and the combination of each group of the first audio sub-data and each group of the second audio sub-data read is used as a group of spliced ​​merged sub-data, and the spliced ​​merged data includes multiple groups of the spliced ​​merged sub-data.

8. The method according to claim 4, characterized in that The at least two audio data in the at least two input buffer queues are transmitted to addresses corresponding to the at least two input buffer queues in the memory respectively, comprising: In response to the storage amount of the first input buffer queue or the second input buffer queue reaching a preset watermark, the first audio data or the second audio data is transferred to the first range address or the second range address.

9. The method according to claim 7, characterized in that: Reading the at least two audio data alternately stored in the memory as the spliced ​​combined data from the at least two digital audio interfaces comprises: In response to each group of the first audio sub-data and each group of the second audio sub-data having been stored in a set of joint sub-range addresses in the memory, a group of first audio sub-data and a group of second audio sub-data are read from the set of joint sub-range addresses in the memory as a set of spliced ​​merged sub-data.

10. The method according to claim 1, characterized in that Also includes: The spliced ​​and merged data is processed using an echo cancellation algorithm to remove background sound, and the audio data includes microphone recording data and audio echo data.

11. The method according to claim 1, characterized in that: Also includes: In response to the audio output instruction, at least two distribution data included in the audio output data are respectively transmitted to addresses corresponding to at least two digital audio interfaces in the memory, so that the at least two distribution data are respectively stored alternately in the memory, and the at least two distribution data correspond to the at least two digital audio interfaces respectively; Transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively; as well as The at least two distribution data in the at least two output buffer queues are outputted respectively through the at least two digital audio interfaces, so as to perform audio playback based on the at least two distribution data.

12. A method for processing audio data, characterized in that: include: In response to the audio output instruction, obtaining audio output data, the audio output data including at least two distribution data, the at least two distribution data respectively corresponding to at least two digital audio interfaces; storing the at least two distribution data of the audio output data in addresses corresponding to the at least two digital audio interfaces in a memory, respectively, so that the at least two distribution data are alternately stored in the memory; Transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively; as well as The at least two distribution data in the at least two output buffer queues are outputted respectively through the at least two digital audio interfaces, so as to perform audio playback based on the at least two distribution data.

13. The method according to claim 12, characterized in that Storing the at least two distribution data of the audio output data respectively in the memory at addresses corresponding to the at least two digital audio interfaces comprises: storing first distribution data of the audio output data in a first range address corresponding to the first digital audio interface in the memory, and storing second distribution data of the audio output data in a second range address corresponding to the second digital audio interface in the memory, The first range address and the second range address are alternately arranged in an interleaved manner.

14. The method according to claim 13, characterized in that Storing the first distribution data of the audio output data at a first range address corresponding to the first digital audio interface in the memory comprises: storing a plurality of groups of first distribution sub-data of the first distribution data at a plurality of groups of first sub-range addresses in the memory respectively, each group of the first distribution sub-data corresponding to a first number of channels, Storing the second distribution data of the audio output data at a second range address corresponding to the second digital audio interface in the memory includes: storing a plurality of groups of second distribution sub-data of the second distribution data at a plurality of groups of second sub-range addresses in the memory, each group of the second distribution sub-data corresponding to a second number of channels, Each group of the first sub-range addresses and each group of the second sub-range addresses are alternately arranged in an interleaved manner, so that a group of second sub-range addresses is immediately followed by a group of first sub-range addresses, the next group of first sub-range addresses is immediately followed by the group of second sub-range addresses, and the next group of second sub-range addresses is immediately followed by the next group of first sub-range addresses.

15. The method according to claim 13, characterized in that Transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively comprises: Driven by the same clock, the first distribution data stored at the first range address in the memory is stored in the first output cache queue corresponding to the first digital audio interface through the first storage controller corresponding to the first digital audio interface, and at the same time, the second distribution data stored at the second range address in the memory is stored in the second output cache queue corresponding to the second digital audio interface through the second storage controller corresponding to the second digital audio interface.

16. The method according to claim 15, characterized in that Outputting the at least two distribution data in the at least two output buffer queues respectively through the at least two digital audio interfaces comprises: Driven by the same clock, the first distribution data in the first output cache queue is output to the outside via the first digital audio interface through the first interface controller corresponding to the first digital audio interface, and at the same time, the second distribution data in the second output cache queue is output to the outside via the second digital audio interface through the second interface controller corresponding to the second digital audio interface, so that the first audio playback device corresponding to the first digital audio interface and the second audio playback device corresponding to the second digital audio interface respectively play audio for the audio output data based on the first distribution data and the second distribution data.

17. The method according to claim 14, characterized in that Transmitting the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces respectively comprises: In response to each group of the first distribution sub-data and each group of the second distribution sub-data having been stored in a set of joint sub-range addresses of the memory, a group of first distribution sub-data and a group of second distribution sub-data stored in the set of joint sub-range addresses of the memory are respectively transmitted to a first output buffer queue corresponding to the first digital audio interface and a second output buffer queue corresponding to the second digital audio interface.

18. The method according to claim 15, characterized in that Outputting the at least two distribution data in the at least two output buffer queues respectively through the at least two digital audio interfaces comprises: In response to the storage capacity of the first output buffer queue or the second output buffer queue reaching a preset watermark, the first distribution data or the second distribution data is transmitted to the first digital audio interface or the second digital audio interface for output.

19. A device for processing audio data, characterized in that: include: The interface module is configured to receive at least two audio data respectively through at least two digital audio interfaces in response to the audio input instruction, and store the at least two audio data respectively in at least two input buffer queues corresponding to the at least two digital audio interfaces; A DMA module is configured to transfer the at least two audio data in the at least two input buffer queues to addresses corresponding to the at least two input buffer queues in a memory, so that the at least two audio data corresponding to the at least two digital audio interfaces are respectively stored alternately in the memory; as well as The application layer module is configured to read the at least two audio data stored alternately from the memory as the spliced ​​and merged data from the at least two digital audio interfaces.

20. A device for processing audio data, characterized in that: include: The application layer module is configured to obtain audio output data in response to the audio output instruction, wherein the audio output data includes at least two distribution data, and the at least two distribution data correspond to at least two digital audio interfaces respectively; A DMA module, configured to store the at least two distribution data of the audio output data at addresses corresponding to the at least two digital audio interfaces in a memory, respectively, so that the at least two distribution data are alternately stored in the memory, respectively, and configured to transmit the at least two distribution data alternately stored in the memory to at least two output buffer queues corresponding to the at least two digital audio interfaces, respectively; as well as The interface module is configured to respectively output the at least two distribution data in the at least two output buffer queues through the at least two digital audio interfaces, so as to perform audio playback based on the at least two distribution data.

21. An electronic device, characterized in that: include: a memory configured to store an executable program; as well as A processor configured to execute the program to perform the method according to any one of claims 1 to 11 or the method according to any one of claims 12 to 18.

Citation Information

Patent Citations

  • Caching device, caching method, computing device and computing method

    CN111008040A

  • Sound recording file generation method and device, storage medium and electronic equipment

    CN118101831A

  • Single-chip data processor

    JP2007087416A

  • Voice recognition device, observation probability calculating device, complex fast fourier transform calculation device and method, cache device, and method of controlling the cache device

    US20040002862A1

  • Audio Processor

    US20090055005A1