Audio processing method, system chip and audio processing system
By aligning audio retrieval data and recording data using DMA retrieval and buffer technology, and combining this with an acoustic echo cancellation algorithm, the high cost and resource waste of hardware ADC retrieval methods are solved, achieving low-cost, high-quality audio processing and speech recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUZHOU ROCKCHIP SEMICON
- Filing Date
- 2024-07-18
- Publication Date
- 2026-06-02
AI Technical Summary
Existing hardware ADC retrieval methods suffer from high hardware costs and resource waste, limiting system scalability and application scenarios. In particular, in systems where the SOC has a built-in audio codec, the I2S pin connection method adds an external ADC, resulting in resource waste.
By using DMA retrieval, audio retrieval data and recording data are aligned using buffers and hardware pointer differences to achieve audio retrieval. This is combined with an acoustic echo cancellation algorithm to filter out background noise and extract the speech signal for speech recognition.
It reduces hardware and debugging costs, improves system scalability and voice communication quality, supports I2SSDI-less SoC system applications, and enhances the performance of voice assistants and automatic speech recognition systems.
Smart Images
Figure CN119049491B_ABST
Abstract
Description
Technical Field
[0001] This disclosure belongs to the field of audio processing technology, and particularly relates to audio processing methods, system chips, and audio processing systems. Background Technology
[0002] Currently, in various applications such as voice assistants, automatic speech recognition (ASR) systems, and voice control devices, hardware ADCs (Analog-to-Digital Converters) or other hardware sampling methods are typically relied upon. These methods have high hardware costs, such as occupying the I2S (Inter-IC Sound) or SDI (Serial Digital Interface) of a SOC (System on Chip) chip, or they limit the scalability and wide application of the system.
[0003] In detail, existing hardware acquisition solutions have several limitations and shortcomings: Acquisition via a hardware ADC is costly, and the added hardware requirements increase the difficulty of system debugging. Adding a pin to the SOC for acquisition is only feasible if the I2S controller has multiple I2SSDI input pins; if there is only one I2SSDI, this solution is not feasible. For systems with a built-in audio codec on the SOC, it may be necessary to connect the I2S pin to external circuitry in conjunction with an ADC for acquisition. This approach adds an external ADC, rendering the SOC's internal ADC unusable, resulting in wasted resources. Summary of the Invention
[0004] This disclosure provides an audio processing method, a system chip, and an audio processing system that can achieve audio re-sampling without additional hardware circuitry.
[0005] In a first aspect, embodiments of this disclosure provide an audio processing method. The audio processing method includes: acquiring audio playback data and writing the audio playback data into a first buffer; caching the audio playback data in the first buffer into a second buffer as audio re-sampling data; outputting the audio playback data in the first buffer to an external source for playback; acquiring audio recording data collected during playback and writing the audio recording data into a third buffer; and generating audio processing data based on the audio re-sampling data in the second buffer and the audio recording data in the third buffer.
[0006] In one implementation of the first aspect, generating audio processing data based on the audio retrieval data in the second buffer and the audio recording data in the third buffer includes: aligning the audio retrieval data and the audio recording data according to a hardware pointer difference and recombining the data to obtain the audio processing data.
[0007] In one implementation of the first aspect, aligning the audio re-acquisition data and the audio recording data according to the hardware pointer difference and recombining the data to obtain the audio processing data includes: determining a playback pointer and a recording pointer based on the DMA pointer and the FIFO pointer of the audio interface; determining the hardware pointer difference based on the playback pointer and the recording pointer; determining the pointer corresponding to the audio re-acquisition data based on the hardware pointer difference; retrieving the audio re-acquisition data from the second buffer according to the pointer corresponding to the audio re-acquisition data; and merging the retrieved audio re-acquisition data and the audio recording data frame by frame according to the preset audio data channel requirements.
[0008] In one implementation of the first aspect, determining the playback pointer and the recording pointer based on the DMA pointer and the audio interface FIFO pointer includes: subtracting the audio interface transmit FIFO pointer from the playback DMA pointer to obtain the playback pointer; and adding the audio interface receive FIFO pointer to the recording DMA pointer to obtain the recording pointer.
[0009] In one implementation of the first aspect, the audio processing method further includes: performing acoustic echo cancellation processing on the echo based on the audio processing data to generate audio data.
[0010] In one implementation of the first aspect, the audio processing method further includes: filtering out background noise based on the result of the acoustic echo cancellation processing to extract a speech signal; and using the extracted speech signal for speech recognition applications.
[0011] Secondly, embodiments of this disclosure provide a system-on-a-chip (SoC). The SoC includes: a first DMA module configured to receive audio playback data and write the audio playback data into a first buffer, and output the audio playback data in the first buffer to an external source for playback; a digital backsampling module configured to cache the audio playback data in the first buffer into a second buffer as audio backsampling data; and a second DMA module configured to receive audio recording data acquired during playback and write the audio recording data into a third buffer, wherein the digital backsampling module is further configured to generate audio processing data based on the audio backsampling data in the second buffer and the audio recording data in the third buffer.
[0012] In one implementation of the second aspect, the system chip further includes a signal processing module configured to perform acoustic echo cancellation processing on the echo based on the audio processing data to generate audio data.
[0013] In one implementation of the second aspect, the system chip further includes: an interface module configured to receive the audio playback data from the first DMA module and output the audio playback data to an audio codec outside the system chip, and to receive the audio recording data from the audio codec and output the audio recording data to the second DMA module.
[0014] In one implementation of the second aspect, the digital acquisition module is implemented by a central processing unit included in the system chip.
[0015] Thirdly, embodiments of this disclosure provide an audio processing system. The audio processing system includes: the system chip; an audio codec electrically coupled to the system chip and configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and to perform analog-to-digital conversion and encoding on the recording signal to generate the audio recording data; a speaker electrically coupled to the audio codec and configured to perform the playback based on the playback signal; and a microphone electrically coupled to the audio codec and configured to capture audio during playback to generate the recording signal.
[0016] According to the audio processing method, system chip, and audio processing system described in this disclosure, audio retrieval is achieved using DMA (Digital Messaging) without the need for additional hardware circuitry, significantly reducing the overall hardware and debugging costs. The system described in this disclosure is also easier to expand and integrate into various application scenarios; for example, it does not require the use of an I2SSDI pin and can be applied to SoC systems without I2SSDI.
[0017] This disclosure synchronously merges the retrieval data from the DMA path with the audio recording data, and uses the merged audio data for audio processing, thereby enabling its application in various fields such as voice assistants, automatic speech recognition (ASR) systems, and voice control devices with lower hardware costs and higher voice communication quality.
[0018] The re-acquisition data provided in this disclosure can provide a reference signal for the playback path of acoustic echo cancellation. Based on this reference signal, the acoustic echo cancellation algorithm can filter out the background noise in the recording and extract the speech signal for subsequent speech recognition.
[0019] The signal provided in this disclosure for the audio algorithm (acoustic echo cancellation algorithm) is audio data that includes the back-sampled signal and the MIC recording signal. Thanks to accurate reference frame calculation, the back-sampled frame and the actual hardware playback position are highly aligned, which can improve the output effect of the acoustic echo cancellation algorithm. Furthermore, due to the high alignment, the reference frame alignment logic of the preprocessing of acoustic echo cancellation can be omitted, thereby saving algorithm overhead. Attached Figure Description
[0020] Figure 1 The diagram shows an application scenario of the audio processing method described in this embodiment.
[0021] Figure 2 The diagram shown is a flowchart illustrating the principle of the audio processing method described in this embodiment.
[0022] Figure 3 The diagram shown is a data processing flowchart of the audio processing method described in the embodiments of this disclosure.
[0023] Figure 4 The diagram shown is a flowchart of the pointer determination process for the audio processing method described in this embodiment.
[0024] Figure 5 The diagram shown is an audio processing schematic of the audio processing method described in the embodiments of this disclosure.
[0025] Figure 6 The diagram shown is a flowchart of the echo cancellation method of the audio processing method described in the embodiments of this disclosure.
[0026] Figure 7 The diagram shown is a speech recognition flowchart of the audio processing method described in this embodiment.
[0027] Figure 8 The diagram shown is a schematic diagram of the system chip described in an embodiment of this disclosure.
[0028] Figure 9 The diagram shown is a structural schematic of the digital data acquisition module described in an embodiment of this disclosure.
[0029] Figure 10 This is another schematic diagram of the system chip described in an embodiment of this disclosure.
[0030] Figure 11 This is another schematic diagram of the system chip described in an embodiment of this disclosure.
[0031] Figure 12 The diagram shown is a structural schematic of the audio processing system described in an embodiment of this disclosure.
[0032] Figure 13 The diagram shown illustrates the working principle of the audio processing system described in this embodiment. Detailed Implementation
[0033] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0034] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0035] This disclosure provides a hardware application scenario for an audio processing method, which will be described below using this application scenario as an example. Figure 1 The illustration shows an application scenario of the audio processing method described in this disclosure. For example... Figure 1 As shown, before the audio playback data is transmitted to the speaker or loudspeaker for playback, the audio processing method described in this disclosure is executed to obtain audio re-acquisition data. Then, the audio re-acquisition data and the audio recording data collected by the microphone are reconstructed to obtain audio processing data. The audio processing data that has completed audio re-acquisition and processing can provide a reference signal for the playback path of the AEC (Acoustic Echo Cancellation) algorithm. Based on this reference signal, the AEC algorithm can filter out the background noise in the recording and extract the speech signal for subsequent speech recognition.
[0036] The technical solutions in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0037] Figure 2 The diagram shown is a flowchart illustrating the principle of the audio processing method according to an embodiment of this disclosure. This embodiment provides an audio processing method, specifically including the following steps S21 to S25.
[0038] In S21, audio playback data is acquired and written into the first buffer.
[0039] Specifically, the CPU (Central Processing Unit) continuously writes the audio playback data to be played into the DMA buffer, i.e., the first buffer. The DMA controller continuously transmits the audio playback data from the DMA buffer to the TX FIFO of the audio interface. The audio interface transmits the data from the TX FIFO to the CODEC (audio codec, which refers to a device or program that can transform a signal or a data stream) for playback according to the protocol clock signal (such as I2S).
[0040] In S22, the audio playback data in the first buffer is cached in the second buffer as audio loopback data.
[0041] Specifically, the data written by the CPU to the DMA buffer is simultaneously cached by the DLP module to achieve audio DMA digital loopback, where DLP module refers to the digital loopback module.
[0042] In S23, the audio playback data in the first buffer is output to the outside for playback.
[0043] In S24, the audio recording data collected during the playback is acquired, and the audio recording data is written into the third buffer.
[0044] Audio recording data refers to MIC (Microphone) recording signals: sound recorded by the microphone (including speaking voice, background music played by speakers, etc.).
[0045] In S25, audio processing data is generated based on the audio retrieval data in the second buffer and the audio recording data in the third buffer.
[0046] In some embodiments, generating audio processing data based on the audio retrieval data in the second buffer and the audio recording data in the third buffer may include: aligning the audio retrieval data and the audio recording data according to a hardware pointer difference and reconstructing the data to obtain the audio processing data.
[0047] Figure 3 This diagram illustrates a data processing flowchart of the audio processing method described in an embodiment of this disclosure. Figure 3 As shown, in some embodiments, aligning the audio retrieval data and the audio recording data according to the hardware pointer difference and recombining the data to obtain the audio processing data specifically includes the following steps S251 to S255.
[0048] In S251, the playback pointer and recording pointer are determined based on the DMA pointer and the FIFO pointer of the audio interface.
[0049] Figure 4 This diagram illustrates the pointer determination flowchart of the audio processing method described in an embodiment of this disclosure. Figure 4 As shown, step S251 specifically includes the following steps S251A to S251B.
[0050] S251A, the playback pointer is obtained by subtracting the audio interface sending FIFO pointer from the playback DMA pointer.
[0051] S251B adds the recording DMA pointer to the audio interface receive FIFO pointer, and uses it as the recording pointer.
[0052] The DMA pointer indicates the location of the audio frame in the hardware DMA. The FIFO pointer of the hardware audio interface indicates the location of the FIFO audio frame in the hardware audio interface (such as I2S). FIFO stands for First In First Out (FIFO buffer) and is located in the audio interface module.
[0053] Specifically, the hardware pointer difference is determined as follows:
[0054] P: Playback pointer = Playback DMA POS - TX FIFO POS;
[0055] C: Recording pointer = Recording DMA POS + RX FIFO POS.
[0056] Next, in S252, the hardware pointer difference is determined based on the playback pointer and the recording pointer.
[0057] Specifically, the hardware pointer difference Delta = P – C represents the difference between playback and recording.
[0058] In S253, the pointer corresponding to the audio re-acquisition data is determined based on the hardware pointer difference.
[0059] Specifically, when playback and recording occur simultaneously, the reference start frame position POS (CUR C + Delta = CUR P, i.e., recording pointer + Delta = playback pointer) corresponding to the actual re-sampling can be quickly calculated based on the hardware pointer difference Delta. Based on this pointer, the re-sampling signal can be retrieved from the mirror buffer. The mirror buffer contains the buffer for playing audio data and is stored in DDR (Double Data Rate) memory, serving as the source of the re-sampling signal in the DLP module.
[0060] In S254, the audio re-acquisition data is retrieved from the second buffer according to the pointer corresponding to the audio re-acquisition data.
[0061] In S255, the extracted audio retrieval data and the audio recording data are merged frame by frame according to the preset audio data channel requirements.
[0062] Specifically, the pre-set audio data channel requirements can be 6 channels of MIC + 2 channels of retransmission, or other audio data arrangement requirements set by the user.
[0063] Figure 5 The diagram shown is an audio processing schematic of the audio processing method described in an embodiment of this disclosure. Figure 5 As shown, during audio playback, the application writes audio playback data to the Playback DMA Buffer. The DLP module buffers the audio playback data from the Playback DMA Buffer into the Playback Mirror Buffer. The DMA module then transfers the audio playback data from the Playback DMA Buffer to the FIFO of the audio interface. The audio interface (e.g., I2S) then transfers the audio playback data from the FIFO to the CODEC for playback. During recording, the audio interface (e.g., I2S) receives recording data from the CODEC into the FIFO. The DMA module transfers the recording data from the FIFO of the audio interface to the Capture DMA Buffer. The DLP module reconstructs the data using hardware pointer differences between the recording data in the Capture DMA Buffer and the audio playback data in the Playback Mirror Buffer. The application then reads the reconstructed audio data from the Capture DMA Buffer.
[0064] like Figure 5 As shown, the DLP module in this disclosure implements a mirror buffer for audio playback data. The mirror buffer data contains the data from the playback buffer. Combined with the DMA pointer and the FIFO pointer of the audio hardware interface, the position of the retrieval reference start frame can be accurately calculated. Then, the CPU reassembles the retrieval signal and the MIC recording signal to form audio data containing both the retrieval and MIC recording signals, which is then provided to the AEC algorithm to remove background noise from the MIC recording signal. This disclosed solution not only saves hardware costs but also provides backward compatibility with chips with weak specifications or low costs, such as those with only one audio input pin.
[0065] The operation of the DLP module is implemented by the CPU, and the specific operation is as follows for different buffers.
[0066] Playback DMA Buffer (Audio Playback DMA Buffer, also known as the First Buffer): The CPU continuously writes the audio playback data to be played into this buffer. The DMA controller continuously transfers the audio playback data from this buffer to the TX FIFO of the audio interface. The audio interface then transfers the data from the TX FIFO to the CODEC (audio codec) for playback according to the protocol clock signal (such as I2S).
[0067] Capture DMA Buffer (Audio Recording DMA Buffer, also known as the third buffer): The audio interface acquires and writes the signal from the CODEC into the RX FIFO of the audio interface according to the protocol clock signal (such as I2S). The DMA controller continuously moves the data in the RX FIFO to the Capture DMA Buffer. The CPU continuously retrieves the audio data in the Capture DMA Buffer and sends it to the application or algorithm for processing.
[0068] Playback Mirror Buffer (also known as the second buffer): Data written by the CPU to the playback DMA buffer is simultaneously cached by the DLP module. This buffer size is larger than the playback DMA buffer (e.g., 1.5 times) to reserve margin and resolve data overlap issues. This mirror buffer is the data source for subsequent processing of re-acquired data by the DLP module.
[0069] DMA POS refers to the DMA hardware pointer, indicating the current position of the buffer pointer for hardware playback or recording; TX / RXFIFO refers to the FIFO sent or received by the audio interface controller (such as I2S), and the FIFO POS can be obtained from the hardware.
[0070] The hardware pointer difference is determined as follows:
[0071] P: Playback pointer = Playback DMA POS - TX FIFO POS;
[0072] C: Recording pointer = Recording DMA POS + RX FIFO POS;
[0073] The hardware pointer difference Delta = P – C represents the difference between playback and recording.
[0074] When playback and recording occur simultaneously, the reference start frame position POS (CUR C + Delta = CUR P, i.e., recording pointer + Delta = playback pointer) corresponding to the actual re-sampling can be quickly calculated based on the hardware pointer difference Delta. Based on this pointer, the re-sampling signal can be retrieved from the mirror buffer.
[0075] Audio data reconstruction: Based on the user's audio data arrangement requirements (e.g., 6-channel MIC + 2-channel re-sampling), the DLP module automatically reprocesses and merges the re-sampling signal and MIC signal frame by frame to form the data arrangement order required by the user. The whole process is transparent to the user-mode application and is compatible with existing audio algorithm frameworks.
[0076] In practical applications, the DLP module also supports an unlimited number of re-sampling channels, enabling complete re-sampling of all playback channels, as well as on-demand re-sampling and mixing of all playback channels.
[0077] In one embodiment, after obtaining the audio data, the method further includes: performing acoustic echo cancellation processing on the echo generated in the voice communication based on the audio data.
[0078] Figure 6 The diagram shown is a flowchart of the echo cancellation process of the audio processing method described in an embodiment of this disclosure. Figure 6 As shown, in addition to steps S21 to S25, the audio processing method also includes step S26, which performs acoustic echo cancellation processing on the echo based on the audio processing data to generate audio data.
[0079] Therefore, this embodiment provides an audio processing method, specifically including the following steps: Acquiring audio playback data to be written to a DMA buffer; the audio playback data is written to the DMA buffer and then sequentially transmitted to the audio interface and audio codec for playback. The acquired audio playback data is cached in a mirror buffer. The audio playback data in the mirror buffer is used as back-sampled data. Input audio recording data is acquired. The back-sampled data and the audio recording data are reconstructed using hardware pointer differences to obtain audio data. Furthermore, acoustic echo cancellation processing is performed on the echo based on the audio processing data to generate audio data.
[0080] In one embodiment, after performing acoustic echo cancellation processing on the echo generated in the voice communication based on the audio data, the method further includes: filtering out the background noise based on the result of the acoustic echo cancellation processing to extract the speech signal; and using the extracted speech signal for speech recognition applications.
[0081] The AEC algorithm is a technique used to handle echo problems in communication systems. Echoes typically occur in voice communication when the audio signal from a speaker is transmitted back through a microphone, causing the other party to hear an echo. The AEC algorithm aims to reduce or eliminate this echo to improve communication quality.
[0082] The signal provided to the audio algorithm in this embodiment is audio data that includes the reconstructed resampled signal and the MIC recording signal. The feature is that, thanks to accurate reference frame calculation, the resampled frame and the actual hardware playback position are highly aligned, which can improve the output effect of the AEC algorithm. Furthermore, due to the high alignment, the reference frame alignment logic of the AEC preprocessing can be omitted, saving algorithm overhead.
[0083] Figure 7 The diagram shown is a speech recognition flowchart of the audio processing method described in an embodiment of this disclosure. Figure 7 As shown, in addition to steps S21 to S26, the audio processing method also includes step S27, which filters out the background noise based on the result of the acoustic echo cancellation processing and extracts the speech signal; and uses the extracted speech signal for speech recognition applications.
[0084] Therefore, this embodiment provides an audio processing method, specifically including the following steps: Acquiring audio playback data to be written to a DMA buffer; the audio playback data is written to the DMA buffer and then sequentially transmitted to the audio interface and audio codec for playback. The acquired audio playback data is cached in a mirror buffer. The audio playback data in the mirror buffer is used as re-sampled data. Input audio recording data is acquired. The re-sampled data and the audio recording data are reconstructed using hardware pointer differences to obtain audio data. Acoustic echo cancellation processing is performed on the echo based on the audio processing data to generate audio data. Furthermore, background noise is filtered out based on the result of the acoustic echo cancellation processing to extract the speech signal; and the extracted speech signal is used for speech recognition applications.
[0085] Figure 8 The diagram shown is a schematic diagram of the system chip described in an embodiment of this disclosure. Figure 8 As shown, this embodiment of the present disclosure provides a system chip, which includes: a digital retrieval module 1, a first DMA module 21, and a second DMA module 22.
[0086] The first DMA module 21 is configured to receive audio playback data and write the audio playback data into a first buffer, and to output the audio playback data in the first buffer to an external source for playback.
[0087] The digital recapture module 1 is configured to cache the audio playback data in the first buffer into a second buffer as audio recapture data.
[0088] The second DMA module 22 is configured to receive audio recording data acquired during playback and write the audio recording data into a third buffer.
[0089] Furthermore, the digital acquisition module 1 is further configured to generate audio processing data based on the audio acquisition data in the second buffer and the audio recording data in the third buffer.
[0090] Figure 9 The diagram shown is a structural schematic of the digital data acquisition module according to an embodiment of this disclosure. Figure 9 As shown, this embodiment of the present disclosure provides a digital acquisition module 1, which specifically includes an acquisition unit 11, a recording data acquisition unit 12, and an audio data reconstruction unit 13.
[0091] The retrieval unit 11 is configured to acquire audio playback data to be written to the DMA buffer. The audio playback data is written to the DMA buffer and then sequentially transmitted to the audio interface and audio codec for playback. The retrieval unit 11 is configured to cache the acquired audio playback data in a mirror buffer and use the audio playback data in the mirror buffer as retrieval data.
[0092] The audio recording data acquisition unit 12 is configured to acquire input audio recording data.
[0093] The audio data reconstruction unit 13 is configured to reconstruct the acquired data and the audio recording data through hardware pointer difference to obtain audio data.
[0094] Figure 10 This diagram illustrates another structural schematic of the system chip described in an embodiment of this disclosure. For example... Figure 10 As shown in this embodiment, in addition to the digital backsampling module 1, the first DMA module 21, and the second DMA module 22, the system chip may also include a signal processing module 3. The signal processing module 3 is configured to perform acoustic echo cancellation processing on the echo based on the audio processing data to generate audio data.
[0095] Figure 11 This is yet another structural schematic diagram of the system chip described in an embodiment of this disclosure. For example... Figure 11As shown, in addition to the digital acquisition module 1, the first DMA module 21, and the second DMA module 22, the system chip may also include an interface module 4. The interface module 4 is configured to receive the audio playback data from the first DMA module 21 and output the audio playback data to an external audio codec, and to receive the audio recording data from the audio codec and output the audio recording data to the second DMA module 22.
[0096] Specifically, during audio playback, the interface module 4 transmits the audio playback data to the audio codec for playback according to the protocol clock signal. During recording, the audio codec collects audio recording data from the microphone, and the interface module 4 obtains the input audio recording data from the audio codec according to the protocol clock signal.
[0097] Figure 12 The diagram shown is a structural schematic of the audio processing system according to an embodiment of this disclosure. Figure 12 As shown in the figure, this disclosure provides an audio processing system, which includes: the system chip described above, an audio codec, a speaker, and a microphone.
[0098] The audio codec is electrically coupled to the system chip. The audio codec is configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and to perform analog-to-digital conversion and encoding on the recording signal to generate the audio recording data.
[0099] The speaker is electrically coupled to the audio codec. The speaker is configured to perform the playback based on the playback signal.
[0100] The microphone is electrically coupled to the audio codec. The microphone is configured to capture data during playback to generate the recorded signal.
[0101] Figure 13 This diagram illustrates the working principle of the audio processing system according to an embodiment of this disclosure. Figure 13As shown, the audio processing system disclosed herein adopts a hardware and software integrated approach. The implementation unit includes: a CPU-controlled DLP module (Digital Recapture Module) 1, a DMA module 2, and an audio interface controller 3 (taking I2S as an example). DMA is a technology in computer architecture used to improve data transfer efficiency. It allows external devices (such as hard drives, network adapters, audio cards, etc.) to directly access computer memory without intervention from the central processing unit (CPU). This reduces the CPU load and increases data transfer speed, especially when large amounts of data need to be transferred between memory and external devices. I2S, or the chip I2S audio interface, is used to connect to the CODEC for recording and audio playback. Simultaneously, in the soft recapture function, it provides a hardware FIFO pointer for calculating the accurate recapture start frame position. In practical applications, this disclosure can broaden the scope of I2S to describe it as an audio interface, and then use I2S as an example.
[0102] Combination Figure 12 When playing audio on the application side, the application writes audio playback data to the Playback DMABuffer. The DLP module 1 caches the audio playback data from the Playback DMA Buffer to the Playback Mirror Buffer. The DMA 2 transfers the audio playback data from the Playback DMA Buffer to the FIFO of the audio interface. The audio interface controller 3 (such as I2S) transfers the audio playback data in the FIFO to the CODEC terminal 4 for playback.
[0103] Combination Figure 12 During recording, CODEC terminal 4 acquires recording data from the MIC microphone. Audio interface controller 3 (such as I2S) receives the recording data from CODEC terminal 4 into FIFO. DMA 2 transfers the recording data from the audio interface FIFO to Capture DMA Buffer. DLP module 1 reconstructs the data by using hardware pointer difference between the recording data in Capture DMA Buffer and the audio playback data in Playback Mirror Buffer. The application reads the reconstructed audio data from Capture DMA Buffer.
[0104] In practical applications, the audio playback or recording application disclosed herein is housed in an electronic device. This electronic device may be a computer comprising all or some of the following components: a memory, a memory controller, one or more processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, speakers, microphones, an input / output (I / O) subsystem, a display screen, other output or control devices, and external ports. The computer includes, but is not limited to, personal computers such as desktop computers, laptops, tablets, smartphones, smart TVs, and personal digital assistants (PDAs). In other embodiments, the electronic device may also be a server, which may be deployed on one or more physical servers based on factors such as functionality and load. It may also be a cloud server comprised of distributed or centralized server clusters; this embodiment does not impose such limitations.
[0105] The scope of protection of the audio processing method described in this disclosure is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this disclosure is included within the scope of protection of this disclosure.
[0106] This disclosure also provides a digital acquisition module or audio processing system, which can implement the audio processing method described in this disclosure. However, the implementation device of the audio processing method described in this disclosure includes, but is not limited to, the structure of the digital acquisition module or audio processing system listed in this embodiment. Any structural modifications and substitutions of the prior art made based on the principles of this disclosure are included within the protection scope of this disclosure.
[0107] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus, modules, systems, or methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or modules or units through some interfaces, and may be electrical, mechanical, or other forms.
[0108] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0109] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.
[0110] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.
[0111] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.
Claims
1. An audio processing method, characterized in that, include: Acquire audio playback data and write the audio playback data into the first buffer; The audio playback data in the first buffer is cached in the second buffer as audio re-sampling data. The second buffer is a playback mirror buffer. The buffer size of the second buffer is larger than that of the first buffer to reserve margin to resolve data overlap. The audio playback data in the first buffer is output to the outside for playback. Acquire the audio recording data collected during playback and write the audio recording data into the third buffer; as well as Based on the audio retrieval data in the second buffer and the audio recording data in the third buffer, the audio retrieval data and the audio recording data are aligned according to the hardware pointer difference and then reconstructed to obtain audio processing data. The process of aligning the audio retrieval data and the audio recording data based on the hardware pointer difference and then reconstructing the data to obtain the audio processing data includes: The playback pointer and recording pointer are determined based on the DMA pointer and the audio interface FIFO pointer, including subtracting the audio interface transmit FIFO pointer from the playback DMA pointer to obtain the playback pointer, and adding the audio interface receive FIFO pointer to the recording DMA pointer to obtain the recording pointer. The hardware pointer difference is determined based on the playback pointer and the recording pointer; The pointer corresponding to the audio retrieval data is determined based on the hardware pointer difference; Based on the pointer corresponding to the audio re-acquisition data, the audio re-acquisition data is retrieved from the second buffer; and According to the pre-set audio data channel requirements, the extracted audio retrieval data and the audio recording data are merged frame by frame.
2. The audio processing method according to claim 1, characterized in that, Also includes: The echo is acoustically canceled based on the audio processing data to generate audio data.
3. The audio processing method according to claim 2, characterized in that, Also includes: Based on the results of the acoustic echo cancellation process, the background noise is filtered out and the speech signal is extracted. as well as The extracted speech signals are used for speech recognition applications.
4. A system-on-a-chip (SoC), characterized in that, include: The first DMA module is configured to receive audio playback data and write the audio playback data into a first buffer, and to output the audio playback data in the first buffer to an external source for playback. The digital back-sampling module is configured to cache the audio playback data in the first buffer into a second buffer as audio back-sampling data. The second buffer is a playback mirror buffer. The buffer size of the second buffer is larger than that of the first buffer to reserve margin to resolve data overlap. as well as The second DMA module is configured to receive audio recording data acquired during playback and write the audio recording data into a third buffer. The digital acquisition module is further configured to align the audio acquisition data and the audio recording data according to the hardware pointer difference and reassemble the data to obtain audio processing data based on the audio acquisition data in the second buffer and the audio recording data in the third buffer. The digital acquisition module is configured as follows: The playback pointer and recording pointer are determined based on the DMA pointer and the audio interface FIFO pointer, including subtracting the audio interface transmit FIFO pointer from the playback DMA pointer to obtain the playback pointer, and adding the audio interface receive FIFO pointer to the recording DMA pointer to obtain the recording pointer. The hardware pointer difference is determined based on the playback pointer and the recording pointer; The pointer corresponding to the audio retrieval data is determined based on the hardware pointer difference; The audio re-acquisition data is retrieved from the second buffer according to the pointer corresponding to the audio re-acquisition data; as well as According to the pre-set audio data channel requirements, the extracted audio retrieval data and the audio recording data are merged frame by frame.
5. The system chip according to claim 4, characterized in that, Also includes: The signal processing module is configured to perform acoustic echo cancellation processing on the echo based on the audio processing data to generate audio data.
6. The system chip according to claim 4, characterized in that, Also includes: The interface module is configured to receive the audio playback data from the first DMA module and output the audio playback data to an external audio codec of the system chip, and to receive the audio recording data from the audio codec and output the audio recording data to the second DMA module.
7. The system chip according to claim 4, characterized in that, The digital acquisition module is implemented by the central processing unit included in the system chip.
8. An audio processing system, characterized in that, include: System chip according to any one of claims 4 to 7; An audio codec is electrically coupled to the system chip and configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and to perform analog-to-digital conversion and encoding on the recording signal to generate the audio recording data; A speaker is electrically coupled to the audio codec and configured to perform the playback based on the playback signal; as well as A microphone is electrically coupled to the audio codec and configured to capture data during playback to generate the recorded signal.