Audio processing method, system chip and audio processing system
Through DMA retrieval technology, the audio retrieval data and recording data are aligned and reorganized, which solves the problems of high costs and resource waste in hardware ADC retrieval methods, and realizes audio retrieval in I2S pin-free system, reducing hardware costs and improving Voice communication quality.
Patent Information
- Application Number
- CN202410968828.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-18
AI Technical Summary
In the prior art, the hardware ADC recovery method has high hardware costs and resource waste, which limits the scalability and application scenarios of the system, especially in SoC systems without I2S pin pins, it is difficult to realize audio recovery.
Through the DMA retrieval method, the audio retrieval data and recorded data are aligned and reorganized by the difference between the buffer and the hardware pointer to generate audio processing data without the need for additional hardware circuits to achieve audio retrieval.
It significantly reduces hardware costs and debugging costs, improves the scalability of the system and voice communication quality, can be applied in systems without I2S pin pins, and improves the output effect of the acoustic echo cancellation algorithm.
Smart Images

Figure CN119049491A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the technical field of audio processing, and in particular, relates to an audio processing method, a system chip, and an audio processing system. Background Art
[0002] At present, in various applications such as voice assistants, automatic speech recognition (ASR) systems, and voice control devices, they usually rely on hardware ADC (Analog-to-Digital Converter) or other hardware acquisition methods. These methods have high hardware costs, such as occupying the SDI (Serial Digital Interface) of the I2S (Inter-IC Sound) of the SOC (System on Chip) chip, or limiting the scalability and wide application of the system.
[0003] In detail, there are many limitations and shortcomings in various hardware recovery solutions in the prior art: the hardware cost of recovery through hardware ADC is relatively high, and because the hardware indicators are relatively high, it may increase the technical debugging difficulty of the system. Recovery can be achieved by adding a SOC pin, which is limited to the I2S controller with multiple I2SSDI input pins. If there is only one I2SSDI, this solution is not feasible. For systems with built-in audio codecs in the SOC, the I2S pin may need to be connected to the SOC external circuit to cooperate with the ADC to achieve recovery. This solution will increase the external ADC, and the ADC inside the SOC cannot be used, resulting in a waste of resources. Summary of the invention
[0004] The embodiments of the present disclosure provide an audio processing method, a system chip, and an audio processing system, which can realize audio capture without additional hardware circuits.
[0005] In a first aspect, an embodiment of the present disclosure provides an audio processing method. The audio processing method includes: obtaining audio playback data and writing the audio playback data into a first buffer; caching the audio playback data in the first buffer into a second buffer as audio playback data; outputting the audio playback data in the first buffer to the outside for playback; obtaining audio recording data collected during the playback and writing the audio recording data into a third buffer; and generating audio processing data based on the audio playback data in the second buffer and the audio recording data in the third buffer.
[0006] In an implementation of the first aspect, generating audio processing data based on the audio recollection data in the second buffer and the audio recording data in the third buffer includes: aligning the audio recollection data and the audio recording data according to a hardware pointer difference and performing data reorganization to obtain the audio processing data.
[0007] In an implementation of the first aspect, aligning the audio capture data and the audio recording data according to the hardware pointer difference and reorganizing the data to obtain the audio processing data includes: determining a play pointer and a recording pointer according to a DMA pointer and a FIFO pointer of an audio interface; determining the hardware pointer difference according to the play pointer and the recording pointer; determining a pointer corresponding to the audio capture data based on the hardware pointer difference; retrieving the audio capture data from the second buffer according to the pointer corresponding to the audio capture data; and merging the retrieved audio capture data and the audio recording data frame by frame according to a preset audio data channel requirement.
[0008] In an implementation of the first aspect, determining a play pointer and a recording pointer based on a DMA pointer and a FIFO pointer of an audio interface includes: subtracting the audio interface send FIFO pointer from the play DMA pointer to obtain the play pointer; and adding the audio interface receive FIFO pointer to the recording DMA pointer to obtain the recording pointer.
[0009] In an implementation manner of the first aspect, the audio processing method further includes: performing acoustic echo cancellation processing on the echo according to the audio processing data to generate audio data.
[0010] In an implementation of the first aspect, the audio processing method further includes: filtering out the playback background sound according to the result of the acoustic echo cancellation processing to extract a speech signal; and using the extracted speech signal for speech recognition application.
[0011] In a second aspect, an embodiment of the present disclosure provides a system chip. The system chip includes: a first DMA module, configured to receive audio playback data and write the audio playback data into a first buffer, and output the audio playback data in the first buffer to the outside for playback; a digital recapture module, configured to cache the audio playback data in the first buffer into a second buffer as audio recapture data; and a second DMA module, configured to receive audio recording data collected during the playback, and write the audio recording data into a third buffer, wherein the digital recapture module is further configured to generate audio processing data based on the audio recapture data in the second buffer and the audio recording data in the third buffer.
[0012] In an implementation manner of the second aspect, the system chip further includes: a signal processing module configured to perform acoustic echo cancellation processing on the echo according to the audio processing data to generate audio data.
[0013] In an implementation of the second aspect, the system chip also includes: an interface module, configured to receive the audio playback data from the first DMA module and output the audio playback data to an audio codec outside the system chip, and to receive the audio recording data from the audio codec and output the audio recording data to the second DMA module.
[0014] In an implementation manner of the second aspect, the digital recovery module is implemented by a central processing unit included in the system chip.
[0015] In a third aspect, the present disclosure provides an audio processing system. The audio processing system includes: the system chip; an audio codec electrically coupled to the system chip and configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and perform analog-to-digital conversion and encoding on the recording signal to generate the audio recording data; a speaker electrically coupled to the audio codec and configured to perform the playback based on the playback signal; and a microphone electrically coupled to the audio codec and configured to collect during the playback to generate the recording signal.
[0016] According to the audio processing method, system chip and audio processing system disclosed in the present invention, no additional hardware circuit is required, but DMA is used to realize audio acquisition, which significantly reduces the hardware cost and debugging cost of the entire system. The system disclosed in the present invention is easier to expand and integrate into various application scenarios. For example, it does not need to occupy the I2SSDI pin and can be used in SoC systems without I2SSDI.
[0017] The present invention synchronously merges the collected data collected by the DMA path with the audio recording data, and uses the merged audio data for audio processing, thereby being applied to various fields such as voice assistants, automatic speech recognition ASR systems, and voice control devices with lower hardware costs and higher voice communication quality.
[0018] The echo data provided by the present disclosure can provide a reference signal of the playback path for acoustic echo cancellation. Based on the reference signal, the acoustic echo cancellation algorithm can filter out the playback background sound in the recording and extract the voice signal for subsequent voice recognition.
[0019] The signal provided to the audio algorithm (acoustic echo cancellation algorithm) disclosed in the present invention is audio data including a re-collected signal and a MIC recording signal. Thanks to the precise reference frame calculation, the re-collected frame and the actual hardware playback position are highly aligned, thereby improving the output effect of the acoustic echo cancellation algorithm. In addition, due to the high alignment, the reference frame alignment logic of the pre-processing of the acoustic echo cancellation can be omitted, thereby saving algorithm overhead. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 Shown is a schematic diagram of an application scenario of the audio processing method described in an embodiment of the present disclosure.
[0021] Figure 2 Shown is a principle flow chart of the audio processing method described in an embodiment of the present disclosure.
[0022] Figure 3 Shown is a data processing flow chart of the audio processing method described in an embodiment of the present disclosure.
[0023] Figure 4 Shown is a pointer determination flow chart of the audio processing method described in an embodiment of the present disclosure.
[0024] Figure 5 Shown is an audio processing schematic diagram of the audio processing method described in an embodiment of the present disclosure.
[0025] Figure 6 Shown is an echo cancellation flow chart of the audio processing method described in an embodiment of the present disclosure.
[0026] Figure 7 Shown is a speech recognition flow chart of the audio processing method described in an embodiment of the present disclosure.
[0027] Figure 8 Shown is a structural principle diagram of the system chip described in an embodiment of the present disclosure.
[0028] Fig. 9 Shown is a structural principle diagram of the digital recovery module described in an embodiment of the present disclosure.
[0029] Fig.10 Shown is another structural principle diagram of the system chip described in an embodiment of the present disclosure.
[0030] Fig.11 Shown is another structural principle diagram of the system chip described in an embodiment of the present disclosure.
[0031] Fig.12 Shown is a structural principle diagram of the audio processing system described in an embodiment of the present disclosure.
[0032] Fig.13 Shown is a diagram of the working principle of the audio processing system described in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] The following is an explanation of the embodiments of the present disclosure by specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0034] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present disclosure in a schematic manner, and thus the illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0035] The embodiments of the present disclosure provide a hardware application scenario of the audio processing method, which will be described below using this application scenario as an example. Figure 1 FIG. 1 is a schematic diagram showing an application scenario of the audio processing method described in an embodiment of the present disclosure. Figure 1 As shown, before the audio playback data is transmitted to the speaker or loudspeaker for playback, the audio processing method described in the present invention is executed to obtain audio capture data, and then the audio capture data is recombined with the audio recording data collected by the microphone to obtain audio processing data. The audio processing data after the audio capture and processing can provide a reference signal of the playback path for the AEC (Acoustic Echo Cancellation) algorithm. Based on the reference signal, the AEC algorithm can filter out the playback background sound in the recording and extract the voice signal for subsequent voice recognition.
[0036] The technical solutions in the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present disclosure.
[0037] Figure 2 The figure shows a principle flow chart of the audio processing method according to the embodiment of the present disclosure. The present embodiment provides an audio processing method, which specifically includes the following steps S21 to S25.
[0038] In S21, audio playback data is acquired and written into a first buffer.
[0039] Specifically, the CPU (Central Processing Unit) continuously writes the audio playback data to be played into the DMA buffer, i.e., the first buffer, and the DMA controller continuously transmits the audio playback data in the DMA buffer to the TX FIFO of the audio interface. The audio interface transmits the data in the TX FIFO to the CODEC end (audio codec, referring to a device or program that can transform a signal or a data stream) for playback according to the protocol clock signal (such as I2S).
[0040] In S22, the audio playback data in the first buffer is cached in a second buffer as audio loopback data.
[0041] Specifically, the CPU writes data into the DMA buffer, and the DLP module will cache the audio playback data at the same time to implement audio DMA digital loopback (Audio DMA Digital Loopback), wherein the DLP module refers to a digital loop back (Digital Loop Back) module.
[0042] In S23, the audio playback data in the first buffer is output to the outside for playback.
[0043] In S24, the audio recording data collected during the playing is acquired, and the audio recording data is written into a third buffer.
[0044] Audio recording data refers to MIC (Microphone) recording signals: the sounds recorded by the microphone (including speaking sounds, background music played by speakers, etc.).
[0045] In S25, audio processing data is generated based on the audio recollected data in the second buffer and the audio recording data in the third buffer.
[0046] In some embodiments, generating audio processing data based on the audio playback data in the second buffer and the audio recording data in the third buffer may include: aligning the audio playback data and the audio recording data according to a hardware pointer difference and performing data reorganization to obtain the audio processing data.
[0047] Figure 3 FIG. 1 is a data processing flow chart of the audio processing method according to an embodiment of the present disclosure. Figure 3 As shown, in some embodiments, aligning the audio capture data and the audio recording data according to the hardware pointer difference and reorganizing the data to obtain the audio processing data specifically includes the following steps S251 to S255.
[0048] In S251, a play pointer and a record pointer are determined according to the DMA pointer and the FIFO pointer of the audio interface.
[0049] Figure 4 FIG. 2 shows a flow chart of a pointer determination method of an audio processing method according to an embodiment of the present disclosure. Figure 4 As shown, step S251 specifically includes the following steps S251A to S251B.
[0050] S251A, subtract the audio interface send FIFO pointer from the play DMA pointer to obtain the play pointer.
[0051] S251B, add the recording DMA pointer to the audio interface receiving FIFO pointer as the recording pointer.
[0052] The DMA pointer refers to the location of the audio frame of the hardware DMA. The FIFO pointer of the hardware audio interface refers to the location of the FIFO audio frame in the hardware audio interface (such as I2S). FIFO refers to FIRST IN FIRST OUT (first in first out buffer), which is located in the audio interface module.
[0053] Specifically, the hardware pointer difference is determined as follows:
[0054] P: Play pointer = Play DMA POS - TX FIFO POS;
[0055] C: Recording pointer = recording DMA POS + RX FIFO POS.
[0056] Next, in S252, the hardware pointer difference is determined according to the play pointer and the recording pointer.
[0057] Specifically, the hardware pointer difference Delta = P – C, that is, the difference between playback and recording.
[0058] In S253, a pointer corresponding to the audio acquisition data is determined based on the hardware pointer difference.
[0059] Specifically, when playback and recording exist at the same time, the hardware pointer difference Delta can be used to quickly calculate the capture reference start frame position POS (CUR C + Delta = CUR P, that is, recording pointer + Delta = playback pointer) corresponding to the actual capture. Based on this pointer, the capture signal can be taken out from the mirror buffer. The mirror buffer contains the buffer for playing audio data, which is stored in the DDR (Double Data Rate) memory and is the source of the capture signal in the DLP module.
[0060] In S254, the audio recollection data is retrieved from the second buffer according to the pointer corresponding to the audio recollection data.
[0061] In S255, the retrieved audio data and the audio recording data are merged frame by frame according to the preset audio data channel requirements.
[0062] Specifically, the preset audio data channel requirement may be MIC 6 channels + recovery 2 channels, or other audio data arrangement requirements set by the user.
[0063] Figure 5 FIG. 2 is a schematic diagram of an audio processing method according to an embodiment of the present disclosure. Figure 5 As shown, during audio playback, the application writes audio playback data to the Playback DMA Buffer, the DLP module caches the audio playback data of the Playback DMA Buffer to the Playback Mirror Buffer, the DMA transfers the audio playback data of the Playback DMA Buffer to the FIFO of the audio interface, and the audio interface (such as I2S) transfers the audio playback data in the FIFO to the CODEC end for playback. During recording, the audio interface (such as I2S) receives the recording data from the CODEC end into the FIFO, the DMA transfers the recording data in the FIFO of the audio interface to the Capture DMA Buffer, the DLP module reassembles the recording data in the Capture DMA Buffer and the audio playback data in the Playback Mirror Buffer through the hardware pointer difference, and the application reads the reassembled audio data from the Capture DMA Buffer.
[0064] like Figure 5 As shown, the DLP module in the present disclosure implements a mirror buffer for audio playback data. The mirror buffer data includes the playback buffer data. Combined with the DMA pointer and the FIFO pointer of the audio hardware interface, the reference start frame position of the recapture can be accurately calculated. Then, the CPU recombines the recapture signal and the MIC recording signal to form audio data containing the recapture and MIC recording signals, and provides it to the AEC algorithm to eliminate the playback background sound in the MIC recording signal. The disclosed solution not only saves hardware costs, but is also forward compatible with chips with weak specifications or low costs, such as those with only one audio input pin.
[0065] The operation of the DLP module is implemented by the CPU. For different buffers, the specific operations are as follows.
[0066] Playback DMA Buffer (audio playback DMA buffer is the first buffer): The CPU continuously writes the audio playback data to be played into the Buffer, and the DMA controller continuously transmits the audio playback data in the Buffer to the TX FIFO of the audio interface. The audio interface transmits the data in the TX FIFO to the CODEC end (audio codec) for playback according to the protocol clock signal (such as I2S).
[0067] Capture DMA Buffer (audio recording DMA buffer is the third buffer): the audio interface collects the signal transmitted by CODEC and writes it into the RX FIFO of the audio interface according to the protocol clock signal (such as I2S). The DMA controller continuously moves the data in the RX FIFO to the Capture DMA Buffer. The CPU continuously takes out the audio data in the Capture DMA Buffer and sends it to the application or algorithm for processing.
[0068] Playback Mirror Buffer (playback mirror buffer, also known as the second buffer): The CPU writes data to the playback DMABuffer, and the DLP module will cache the data at the same time. The buffer size is larger than the playback DMA Buffer (for example: 1.5 times) to reserve margin to solve the data overlap problem. The mirror buffer is the data source for the subsequent DLP module to process the recovered data.
[0069] DMA POS refers to the DMA hardware pointer, indicating the buffer pointer position of the current hardware playback or recording; TX / RXFIFO refers to the send or receive FIFO of the audio interface controller (such as I2S), and the FIFO POS can be obtained from the hardware.
[0070] The hardware pointer difference is determined as follows:
[0071] P: Play pointer = Play DMA POS - TX FIFO POS;
[0072] C: Recording pointer = recording DMA POS + RX FIFO POS;
[0073] The hardware pointer difference Delta = P – C, which is the difference between playback and recording.
[0074] When playback and recording exist at the same time, the acquisition reference start frame position POS (CUR C + Delta = CUR P, that is, recording pointer + Delta = playback pointer) corresponding to the actual acquisition can be quickly calculated based on the hardware pointer difference Delta. Based on this pointer, the acquisition signal can be taken out from the mirror buffer.
[0075] Audio data reorganization: According to the audio data arrangement requirements set by the user (such as MIC 6 channels + re-collection 2 channels), the DLP module automatically reprocesses and merges the re-collection signal and the MIC signal frame by frame to form the data arrangement order required by the user. The whole process is transparent to user-state applications and is compatible with existing audio algorithm frameworks.
[0076] In actual applications, the DLP module also supports unlimited capture channels, which can completely capture all playback channels and capture and mix all playback channels on demand.
[0077] In one embodiment, after the step of obtaining audio data, the method further includes: performing acoustic echo cancellation processing on the echo generated in the voice communication according to the audio data.
[0078] Figure 6 FIG. 2 shows an echo cancellation flow chart of the audio processing method according to an embodiment of the present disclosure. Figure 6 As shown, in addition to steps S21 to S25, the audio processing method further includes step S26, performing acoustic echo cancellation processing on the echo according to the audio processing data to generate audio data.
[0079] Therefore, this embodiment provides an audio processing method, which specifically includes the following steps. The audio playback data to be written into the DMA buffer is obtained; the audio playback data is written into the DMA buffer, and then transmitted to the audio interface and the audio codec in sequence for playback. The obtained audio playback data is cached in a mirror buffer. The audio playback data in the mirror buffer is used as the re-collected data. The input audio recording data is obtained. The re-collected data and the audio recording data are reorganized by hardware pointer difference to obtain audio data. In addition, the echo is subjected to acoustic echo cancellation processing according to the audio processing data to generate audio data.
[0080] In one embodiment, after the step of performing acoustic echo cancellation processing on the echo generated in the voice communication according to the audio data, the method further includes: filtering out the playback background sound according to the result of the acoustic echo cancellation processing to extract the voice signal; and using the refined voice signal for voice recognition application.
[0081] AEC algorithm is a technology used to deal with echo problems in communication systems. Echo usually occurs in voice communication when the audio signal of the speaker is transmitted back again through the microphone, resulting in an echo heard by the other party. AEC algorithm aims to reduce or eliminate this echo to improve the quality of communication.
[0082] The signal provided to the audio algorithm by the disclosed embodiment is audio data after reorganization of the recaptured signal and the MIC recording signal, and has the following characteristics: thanks to the precise reference frame calculation, the recaptured frame and the actual hardware playback position are highly aligned, which can improve the output effect of the AEC algorithm, and due to the high alignment, the reference frame alignment logic of the AEC pre-processing can be omitted, saving algorithm overhead.
[0083] Figure 7 FIG. 2 shows a flow chart of speech recognition of the audio processing method according to an embodiment of the present disclosure. Figure 7 As shown, in addition to steps S21 to S26, the audio processing method further includes step S27, filtering out the background sound according to the result of the acoustic echo cancellation processing to extract the voice signal; and using the extracted voice signal for voice recognition application.
[0084] Therefore, this embodiment provides an audio processing method, which specifically includes the following steps. The audio playback data to be written into the DMA buffer is obtained; the audio playback data is written into the DMA buffer, and then transmitted to the audio interface and the audio codec in sequence for playback. The obtained audio playback data is cached in a mirror buffer. The audio playback data in the mirror buffer is used as the re-collection data. The input audio recording data is obtained. The re-collection data and the audio recording data are reorganized by hardware pointer difference to obtain audio data. The echo is subjected to acoustic echo cancellation processing according to the audio processing data to generate audio data. In addition, the playback background sound is filtered out according to the result of the acoustic echo cancellation processing to extract the voice signal; and the refined voice signal is used for voice recognition application.
[0085] Figure 8 FIG. 1 is a schematic diagram showing the structure of the system chip according to an embodiment of the present disclosure. Figure 8 As shown, an embodiment of the present disclosure provides a system chip, which includes: a digital recovery module 1 and a first DMA module 21 and a second DMA module 22.
[0086] The first DMA module 21 is configured to receive audio play data and write the audio play data into a first buffer, and output the audio play data in the first buffer to the outside for playing.
[0087] The digital recapture module 1 is configured to cache the audio playback data in the first buffer into a second buffer as audio recapture data.
[0088] The second DMA module 22 is configured to receive the audio recording data collected during the playing, and write the audio recording data into the third buffer.
[0089] In addition, the digital recapture module 1 is further configured to generate audio processing data based on the audio recapture data in the second buffer and the audio recording data in the third buffer.
[0090] Fig. 9 The figure shows the structural principle diagram of the digital recovery module according to the embodiment of the present disclosure. Fig. 9 As shown, the embodiment of the present disclosure provides a digital recovery module 1, which specifically includes a recovery unit 11, a recording data acquisition unit 12 and an audio data reorganization unit 13.
[0091] The retrieving unit 11 is configured to obtain the audio playback data to be written into the DMA buffer, and the audio playback data is written into the DMA buffer and then transmitted to the audio interface and the audio codec in sequence for playback. The retrieving unit 11 is configured to cache the obtained audio playback data into the mirror buffer, and use the audio playback data in the mirror buffer as the retrieving data.
[0092] The recording data acquiring unit 12 is configured to acquire input audio recording data.
[0093] The audio data reassembly unit 13 is configured to reassemble the retrieved data and the audio recording data through a hardware pointer difference to obtain audio data.
[0094] Fig.10 FIG. 2 shows another structural principle diagram of the system chip according to an embodiment of the present disclosure. Fig.10 As shown, in the embodiment of the present disclosure, the system chip may include a signal processing module 3 in addition to the digital echo module 1 and the first DMA module 21 and the second DMA module 22. The signal processing module 3 is configured to perform acoustic echo cancellation processing on the echo according to the audio processing data to generate audio data.
[0095] Fig.11 FIG. 2 shows another structural principle diagram of the system chip according to an embodiment of the present disclosure. Fig.11As shown, the system chip includes, in addition to the digital acquisition module 1, the first DMA module 21, and the second DMA module 22, an interface module 4. The interface module 4 is configured to receive the audio playback data from the first DMA module 21, and output the audio playback data to the audio codec outside the system chip, and receive the audio recording data from the audio codec, and output the audio recording data to the second DMA module 22.
[0096] Specifically, during audio playback, the interface module 4 transmits the audio playback data to the audio codec for playback according to the protocol clock signal. During recording, the audio codec collects audio recording data from the microphone, and the interface module 4 obtains the input audio recording data from the audio codec according to the protocol clock signal.
[0097] Fig.12 FIG. 2 is a schematic diagram showing the structure of the audio processing system according to an embodiment of the present disclosure. Fig.12 As shown, an embodiment of the present disclosure provides an audio processing system, which includes: the system chip described above, an audio codec, a speaker, and a microphone.
[0098] The audio codec is electrically coupled to the system chip and is configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and perform analog-to-digital conversion and encoding on a recording signal to generate the audio recording data.
[0099] The speaker is electrically coupled to the audio codec and is configured to perform the playback based on the playback signal.
[0100] The microphone is electrically coupled to the audio codec and is configured to collect data during the playback to generate the recording signal.
[0101] Fig.13 FIG. 2 is a diagram showing the working principle of the audio processing system according to an embodiment of the present disclosure. Fig.13As shown, the audio processing system of the present disclosure adopts a combination of software and hardware, and the implementation unit includes: a DLP module (digital capture module) 1 controlled by a CPU, DMA 2, and an audio interface controller 3 (taking I2S as an example). DMA is a technology used in computer architecture to improve data transmission efficiency. It allows external devices (such as hard drives, network adapters, audio cards, etc.) to directly access computer memory without the intervention of the central processing unit (CPU). This can reduce the burden on the CPU and increase the speed of data transmission, especially when a large amount of data needs to be transferred between the memory and external devices. I2S is a chip I2S audio interface, which is used to connect to the CODEC end for recording and audio playback. At the same time, a hardware FIFO pointer is provided in the soft capture function to calculate the accurate capture start frame position. In practical applications, the present disclosure can expand the scope of I2S and describe it as an audio interface, and then take I2S as an example.
[0102] Combination Fig.12 When audio is played on the application side, the application writes the audio playback data to the Playback DMA Buffer, the DLP module 1 caches the audio playback data of the Playback DMA Buffer to the PlaybackMirror Buffer, DMA 2 transfers the audio playback data of the Playback DMA Buffer to the FIFO of the audio interface, and the audio interface controller 3 (such as I2S) transfers the audio playback data in the FIFO to the CODEC end 4 for playback.
[0103] Combination Fig.12 During recording, CODEC terminal 4 collects recording data from the MIC microphone, audio interface controller 3 (such as I2S) receives the recording data from CODEC terminal 4 into FIFO, DMA 2 transfers the recording data in the FIFO of the audio interface to Capture DMA Buffer (data capture buffer), DLP module 1 reassembles the recording data in Capture DMA Buffer and the audio playback data in Playback Mirror Buffer through the hardware pointer difference, and the application reads the reassembled audio data from Capture DMA Buffer.
[0104] In practical applications, the application (application) for audio playback or recording disclosed in the present invention is set in an electronic device, and the electronic device can be a computer including all or part of the components such as a memory, a storage controller, one or more processing units (CPU), a peripheral interface, an RF circuit, an audio circuit, a speaker, a microphone, an input / output (I / O) subsystem, a display screen, other output or control devices, and an external port; the computer includes but is not limited to personal computers such as desktop computers, laptops, tablet computers, smart phones, smart TVs, and personal digital assistants (PDAs). In other embodiments, the electronic device can also be a server, which can be arranged on one or more physical servers according to various factors such as function and load, or can be a cloud server composed of distributed or centralized server clusters, which is not limited in this embodiment.
[0105] The protection scope of the audio processing method described in the embodiment of the present disclosure is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.
[0106] The embodiments of the present disclosure also provide a digital acquisition module or an audio processing system, which can implement the audio processing method described in the present disclosure accordingly. However, the implementation device of the audio processing method described in the present disclosure includes but is not limited to the structure of the digital acquisition module or the audio processing system listed in the embodiments. All structural deformations and replacements of the prior art made according to the principles of the present disclosure are included in the protection scope of the present disclosure.
[0107] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices, modules, systems or methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.
[0108] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure. For example, the functional modules / units in the various embodiments of the present disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0109] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0110] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.
[0111] The above embodiments are merely illustrative of the principles and effects of the present disclosure, and are not intended to limit the present disclosure. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present disclosure. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present disclosure shall still be covered by the claims of the present disclosure.
Claims
1. An audio processing method, characterized in that: include: Acquire audio playback data, and write the audio playback data into the first buffer; Cache the audio playback data in the first buffer into the second buffer as audio playback data; Outputting the audio playback data in the first buffer to the outside for playback; Acquire the audio recording data collected during the playing, and write the audio recording data into a third buffer; as well as Audio processing data is generated based on the audio recollected data in the second buffer and the audio recording data in the third buffer.
2. According to the audio processing method of claim 1, generating audio processing data based on the audio sampling data in the second buffer and the audio recording data in the third buffer comprises: The audio recollected data and the audio recording data are aligned and data reorganized according to the hardware pointer difference to obtain the audio processing data.
3. The audio processing method according to claim 2, characterized in that: Aligning the audio recollected data and the audio recording data according to the hardware pointer difference and reorganizing the data to obtain the audio processing data includes: Determine the play pointer and the recording pointer according to the DMA pointer and the FIFO pointer of the audio interface; Determine the hardware pointer difference according to the play pointer and the recording pointer; Determine the pointer corresponding to the audio acquisition data based on the hardware pointer difference; fetching the audio recollection data from the second buffer according to the pointer corresponding to the audio recollection data; and According to the preset audio data channel requirements, the retrieved audio data and the audio recording data are merged frame by frame.
4. The audio processing method according to claim 3, characterized in that: Determining the play pointer and recording pointer based on the DMA pointer and the FIFO pointer of the audio interface includes: Subtract the audio interface send FIFO pointer from the play DMA pointer to obtain the play pointer; and The recording DMA pointer plus the audio interface receiving FIFO pointer is used as the recording pointer.
5. The audio processing method according to claim 1, characterized in that: Also includes: The echo is subjected to acoustic echo cancellation processing according to the audio processing data to generate audio data.
6. The audio processing method according to claim 5, characterized in that: Also includes: According to the result of the acoustic echo cancellation processing, the background sound is filtered out to extract the speech signal; as well as The extracted speech signal is used for speech recognition application.
7. A system chip, characterized in that: include: A first DMA module is configured to receive audio playback data and write the audio playback data into a first buffer, and output the audio playback data in the first buffer to an external device for playback; A digital recollection module is configured to cache the audio playback data in the first buffer into a second buffer as audio recollection data; as well as The second DMA module is configured to receive the audio recording data collected during the playing and write the audio recording data into the third buffer, The digital retrieval module is further configured to generate audio processing data based on the audio retrieval data in the second buffer and the audio recording data in the third buffer.
8. The system chip according to claim 7, characterized in that: Also includes: The signal processing module is configured to perform acoustic echo cancellation processing on the echo according to the audio processing data to generate audio data.
9. The system chip according to claim 7, characterized in that: Also includes: The interface module is configured to receive the audio playback data from the first DMA module and output the audio playback data to the audio codec outside the system chip, and to receive the audio recording data from the audio codec and output the audio recording data to the second DMA module.
10. The system chip according to claim 7, characterized in that: The digital acquisition module is implemented by a central processing unit included in the system chip.
11. An audio processing system, characterized in that: include: The system chip according to any one of claims 7 to 10; An audio codec, electrically coupled to the system chip, and configured to decode and perform digital-to-analog conversion on the audio playback data to generate a playback signal, and perform analog-to-digital conversion and encoding on a recording signal to generate the audio recording data; a speaker electrically coupled to the audio codec and configured to perform the playback based on the playback signal; as well as A microphone is electrically coupled to the audio codec and is configured to collect audio during the playback to generate the recording signal.
Citation Information
Patent Citations
Audio recording method and system thereof
CN102420010A
Method and system for reducing delay from sound input to sound output based on DMA technology
CN105389156A
Audio processing method and system
CN106325804A
Echo cancellation signal processing method and medium
CN112562709A
Echo cancellation time delay estimation method and echo cancellation method
CN113409808A
