An audio processing method, apparatus, electronic device, and program product
By configuring the read state machine of the external storage controller to provide sampling clocks for multiple digital microphones and processing PDM data in parallel, the problem of synchronous acquisition of multiple microphones in low-cost chips is solved, achieving efficient multi-channel synchronous latching and improved stability.
Patent Information
- Application Number
- CN202611362881.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies make it difficult to achieve synchronous acquisition of multiple digital microphones in low-cost chips, especially without a dedicated controller, which leads to phase shift between channels and audio sampling distortion, failing to meet the requirements of multi-channel synchronous sound pickup.
By configuring the read state machine of the external storage controller to run continuously in a loop, the RD read pulse is used as the sampling clock for multiple digital microphones to acquire PDM single-bit data from multiple microphones in parallel. The signal is then processed by cascaded integral comb filtering and finite impulse response filtering to generate PCM samples.
Multi-channel synchronous latching sampling was implemented, which solved the problems of sampling asynchrony and phase deviation, reduced resource consumption and hardware costs, and improved the synchronization and stability of multi-channel microphone arrays.
Smart Images

Figure CN123123784A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio processing technology, and more particularly to an audio processing method, apparatus, electronic device, and program product. Background Technology
[0002] With the rapid popularization of audio technologies such as voice interaction, far-field sound pickup, and array noise reduction, the application scenarios of multi-channel digital microphone arrays are constantly expanding, such as professional recording equipment, smart homes, portable audio terminals, and security monitoring.
[0003] Currently, most mainstream digital microphones use single-bit pulse density modulation (PDM) bitstreams as their output. Their operation relies on a continuously provided high-precision sampling clock at the megahertz level. After latching the bitstream data via the clock edge, the digital microphone then performs multi-stage decimation filtering to recover standard pulse code modulation (PCM) audio data. Mid-to-high-end audio processing chips typically integrate dedicated digital microphone controllers, directly handling clock output, bitstream sampling, and pre-stage decimation filtering to ensure the stability of single-channel or a few-channel audio acquisition. However, many resource-constrained chips designed for low-cost scenarios often lack dedicated digital microphone controllers due to area and cost constraints. They only retain basic general-purpose peripherals such as external memory controllers, universal input / output interfaces, and serial interfaces, making it difficult to directly adapt to the acquisition needs of multiple PDM digital microphones.
[0004] For multi-channel digital microphone acquisition solutions without dedicated controller chips, PDM bitstream reading is often achieved by using software-flipped analog clocks via general-purpose input / output interfaces and serial interface polling sampling. These solutions are highly susceptible to processor real-time scheduling capabilities and interrupt jitter, making it difficult to stably generate the megahertz-level high-frequency sampling clock required for digital microphones. Furthermore, they cannot guarantee synchronous latching of up to 16 channels at the same sampling moment, easily leading to issues such as inter-channel phase shift and audio sampling distortion. Even in mainstream chips that integrate dedicated digital microphone controllers, the number of hardware paths typically limits support to only 2 to 4 audio acquisition paths, failing to meet the synchronous pickup requirements of multi-microphone arrays. Summary of the Invention
[0005] This invention provides an audio processing method, apparatus, electronic device, and program product.
[0006] In a first aspect, embodiments of the present invention provide an audio processing method, the method comprising: Configure an external storage controller to make the read state machine inside the external storage controller run continuously in a loop; the read state machine generates a read cycle every time it completes a full read state cycle, outputs a read control RD read pulse in the corresponding read cycle, and performs a multi-bit width read operation; wherein, the RD read pulse is multiplexed as the sampling clock of the multi-channel digital microphone DMIC; Within each read cycle, pulse density modulation (PDM) single bits output by multiple digital microphones (DMICs) are acquired in parallel to obtain a single multi-bit sample word. Acquire multiple sample words obtained from multiple consecutive read cycles and assemble them into a sample word set in chronological order; The sample word set is decomposed into channels according to the bit correspondence to obtain the PDM bit stream corresponding to each microphone channel.
[0007] In one possible implementation, the method further includes: The PDM bitstream of each microphone channel is decimated, filtered, and frequency response compensated to obtain the filtered signal. The step of performing decimation filtering and frequency response compensation processing on the PDM bitstream of each microphone channel to obtain the filtered signal includes: For each microphone channel's PDM bitstream, the same set of decimation filtering parameters and the same set of frequency response compensation parameters are used; wherein, the decimation filtering parameters are used for cascaded integral comb filtering, and the frequency response compensation parameters are used for finite impulse response filtering; The PDM bitstream of each microphone channel is decimated and downsampled by cascaded integral comb filtering, converting the high-frequency PDM bitstream into an intermediate audio signal with a preset audio sampling rate. Frequency response compensation is performed on the intermediate audio signal by finite impulse response filtering to correct the passband attenuation introduced by cascaded integral comb filtering, thereby obtaining the filtered signal of each microphone channel.
[0008] In one possible implementation, the method further includes: The filtered signal is subjected to DC component suppression and gain adjustment processing to generate pulse code modulation (PCM) samples corresponding to each microphone channel.
[0009] In one possible implementation, the step of decomposing the sampled word set into channels according to bit-to-bit correspondences to obtain the PDM bitstream corresponding to each microphone channel includes: Each multi-bit sample word in the sample word set is traversed in chronological order. Bits corresponding to the same microphone channel are extracted from each multi-bit sample word. The extracted bits are then concatenated in chronological order to form the PDM bit stream corresponding to the same microphone channel.
[0010] In one possible implementation, configuring the external storage controller to continuously cycle through its internal read state machine includes: Perform a one-time register configuration on the external storage controller to set the read timing parameters to match the PDM timing of the multiplexed DMIC; and, The chip select channel of the external storage controller is configured to use a fixed address access mode or an address automatic wraparound burst access mode, so that the read state machine can autonomously and continuously generate read cycles based on the configured read timing parameters after the configuration is completed.
[0011] In one possible implementation, acquiring the multi-bit sampled word obtained from multiple consecutive read cycles includes: Each time the external storage controller completes a read cycle, a direct memory access (DMA) request is generated, triggering the DMA to move the single multi-bit sample word generated in the read cycle to the circular buffer for storage in chronological order. The sample words corresponding to multiple consecutive read cycles are read from the circular buffer to form the sample word set.
[0012] In one possible implementation, after generating the PCM samples corresponding to each microphone channel, the method further includes: Determine audio storage requirements; Based on the audio storage requirements, a target audio processing method is determined, and the PCM samples corresponding to each microphone channel are processed according to the target audio processing method to obtain audio data; The target audio processing method is to perform WAV format encapsulation processing on the PCM sample value, or to perform MP3 compression encoding processing on the PCM sample value.
[0013] Secondly, embodiments of the present invention provide an audio processing device, the device including an external storage controller and a multi-channel digital microphone (DMIC); The external storage controller includes a read state machine; The RD output terminal of the external storage controller is electrically connected to the clock input terminal of the multi-channel digital microphone (DMIC) so as to use the RD read pulse as the sampling clock of the multi-channel digital microphone (DMIC). The data output terminals of the multi-channel digital microphone (DMIC) are electrically connected one-to-one with the data bus of the external storage controller. The audio processing device is used to perform the steps included in any of the methods in the first aspect.
[0014] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps included in any of the methods in the first aspect.
[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium including program code, which, when executed on an electronic device, causes the electronic device to perform the steps included in any of the methods in the first aspect.
[0016] Fifthly, embodiments of the present invention provide a computer program product, the computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform the steps included in any of the methods in the first aspect.
[0017] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: In this embodiment of the invention, an external storage controller is innovatively reused. By configuring the read state machine of the external storage controller to run continuously in a loop, the read cycle RD pulse generated by the hardware autonomously in a loop is used as the unified sampling clock for multiple digital microphones. Relying on the inherent high precision and high stability of hardware timing, the high-frequency clock at the megahertz level required by the microphones is stably output, completely eliminating the jitter and instability problems caused by traditional software analog clocks. This ensures the accuracy and stability of PDM bitstream sampling from the hardware timing level. At the same time, this invention uses the multi-bit width read mechanism of the external storage controller to acquire PDM single-bit data output from multiple digital microphones in parallel within a single read cycle and combine them into multi-bit sampling words. Then, by sampling in continuous read cycles, a set of sampling words is constructed and the bit correspondence is combined to complete the multi-channel decomposition, obtaining independent PDM bitstreams for each channel. This achieves true multi-channel simultaneous synchronous latching sampling, effectively solving the defects of asynchronous sampling and large phase deviation in traditional multi-channel acquisition schemes, and breaking through the bottleneck of limited channel number of dedicated controllers.
[0018] Furthermore, this invention fully reuses existing external storage controllers to complete multi-channel microphone synchronous acquisition, eliminating the need for dedicated digital microphone controllers and additional hardware costs. Stable and continuous multi-channel PDM data acquisition can be achieved simply through parameter configuration, greatly reducing the cost and resource consumption of audio acquisition on resource-constrained chips. It is suitable for various low-cost embedded audio acquisition scenarios and significantly improves the synchronization, stability, and versatility of multi-channel microphone array acquisition.
[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practice. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention, but do not constitute an undue limitation of the invention.
[0022] Figure 1 This is a flowchart illustrating an audio processing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the hardware connection between an external storage controller and a digital microphone, provided as an embodiment of the present invention. Figure 3 A schematic diagram of the read cycle timing after multiplexing is provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an audio processing method provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of an audio processing device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. Without conflict, the embodiments and features in the embodiments of this invention can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. The term "comprising" and any variations thereof in the specification and claims of this invention are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0024] In this embodiment of the invention, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.
[0025] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of the present invention, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of the present invention, and do not imply that the inventors have used or necessarily used such solutions.
[0026] Currently, digital microphone acquisition solutions in the industry have formed three mainstream implementation paths after long-term technological iteration. Each path has its own focus and obvious application boundaries. As multi-channel array sound pickup scenarios become more widespread, the common shortcomings of existing solutions are gradually becoming apparent.
[0027] The first type of solution integrates a dedicated digital microphone controller within the chip. This dedicated hardware directly handles sampling clock generation, bitstream latching, decimation filtering, and some formatting tasks. The technology for this type of solution is already quite mature. However, the number of channels supported by the controller is usually a fixed configuration at the factory, leaving very limited room for future functional expansion. Most chips on the market that integrate this type of controller only provide 2 or 4 interfaces. When the product needs to achieve array acquisition of 16 or more channels, the chip's native hardware resources will show a significant gap.
[0028] The second approach utilizes general-purpose input / output interfaces, serial interfaces, or software timer interrupts to simulate the timing of a digital microphone through code. The biggest advantage of this approach is its rapid deployment during the prototyping phase without requiring additional hardware modifications. However, the entire sampling cycle relies entirely on the processor's response speed and the system's interrupt scheduling capabilities. The continuity of the output clock and the synchronization consistency between multiple channels have inherent weaknesses, especially when the system load fluctuates significantly, making it impossible to guarantee stable synchronization and clock accuracy across multiple data streams. In practical applications, the processor often needs to handle other tasks simultaneously, such as display refresh, network communication, and file system access. The sampling cycle is easily interrupted by various unexpected tasks, and digital microphones are highly sensitive to such timing disturbances. A single interrupt timeout can cause local audio bitstream misalignment, and even if the system subsequently resumes normal operation, this erroneous data cannot be automatically repaired.
[0029] The third type of solution involves adding programmable logic devices, dedicated acquisition chips, or small coprocessors outside the main control chip. The external devices first complete multi-channel parallel sampling and preliminary buffering, and then the processed data is sent back to the main control chip. This type of solution can indeed support a larger channel scale, but it also directly brings a series of problems such as an increase in the number of devices, complex power supply design, increased wiring difficulty, and increased debugging workload. The design difficulty of system stability and the overall hardware cost will also increase simultaneously. The additional FPGA logic development work further raises the threshold for implementation. For projects that are cost-sensitive, have limited hardware board area, or have tight product development cycles, it is difficult to implement quickly.
[0030] From a theoretical perspective, these three approaches correspond to dedicated controller approaches, software simulation approaches, and external parallel acquisition approaches, respectively. While their technical focuses differ, they all essentially address the same core problem: how to stably generate a high-precision high-frequency clock and ensure that each data stream is latched at the correct sampling edge. In practical implementation, three core tasks must be completed: providing a continuous, stable, and accurate clock signal to the digital microphone; acquiring each pulse density modulation bitstream at fixed times; and restoring the high-frequency single-bit bitstream into directly usable pulse code modulation audio data. The digital microphone itself outputs a pulse density modulation bitstream, not directly playable or storeable audio sample values. Its normal operation relies entirely on the continuous drive of an external clock; the corresponding bit value is only output when each sampling edge arrives. This means that existing solutions can basically meet the needs of single-channel and dual-channel applications. However, when the number of channels increases to 8, 16, or even more, the importance of channel synchronization significantly increases. In typical application scenarios such as array pickup, echo cancellation, beamforming, and sound source localization, if the sampling reference of each channel cannot be kept completely consistent, the backend algorithm will continuously amplify these subtle timing errors, ultimately directly affecting the actual use effect of the product.
[0031] Multi-channel digital microphone acquisition is not simply a matter of repeatedly deploying 16 channels of a single-channel acquisition solution. It requires prioritizing the resolution of two core issues: a shared clock and synchronized sampling. Only when all channels operate on the same clock and data latching is completed within the same hardware sampling window can subsequent filtering operations and various algorithmic processing have a reliable underlying foundation. Furthermore, many existing solutions suffer from a disconnect between front-end sampling and back-end format processing. The front end only receives the pulse density modulation bitstream; subsequent tasks such as filtering, gain adjustment, pulse code modulation generation, and audio file storage need to be separated and processed separately. The more fragmented the business chain, the more difficult it is to troubleshoot and debug, and it hinders the development of a stable, mass-producible engineering solution.
[0032] In summary, the industry currently lacks an integrated solution that can fully utilize existing peripheral resources of the chip, ensure multi-channel synchronous sampling at the hardware level, and directly connect to subsequent audio processing links within the chip.
[0033] In view of this, this invention provides an audio processing method that innovatively reuses an external storage controller. By configuring the external storage controller's read state machine to run continuously in a loop, the read cycle RD pulse generated by the hardware's autonomous loop is used as the unified sampling clock for multiple digital microphones. Relying on the inherent high precision and stability of hardware timing, the method stably outputs the megahertz-level high-frequency clock required by the microphones, completely eliminating the jitter and instability problems caused by traditional software analog clocks. This ensures the accuracy and stability of PDM bitstream sampling from the hardware timing level. Simultaneously, within a single read cycle, this invention uses the external storage controller's multi-bit width read mechanism to parallelly acquire single-bit PDM data from multiple digital microphones and combine them into multi-bit sample words. Then, through continuous read cycle sampling, a set of sample words is constructed, and the bit correspondence is combined to complete multi-channel decomposition, obtaining independent PDM bitstreams for each channel. This achieves true multi-channel simultaneous synchronous latching sampling, effectively solving the defects of asynchronous sampling and large phase deviation in traditional multi-channel acquisition schemes, and breaking through the bottleneck of limited channel quantity of dedicated controllers.
[0034] Furthermore, this invention fully reuses existing external storage controllers to complete multi-channel microphone synchronous acquisition, eliminating the need for dedicated digital microphone controllers and additional hardware costs. Stable and continuous multi-channel PDM data acquisition can be achieved simply through parameter configuration, greatly reducing the cost and resource consumption of audio acquisition on resource-constrained chips. It is suitable for various low-cost embedded audio acquisition scenarios and significantly improves the synchronization, stability, and versatility of multi-channel microphone array acquisition.
[0035] In this embodiment of the invention, for the purpose of better introducing the invention, the technical names involved in the invention are described below: 1. External Memory Controller (NEMC): It can generate configurable asynchronous read / write timing and provide a parallel data bus; in this invention, it is used to output a sampling clock to synchronously complete the acquisition of multiple digital audio bitstreams.
[0036] 2. Digital Microphone (DMIC): Outputs a pulse density modulated digital bit stream synchronized with the clock. It generally requires a dedicated controller or equivalent timing logic to complete the sampling and decimation filtering process.
[0037] 3. Direct Memory Access (DMA) module: It can realize the continuous transfer of sampled data without the processor moving the data one by one. In this solution, the 16-bit sampled words read continuously are automatically moved to the circular buffer, reducing the CPU load and improving the CPU's data processing response speed.
[0038] 4. Pulse Density Modulation (PDM): The standard output bus format for digital microphones.
[0039] 5. Pulse Code Modulation (PCM): Audio data obtained after decimation and filtering of a pulse density modulated bitstream can be directly used for playback or storage.
[0040] 6. Cascaded Integrator-Comb 3 (CIC3): An algorithm for decimation filtering of high-frequency bitstreams output from digital microphones.
[0041] 7. Finite Impulse Response (FIR): Set after the output of the three-stage cascaded integral comb filter algorithm, it is used for tone correction and noise suppression.
[0042] 8. Circular buffer: A circular storage space used to buffer continuously sampled word streams for subsequent algorithms to read and process sequentially.
[0043] 9. Waveform Audio File Format (WAV): A standard file format for encapsulating audio sample data, used to store audio data along with related information such as sampling rate, number of channels, and data bit width.
[0044] 10. MP3: A standard format that uses lossy compression to encode audio data, which can reduce storage space and transmission bandwidth overhead while preserving the main audio information.
[0045] 11. Read control signal: Output by the external storage controller in each read cycle, serving as the sampling clock for the digital microphone.
[0046] 12.16-bit data bus: A single read operation can acquire the level status of 16 peripheral devices in parallel and receive the current bit value of 16 audio bit streams. Each bit of the bus corresponds to one acquisition channel.
[0047] 13. Decimation Filtering Module: This module converts the high-frequency pulse density modulated bitstream output from the digital microphone into usable pulse code modulated audio data with a low sampling rate. The conversion process filters out high-frequency noise and invalid spectral components, preserving the effective audio bandwidth. The three-stage cascaded integral comb filtering and decimation are optional algorithms and can be replaced with others depending on actual needs.
[0048] 14. Format Conversion Module: Performs DC suppression, gain adjustment, and other processing on the audio data after extraction and filtering, and organizes it into a standard waveform audio file format that the system and algorithm can recognize; it can also adapt the sampling rate and number of channels according to the target encoding parameters to generate target compressed audio files such as MP3 for storage and transmission.
[0049] To further illustrate the audio processing method provided in the embodiments of the present invention, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present invention provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of the present invention. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing application environment).
[0050] The following combination Figure 1 The flowchart shown illustrates the audio processing method in this embodiment of the invention. The specific implementation process of this method is as follows: Step 101: Configure the external storage controller so that the read state machine inside the external storage controller runs continuously in a loop; the read state machine generates a read cycle every time it completes a full read state cycle, outputs the read control RD read pulse in the corresponding read cycle, and performs a multi-bit width read operation; wherein, the RD read pulse is multiplexed as the sampling clock of the multi-channel digital microphone DMIC.
[0051] In this embodiment of the invention, the conventional read operation behavior of the general external storage controller is reinterpreted as a fixed-beat multi-channel audio synchronous sampling behavior. Without the need to add a new dedicated audio acquisition peripheral, the high-synchronization multi-channel digital microphone (DMIC) array acquisition is achieved through the functional multiplexing and reconstruction of hardware interfaces and timing.
[0052] In traditional scenarios, the external memory controller is only used for parallel access to external static memory (SRAM / PSRAM / NOR Flash, etc.), and automatically generates read control pulses and data latch timing that conform to memory specifications based on the on-chip hardware state machine.
[0053] In this embodiment of the invention, based on a full understanding of the timing characteristics of the external storage controller (taking NEMC, i.e., a general-purpose external storage controller, as an example), its read control signal, data bus, and DMA transfer path are functionally reorganized so that it can simultaneously undertake three core audio acquisition capabilities: sampling timing generator, parallel data sampler, and high-speed data transfer entry.
[0054] The core multiplexing logic is as follows: The external memory controller can continuously generate RD read control pulses through register configuration. In this invention, these pulses are no longer merely used as "memory read enable" signals, but are directly multiplexed as the common sampling clock for all digital microphones. Synchronously multiplexed with the clock is the controller's multi-bit parallel data bus, which can be 8 bits, 16 bits, etc. The following embodiments of this invention will use 16 bits as an example. Whenever an RD read pulse arrives, the 16 digital microphones output 1 bit of PDM (Pulse Density Modulation) data at the current sampling moment to data bits D0~D15 respectively. The controller then completes parallel latching of the 16 bits of data within the same read data window. Thus, a typical 16-bit memory read cycle is directly converted into a synchronous sampling cycle for the 16 digital microphones.
[0055] From the perspective of data stream processing, the interface acquires a continuous 16-bit parallel sampling word. Each bit of the word corresponds to one bit of the PDM sampling result of one DMIC channel. Subsequently, the sampling word only needs to be split into 16 independent PDM bit streams bit by bit, which can then be sent to the back-end PDM filtering and audio format conversion stage to complete the complete acquisition link of multiple audio channels.
[0056] To achieve the aforementioned parallel sampling multiplexing, the hardware connection adopts a fully parallel topology structure of "collinear clock and independent bit lines," as shown in the connection diagram below. Figure 2 As shown. The specific connection rules are as follows: Clock signal multiplexing: The RD read control signal of the external storage controller is used as the output terminal and directly connected to the clock input terminal of the 16-channel digital microphone. Since the 16 microphones share the same source clock, all channels are triggered for sampling and data output at the same clock edge. This structure naturally has sub-microsecond-level inter-channel synchronization accuracy, which is fully suitable for acquisition scenarios such as microphone arrays that have extremely high requirements for multi-channel synchronization. Parallel data signal access: The PDM data output terminals of the 16-channel digital microphone are independently connected to the D0 to D15 bit lines of the parallel data bus of the external storage controller. The data input of each DMIC is not serial, but independent channels and parallel access. In any read cycle, the D[15:0] bus will simultaneously present 1 bit PDM data output by each of the 16 microphones; the controller only needs to complete one 16-bit width read operation to complete the synchronous sampling of 16 single-bit PDM data.
[0057] The key point of this connection method is that "one bit line corresponds to one independent channel", which is different from the traditional serial audio interface mode of transmitting multiple data on the same line through time-division scheduling. The advantage of the fully parallel structure is that the channel mapping relationship is intuitive and fixed, all channel data latching actions are completely unified, the back-end data bit-by-bit splitting logic is extremely simple, and it fundamentally avoids the deviation in sampling time between channels caused by time-division transmission scheduling.
[0058] The processing logic for unused signals: The address lines and chip select signal lines of the external memory controller do not undertake core audio functions in this invention. In actual implementation, the chip select space used can be fixedly mapped to a certain continuously readable address range, and the address lines can maintain a fixed address access or repeated wraparound access mode. The core objective is to ensure that the read state machine inside the controller continues to operate, thereby stably and continuously outputting the RD read pulse sequence.
[0059] If the connected digital microphone has specific requirements for the direction of the input clock edge, such as sampling on the rising edge and outputting data on the falling edge, or sampling on the falling edge and outputting data on the rising edge, it can be flexibly adapted in three ways: First, adjust the read timing parameters of the external memory controller (NEMC) to change the phase of the RD pulse and the position of the data sampling edge; second, adjust the data sampling phase offset inside the controller; third, add simple inverting logic outside the clock link without modifying the core parallel connection topology.
[0060] In this embodiment of the invention, the external memory controller operates based on its on-chip clock and generates complete read cycles through a hardware state machine. Therefore, the RD read signal is not generated by software successively toggling I / O levels, but rather by a regular pulse sequence automatically output according to register configuration. This hardware characteristic makes it more suitable as a clock source for a digital microphone than software-simulated I / O toggling or general-purpose timer PWM output.
[0061] Digital microphones require a continuous, regular, and stable low-jitter clock for proper operation, characteristics that an external memory controller possesses. Therefore, by configuring the controller to continuously initiate read operations, the RD read signal will maintain a fixed, reciprocating rhythm, eliminating the need for processor intervention in every clock cycle. The processor only needs to complete the register configuration once; subsequent sampling clock generation and data latching can be handled autonomously by the hardware, resulting in near-zero CPU utilization.
[0062] In traditional memory scenarios, read timing parameters include address setup time, address hold time, read enable pulse width, data hold time, and bus turnaround time, used to meet the read / write timing requirements of external memory devices. In this solution, however, these parameters are reused to adjust the period, width, and phase relationship of the RD read pulse. The multiplexed read cycle timing and parameter definitions are as follows: Figure 3 As shown. Among them, Figure 3 The core timing parameters are defined as follows: CKO: Internal operating clock of the external memory controller NEMC. CS#: Chip select pin of the external memory controller NEMC (not used in this invention). RD#: Read signal output pin of the external memory controller NEMC (used to simulate the clock output of the dmic in this invention). A[15:0]: Address output pin of the external memory controller NEMC (not used in this invention). D[15:0]: Data acquisition pin of the external memory controller NEMC (used for data acquisition of the dmic in this invention). Tras: Setup time of the RD read signal. T1: Fixed delay (single cycle) before RD read access. Toe: Hold time for RD read data.
[0063] By combining and adjusting the above parameters, a clock configuration that meets the requirements of the target DMIC can be flexibly configured. For example, if the read enable time is set to be shorter, the duty cycle of the output RD pulse will be narrower; if the bus turnaround time is set to be larger, the interval between two adjacent read pulses will be longer, resulting in a lower final output sampling clock frequency. During parameter configuration, two core indicators should be prioritized: first, the output clock frequency must meet the PDM clock input range requirements of the DMIC; second, the clock edge position must be stable and the jitter repeatable. For digital microphones, small deviations in clock frequency can usually be compensated for by adjusting the multiplier of the backend PDM decimation filter, but large clock edge jitter will directly disrupt the stability of the microphone's internal modulation logic and output bitstream, leading to sampling failure.
[0064] In this embodiment of the invention, after completing the hardware connection and timing parameter adaptation, the controller can enter a continuous multi-channel sampling state through initialization configuration. Specifically, the external storage controller needs to be specifically configured to enable its internal read state machine to enter a continuous loop operation state. This read state machine is a native hardware state machine within the external storage controller, originally used to automatically complete the entire process of read operations according to the memory protocol, including a series of timing actions such as address output, chip select pull-low, read enable output, data sampling, and bus release. In this invention, the continuous operation of this read state machine no longer serves the reading of memory data, but is instead redefined as a "timing generator".
[0065] In this embodiment of the invention, each time the read state machine completes a full read state cycle, a standard read cycle is generated. Within each read cycle, the state machine outputs read control RD pulses according to a predetermined timing sequence and performs a multi-bit width read operation on the data bus. That is, in this invention, the RD pulse is no longer merely used as a memory read enable signal, but is multiplexed as the common sampling clock for multiple digital microphones (DMICs). Each RD pulse corresponds to a sampling moment, and each read cycle corresponds to a complete sampling cycle, thereby transforming the memory read operation into an audio sampling action.
[0066] In some implementations, configuring an external storage controller to continuously cycle its internal read state machine specifically includes the following two configuration operations: The first aspect is the one-time register configuration and timing parameter matching.
[0067] A one-time register configuration is performed on the external storage controller to set the read timing parameters to match the PDM timing requirements of the multiplexed DMIC. "One-time" here means that the configuration operation only needs to be performed once during the system initialization phase. After configuration, the external storage controller can operate autonomously without requiring software intervention from the processor in each sampling cycle.
[0068] Read timing parameters include, but are not limited to, address setup time, address hold time, read enable width, data hold time, and bus turnaround time. In traditional memory applications, these parameters are used to match the read / write timing requirements of the memory device to ensure that data can be read correctly. In this invention, however, these parameters are reused to adjust the period, width, and phase relationship of the RD read pulse so that its final clock pattern is completely consistent with the clock input requirements of the target digital microphone.
[0069] Specifically, by adjusting the read enable width parameter, the duration of the high or low level of the RD pulse can be controlled, thereby matching the clock duty cycle requirements of the digital microphone; by adjusting the bus turnaround time parameter, the interval between two adjacent RD pulses can be controlled, thereby adjusting the overall sampling clock frequency; by adjusting the data setup and hold time parameters, the phase relationship between the RD edge and the data output edge can be ensured to meet the output timing constraints of the digital microphone, guaranteeing the reliability of data latching.
[0070] When configuring the above parameters, this invention prioritizes two core indicators: first, the overall frequency of the RD signal meets the frequency requirements of the PDM clock; and second, the edge position of the RD signal is stable and repeatable. For digital microphones, small frequency deviations can usually be accommodated by adjusting the decimation filter rate in the subsequent stage. However, if there is significant jitter at the clock edge, it will directly disrupt the stability of the bitstream sampling, leading to bitstream misalignment or even data loss. Since the RD signal in this invention originates from a hardware state machine rather than software simulation, its edge jitter is naturally lower than that of a software flip-flop scheme, which is a crucial basis for the stable operation of this solution.
[0071] In an exemplary embodiment of the present invention, the RD read clock frequency can be configured to 2.4MHz, and combined with a subsequent 50x decimation filter to obtain a standard PCM sampling rate of 48kHz; alternatively, the RD read clock can be configured to 3.072MHz, and combined with a 64x decimation, also outputting a 48kHz sampling rate. Both of these parameter combinations are common configurations in the audio field, facilitating direct interface with subsequent algorithms or standard audio file formats. If the target sampling rate is not 48kHz, but other audio sampling rates such as 16kHz, 24kHz, or 32kHz, it can also be achieved by adjusting the combination of the RD frequency and the decimation factor accordingly. For example, when the application scenario is only for speech recognition, a 16kHz sampling rate is usually sufficient; in this case, the front-end PDM clock frequency can be appropriately reduced or the decimation factor increased to achieve a lower system processing burden.
[0072] The second aspect is address access mode configuration and continuous read cycle generation.
[0073] In this embodiment of the invention, the address access mode of the chip select channel corresponding to the external storage controller is configured as a fixed address access mode or a burst access mode with automatic address wraparound. The purpose of this configuration is to enable the read state machine to autonomously and continuously generate read cycles based on the aforementioned pre-set read timing parameters after configuration, without requiring the processor to initiate read instructions cycle by cycle.
[0074] In the implementation using the fixed address access mode, the external memory controller is configured to continuously access the same fixed memory address. Since the chip select space corresponding to this address is set to a continuously readable state, each read operation completes the entire read cycle timing normally, and the RD pulse is output normally. Functionally, repeatedly reading the same address does not generate new memory data, but for this invention, the focus is not on what data is read, but on the continuous operation of the read state machine and the continuous output of the RD pulse. As long as the read state machine continuously executes the read operation of the same address, the RD signal will be continuously output at a fixed beat, thereby providing a continuous and stable clock source for the digital microphone.
[0075] In the implementation of the burst access mode using automatic address wraparound, the external memory controller is configured to perform continuous burst read operations within an address range. When the end of the address range is reached, the address automatically wraps back to the beginning of the range, forming a cyclic read. The advantage of this mode is that it can leverage the burst transfer capability of the external memory controller to further reduce the processor's intervention frequency. The cyclic wraparound of the address within the preset range ensures that the read operation can continue indefinitely, and the RD pulse sequence also extends indefinitely.
[0076] Whether using a fixed address access mode or a burst access mode with automatic address wrap-around, the core design principle remains the same: leveraging the external memory controller's native address generation and read state machine mechanism, the hardware maintains the continuity of read operations, thereby continuously outputting regular RD clock pulses without consuming processor resources. Once configured, the entire sampling clock generation process is completed automatically by the hardware; the processor only needs to handle initial configuration and subsequent data transfer and processing, without intervening in the sampling clock itself. This feature fundamentally solves the problem of sampling clock interference caused by system load in software simulation schemes.
[0077] After the above configuration is completed, the external storage controller enters the continuous sampling mode: the read state machine operates cyclically according to the configured timing parameters, the RD pin continuously outputs a regular pulse sequence that matches the clock requirements of the digital microphones, and the 16-bit data bus simultaneously latches 1 bit of PDM data output by each of the 16 digital microphones within the sampling window of each read cycle. Thus, a normal 16-bit memory read cycle is completely transformed into a synchronous sampling cycle of 16 digital microphones.
[0078] Step 102: In each read cycle, acquire the pulse density modulation (PDM) single bit output by each of the multiple digital microphones (DMICs) in parallel to obtain a single multi-bit sample word.
[0079] In this embodiment of the invention, the aforementioned parallel sampling action is automatically completed at the hardware level by the read state machine built into the external storage controller, without processor intervention throughout the sampling process. Specifically, when the read state machine enters the preset data sampling window for each read cycle, the 16 digital microphones, under the unified synchronous triggering of the current RD clock edge, drive their latest generated PDM modulation single-bit results to the independent data bit lines corresponding to D0 to D15 respectively, and maintain sufficient setup and hold times under hardware timing constraints to ensure that the data of each channel is stable and valid within the sampling window; subsequently, the external storage controller performs a unified parallel latching operation on the 16-bit data bus within the same read window according to the pre-configured data sampling phase, packaging the single-bit data on the 16 independent bit lines into a complete multi-bit sampling word at once, and storing it in the data register inside the controller.
[0080] The aforementioned parallel acquisition mechanism of "output triggered by the same clock edge and parallel latching within the same sampling window" fundamentally guarantees strict synchronization of sampling times among multiple channels at the hardware timing level, completely avoiding the inherent inter-channel phase offset problem in traditional serial time-division acquisition mode. In subsequent processing, only bitmasking and shifting operations need to be performed on the multi-bit sample word to quickly extract multiple independent PDM bit streams corresponding one-to-one with the hardware bit lines, without the need for additional complex channel data alignment and timestamp calibration operations.
[0081] Step 103: Obtain the multi-bit sampled words obtained from multiple consecutive read cycles, and assemble them into a sampled word set in chronological order.
[0082] In this embodiment of the invention, the direct memory access (DMA) request generated by the external storage controller after each read cycle triggers the DMA to move the single multi-bit sample word generated in the read cycle to the circular buffer for storage in chronological order; from the circular buffer, sample words corresponding to multiple consecutive read cycles are read to form a sample word set.
[0083] In this embodiment of the invention, considering that the front-end PDM sampling cycle is high, if the CPU is involved in the transfer of each 16-bit sample word, it will cause the CPU to experience continuous and frequent interrupts and a large amount of memory access pressure, which will seriously squeeze the algorithm processing bandwidth. Therefore, the present invention adopts the direct memory access (DMA) method to complete the automatic transfer of sampled data.
[0084] In this embodiment of the invention, the external storage controller generates a DMA request every time a read cycle is completed, triggering the DMA controller to automatically move the single multi-bit sample word generated in the read cycle to a pre-allocated circular buffer in the system memory for storage in chronological order; the DMA only undertakes the pure data transfer function and does not participate in the generation and control of the sampling clock, so it will not cause any interference or damage to the front-end sampling rhythm.
[0085] In continuous acquisition scenarios, the aforementioned circular buffer is a crucial intermediate layer connecting the high-speed sampling front-end and back-end processing algorithms. On one hand, it transforms the high-speed, continuous, and uninterrupted front-end sampling word stream into a buffered data format that can be read in blocks by back-end filtering, noise reduction, and audio encapsulation algorithms, effectively decoupling the sampling and processing processes on the time axis. As long as the depth of the circular buffer is designed reasonably, the front-end sampling hardware and the back-end processing algorithms do not need to strictly synchronize to complete their respective tasks at the same time, allowing the system to achieve good throughput elasticity. On the other hand, in addition to storing the original sampled words, the circular buffer also plays an important role in "absorbing processing jitter." Since the actual execution time of back-end filtering operations and file encapsulation operations will vary with system load fluctuations and cannot remain constant, the circular buffer can provide the necessary buffer margin to prevent back-end sampling from being blocked immediately when the subsequent processing occasionally slows down, thereby ensuring the continuous and stable operation of the entire acquisition link.
[0086] In this embodiment of the invention, after the DMA completes the sample word transfer, sample words corresponding to multiple consecutive read cycles can be read from the circular buffer as needed to form a sample word set for subsequent PDM demodulation and digital signal processing.
[0087] Step 104: Decompose the sampled word set into channels according to the bit correspondence to obtain the PDM bit stream corresponding to each microphone channel.
[0088] In this embodiment of the invention, the circular buffer does not store the complete audio frame of a certain channel, but a string of 16-bit parallel sampling words arranged strictly in time order. Each sampling word has a corresponding bit that is mapped one-to-one with the physical bit line of the 16 digital microphones. That is, the nth bit corresponds to the PDM single bit output by the nth DMIC at that sampling time.
[0089] In this embodiment of the invention, each multi-bit sample word in the sample word set is traversed in chronological order, bits corresponding to the same microphone channel are extracted from each multi-bit sample word, and the extracted bits are concatenated in chronological order to form a PDM bit stream corresponding to the same microphone channel.
[0090] In other words, when reading data from the circular buffer, the corresponding channel bits can be quickly extracted from each sample word through bitmasking and shifting operations, and then concatenated in time order to reconstruct the independent PDM bitstreams. This splitting process involves only simple bit operations, requiring no channel rearrangement or timestamp alignment, resulting in extremely low computational overhead. After splitting, the bitstream of each channel is independently fed into its corresponding decimation filtering link for subsequent downsampling and frequency response correction processing, thereby achieving parallel decoding of 16 audio channels.
[0091] In this embodiment of the invention, after step 104, the PDM bitstream of each microphone channel can be subjected to decimation filtering and frequency response compensation processing to obtain the filtered signal.
[0092] In some implementations, the filtered signal may be determined using, but is not limited to, the following steps: Step A: For the PDM bitstream of each microphone channel, use the same set of decimation filtering parameters and the same set of frequency response compensation parameters; wherein, the decimation filtering parameters are used for cascaded integral comb filtering, and the frequency response compensation parameters are used for finite impulse response filtering.
[0093] In this embodiment of the invention, to simplify the hardware implementation and software maintenance of multi-channel parallel processing, all 16 microphone channels adopt the same set of filtering parameter configurations, that is, all channels share the same set of decimation filtering parameters and the same set of frequency response compensation parameters. Specifically, the decimation filtering parameters correspond to the core configuration of cascaded integrator comb (CIC) filtering, including key parameters such as decimation factor, order, and differential delay, which are used to determine the downsampling ratio and overall transmission characteristics of the PDM bitstream from high frequency to audio baseband; the frequency response compensation parameters correspond to the tap coefficients and order of finite impulse response (FIR) filtering, which are used to accurately correct the frequency response after CIC filtering. Adopting a unified parameter scheme for all channels can significantly reduce the coefficient storage space, as multiple channels can complete the calculation by sharing the same filter coefficient table; on the other hand, it also ensures that each channel maintains a high degree of consistency in amplitude and phase response, which is beneficial to ensuring the amplitude and phase consistency of the multi-channel array and providing a good input foundation for subsequent audio algorithms that rely on channel matching, such as beamforming and sound source localization.
[0094] Step B: Perform decimation downsampling on the PDM bitstream of each microphone channel using a cascaded integral comb filter, converting the high-frequency PDM bitstream into an intermediate audio signal with a preset audio sampling rate.
[0095] In this embodiment of the invention, the cascaded integrator-comb (CIC) filter, as the first stage of the decimation link, undertakes the main downsampling task. The PDM bitstream output from a digital microphone typically operates at a high sampling rate in the megahertz range, with a bit width of only 1 bit, making it unsuitable for direct use in conventional audio processing. The CIC filter, utilizing its simple structure consisting only of a cascaded integrator and comb, without the need for multipliers, performs high-rate decimation on each PDM bitstream, rapidly downsampling the high-frequency 1-bit data stream to a preset audio sampling rate of tens of kilohertz, outputting a multi-bit intermediate audio signal. For 16 parallel channels, the advantages of the CIC structure are particularly prominent—its operations require only adders and delay units, without consuming hardware multiplier resources. The computational load and memory usage increase gradually when implemented in parallel, making it ideal for deployment on resource-constrained embedded controllers. After CIC decimation, each channel obtains a multi-bit intermediate sample value consistent with the preset audio sampling rate. However, due to the inherent sinx / x amplitude frequency characteristics of CIC, the signal still exhibits a noticeable high-frequency droop within the passband, requiring further compensation.
[0096] Step C: Perform frequency response compensation on the intermediate audio signal using finite impulse response filtering to correct the passband attenuation introduced by cascaded integral comb filtering, and obtain the filtered signal for each microphone channel.
[0097] In this embodiment of the invention, the Finite Impulse Response (FIR) filter, as the second stage of the decimation link, is specifically designed to correct the passband attenuation introduced by the CIC filter. Because the amplitude-frequency response of the CIC filter exhibits a sinx / x-shaped roll-off characteristic, there is a significant amplitude droop at the high-frequency end within the passband, resulting in an uneven frequency response. Direct use of this filter would affect the high-frequency fidelity of the audio signal. The FIR compensation filter performs sample-by-sample convolution operations on the intermediate audio signal output by the CIC filter according to pre-designed compensation coefficients, accurately compensating for the amplitude attenuation in the high-frequency band, making the frequency response within the entire audio passband flatter, while further suppressing out-of-band noise and image spectrum components generated during downsampling. In the 16-channel parallel implementation, each channel shares the same set of FIR coefficients, independently performing filtering operations on its own intermediate audio signal, outputting a filtered signal with corrected frequency response. This signal possesses a standard audio sampling rate and a flat passband characteristic, and can be directly fed into subsequent post-processing stages such as DC suppression and gain adjustment, ultimately forming PCM audio samples that meet application requirements.
[0098] In this embodiment of the invention, after obtaining the filtered signal, the filtered signal can be further processed by DC component suppression and gain adjustment to generate pulse code modulation (PCM) samples corresponding to each microphone channel.
[0099] In this embodiment of the invention, after CIC extraction and FIR compensation filtering, the signal of each channel further undergoes two post-processing stages: DC component suppression and gain adjustment, ultimately forming a standard PCM sample output. The DC suppression stage uses a first-order or higher-order high-pass filter structure to filter out low-frequency DC offsets introduced by device bias, power supply noise, and environmental temperature drift, preventing invalid bias from being amplified during subsequent gain amplification and thus crowding out the effective dynamic range. The gain adjustment stage plays a dual role: firstly, it performs independent channel gain correction for the sensitivity differences between different microphone channels, ensuring that the 16 PCM outputs have a consistent amplitude scale at the same sound pressure level, improving the channel consistency of the array processing; secondly, it adjusts the output level to a target range suitable for encoding, storage, or algorithm input through overall gain control, preventing both excessively small signals that waste dynamic range and excessively large signals that cause clipping distortion.
[0100] After all processing is complete, each channel outputs a stable PCM sample value, using a 16-bit fixed-point format by default to meet the needs of most voice acquisition products.
[0101] In this embodiment of the invention, after CIC extraction, FIR compensation, DC suppression, and gain adjustment are all completed, each channel can output a stable PCM sample value. This sample value defaults to a 16-bit fixed-point format to meet the general needs of most voice acquisition products. That is, a 16-bit width is sufficient to cover the dynamic range requirements of typical application scenarios such as voice calls, voice recognition, and conference recording, and is naturally compatible with mainstream audio codec frameworks and storage formats. For high-precision applications requiring a larger dynamic range, such as professional audio recording and acoustic testing, a 24-bit or 32-bit fixed-point format can be retained depending on the actual needs of subsequent modules. By retaining more low-bit details, a better signal-to-noise ratio and a wider dynamic range can be obtained, avoiding additional distortion introduced due to quantization truncation accumulation during subsequent multi-stage processing. The output bit width can be flexibly set through register configuration during system initialization without modifying the hardware architecture.
[0102] In this embodiment of the invention, the PCM sample value serves as both the final output data of the front-end acquisition link and the intermediate standard format connecting the front-end sampling and the back-end encapsulation, acting as a crucial bridge between them. The front-end sampling and filtering links do not need to be directly adapted to specific file formats such as WAV or MP3; they only need to stably output standard PCM samples. Similarly, the back-end file encapsulation, encoding compression, or speech algorithm modules do not need to re-understand and process complex PDM bitstreams; they can all operate using the PCM format as their input. This layered architecture centered on PCM achieves excellent decoupling between the front-end sampling and back-end processing parts, allowing them to evolve and be debugged independently, significantly reducing the overall system design complexity and maintenance costs.
[0103] In this embodiment of the invention, after generating the PCM sample values corresponding to each microphone channel, the audio storage requirements can also be obtained; based on the audio storage requirements, the target audio processing method is determined, and the PCM sample values corresponding to each microphone channel are processed according to the target audio processing method to obtain audio data.
[0104] The target audio processing methods include two optional paths: one is to perform WAV format encapsulation processing on PCM samples, suitable for scenarios where audio quality is prioritized and link verification is easy. WAV files have a simple and transparent structure, with key parameters such as sampling rate, bit width, number of channels, and data length explicitly written in the file header. After exporting, the waveform, spectrum, and time domain alignment can be directly viewed on the host computer, facilitating R&D debugging and effect verification. The other method is to perform MP3 compression encoding processing on PCM samples, suitable for scenarios sensitive to storage space or transmission bandwidth. MP3, as a lossy compression format, can significantly reduce data size within an acceptable range of audio quality loss, making it suitable for applications such as long-duration recording and network transmission.
[0105] WAV is essentially a standard format encapsulation of PCM, while MP3 further performs frame division, frequency domain analysis, quantization, and bitstream packaging on top of PCM. Both use PCM as a unified input, so as long as the PCM layer output is stable, the output layer can freely switch between WAV and MP3 according to project requirements. PCM data can also be directly used as input for subsequent audio algorithms such as speech recognition, beamforming, and sound source localization, providing good configuration flexibility.
[0106] To facilitate understanding of the resource requirements and parameter relationships of this invention, a configuration example is given below.
[0107] Assuming the front end uses 16 digital microphones and the target output is 48kHz, 16-bit PCM, you can choose a 3.072MHz PDM clock with 64x decimation, or you can choose a 2.4MHz PDM clock with 50x decimation.
[0108] In this configuration, a 16-bit sample word is acquired each PDM clock cycle. Taking 3.072MHz as an example, the front end generates 3.072 million 16-bit parallel sample words per second; however, these sample words are not the final 16-channel PCM, but rather the raw sampling results of 16 single-bit streams. After decimation by 64 times, each channel ultimately yields a 48kHz PCM sample value. If the output format is 16-bit, 16-channel, the total PCM output data rate is approximately 48000×16×16 bit / s, or 12.288 Mbit / s, which is approximately 1.536 MB / s.
[0109] If the application target changes from 16-channel array recording to speech recognition, the output requirements can be significantly reduced. For example, some channels can be retained, the sampling rate reduced, or only a single-channel mix can be output. This not only reduces buffer pressure but also reduces file system writes and post-encoding pressure.
[0110] See Figure 4 The diagram shown is a schematic representation of an audio processing method according to an example of the present invention.
[0111] exist Figure 4 In this invention, after acquiring the multi-channel pulse density modulation (PDM) bitstream, the on-chip back-end signal processing link executes as follows: First, the PDM bitstream input from the digital microphone (DMIC) is received and sent to the three-stage cascaded integral comb filter (CIC3) module to complete decimation and sampling rate conversion; the signal processed by the CIC3 module is then sent to the finite impulse response (FIR) filter to complete sound quality correction and noise suppression, further smoothing the audio response; subsequently, it enters the gain adjustment module to complete the volume amplitude adjustment, obtain effective pulse code modulation (PCM) sample values, and output standard audio sample values for subsequent packaging and encoding.
[0112] After obtaining the PCM audio data, the system can proceed to the format conversion branch: on one hand, it can perform WAV encapsulation in waveform audio file format, generating a WAV file by writing to the RIFF / fmt / data structure; on the other hand, it can perform MP3 encoding, performing lossy compression encoding on the PCM data to generate an MP3 file. It should be understood that... Figure 4 The CIC and FIR filtering architectures, WAV and MP3 output formats shown are merely exemplary embodiments and do not constitute a limitation on the scope of protection of this invention. Those skilled in the art can replace the filtering algorithm, add custom post-processing modules, and select other audio output formats according to actual product requirements.
[0113] In summary, compared to existing multi-channel digital microphone acquisition solutions, this invention constructs an on-chip end-to-end acquisition architecture by reusing the chip's native peripheral hardware resources. This architecture offers significant technical advantages in hardware adaptability, channel synchronization accuracy, operational stability, channel scalability, engineering mass production versatility, and scenario adaptability, as detailed below: 1. Reuse of native hardware resources, low modification cost, and short implementation cycle.
[0114] This invention fully utilizes the existing external memory controller hardware resources of the chip, eliminating the need for a dedicated digital microphone (DMIC) controller, programmable logic devices, dedicated acquisition chips, coprocessors, or other external acquisition hardware. For mature chip platforms that have already completed tape-out, this invention can quickly integrate multi-channel digital microphone synchronous acquisition functionality with minimal changes to the existing hardware architecture. This significantly reduces the hardware modification costs, material costs, and project development timelines associated with adding audio acquisition capabilities, and adapts to the upgrade and iteration needs of various existing mature chip platforms.
[0115] 2. Hardware-level unified timing sampling, high channel synchronization accuracy, suitable for high-end array acoustic scenarios.
[0116] This invention leverages the inherent timing characteristics of the built-in hardware state machine in the external storage controller to construct a globally unified sampling timing reference. Multiple digital microphones can share the same read control signal as the sampling trigger clock, and the pulse density modulation (PDM) bitstream data from all channels can be latched in parallel within the same hardware read window. This ensures, from a hardware perspective, that all acquisition channels have a completely consistent sampling time reference, thoroughly avoiding the inter-channel phase shift and timing deviation problems inherent in traditional software time-division multiplexing and multi-interface serial acquisition schemes. This high-precision amplitude and phase consistency characteristic perfectly adapts to high-end array audio applications with stringent requirements for channel synchronization, such as beamforming, array noise reduction, sound source localization, and acoustic imaging, possessing irreplaceable core technological advantages.
[0117] 3. Purely hardware-driven timing mechanism with strong anti-interference capabilities and stable sampling operation.
[0118] The sampling clock of this invention is independently generated and maintained by the internal hardware state machine of the external storage controller, completely independent of analog timing methods such as CPU software toggling I / O ports or timed interrupt polling. Even when various tasks such as display refresh, network communication, file system access, and intensive interrupt handling are running synchronously in the background, the clock beat and data latching logic of the front-end acquisition will not be affected by task preemption, interrupt delay, or system scheduling jitter. It can maintain a continuous, uniform, and stable sampling rhythm for a long time, eliminating problems such as audio bitstream misalignment, data frame loss, and timing abnormalities that are prone to occur in software simulation schemes, and ensuring the long-term stable operation of the audio acquisition link.
[0119] 4. The architecture is flexible and expandable, with strong channel adaptability and a good balance between cost and performance.
[0120] The parallel acquisition architecture of this invention possesses extremely strong scalability and is not limited to a fixed number of channels. Based on the chip's data bus width characteristics and relying on a unified parallel sampling principle, an 8-bit data bus can achieve 8-channel synchronous acquisition, and a 16-bit data bus can achieve 16-channel synchronous acquisition. By expanding the chip's parallel read interface and widening the data bus width, it can be flexibly expanded to 32, 64, or even more channels for synchronous acquisition. Simultaneously, this solution eliminates the need for complex external data splicing logic and multi-interface time-division cascading scheduling architecture. It achieves large-scale synchronous acquisition of multiple channels with extremely low hardware costs, balancing hardware simplicity with high-channel acquisition performance, resulting in an excellent cost-performance ratio.
[0121] 5. The on-chip fully closed-loop integrated link has a high degree of integration and low difficulty in debugging and mass production.
[0122] This invention constructs a fully closed-loop on-chip processing chain from signal acquisition, data transmission, signal processing to audio output. Raw PDM sampling data is automatically moved to a circular buffer via the Direct Memory Access (DMA) module. Subsequent core processing steps such as decimation filtering, gain control, PCM format processing, and audio encapsulation are all completed within the chip, significantly reducing reliance on external components and avoiding the problems of fragmented front-end sampling and back-end processing chains and complex cross-chip interactions among multiple components in traditional solutions. Furthermore, the sampling clock path, data latch path, and signal processing path of this invention are all fixed hardware pathways within the chip. Mass production debugging and fault diagnosis can achieve layered and modular localization, solving the pain points of scattered fault points, high debugging difficulty, and poor mass production stability in external multi-component solutions. This results in extremely high engineering feasibility and mass production reliability.
[0123] 6. The software and hardware are highly customizable, with wide adaptability to various scenarios and strong general compatibility.
[0124] The core acquisition architecture of this invention is highly versatile, with a fixed hardware structure that can be adapted to different product requirements through pure software configuration. At the hardware level, it is not limited to standard external storage controller peripherals; all on-chip parallel peripherals with hardware state machine timing generation capabilities and parallel data latching capabilities can reuse the core concept of this invention to achieve synchronous acquisition of multiple PDM signals. At the algorithm level, the back-end filtering architecture is not limited to the CIC3 and FIR two-stage filtering schemes; various decimation filtering algorithms can be replaced according to performance requirements, and post-processing algorithms such as echo cancellation, automatic gain control, and array adaptive weighting can be flexibly expanded. At the output level, it can output multi-channel raw audio data, single-channel mixed data, and standard PCM data streams as needed, and can also package and generate various audio files such as WAV and MP3, comprehensively adapting to various audio product scenarios such as acoustic measurement, speech recognition, array sound source analysis, and conventional recording.
[0125] Based on the same inventive concept, this invention provides an audio processing device, which includes an external storage controller and a multi-channel digital microphone (DMIC). The external storage controller includes a read state machine. The RD output terminal of the external storage controller is electrically connected to the clock input terminal of the multi-channel digital microphone (DMIC) to use the RD read pulse as the sampling clock of the multi-channel digital microphone (DMIC). The data output terminals of the multi-channel digital microphone (DMIC) are electrically connected one-to-one with the data bus of the external storage controller. This audio processing device can realize the functions corresponding to the aforementioned audio processing method.
[0126] Based on the same inventive concept, embodiments of the present invention provide an audio processing apparatus capable of implementing the functions corresponding to the aforementioned audio processing method. This audio processing apparatus can be a hardware structure, a software module, or a hardware structure plus a software module. The audio processing apparatus can be implemented by a chip system, which can consist of chips or include chips and other discrete components. Please refer to [link to previous document]. Figure 5 As shown, the audio processing device 500 includes: Configuration unit 501 is used to configure the external storage controller so that the read state machine inside the external storage controller runs continuously in a loop; the read state machine generates a read cycle every time it completes a full read state cycle, outputs a read control RD read pulse in the corresponding read cycle, and performs a multi-bit width read operation; wherein, the RD read pulse is multiplexed as the sampling clock of the multi-channel digital microphone DMIC. The acquisition unit 502 is used to acquire the pulse density modulation (PDM) single bit output by each of the multiple digital microphones (DMICs) in parallel during each read cycle to obtain a single multi-bit sample word. The component unit 503 is used to acquire multiple sampled words obtained from multiple consecutive read cycles and assemble them into a sampled word set in chronological order. The acquisition unit 504 is used to decompose the sampling word set into channels according to the bit correspondence to obtain the PDM bit stream corresponding to each microphone channel.
[0127] In one possible implementation, the device further includes a filter compensation unit 505, for: For each microphone channel's PDM bitstream, decimation filtering and frequency response compensation are performed to obtain the filtered signal; The step of performing decimation filtering and frequency response compensation processing on the PDM bitstream of each microphone channel to obtain the filtered signal includes: For each microphone channel's PDM bitstream, the same set of decimation filtering parameters and the same set of frequency response compensation parameters are used; wherein, the decimation filtering parameters are used for cascaded integral comb filtering, and the frequency response compensation parameters are used for finite impulse response filtering; The PDM bitstream of each microphone channel is decimated and downsampled by cascaded integral comb filtering, converting the high-frequency PDM bitstream into an intermediate audio signal with a preset audio sampling rate. Frequency response compensation is performed on the intermediate audio signal by finite impulse response filtering to correct the passband attenuation introduced by cascaded integral comb filtering, thereby obtaining the filtered signal of each microphone channel.
[0128] In one possible implementation, the device further includes an adjustment unit 506 for: The filtered signal is subjected to DC component suppression and gain adjustment processing to generate pulse code modulation (PCM) samples corresponding to each microphone channel.
[0129] In one possible implementation, the obtaining unit 504 is configured to: Each multi-bit sample word in the sample word set is traversed in chronological order. Bits corresponding to the same microphone channel are extracted from each multi-bit sample word. The extracted bits are then concatenated in chronological order to form the PDM bit stream corresponding to the same microphone channel.
[0130] In one possible implementation, the configuration unit 501 is configured to: Perform a one-time register configuration on the external storage controller to set the read timing parameters to match the PDM timing of the multiplexed DMIC; and, The chip select channel of the external storage controller is configured to use a fixed address access mode or an address automatic wraparound burst access mode, so that the read state machine can autonomously and continuously generate read cycles based on the configured read timing parameters after the configuration is completed.
[0131] In one possible implementation, the component 503 is used for: Each time the external storage controller completes a read cycle, a direct memory access (DMA) request is generated, triggering the DMA to move the single multi-bit sample word generated in the read cycle to the circular buffer for storage in chronological order. The sample words corresponding to multiple consecutive read cycles are read from the circular buffer to form the sample word set.
[0132] In one possible implementation, the apparatus further includes an encoding processing unit 507, configured to: Determine audio storage requirements; Based on the audio storage requirements, a target audio processing method is determined, and the PCM samples corresponding to each microphone channel are processed according to the target audio processing method to obtain audio data; The target audio processing method is to perform WAV format encapsulation processing on the PCM sample value, or to perform MP3 compression encoding processing on the PCM sample value.
[0133] All relevant content of each step involved in the aforementioned audio processing method embodiments can be referenced to the functional description of the corresponding functional module of the electronic device in the embodiments of the present invention, and will not be repeated here.
[0134] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single controller, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0135] Based on the same inventive concept, embodiments of the present invention provide an electronic device, please refer to... Figure 6 As shown, the electronic device includes at least one processor 601 and a memory 602 connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 Taking the connection between processor 601 and memory 602 via bus 600 as an example, bus 600 in... Figure 6 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 600 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 6 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. In addition, the electronic device includes a communication interface 603 for receiving or transmitting data.
[0136] In this embodiment of the invention, the memory 602 stores instructions that can be executed by at least one processor 601. By executing the instructions stored in the memory 602, at least one processor 601 can perform the steps included in the aforementioned audio processing method.
[0137] The processor 601 is the control center of the electronic device. It can connect to various parts of the electronic device through various interfaces and lines. By running or executing instructions stored in the memory 602 and calling data stored in the memory 602, it can monitor the various functions and data processing of the electronic device as a whole.
[0138] Optionally, processor 601 may include one or more processing units. Processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip; in some embodiments, they may be implemented separately on independent chips.
[0139] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the various methods, steps, and logic block diagrams of the present invention embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods invented in conjunction with the present invention embodiments can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.
[0140] Memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 602 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In embodiments of the present invention, memory 602 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.
[0141] By designing and programming the processor 601, the code corresponding to the audio processing method described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned audio processing method when running. How to design and program the processor 601 is a well-known technique to those skilled in the art, and will not be described in detail here.
[0142] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium including program code, which, when the program product is run on an electronic device, is used to cause the electronic device to perform the steps of the aforementioned audio processing method.
[0143] In some possible implementations, aspects of the audio processing method provided by the present invention can also be implemented as a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps in the audio processing method according to various exemplary embodiments of the present invention described above.
[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0145] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0148] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An audio processing method, characterized in that, The method includes: Configure an external storage controller to make the read state machine inside the external storage controller run continuously in a loop; the read state machine generates a read cycle every time it completes a full read state cycle, outputs a read control RD read pulse in the corresponding read cycle, and performs a multi-bit width read operation; wherein, the RD read pulse is multiplexed as the sampling clock of the multi-channel digital microphone DMIC; Within each read cycle, pulse density modulation (PDM) single bits output by multiple digital microphones (DMICs) are acquired in parallel to obtain a single multi-bit sample word. Acquire multiple sample words obtained from multiple consecutive read cycles and assemble them into a sample word set in chronological order; The sample word set is decomposed into channels according to the bit correspondence to obtain the PDM bit stream corresponding to each microphone channel.
2. The method according to claim 1, characterized in that, The method further includes: The PDM bitstream of each microphone channel is decimated, filtered, and frequency response compensated to obtain the filtered signal. The step of performing decimation filtering and frequency response compensation processing on the PDM bitstream of each microphone channel to obtain the filtered signal includes: For each microphone channel's PDM bitstream, the same set of decimation filtering parameters and the same set of frequency response compensation parameters are used; wherein, the decimation filtering parameters are used for cascaded integral comb filtering, and the frequency response compensation parameters are used for finite impulse response filtering; The PDM bitstream of each microphone channel is decimated and downsampled by cascaded integral comb filtering, converting the high-frequency PDM bitstream into an intermediate audio signal with a preset audio sampling rate. Frequency response compensation is performed on the intermediate audio signal by finite impulse response filtering to correct the passband attenuation introduced by cascaded integral comb filtering, thereby obtaining the filtered signal of each microphone channel.
3. The method according to claim 2, characterized in that, The method further includes: The filtered signal is subjected to DC component suppression and gain adjustment processing to generate pulse code modulation (PCM) samples corresponding to each microphone channel.
4. The method according to any one of claims 1-3, characterized in that, The step of decomposing the sampled word set according to the bit correspondence to obtain the PDM bit stream corresponding to each microphone channel includes: Each multi-bit sample word in the sample word set is traversed in chronological order. Bits corresponding to the same microphone channel are extracted from each multi-bit sample word. The extracted bits are then concatenated in chronological order to form the PDM bit stream corresponding to the same microphone channel.
5. The method according to claim 1, characterized in that, The configuration of the external storage controller, enabling the read state machine within the external storage controller to run continuously in a loop, includes: Perform a one-time register configuration on the external storage controller to set the read timing parameters to match the PDM timing of the multiplexed DMIC; and, The chip select channel of the external storage controller is configured to use a fixed address access mode or an address automatic wraparound burst access mode, so that the read state machine can autonomously and continuously generate read cycles based on the configured read timing parameters after the configuration is completed.
6. The method according to claim 1, characterized in that, The acquisition of multi-bit sampled words obtained from multiple consecutive read cycles includes: Each time the external storage controller completes a read cycle, a direct memory access (DMA) request is generated, triggering the DMA to move the single multi-bit sample word generated in the read cycle to the circular buffer for storage in chronological order. The sample words corresponding to multiple consecutive read cycles are read from the circular buffer to form the sample word set.
7. The method according to claim 3, characterized in that, After generating the PCM samples corresponding to each microphone channel, the method further includes: Determine audio storage requirements; Based on the audio storage requirements, a target audio processing method is determined, and the PCM samples corresponding to each microphone channel are processed according to the target audio processing method to obtain audio data; The target audio processing method is to perform WAV format encapsulation processing on the PCM sample value, or to perform MP3 compression encoding processing on the PCM sample value.
8. An audio processing apparatus, characterized in that, Includes an external storage controller and a multi-channel digital microphone (DMIC); The external storage controller includes a read state machine; The RD output terminal of the external storage controller is electrically connected to the clock input terminal of the multi-channel digital microphone (DMIC) so as to use the RD read pulse as the sampling clock of the multi-channel digital microphone (DMIC). The data output terminals of the multi-channel digital microphone (DMIC) are electrically connected one-to-one with the data bus of the external storage controller. The audio processing device is used to perform the audio processing method according to any one of claims 1-7.
9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps included in any one of claims 1-7.
10. A computer program product, the computer program product comprising: Computer program code that, when run on an electronic device, causes the electronic device to perform the steps included in any of the methods of claims 1-7.
