An audio processor, an audio processing method, a chip, a chip module, and a terminal.

Abstract

This disclosure provides an audio processor, an audio processing method, a chip, a chip module, and a terminal. The audio processor includes: a USB module coupled to a microphone array and receiving audio data from the microphone array; an application processor configured to be in a deep sleep mode when not woken up, and to receive and process the audio data after being woken up; and a USB audio processor module coupled to both the USB module and the application processor, and configured to at least: perform key audio detection on the audio data while the application processor is in the deep sleep mode, and wake up the application processor when key audio is detected; and be turned off after the application processor is woken up. This disclosure enables ultra-low power consumption in high-bandwidth microphone array scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of audio acquisition and processing technology, and in particular to an audio processor, an audio processing method, a chip, a chip module, and a terminal. Background Technology

[0002] Audio localization and tracking technology, based on acoustic principles and signal processing techniques, is used to determine the location of sound sources and track their movement. Developments in this field are closely linked to advancements across multiple industries. Here are some applications of audio localization and tracking technology in various industry contexts: For example, in the security industry, intelligent monitoring systems can be combined with video surveillance systems to pinpoint the precise location of events by identifying specific sounds (such as gunshots or screams), improving response speed and efficiency. In military and defense, battlefield sound monitoring can be used to monitor enemy positions and activities, such as locating gunshots and vehicle sounds, enhancing battlefield situational awareness. For drone monitoring, audio localization tracks drone flight paths and identifies and tracks unidentified flying objects. In smart homes and IoT-based home security, audio localization allows smart home systems to identify and locate abnormal sounds, such as breaking glass or fire alarms, automatically triggering security alerts. Finally, audio localization helps smart devices identify user locations, providing more personalized voice services, such as directional sound playback and voice recognition. Wildlife conservation in scientific research: In the field of bioscience, this technology is used to monitor and track the sounds of wild animals, helping to study animal behavior and protect ecosystems.

[0003] Especially in the military, security, and scientific research fields, there are extremely high requirements for sound capture. In these cases, it is usually necessary to use ultra-large microphone arrays (such as an array of 80 microphones) combined with high bit width and high sampling rate to accurately capture sound. In this scenario, the amount of audio data is very large, and high-speed interfaces such as Universal Serial Bus (USB) are usually required to meet the data transmission requirements.

[0004] Traditional microphone array product architecture based on high-speed USB transmission, such as Figure 1As shown, it mainly consists of three parts: a microphone array, a USB module, and an application processor (AP). The USB module and AP are components of the host chip. The audio data processing flow is as follows: 1. The microphone array acquires audio data via an analog-to-digital converter (ADC); 2. The audio data is transmitted to the host chip via the USB interface; 3. The host chip's USB module (located in the AON domain) receives the audio data and notifies the AP via an interrupt. AON stands for Always On, used to describe functions or systems that continue to operate even when most device functions are off or in low-power mode; 4. After receiving the interrupt from the USB module, the AP retrieves the audio data in the interrupt handler function; 5. The AP executes an audio recognition algorithm to determine whether the received audio is critical audio (such as gunshots, specific words, etc.) and calculates information such as the direction of the sound source.

[0005] As can be seen from the above, in traditional solutions, because the AP needs to receive audio data at all times, it is forced to remain in an active state and cannot enter a deep sleep mode. This limitation significantly affects the energy efficiency of the entire system and also restricts the battery life of battery-powered devices. Summary of the Invention

[0006] A brief overview of embodiments of this disclosure is provided below to provide a basic understanding of certain aspects of this disclosure. It should be understood that this overview is not an exhaustive summary of this disclosure. It is not intended to identify key or essential parts of this disclosure, nor is it intended to limit the scope of this disclosure. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0007] Given the limitations of existing technology, this disclosure aims to provide a method for an AP to enter deep sleep when no specific sound is occurring, thereby reducing the overall system power consumption and improving product battery life.

[0008] According to a first aspect of this disclosure, an audio processor is provided, comprising:

[0009] The USB module is coupled to and receives audio data from the microphone array.

[0010] The application processor is configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after being awakened;

[0011] The USB audio processor module, coupled to both the USB module and the application processor, is configured at least as follows:

[0012] While the application processor is in the deep sleep mode, key audio detection is performed on the audio data, and the application processor is woken up upon detection of key audio; and

[0013] It is shut down after the application processor is woken up.

[0014] In some embodiments, the key audio includes at least keyword audio, specific event sound, or a superposition of both acoustic events.

[0015] In some embodiments, when the application processor meets the deep sleep conditions, the application processor enters the deep sleep mode, wherein the deep sleep conditions are: the application processor has no tasks to execute, or the judgment criteria for entering deep sleep are met according to a preset power consumption strategy.

[0016] In some embodiments, the USB audio processor module includes a USB processor and an audio preprocessor, wherein:

[0017] The USB processor is configured to communicate with the USB module and transmit the audio data from the USB module to the audio preprocessor;

[0018] The audio preprocessor, coupled to the USB processor, is configured to calculate a similarity score between the audio data and preset key audio, and to wake up the application processor when the similarity score exceeds a preset threshold.

[0019] In some embodiments, the USB processor is a hardware processor, which includes a storage unit for storing USB operation instructions, and the hardware processor transmits the audio data from the USB module to the audio preprocessor according to the USB operation instructions.

[0020] In some embodiments, the audio preprocessor is used to preprocess the audio data; the audio preprocessor includes an audio data input unit, a preset key audio storage unit, a key audio wake-up unit, and a score threshold comparator; wherein, the audio data input unit is used to receive the audio data from the USB processor, the preset key audio storage unit is used to store the preset key audio, the key audio wake-up unit is used to calculate the similarity score between the audio data and the preset key audio, and the score threshold comparator is used to compare the similarity score with a preset threshold; if the similarity score exceeds the preset threshold, the application processor is woken up.

[0021] According to a second aspect of this disclosure, an audio processing method is provided, comprising:

[0022] The audio data is received from the microphone array via the USB module.

[0023] When the application processor is in deep sleep mode, the USB audio processor module performs key audio detection on the audio data. If key audio is detected, the application processor is woken up and the USB audio processor module is turned off.

[0024] Once the application processor is woken up, it receives and processes the audio data.

[0025] In some embodiments, performing key audio detection on the audio data includes:

[0026] Calculate the similarity score between the audio data and the preset key audio;

[0027] When the similarity score exceeds a preset threshold, it is determined that key audio has been detected.

[0028] According to a third aspect of this disclosure, a chip is provided that integrates a USB module, a USB audio processor module, and an application processor on the same silicon die, wherein the USB module, the USB audio processor module, and the application processor are configured to work together to perform all or part of the functions of the audio processor as described in the first aspect of this disclosure.

[0029] According to a fourth aspect of this disclosure, a chip module is provided, comprising:

[0030] Microphone array;

[0031] The chip as described in the third aspect of this disclosure;

[0032] The chip is coupled to the microphone array to receive the audio data from the microphone array.

[0033] According to a fifth aspect of this disclosure, a terminal is provided, including a chip module as described in a fourth aspect of this disclosure.

[0034] The aforementioned chips, chip modules, and terminals can be packaged or cascaded sequentially.

[0035] This disclosure introduces a USB audio processor module, enabling the audio processing system to enter a deep sleep mode when non-critical audio is present, thereby significantly reducing power consumption and extending device battery life.

[0036] This disclosure is particularly suitable for applications in drone-based duty operations and wildlife conservation where tasks can be performed only after a specific event (sound) occurs, significantly extending mission duration. Furthermore, it can also achieve energy conservation and emission reduction in products using direct power supply (not battery-powered or combined with solar power) in industries such as security, smart homes and IoT, and military and defense. Attached Figure Description

[0037] This disclosure can be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which the same or similar reference numerals are used throughout the drawings to denote the same or similar parts. These drawings, together with the following detailed description, are incorporated in and form part of this specification, and are used to further illustrate preferred embodiments of the disclosure and explain the principles and advantages of the disclosure. In the drawings:

[0038] Figure 1 Architecture diagram of existing microphone array products;

[0039] Figure 2 This is a product architecture diagram of a microphone array according to an embodiment of the present disclosure;

[0040] Figure 3 This is an internal architecture diagram of a USB audio processor according to an embodiment of the present disclosure;

[0041] Figure 4 This is an internal architecture diagram of an audio processor according to an embodiment of the present disclosure;

[0042] Figure 5 This is a flowchart of the processing of sound after it is collected by a microphone array. Detailed Implementation

[0043] Embodiments of this disclosure will now be described with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of this disclosure may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that, for clarity, representations and descriptions of components and processes unrelated to this disclosure and known to those skilled in the art have been omitted from the drawings and description.

[0044] In traditional large microphone array products based on USB audio, the AP subsystem needs to continuously receive audio data and remains active, unable to enter a deep sleep mode, resulting in high power consumption. This disclosure proposes a low-power system for implementing a high-bandwidth microphone array using USB. By adding a USB Audio Processor (UAP) module, the AP remains in deep sleep until a key audio signal (such as a specific wake word) is detected, and is only awakened for subsequent processing after the key audio signal is detected, thus significantly reducing the average system power consumption. The UAP module ensures the system is in deep sleep mode before the key audio signal (word) appears, thereby reducing the overall power consumption of the product.

[0045] Specifically, see Figure 2 This disclosure provides an audio processor, which is a low-power system based on a USB high-bandwidth microphone array. For example... Figure 2 As shown, the audio processor 10 includes a USB module 101, an AP 103, and a UAP module 102. The audio processor 10 can also be referred to as a host chip.

[0046] USB module 101 is coupled to microphone array 11 and receives audio data from microphone array 11.

[0047] In some embodiments, the microphone array 11 may include N microphones, where N≥2, and each microphone has a sampling rate of not less than 48 kHz and a bit width of not less than 16 bits, but is not limited thereto.

[0048] The USB Module 101 is compatible with USB 2.0, USB 3.x, or USB Type-C.

[0049] AP 103 is configured to be in deep sleep mode when not awakened, and to receive and process the audio data upon awakening. AP 103 enters deep sleep mode when deep sleep conditions are met. The deep sleep conditions are: AP 103 has no tasks to perform, or the AP 103 meets the criteria for entering deep sleep according to a preset power consumption strategy.

[0050] AP 103 can use a conventional multi-core application processor in the field. Its deep sleep / wake-up control interface and register definition are adapted to the interrupt output pins of UAP module 102. For specific implementation details, please refer to the publicly available SoC technical manual. It will not be described in detail here.

[0051] The UAP module 102 is coupled to both the USB module 101 and the AP 103. The UAP module 102 is configured to: perform key audio detection on the audio data while the AP 103 is in the deep sleep mode, and wake up the application processor upon detecting key audio; and to shut down the UAP module 102 after the AP 103 is woken up. The key audio includes at least keyword audio, specific event sounds, or a superposition of both acoustic events.

[0052] like Figure 3 As shown, the UAP module 102 includes a USB processor 1021 and an audio preprocessor 1022.

[0053] USB processor 1021 is a hardware processor used to communicate with USB module 101 to transfer audio data from USB module 101 to audio preprocessor 1022. This hardware processor has an internal storage unit (e.g., RAM) for storing USB operation instructions. When the state machine of this hardware processor is running, it first retrieves the USB operation instructions from the storage unit, then parses them, and finally executes the corresponding tasks based on the parsed USB operation instructions. USB operation instructions typically include operations such as read, write, data save, and data comparison.

[0054] Furthermore, the USB processor 1021 in this embodiment is a pure hardware logic or a dedicated state machine containing only microcode, without containing a field-programmable microcontroller (MCU) or digital signal processing (DSP) core.

[0055] An audio preprocessor 1022 is used to preprocess audio data from the microphone array 11. The audio preprocessor 1022 is coupled to a USB processor 1021 and configured to calculate a similarity score between the audio data and preset key audio, and to wake up the AP 103 when the similarity score exceeds a preset threshold.

[0056] The structure of the audio preprocessor 1022 is as follows Figure 4 As shown, it includes an audio data input unit 1023, a preset key audio storage unit 1024, a key audio wake-up unit 1025, and a score threshold comparator 1026. Key audio may be, for example, keywords or specific event sounds. The workflow of the audio preprocessor 1022 is as follows: the key audio wake-up unit 1025 takes the audio data collected by the microphone array and the preset key audio as input, performs internal calculations, and outputs a score (i.e., audio similarity; the higher the score, the higher the similarity). Finally, the score is sent to the score threshold comparator 1026 for comparison. If the score exceeds the preset threshold, an interrupt signal is issued to wake up the AP 103.

[0057] The audio data input unit 1023 receives audio data collected by the microphone array from the USB processor 1021. For example, the USB processor 1021 writes a frame of audio data into the input buffer of the audio preprocessor 1022 via a 32-bit AHB-Lite bus or a dedicated First-In First-Out (FIFO) interface using Direct Memory Access (DMA). After writing is complete, it pulls the frame_done signal (a marker indicating audio data transfer completion) high to trigger the audio preprocessor 1022 to begin processing.

[0058] The preset key audio storage unit 1024 can store one or more sets of key tones. For example, the preset key audio storage unit 1024 is a 2kB to 8kB single-port static random access memory (SRAM), which can store 1 to 32 sets of 16kHz / 16-bit key audio templates. The template length can be configured from 0.5s to 5s, not exceeding the capacity of the preset key audio storage unit 1024.

[0059] The score threshold comparator 1026 issues an interrupt signal to wake up AP 103. For example, the score threshold comparator 1026 outputs a wake-up interrupt signal, which is sent to the interrupt input of AP 103 via an internal general-purpose input / output (I / O) pin or a dedicated wake-up pin. The signal is triggered on the rising edge, and the interrupt channel remains enabled in the deep sleep domain of AP 103 and cannot be masked by software.

[0060] Figure 5 The processing flow of sound after it is collected by microphone array 11 is shown.

[0061] (1) Microphone array 11 samples the sound through N ADCs at 96KHz and 24-bit width (the sampling rate and bit width are not mandatory, this is just an example) to obtain N digital audio data. The next step is to execute step (2).

[0062] (2) The microphone array 11 reports N channels of digital audio data to the audio processor 10 via USB. The next step is to execute step (3).

[0063] (3) The USB module 101 of the audio processor 10 receives audio data. The next step is to execute step (4).

[0064] (4) If AP 103 is in deep sleep, proceed to step (5); otherwise, proceed to step (8).

[0065] (5) Based on the above regarding Figure 4 As described, UAP module 102 obtains audio data from USB module 101 and executes a key audio (word) detection algorithm. If the similarity score output by the algorithm exceeds a preset threshold, step (6) is executed; otherwise, step (7) is executed.

[0066] (6) UAP module 102 wakes up AP 103 via an interrupt and reports information such as the sound source to AP 103. After AP 103 is woken up, AP 103 acquires and processes the audio data (in conjunction with other modules of the system to perform tasks). The audio data no longer flows into UAP module 102, and UAP module 102 is then turned off.

[0067] (7) Maintain the existing data flow path, that is, process it by UAP module 102.

[0068] (8) AP 103 acquires and processes audio data. The next step is to execute step (9).

[0069] (9) Determine whether AP 103 meets the deep sleep conditions. If it does, proceed to step (10); otherwise, proceed to step (11). The deep sleep conditions are configured as follows: AP 103's task queue is empty (no tasks to execute) and the power consumption policy register PWR_CTL[DEEP_SLEEP_EN] = 1 (where PWR_CTL[DEEP_SLEEP_EN] is a system control register bit of AP 103, which can be set to 1 by software and reset to the default value of 0), or the system power consumption management unit determines that it can enter deep sleep.

[0070] (10) Turn on UAP module 102, and the audio data is taken over and processed by UAP module 102. AP 103 enters deep sleep.

[0071] (11) Maintain the existing data flow path, that is, process it by AP 103.

[0072] This disclosure also provides an audio processing method, which is a low-power method based on a USB high-bandwidth microphone array. The audio processing method includes:

[0073] Step 1: Receive audio data collected by the microphone array via the USB module;

[0074] Step 2: When the AP is in deep sleep mode, the UAP module performs key audio detection on the audio data from Step 1;

[0075] Step 3: If a key audio signal is detected, wake up the AP and shut down the UAP module; if no key audio signal is detected, maintain the AP's deep sleep mode.

[0076] Step 4: After the AP is woken up, the AP receives and processes the audio data;

[0077] Step 5: When the AP has no tasks to perform, the UAP module takes over the audio stream again, and the AP enters deep sleep mode until the next critical audio trigger.

[0078] This disclosure also provides a UAP module, which uses the UAP module as an independent coprocessor IP core and is coupled to a third-party AP via a standard bus.

[0079] This disclosure also provides a chip that integrates a USB module, a UAP module, and an AP within a single System on a Chip (SoC). The USB module, UAP module, and AP are configured to work together to implement all or part of the functions of the audio processor as described in this disclosure.

[0080] This disclosure also provides a chip module that encapsulates the above-mentioned SoC, microphone array, and power management unit into a standard module.

[0081] This disclosure also provides a terminal, including the aforementioned UAP module, chip, or chip module, which can be applied to drones, field monitoring stations, smart speakers, etc., to achieve low-power audio monitoring.

[0082] In traditional architectures, the average power consumption of the access point (AP) accounts for more than 70% of the system's total power consumption. By adopting the implementation scheme disclosed herein, the AP current during non-event periods can be reduced to the microampere level, increasing the overall battery life by 3 to 10 times; it incurs no additional overhead on USB bandwidth and is compatible with existing host drivers; keywords and thresholds are configurable, facilitating scenario reuse.

[0083] This disclosure proposes a low-power solution for implementing a high-bandwidth microphone array based on USB, as described above. By introducing a UAP module, the system enters a deep sleep mode before critical audio occurs, significantly reducing power consumption and improving device battery life. Furthermore, the UAP module of this disclosure can be directly embedded into the SoC as an IP hard core, eliminating the need for an external DSP or Field-Programmable Gate Array (FPGA), saving on bill of materials (BOM) costs and printed circuit board (PCB) area. In deep sleep scenarios, the UAP module independently manages power consumption at the SoC level of 3mW, reducing AP current to the µA level, meeting the battery life requirements of wearable devices, drones, and other battery-powered devices for "weekly (≥7 days)" or even "monthly (≥30 days)" battery life. The SoC natively supports USB 3.2 Gen1×1 or USB-C, and can simultaneously carry 80 channels of 96kHz / 24bit audio streams, providing a "single-chip" solution for security, military reconnaissance, and scientific research-grade microphone arrays. The SoC can integrate TrustZone / TEE to store keyword templates and algorithm weights in secure SRAM; it also reserves a programmable Artificial Intelligence (AI) engine interface for easy over-the-air (OTA) algorithm upgrades. Leveraging mature 28nm / 22nm / 12nm FinFET processes or equivalent FD-SOI / CMOS processes, this disclosure enables the integrated "microphone array + audio preprocessing + main control computing + security subsystem" on a single SoC, significantly reducing the development threshold for terminal manufacturers and forming a replicable chip-level platform ecosystem.

[0084] In summary, this disclosure introduces a UAP module that enables the system to enter a deep sleep mode when non-critical audio occurs, thereby significantly reducing power consumption and extending device battery life. This solution is particularly suitable for fields such as drones, wildlife conservation, security, and smart homes.

[0085] The methods disclosed herein are not limited to being executed in the chronological order described in the specification, but can also be executed in other chronological orders, in parallel, or independently. Therefore, the execution order of the methods described in this specification does not limit the technical scope of this disclosure.

[0086] Although this disclosure has been presented above with reference to specific embodiments thereof, it should be understood that all the embodiments and examples described above are exemplary and not restrictive. Those skilled in the art can devise various modifications, improvements, or equivalents to this disclosure within the spirit and scope of the appended claims. Such modifications, improvements, or equivalents should also be considered to be included within the scope of protection of this disclosure.

Claims

1. An audio processor, characterized in that, include: The USB module is coupled to and receives audio data from the microphone array. The application processor is configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after being awakened; The USB audio processor module, coupled to both the USB module and the application processor, is configured at least as follows: When the application processor is in the deep sleep mode, key audio detection is performed on the audio data, and the application processor is woken up when key audio is detected; as well as The application processor is turned off after it is woken up; The USB audio processor module includes a USB processor and an audio preprocessor, wherein: The USB processor is configured to communicate with the USB module and transmit the audio data from the USB module to the audio preprocessor; the USB processor is pure hardware logic or a dedicated state machine containing only microcode, without containing an MCU or DSP core. The audio preprocessor, coupled to the USB processor, is configured to calculate a similarity score between the audio data and preset key audio, and to wake up the application processor when the similarity score exceeds a preset threshold.

2. The audio processor according to claim 1, characterized in that, The key audio includes at least keyword audio, specific event sound, or a superposition of both acoustic events.

3. The audio processor according to claim 1, characterized in that, When the application processor meets the deep sleep conditions, the application processor enters the deep sleep mode, wherein the deep sleep conditions are: the application processor has no tasks to execute, or the judgment criteria for entering deep sleep are met according to a preset power consumption strategy.

4. The audio processor according to claim 1, characterized in that, The USB processor includes a storage unit for storing USB operation instructions, and the USB processor transmits the audio data from the USB module to the audio preprocessor according to the USB operation instructions; The audio preprocessor is used to preprocess the audio data; the audio preprocessor includes an audio data input unit, a preset key audio storage unit, a key audio wake-up unit, and a score threshold comparator; wherein, the audio data input unit is used to receive the audio data from the USB processor, the preset key audio storage unit is used to store the preset key audio, the key audio wake-up unit is used to calculate the similarity score between the audio data and the preset key audio, and the score threshold comparator is used to compare the similarity score with a preset threshold; if the similarity score exceeds the preset threshold, the application processor is woken up.

5. An audio processing method, characterized in that, The audio processor according to any one of claims 1-4 comprises: The audio data is received from the microphone array via the USB module. When the application processor is in deep sleep mode, the USB audio processor module performs key audio detection on the audio data. If key audio is detected, the application processor is woken up and the USB audio processor module is turned off. Once the application processor is woken up, it receives and processes the audio data.

6. The audio processing method according to claim 5, characterized in that, The key audio detection of the audio data includes: Calculate the similarity score between the audio data and the preset key audio; When the similarity score exceeds a preset threshold, it is determined that key audio has been detected.

7. A chip, characterized in that, The USB module, USB audio processor module, and application processor are integrated on the same silicon chip; The USB module is coupled to and receives audio data from the microphone array. The application processor is configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after being awakened; The USB audio processor module, coupled to both the USB module and the application processor, is configured at least as follows: While the application processor is in the deep sleep mode, key audio detection is performed on the audio data, and the application processor is woken up upon detection of key audio; and The application processor is turned off after it is woken up; The USB audio processor module includes a USB processor and an audio preprocessor, wherein: The USB processor is configured to communicate with the USB module and transmit the audio data from the USB module to the audio preprocessor; the USB processor is pure hardware logic or a dedicated state machine containing only microcode, without containing an MCU or DSP core. The audio preprocessor, coupled to the USB processor, is configured to calculate a similarity score between the audio data and preset key audio, and to wake up the application processor when the similarity score exceeds a preset threshold.

8. A chip module, characterized in that, include: Microphone array; The chip as described in claim 7; The chip is coupled to the microphone array to receive the audio data from the microphone array.

9. A terminal, characterized in that, Includes the chip module as described in claim 8.

Images