Audio processor, audio processing method, chip, chip module and terminal

Abstract

The invention provides an audio processor, an audio processing method, a chip, a chip module and a terminal. An audio processor includes a USB module coupled with a microphone array and receiving audio data from the microphone array; the application processor is configured to be in a deep sleep mode when the application processor is not awakened, and receives and processes the audio data after the application processor is awakened; the USB audio processor module is respectively coupled with the USB module and the application processor, and is at least configured to execute key audio detection on the audio data when the application processor is in the deep sleep mode, and awaken the application processor when key audio is detected; and closing the application processor after the application processor is awakened. According to the invention, ultra-low power consumption on duty in a high-bandwidth microphone array scene can be realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio acquisition and processing, and in particular to an audio processor, an audio processing method, a chip, a chip module, and a terminal. Background Art

[0002] Audio positioning and tracking technology is based on acoustic principles and signal processing techniques to determine the location of sound sources and track their movement. Developments in this field are closely tied to advancements across multiple industries. The following are some applications of audio positioning and tracking technology in different industry contexts: For example, in the security industry, intelligent surveillance systems: Audio positioning and tracking technology can be combined with video surveillance systems to pinpoint the precise location of an incident by identifying specific sounds (such as gunshots and screams), improving response speed and efficiency. In the military and defense industry, battlefield sound monitoring: In the military, audio positioning technology can be used to monitor enemy positions and activities, such as locating gunshots and vehicle sounds, enhancing battlefield situational awareness. Drone monitoring uses audio positioning to track drone flight paths and identify and track unidentified flying objects. Smart home and IoT-based home security: Using audio positioning technology, smart home systems can identify and locate unusual sounds, such as breaking glass and fire alarms, automatically triggering security alerts. And in intelligent voice assistants, audio positioning helps smart devices identify the user's location, providing more personalized voice services, such as directional audio playback and voice recognition. Wildlife conservation for scientific research: In the biological sciences, this technology is used to monitor and track wildlife sounds, helping to study animal behavior and protect ecosystems.

[0003] In the military, security, and scientific research fields, sound capture is extremely demanding. This often requires ultra-large microphone arrays (e.g., 80 microphones) combined with high bit width and high sampling rates to accurately capture sound. In these scenarios, the volume of audio data is extremely large, often requiring high-speed interfaces such as the Universal Serial Bus (USB) for data transmission.

[0004] The traditional microphone array product architecture based on USB high-speed transmission is as follows Figure 1As shown in the figure, it primarily consists of three parts: a microphone array, a USB module, and an application processor (AP). The USB module and AP are components of the host chip. The audio data processing flow is as follows: 1. The microphone array captures audio data via an analog-to-digital converter (ADC); 2. The audio data is transmitted to the host chip via the USB interface; 3. Upon receiving the audio data, the host chip's USB module (located in the AON domain) notifies the AP via an interrupt. AON stands for Always On, describing features or systems that remain operational even when most device functions are shut down or in low-power mode. 4. Upon receiving an interrupt from the USB module, the AP retrieves the audio data in its interrupt handler. 5. The AP executes an audio recognition algorithm to determine whether the received audio is critical (such as gunshots or specific nouns) and calculate information such as the direction of the sound source.

[0005] As can be seen from the above, in traditional solutions, because the AP needs to receive audio data at all times, it is forced to remain active and cannot enter deep sleep mode. This limitation significantly affects the energy efficiency of the entire system and also restricts the battery life of battery-powered devices. Summary of the Invention

[0006] A brief overview of embodiments of the present disclosure is provided below to provide a basic understanding of certain aspects of the disclosure. It should be understood that the following overview is not an exhaustive overview of the disclosure. It is not intended to identify key or important portions of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to present certain concepts in a simplified form as a prelude to a more detailed description that will be discussed later.

[0007] In view of the limitations of the existing technology, the present disclosure aims to provide a method for enabling an AP to enter deep sleep when no specific sound occurs, thereby reducing the energy consumption of the entire system and improving the battery life of the product.

[0008] According to a first aspect of the present disclosure, there is provided an audio processor, comprising: A USB module coupled to the microphone array and receiving audio data from the microphone array; an application processor, configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after being awakened; A USB audio processor module is coupled to the USB module and the application processor, and is at least configured to: performing critical audio detection on the audio data while the application processor is in the deep sleep mode, and waking up the application processor when critical audio is detected; and the application processor is turned off after being woken up.

[0009] In some embodiments, the critical audio includes at least keyword audio, specific event sound, or superimposed acoustic events of both.

[0010] In some embodiments, the application processor enters the deep sleep mode when the application processor satisfies a deep sleep condition, wherein the deep sleep condition is that the application processor has no executable task, or satisfies a criterion for entering deep sleep according to a preset power consumption strategy.

[0011] In some embodiments, the USB audio processor module includes a USB processor and an audio preprocessor, wherein: the USB processor is configured to communicate with the USB module and transmit the audio data from the USB module to the audio preprocessor; the audio preprocessor, coupled with the USB processor, is configured to calculate a similarity score between the audio data and a preset critical audio, and wake up the application processor when the similarity score exceeds a preset threshold.

[0012] In some embodiments, the USB processor is a hardware processor, and the hardware processor includes a storage unit storing USB operation instructions, and the hardware processor transmits the audio data from the USB module to the audio preprocessor according to the USB operation instructions.

[0013] In some embodiments, the audio preprocessor is configured to preprocess the audio data, and the audio preprocessor includes an audio data input unit, a preset critical audio storage unit, a critical audio wake-up unit, and a score threshold comparator, wherein the audio data input unit is configured to receive the audio data from the USB processor, the preset critical audio storage unit is configured to store the preset critical audio, the critical audio wake-up unit is configured to calculate the similarity score between the audio data and the preset critical audio, and the score threshold comparator is configured to compare the similarity score with a preset threshold, and wake up the application processor if the similarity score exceeds the preset threshold.

[0014] According to a second aspect of the present disclosure, an audio processing method is provided, including: receiving audio data collected by a microphone array through a USB module; detecting key audio from the audio data by the USB audio processor module, and if key audio is detected, waking up the application processor and shutting down the USB audio processor module; after the application processor is woken up, receiving and processing the audio data by the application processor.

[0015] In some embodiments, the detecting key audio from the audio data comprises: calculating a similarity score between the audio data and preset key audio; if the similarity score exceeds a preset threshold, determining that key audio is detected.

[0016] According to a third aspect of the present disclosure, a chip is provided, which integrates a USB module, a USB audio processor module and an application processor on the same silicon chip, wherein the USB module, the USB audio processor module and the application processor are configured to work cooperatively to realize all or part of the functions of the audio processor as described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a chip module is provided, comprising: a microphone array; a chip as described in the third aspect of the present disclosure; wherein the chip is coupled to the microphone array to receive the audio data from the microphone array.

[0018] According to a fifth aspect of the present disclosure, a terminal is provided, comprising the chip module as described in the fourth aspect of the present disclosure.

[0019] The chip, the chip module and the terminal described above can be used in turn or in cascade.

[0020] The present disclosure introduces the USB audio processor module, so that the audio processing system enters deep sleep mode when non-key audio occurs, thereby significantly reducing power consumption and prolonging the device's endurance time.

[0021] The present disclosure is particularly suitable for the demand of executing tasks again when specific events (sounds) occur in the fields of unmanned aerial vehicle duty and wildlife protection, and can greatly prolong the task execution time. In addition, in the products of the security, smart home and Internet of Things, military and national defense industries, which are directly powered by power supply (not battery power supply and solar power supply), energy saving and emission reduction effects can also be achieved. BRIEF DESCRIPTION OF DRAWINGS

[0022] The present disclosure can be better understood with reference to the following description in conjunction with the accompanying drawings, in which like reference signs refer to like elements in the several figures. The accompanying drawings, together with the following detailed description, explain and illustrate preferred embodiments of the present disclosure. Figure 1 is a schematic diagram of a microphone array product architecture; Figure 2 is a schematic diagram of a microphone array product architecture according to an embodiment of the present disclosure; Figure 3 is a schematic diagram of the internal architecture of a USB audio processor according to an embodiment of the present disclosure; Figure 4 is a schematic diagram of the internal architecture of an audio processor according to an embodiment of the present disclosure; Figure 5 is a schematic diagram of the processing flow of sound collected by a microphone array. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of the present disclosure can be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that, for the purpose of clarity, representations and descriptions of components and processes that are irrelevant to the present disclosure and known to those of ordinary skill in the art are omitted from the drawings and the description.

[0024] In a conventional large microphone array product based on USB audio, the AP subsystem needs to continuously receive audio data and always remains active, and cannot enter a deep sleep mode, resulting in high power consumption. Embodiments of the present disclosure propose a low-power system for a high-bandwidth microphone array implemented by USB, which adds a USB audio processor (UAP) module to enable the AP to remain in a deep sleep mode before a key audio (such as a specific wake-up word) appears, and only wake up the AP for subsequent processing after detecting the key audio, thereby significantly reducing the average power consumption of the system. The UAP module enables the system to be in a deep sleep mode before the key audio (word) appears, thereby reducing the overall power consumption of the product.

[0025] Specifically, referring to Figure 2 Embodiments of the present disclosure provide an audio processor for a low-power system of a high-bandwidth microphone array based on USB. As shown in Figure 2 The audio processor 10 includes a USB module 101, an AP 103, and a UAP module 102. The audio processor 10 can also be referred to as a host chip.

[0026] The USB module 101 is coupled to the microphone array 11 and receives audio data from the microphone array 11 .

[0027] In some embodiments, the microphone array 11 may include N microphones, where N≥2, and the sampling rate of each microphone is not less than 48 kHz and the bit width is not less than 16 bits, but the present invention is not limited thereto.

[0028] The USB module 101 is compatible with USB 2.0, USB 3.x, or USB Type-C.

[0029] AP 103 is configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after awakening. AP 103 enters deep sleep mode when deep sleep conditions are met. Deep sleep conditions are: AP 103 has no tasks to perform, or meets the criteria for entering deep sleep according to a preset power consumption policy.

[0030] AP 103 can adopt a conventional multi-core application processor in the field, and its deep sleep / wake-up control interface and register definition are adapted to the interrupt output pin of UAP module 102. For specific implementation, please refer to the public SoC technical manual and will not be repeated here.

[0031] The UAP module 102 is coupled to the USB module 101 and the AP 103, respectively. The UAP module 102 is configured to: when the AP 103 is in the deep sleep mode, perform key audio detection on the audio data and wake up the application processor upon detecting key audio; and shut down the UAP module 102 after the AP 103 is woken up. Key audio includes at least keyword audio, specific event sounds, or a combination of both.

[0032] like Figure 3 As shown, the UAP module 102 includes a USB processor 1021 and an audio pre-processor 1022 .

[0033] USB processor 1021 is a hardware processor used to communicate with USB module 101 and transfer audio data from USB module 101 to audio preprocessor 1022. This hardware processor has an internal storage unit (e.g., RAM) for storing USB operation instructions. When the hardware processor's state machine is running, it first extracts USB operation instructions from the storage unit, then parses them, and finally executes the corresponding task based on the parsed USB operation instructions. USB operation instructions are generally categorized into operations such as reading, writing, data saving, and data comparison.

[0034] In addition, the USB processor 1021 of this embodiment is pure hardware logic or a dedicated state machine containing only microcode, and does not include a field-programmable microcontroller (MCU) or digital signal processing (DSP) core.

[0035] The audio preprocessor 1022 is configured to preprocess the audio data from the microphone array 11. The audio preprocessor 1022 is coupled to the USB processor 1021 and is configured to calculate a similarity score between the audio data and a preset key audio, and wake up the AP 103 when the similarity score exceeds a preset threshold.

[0036] The structure of the audio preprocessor 1022 is as follows Figure 4 As shown, it includes an audio data input unit 1023, a preset key audio storage unit 1024, a key audio wake-up unit 1025, and a score threshold comparator 1026. Key audio can be, for example, a keyword or a specific event sound. The workflow of the audio preprocessor 1022 is as follows: the key audio wake-up unit 1025 takes the audio data collected by the microphone array and the preset key audio as input, outputs a score (i.e., audio similarity after internal calculation; the higher the score, the higher the similarity), and finally sends the score to the score threshold comparator 1026 for comparison. If the score exceeds the preset threshold, an interrupt signal is issued to wake up the AP 103.

[0037] The audio data input unit 1023 receives audio data collected by the microphone array and transmitted from the USB processor 1021. For example, the USB processor 1021 writes a frame of audio data into the input buffer of the audio preprocessor 1022 using Direct Memory Access (DMA) via a 32-bit AHB-Lite bus or a dedicated First-In First-Out (FIFO) interface. After the write is complete, the USB processor 1021 raises the frame_done signal (which indicates the completion of the audio data transfer) to trigger the audio preprocessor 1022 to begin processing.

[0038] The preset key audio storage unit 1024 can store one or more key audio groups. For example, the preset key audio storage unit 1024 is a 2kB to 8kB single-port static random access memory (SRAM), which can store 1 to 32 groups of 16kHz / 16-bit key audio templates. The template length can be configured to be 0.5s to 5s, subject to the capacity of the preset key audio storage unit 1024.

[0039] The score threshold comparator 1026 issues an interrupt signal to wake up the AP 103. For example, the score threshold comparator 1026 outputs a wake-up interrupt signal to the interrupt input terminal of the AP 103 through a chip internal general-purpose input / output (I / O) pin or a dedicated wake-up pin, the rising edge triggers, and the interrupt channel in the deep sleep domain of the AP 103 is enabled and cannot be shielded by software.

[0040] Figure 5 The processing flow after the sound is collected by the microphone array 11 is shown.

[0041] (1) The microphone array 11 samples the sound through N-channel ADC at 96KHz, 24bit bit width (the sampling rate and bit width are not mandatory, which is only an example here). N-channel digital audio data is obtained. Next, step (2) is performed.

[0042] (2) The microphone array 11 reports the N-channel digital audio data to the audio processor 10 through USB. Next, step (3) is performed.

[0043] (3) The USB module 101 of the audio processor 10 receives the audio data. Next, step (4) is performed.

[0044] (4) If the AP 103 is in deep sleep, step (5) is performed, otherwise step (8) is performed.

[0045] (5) According to the above description of Figure 4 , the UAP module 102 obtains the audio data from the USB module 101 and performs a key audio (word) detection algorithm. If the similarity score output by the algorithm exceeds the preset threshold, step (6) is performed, otherwise step (7) is performed.

[0046] (6) The UAP module 102 wakes up the AP 103 through an interrupt, and reports the sound source and other information to the AP 103. After the AP 103 is woken up, the audio data is obtained by the AP 103 and processed (jointly with other modules of the system to perform tasks), and the audio data no longer flows into the UAP module 102, and the UAP module 102 is closed at this time.

[0047] (7) Maintain the existing data flow path, that is, processed by the UAP module 102.

[0048] (8) The AP 103 obtains the audio data and processes it. Next, step (9) is performed.

[0049] (9) Determine whether the AP 103 meets the deep sleep condition. If yes, execute step (10). If not, execute step (11). The deep sleep condition is configured as: the task queue of the AP 103 is empty (no executable task) and the power consumption strategy register PWR_CTL[DEEP_SLEEP_EN]=1 (where PWR_CTL[DEEP_SLEEP_EN] is a system control register bit of the AP 103, which can be set to 1 by software, and the default value is 0), or the system power management unit determines that the deep sleep can be entered.

[0050] (10) Start the UAP module 102, and the audio data is taken over and processed by the UAP module 102, and the AP 103 enters deep sleep.

[0051] (11) Maintain the existing data flow channel, that is, processed by the AP 103.

[0052] The embodiments of the present disclosure also provide an audio processing method, which is a low-power method based on a USB high-bandwidth microphone array. The audio processing method comprises: Step 1: receiving audio data collected by a microphone array through a USB module; Step 2: when the AP is in a deep sleep mode, performing key audio detection on the audio data of step 1 by a UAP module; Step 3: if key audio is detected, waking up the AP and closing the UAP module; if key audio is not detected, maintaining the deep sleep mode of the AP; Step 4: after the AP is woken up, receiving and processing the audio data by the AP; Step 5: when the AP has no executable task, taking over the audio stream by the UAP module, and the AP enters a deep sleep mode until the next key audio trigger.

[0053] The embodiments of the present disclosure also provide a UAP module, which takes the UAP module as an independent coprocessor IP core, and is coupled with a third-party AP through a standard bus.

[0054] The embodiments of the present disclosure also provide a chip, which integrates a USB module, a UAP module and an AP in a single System on a chip (SoC). The USB module, the UAP module and the AP are configured to work cooperatively to realize all or part of the functions of the audio processor as described in the present disclosure.

[0055] The embodiments of the present disclosure also provide a chip module, which packages the above-mentioned SoC, a microphone array and a power management unit into a standard module.

[0056] The embodiments of the present disclosure also provide a terminal comprising the UAP module, the chip or the chip module, which can be applied to a drone, a field monitoring station, an intelligent sound box and the like, and realizes low-power audio value guard.

[0057] In the traditional architecture, the average power consumption of the AP accounts for more than 70% of the system. After adopting the implementation scheme of the present disclosure, the AP current can be reduced to the microampere level during the non-event period, and the whole machine endurance is improved by 3-10 times; there is no additional overhead for the USB bandwidth, and it is compatible with the existing host driver; the keywords and thresholds can be configured, which is convenient for scene reuse.

[0058] The present disclosure proposes a low-power scheme for implementing a high-bandwidth microphone array based on USB through the above scheme. By introducing the UAP module, the system enters a deep sleep mode before the key audio appears, thereby significantly reducing power consumption and improving device endurance. In addition, the UAP module of the present disclosure can be directly embedded as an IP hard core in SoC, without the need for external DSP or Field-Programmable Gate Array (FPGA), saving Bill of Materiel (BOM) cost and Printed Circuit Board (PCB) area. In the deep sleep scene, the UAP module independently guards with a power consumption of 3mW, and the AP current is reduced to the microampere level, meeting the needs of battery-powered devices such as wearable devices and drones for "weekly level (≥7 days)" or even "monthly level (≥30 days)" endurance. The SoC natively supports USB 3.2 Gen1x1 or USB-C, and can simultaneously carry 80 channels of 96kHz / 24bit audio stream, providing a "single-chip" level solution for security, military reconnaissance, and scientific research microphone arrays. The SoC can integrate TrustZone / TEE, store keyword templates and algorithm weights in secure SRAM, and at the same time reserve an Artificial Intelligence (AI) engine interface for subsequent algorithm Over the Air (OTA) upgrade. Relying on mature 28nm / 22nm / 12nm FinFET process or equivalent FD-SOI / CMOS process, the present disclosure can realize the integration of "microphone array + audio preprocessing + main control calculation + security subsystem" on a single SoC, significantly reducing the development threshold of terminal manufacturers, and forming a replicable chip-level platform ecosystem.

[0059] In summary, the present disclosure introduces the UAP module to make the system enter a deep sleep mode when non-key audio appears, thereby significantly reducing power consumption and prolonging device endurance. This scheme is particularly suitable for drones, wildlife protection, security, smart home and the like.

[0060] The method of the present disclosure is not limited to be executed in the time sequence described in the specification, but can be executed in other time sequences, in parallel, or independently. Therefore, the execution order of the method described in the specification does not constitute a limitation on the technical scope of the present disclosure.

[0061] Although the present disclosure has been disclosed by the description of the specific embodiments of the present disclosure above, it should be understood that all the embodiments and examples described above are exemplary, not limiting. Those skilled in the art can design various modifications, improvements or equivalents of the present disclosure within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present disclosure.

Claims

1. An audio processor, characterized in that: include: A USB module coupled to the microphone array and receiving audio data from the microphone array; an application processor, configured to be in a deep sleep mode when not awakened, and to receive and process the audio data after being awakened; A USB audio processor module is coupled to the USB module and the application processor, and is at least configured to: When the application processor is in the deep sleep mode, performing key audio detection on the audio data, and waking up the application processor when key audio is detected; as well as being shut down after the application processor is awakened; The USB audio processor module includes a USB processor and an audio preprocessor, wherein: The USB processor is configured to communicate with the USB module and transmit the audio data from the USB module to the audio preprocessor; the USB processor is pure hardware logic or a dedicated state machine containing only microcode curing, and does not include an MCU or DSP core; The audio preprocessor is coupled to the USB processor and is configured to calculate a similarity score between the audio data and a preset key audio, and wake up the application processor when the similarity score exceeds a preset threshold.

2. The audio processor according to claim 1, wherein The key audio at least includes keyword audio, specific event sound, or a superimposed acoustic event of the two.

3. The audio processor according to claim 1, wherein When the application processor meets a deep sleep condition, the application processor enters the deep sleep mode, wherein the deep sleep condition is: the application processor has no executable tasks, or meets a judgment criterion for entering deep sleep according to a preset power consumption policy.

4. The audio processor according to claim 1, wherein The USB processor is a hardware processor, comprising a storage unit for storing USB operation instructions, and the hardware processor transmits the audio data from the USB module to the audio preprocessor according to the USB operation instructions; The audio preprocessor is used to preprocess the audio data; the audio preprocessor includes an audio data input unit, a preset key audio storage unit, a key audio wake-up unit and a score threshold comparator; wherein, the audio data input unit is used to receive the audio data from the USB processor, the preset key audio storage unit is used to store the preset key audio, the key audio wake-up unit is used to calculate the similarity score between the audio data and the preset key audio, and the score threshold comparator is used to compare the similarity score with a preset threshold. If the similarity score exceeds the preset threshold, the application processor is woken up.

5. An audio processing method, characterized in that: The audio processor according to any one of claims 1 to 4, comprising: Receive audio data collected by the microphone array through the USB module; When the application processor is in a deep sleep mode, the USB audio processor module performs a key audio detection on the audio data, and if a key audio is detected, the application processor is woken up and the USB audio processor module is shut down; After the application processor is awakened, the application processor receives and processes the audio data.

6. The audio processing method according to claim 5, characterized in that: The performing key audio detection on the audio data comprises: Calculating a similarity score between the audio data and a preset key audio; When the similarity score exceeds a preset threshold, it is determined that key audio is detected.

7. A chip, characterized in that: Integrates USB module, USB audio processor module and application processor on the same silicon chip; The USB module, the USB audio processor module and the application processor are configured to work together to implement all or part of the functions of the audio processor according to any one of claims 1 to 4.

8. A chip module, characterized in that: include: microphone array; The chip according to claim 7; The chip is coupled to the microphone array to receive the audio data from the microphone array.

9. A terminal, characterized in that: Comprising the chip module as claimed in claim 8.

Images