Method and device for continuous recording and electronic equipment

Abstract

The invention discloses a continuous recording method and device and electronic equipment. The method comprises the steps that a low-power-consumption power domain of the system is configured to control the system to enter a sleep mode, a microphone of the low-power-consumption power domain continuously collects audio data, a continuous recording controller of the low-power-consumption power domain carries out abnormal sound detection on the audio data to obtain a detection result, and if the detection result is that abnormal sound is detected, the system is awakened. According to the invention, the low-power-consumption power domain is used for continuous recording, after the system enters the dormant state, the low-power-consumption power domain still keeps working, and abnormal sound detection can be carried out on the audio data collected by continuous recording, so that continuous recording is realized in a low-power-consumption mode, and the requirement of long-time use of battery products is met.

Description

Technical Field

[0001] This invention relates to the field of recording technology, and in particular to methods, apparatus, and electronic devices for continuous recording. Background Technology

[0002] Currently, there is an increasing demand for battery-powered products to detect and wake up specific sounds in low-power mode. However, most products do not support low-power recording during system sleep and require continuous recording in normal system operation mode, resulting in high power consumption (at least 300mW), which cannot meet the needs of battery-powered products for long-term use. Summary of the Invention

[0003] The present invention provides a method and apparatus for continuous recording, and an electronic device that enables continuous recording in a low-power mode.

[0004] In one aspect of the invention, a method for continuous recording is provided. The method includes: configuring a low-power power domain of the system to control the system to enter a sleep mode; continuously acquiring audio data using a microphone in the low-power power domain; performing abnormal sound detection on the audio data using a continuous recording controller in the low-power power domain to obtain a detection result; and waking up the system if the detection result indicates that an abnormal sound has been detected.

[0005] In another aspect of the invention, an apparatus for continuous recording is provided. The apparatus includes: a low-power power domain configured to continue operating after the system enters a sleep mode; and includes: an audio acquisition and processing module configured to continuously acquire audio signals and convert the audio signals into audio data having a predetermined format; and a continuous recording control module configured to perform abnormal sound detection on the audio data to obtain a detection result, and if the detection result indicates that an abnormal sound has been detected, to wake up the system.

[0006] In another aspect of the invention, an electronic device is provided. The electronic device includes a memory configured to store information associated with audio; and at least one processor electrically coupled to the memory and configured to perform the method described above for continuous recording.

[0007] According to the technical solution of the present invention, a low-power power domain is configured in the system to control the system to enter sleep mode. The microphone in the low-power power domain continuously collects audio data, and the continuous recording controller in the low-power power domain performs abnormal sound detection on the audio data and obtains the detection result. If the detection result is that an abnormal sound is detected, the system is woken up. In this way, the low-power power domain is used for continuous recording. After the system enters sleep mode, the low-power power domain continues to work and can also perform abnormal sound detection on the audio data collected by continuous recording, thereby realizing continuous recording in low-power mode and meeting the needs of battery-powered products for long-term use. Attached Figure Description

[0008] Figure 1 This is a flowchart of a method for continuous recording according to an embodiment of the present invention; Figure 2 This is a schematic diagram of an upgraded AOV containing continuous sound according to an embodiment of the present invention; Figure 3 This is a schematic diagram of graded abnormal sound detection according to an embodiment of the present invention; Figure 4 This is a schematic diagram of AI model detection according to an embodiment of the present invention; Figure 5 This is a block diagram of the feature extraction part of the AI ​​model according to an embodiment of the present invention; Figure 6 A flowchart illustrating the signal processing performed on audio data input from a single microphone according to an embodiment of the present invention; Figure 7 A flowchart illustrating the signal processing performed on audio data input from multiple microphones according to an embodiment of the present invention; Figure 8 This is a flowchart of a method for continuous recording according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a device for continuous recording according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0009] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0010] In existing technologies, many battery-powered products, such as battery-powered cameras, already support AOV (Always On Video) technology. This means that the camera can continuously record video in a low frame rate mode (such as 1 frame per second) while in a low-power state, thus achieving a longer working time on a full charge. However, due to technological limitations, these cameras that support AOV technology usually do not have a recording function in low-power mode, and therefore cannot continuously capture sound.

[0011] To address at least the aforementioned technical problems, this disclosure provides a method for continuous recording. According to this disclosure, a low-power power domain of the system is configured to control the system into a sleep mode. Audio data is continuously acquired by a microphone within the low-power power domain, and an abnormal sound detection is performed on the audio data by a continuous recording controller within the low-power power domain. If an abnormal sound is detected, the system is woken up. In this way, continuous recording is achieved using the low-power power domain. Even after the system enters sleep mode, the low-power power domain remains operational and can still detect abnormal sounds in the continuously acquired audio data, thus enabling continuous recording in low-power mode and meeting the needs of battery-powered products for extended use.

[0012] In the following, the technical solutions according to this disclosure will be described with reference to specific embodiments and in conjunction with the accompanying drawings.

[0013] Figure 1 This is a flowchart illustrating a method 100 for continuous recording according to an embodiment of the present disclosure. (Refer to...) Figure 1 The method 100 includes the following steps 102 to 108.

[0014] In step 102, configure the system's low-power power domain to control the system to enter sleep mode.

[0015] In some embodiments, modules in the low-power power domain are enabled, and the audio data format required for recording is configured. The acquired audio signal is converted into audio data according to the audio data format; the audio data is then transferred to non-volatile memory via direct memory access. Furthermore, signal processing is performed on the data in the non-volatile memory. If the above process works normally, the system is controlled to enter a sleep mode.

[0016] In this way, low-power power domains are used to ensure that battery-powered products can operate for extended periods of time.

[0017] In step 104, audio data is continuously collected by the microphone in the low-power power domain.

[0018] In some embodiments, the audio signal acquired by the microphone is converted into audio data in Pulse Code Modulation (PCM) format by a low-power analog-to-digital converter (ADC) in the low-power power domain, the ADC having a low-precision state. Furthermore, the Pulse Code Modulation format audio data is transferred to static random access memory (Audio SRAM) by a low-power serial audio interface (Low Power SAI) and a low-power direct memory access (Low Power DMA) in the low-power power domain.

[0019] In this way, low-power analog-to-digital converters, low-power digital audio interfaces, and low-power direct memory access in the low-power power domain are used to store audio data in static random access memory, which facilitates subsequent audio data processing. Using a low-power microcontroller to process the audio data can effectively improve audio quality before abnormal sound detection, ensuring the reliability of abnormal sound detection. In addition, the low-power analog-to-digital converter is in a low-precision state, which can further reduce power consumption.

[0020] In some embodiments, if the microphone is a single microphone, the low-power microcontroller in the low-power power domain performs preprocessing DC filtering on the audio data in the static random access memory to obtain filtered data. The low-power microcontroller then performs noise reduction processing on the filtered data to obtain noise-reduced data. The low-power microcontroller uses automatic gain control (AGC) to adjust the amplitude of the noise-reduced data to obtain amplitude-adjusted data. Furthermore, the low-power microcontroller uses an equalizer to adjust the frequency response of the amplitude-adjusted data to obtain processed audio data.

[0021] In this way, when a single microphone is used to collect audio data, the audio data is preprocessed by DC filtering and noise reduction to improve the overall signal-to-noise ratio. Then, the amplitude is adjusted by automatic gain control technology and the frequency response is adjusted by an equalizer, which can improve the detection accuracy for subsequent abnormal sound detection.

[0022] In some embodiments, if the microphone is a multi-microphone, the low-power microcontroller in the low-power power domain performs preprocessing DC filtering on the audio data in the static random access memory to obtain filtered data. The low-power microcontroller then performs beamforming on the filtered data to obtain beamformed data. Next, the low-power microcontroller performs noise reduction on the beamformed data to obtain noise-reduced data. Finally, the low-power microcontroller uses automatic gain control to adjust the amplitude of the noise-reduced data to obtain amplitude-adjusted data. Furthermore, the low-power microcontroller uses an equalizer to adjust the frequency response of the amplitude-adjusted data to obtain processed audio data.

[0023] In this way, when multiple microphones (such as a microphone array) are used to collect audio data, the low-power microcontroller performs beamforming processing before the noise reduction step, which can further improve the signal strength for specific pickup angles, thereby improving the overall accuracy of pickup detection.

[0024] In step 106, the Always On Audio (AOA) controller of the low-power power domain performs abnormal sound detection on the audio data and obtains the detection result.

[0025] In some embodiments, the audio data in the static random access memory is processed by a low-power microcontroller (Low Power MCU) in the low-power power domain to obtain processed audio data.

[0026] In some embodiments, the continuous recording controller in the low-power power domain performs abnormal sound detection on the processed audio data to obtain a detection result. In some embodiments, the continuous recording controller in the low-power power domain performs initial signal detection on the processed audio data to obtain an initial detection result. If the initial detection result indicates that an abrupt signal has been detected, an artificial intelligence (AI) model is used to perform abnormal sound detection on the abrupt signal to obtain a detection result. The AI ​​model includes a convolutional neural network (CNN), a recurrent neural network (RNN), and a fully connected layer connected in sequence.

[0027] In this way, an initial signal detection is performed. If an abrupt signal is detected, an artificial intelligence model is used to detect abnormal sounds, achieving hierarchical detection. The initial signal detection can be achieved using a low-complexity algorithm. The initial signal detection filters out most relatively quiet states, avoiding repeated triggering of the relatively power-intensive artificial intelligence model detection, thus reducing system power consumption while ensuring detection accuracy. Furthermore, the artificial intelligence model includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected in sequence. The fully connected layer is used to fuse the audio features extracted by the convolutional neural network and the recurrent neural network, thereby mapping the probability of different types of audio, making the output detection results more accurate and reducing the false positive rate (waking up when it should not) and the false negative rate (waking up when it should not).

[0028] In step 108, if the detection result indicates that an abnormal sound has been detected, the system is then awakened.

[0029] In some embodiments, after step 108, the method 100 further includes: copying the audio data and the detection result from the static random access memory to the memory of the general power domain of the system; recording the state of the continuous recording controller; obtaining the abnormal sound event detection result; if the abnormal sound event detection result indicates the presence of an abnormal sound event, clearing the recording frame count during the wake-up period and returning to the state of recording the continuous recording controller; if the abnormal sound event detection result indicates the absence of an abnormal sound event, determining whether the recording frame count during the wake-up period meets the sleep threshold; if it does, returning to the state of the system entering sleep mode; and if it does not, returning to the state of recording the continuous recording controller. In some embodiments, the memory of the general power domain is double data rate synchronous dynamic random access memory (DDR).

[0030] In this way, after waking up the system, the audio data and detection results in the static random access memory are immediately copied to the memory in the general power domain of the system, so that the user can obtain relevant audio information and obtain the detection results of abnormal sound events, and determine whether to return to the low power state based on the detection results of abnormal sound events.

[0031] In some embodiments, after step 108, the method 100 further includes: controlling the low-power analog-to-digital converter to switch from the low-precision state to a normal-precision state.

[0032] In this way, since the initial signal detection tracks the relative changes in the signal envelope, while the artificial intelligence model detection detects the cepstral distribution characteristics, both detection algorithms focus on relative feature variables. Therefore, while maintaining a certain level of accuracy in relative features, the low-power analog-to-digital converter can appropriately reduce the quantization accuracy and signal-to-noise ratio of absolute variables. Thus, during the wake-up detection period, the low-power analog-to-digital converter is put into a low-precision state through parameter configuration, and then put into normal accuracy after the abnormal sound wake-up, which can effectively reduce power consumption.

[0033] Figure 2 This is a schematic diagram illustrating an upgraded AOV (Area of ​​Sound) incorporating continuous sound according to an embodiment of the present invention. (Refer to...) Figure 2 This invention implements AOA (Automatic Audio Detection) functionality for power-sensitive battery-based products and adds "ears" to the aforementioned silent battery-based cameras, enabling the silent video recorded by AOV (Automated Optical Video Recording) to include the sounds of the scene. Furthermore, this invention, based on continuous recording AOA, configures abnormal sound detection (AI-Audio Detection, AAD). Through the audio neural network module of the AAD within the AOA controller, it can detect specific sounds, such as human voices, the sound of shattering glass, dog barks, alarms, and crying, and wake the system accordingly. Moreover, this AAD function can be turned on or off at any time, and a specified network model can be switched in real time to detect different sounds. Upon camera wake-up, the low-power recording state immediately stops, and the length and status of the latest recording are recorded. The length of this low-power recording depends on the memory allocated to the low-power recording mode after sleep mode. Typically, to match the AOV wake-up cycle, such as a requirement of waking up once per second, memory for a 1-second low-power recording length can be allocated. After the system wakes up, the latest recording data from the low-power state and the current video frame are encoded and encapsulated together into a multimedia format file (such as a .mp4 file).

[0034] Normal recording mode requires the CPU and system to be working normally, with many power modules activated, resulting in high overall power consumption (at least 300mW), which is insufficient for the extended use of battery-powered products. In contrast, AOA ultra-low power recording mode allows the CPU and system to shut down, keeping only the AOA recording circuitry active, reducing overall power consumption to around 2.5mW, thus meeting the needs of battery-powered products for prolonged use.

[0035] Figure 3 This is a schematic diagram illustrating graded abnormal sound detection. (Refer to...) Figure 3 The initial signal detection, as the first level of detection, uses a low-complexity algorithm to track the audio signal envelope, statistically analyze the amplitude of stable noise, and identify the starting point of abrupt signals before triggering the second-level AI model detection. This first-level initial detection filters out most relatively quiet states, avoiding repeated triggering of the relatively power-intensive AI model detection.

[0036] Figure 4 This is a schematic diagram illustrating AI model detection. (Refer to...) Figure 4 After the initial signal detection detects an abrupt signal, it triggers the AI ​​model for detection. The AI ​​model's inference is hardened, using a dedicated computing unit for inference. During inference, dedicated RAM (Random Access Memory) is used to store intermediate variables, avoiding latency and power consumption issues caused by repeated data exchange with external storage. A CNN, RNN, and fully connected layer are sequentially connected to detect the input features, outputting different abnormal sound probabilities at the frame level. Post-processing is then performed, and the final detection result is output by statistically analyzing the abnormal sound frame probabilities of different durations. It should be noted that abnormal sounds, regardless of whether they are human voices or not, are defined as sounds with features of interest to the user. The CNN has three 5x2 convolutional layers (conv), the RNN has three recurrent neural network layers (rnn), and after the RNN, there is a one-dimensional convolutional layer (1x1conv), a fully connected layer (fullcon), and a classification layer (class).

[0037] The abnormal sound detection method of this invention employs streaming time-series processing to improve real-time performance. Audio signals are streamed in 16ms frames, with processing occurring simultaneously with data acquisition. This not only reduces the consumption of historical data storage but also enhances real-time performance. Furthermore, this invention introduces an RNN into a CNN, utilizing the long-term memory effect of RNNs to compensate for the short-term feature limitations of streaming processing, thus avoiding the problem of excessively short input feature time inherent in streaming processing.

[0038] Figure 5 This is a block diagram showing the feature extraction part of the AI ​​model. (Refer to...) Figure 5 After feature extraction, the extracted features are sequentially subjected to batch normalization (BN normalization), CNN processing, RNN processing, and activation before being input into a fully connected layer. Finally, probability normalization (Softmax sigmoid) is performed. This invention adds time-domain and frequency-domain RNNs sequentially to fuse the correlation information between the time and frequency dimensions, respectively. Finally, a fully connected layer fuses the time and frequency dimensions to map the audio probabilities of each category. Furthermore, for most network models, the normalization part is usually located at the beginning of the network, which is more conducive to the subsequent convolutional processing.

[0039] Figure 6 This is a flowchart illustrating the signal processing performed on audio data input from a single microphone. Figure 7 This is a flowchart illustrating the signal processing performed on audio data from multiple microphone inputs.

[0040] Reference Figure 6When audio data is input from a single microphone and a single PCM channel, the audio data undergoes preprocessing filtering, followed by noise reduction (ANR) on the filtered data. Automatic gain control (AGC) is then used to adjust the amplitude of the noise-reduced data, and finally, an equalizer (EQ) is used to adjust the frequency response of the amplitude-adjusted data, resulting in processed audio data. This further improves the accuracy of subsequent AAD (Automatic Amplifier Detection) stages.

[0041] Reference Figure 7 If the audio data is input from a multi-microphone, multi-channel PCM, the audio data input from the multi-channel is pre-processed and filtered, and then the filtered data is processed by beamforming, which can further improve the signal strength for a specific pickup angle, thereby improving the overall pickup detection accuracy. Next, the beamformed data is processed for noise reduction, and then automatic gain control technology is used to adjust the amplitude of the noise-reduced data. Finally, an equalizer is used to adjust the frequency response of the amplitude-adjusted data to obtain the processed audio data.

[0042] Figure 8 This is a flowchart illustrating a method for continuous recording according to an embodiment of the present invention. (Refer to...) Figure 8 The method includes steps 201 to 212.

[0043] In step 201, the system's low-power power domain is configured to control the system to enter sleep mode.

[0044] In some embodiments, modules in the low-power power domain are enabled, and the audio data format required for recording is configured. The acquired audio signal is converted into audio data according to the audio data format. The audio data is transferred to non-volatile memory via direct memory access. Signal processing is performed on the data in the non-volatile memory. If the above process works normally, the control system enters sleep mode.

[0045] In some embodiments, the control system enters sleep mode by shutting down the general power domain.

[0046] In some embodiments, relevant hardware control modules, including a clock module (RCOSC), a Low Power MCU, a microphone, a Low Power ADC, a Low Power SAI, and a Low Power DMA, are enabled, and the PCM format required for recording is configured, such as a sampling rate of 16kHz, single channel, and a bit depth of 16bit.

[0047] In step 202, audio data is continuously acquired by the microphone in the low-power power domain.

[0048] In step 203, the audio signal acquired by the microphone is converted into audio data in PCM format by the Low Power ADC of the low power domain, which has a low precision state.

[0049] In step 204, the PCM format audio data is transferred to the Audio SRAM by the Low Power SAI and Low Power DMA of the low power domain.

[0050] In some embodiments, the Low Power SAI in the low-power power domain communicates with the front-end Low Power ADC via the I2S (Inter-IC Sound) protocol to control and acquire the PCM format audio data returned by the Low Power ADC, and temporarily stores it in a FIFO (First-In-First-Out) buffer. The Low Power DMA in the low-power power domain transfers the PCM format audio data from the FIFO buffer to the Audio SRAM.

[0051] In step 205, the audio data in the Audio SRAM is processed by the Low Power MCU in the low power domain to obtain the processed audio data.

[0052] In some embodiments, if the microphone is a single microphone, the audio data in the audio SRAM is preprocessed by a Low Power MCU in the low-power power domain using DC filtering to obtain filtered data. The Low Power MCU then performs noise reduction (ANR) on the filtered data to obtain noise-reduced data. The Low Power MCU then uses Automatic Gain Control (AGC) technology to adjust the amplitude of the noise-reduced data to obtain amplitude-adjusted data. Finally, the Low Power MCU uses an equalizer (EQ) to adjust the frequency response of the amplitude-adjusted data to obtain processed audio data.

[0053] In some embodiments, if the microphone is a multi-microphone setup, the low-power MCU in the low-power domain performs preprocessing DC filtering on the audio data in the audio SRAM to obtain filtered data. The low-power MCU then performs beamforming on the filtered data to obtain beamformed data. Next, the low-power MCU performs noise reduction on the beamformed data to obtain noise-reduced data. Finally, the low-power MCU uses automatic gain control to adjust the amplitude of the noise-reduced data to obtain amplitude-adjusted data. Finally, the low-power MCU uses an equalizer to adjust the frequency response of the amplitude-adjusted data to obtain the processed audio data.

[0054] In step 206, the AOA controller in the low-power power domain performs abnormal sound detection on the processed audio data and obtains the detection results.

[0055] In some embodiments, the AAD in the AOA controller of the low-power power domain performs initial signal detection on the processed audio data to obtain an initial detection result. If the initial detection result indicates that an abrupt signal has been detected, an AI model is used to detect abnormal sound from the abrupt signal to obtain a detection result. The AI ​​model includes a CNN, an RNN, and a fully connected layer connected in sequence.

[0056] In step 207, determine whether the detection result is an abnormal sound. If yes, wake up the system; otherwise, return to step 206.

[0057] In some embodiments, waking up the system specifically involves: enabling the general power domain.

[0058] In step 208, the audio data and detection results in the Audio SRAM are copied to the DDR in the system's general power domain.

[0059] In some embodiments, the method may further include: updating the producer and consumer pointers of the audio loop buffer; and passing the audio data and detection results (including event information) corresponding to the consumer pointer in user mode to the user. The user receives the latest audio data and event information and can then perform further post-processing, such as more refined backend audio algorithm processing or audio encoding, encapsulating it together with the video data in a media format file.

[0060] In step 209, the status of the AOA controller is recorded.

[0061] In step 210, the abnormal sound event detection results are obtained.

[0062] In step 211, if the abnormal sound event detection result indicates the presence of an abnormal sound event, the recording frame count during the wake-up period is cleared, and the process returns to step 209.

[0063] In step 212, if the abnormal sound event detection result is that there is no abnormal sound event, then it is determined whether the recording frame count during the wake-up period meets the sleep threshold. If it does, then return to step 205; if it does not, then return to step 209.

[0064] In some embodiments, the method may further include: controlling the Low Power ADC to switch from a low-precision state to a normal-precision state.

[0065] In some embodiments, configurable parameters are also provided, which can be configured according to customer scenarios to make trade-offs between power consumption, latency, and accuracy, thereby increasing product flexibility. Taking the sound of breaking glass and dog barking as examples, with a signal-to-noise ratio of -3dB to 15dB, a signal amplitude of -12dB to -30dB, and a latency of 200ms, the algorithm simulation detection shows a false negative rate of 9.5% and a false positive rate of 4.5%. From the power consumption simulation results, in standby mode, the AOA controller's power consumption for initial detection + feature extraction + model detection, with the three working simultaneously, averages 0.25mW. Furthermore, users can customize the selection of sounds of interest for detection.

[0066] The method described above can be applied to battery-powered companion robots, action cameras, and eyeglasses. Battery-powered companion robots need to detect and wake up from sleep mode by human voice. This invention supports not only the detection of abnormal sounds (such as the sound of breaking glass in a home environment) but also the detection and wake-up of human voice. Action cameras enter deep sleep during use. If the user says a specific wake-up phrase, such as "I want to take a picture," this invention can detect this as a target abnormal sound and wake up the system. The wake-up phrase "I want to take a picture" is then sent to the backend algorithm for secondary recognition. If it is determined to be valid, the action camera is triggered to take a picture. Eyeglasses have extremely limited requirements for size and power consumption, making this invention suitable for voice wake-up and interaction on eyeglasses.

[0067] According to another aspect of the invention, Figure 9 This is a block diagram illustrating an apparatus 300 for continuous recording according to an embodiment of the present invention. (Refer to...) Figure 9The apparatus 300 for continuous recording includes a low-power power domain 302, which is configured to continue operating after the system enters a sleep mode. The low-power power domain 302 includes an audio acquisition and processing module and a continuous recording control module. The audio acquisition and processing module is configured to continuously acquire audio signals and convert the audio signals into audio data with a predetermined format. The continuous recording control module is configured to perform abnormal sound detection on the audio data to obtain a detection result; if the detection result indicates that an abnormal sound has been detected, the system is woken up.

[0068] Reference Figure 9 The device 300 for continuous recording also includes a general power domain 304, which is configured to shut down after the system enters a sleep mode. The general power domain 304 and the low-power power domain 302 are located in a system-on-a-chip (SoC).

[0069] In some embodiments, the low-power power domain 302 includes modules such as Low Power ADC, Low Power SAI, Low Power DMA, Low Power MCU, RCOSC, Audio SRAM, and AOA. Acoustic signals can be input through a single microphone or multi-microphone array, converted into analog signals, and then transmitted to the Low Power ADC. The data is then sequentially transmitted to Low Power SAI, Low Power DMA, Audio SRAM, Low Power MCU, and AOA. The RCOSC is the clock module built into the SoC, providing a recording reference clock. It is an oscillator generated by a circuit composed of resistor R and capacitor C, which has the advantage of lower power consumption compared to traditional crystal oscillators. The Low Power ADC can convert analog electrical signals into discrete digital signals and output them in the commonly used audio pulse code modulation (PCM) format. Low Power SAI is a commonly used digital audio interface on the SoC. It can communicate with the front-end ADC module via the I2S (Inter-IC Sound) protocol to control and acquire the PCM audio data returned by the ADC, and temporarily store it in a FIFO (First In, First Out) buffer. Low-Power DMA enables the SoC to periodically and continuously and accurately move PCM data from the source address (SAI FIFO) to the target address (Audio SRAM) without CPU or MCU intervention. The Low-Power MCU, a general-purpose processing unit operating in the low-power power domain, can perform signal processing at the AOA / AAD front end to improve overall audio signal quality. After system sleep, the DDR enters a self-refresh inaccessible state, requiring the Audio SRAM to store key data such as the acquired audio, DMA run list, AOA / AAD module firmware, and signal processing data. The AAD module in the AOA controller can detect abnormal sounds; when a target sound is detected, it can act as a wake-up source to wake the system, reducing the power consumption of the low-power power domain 302 to approximately 2.5mW.

[0070] In some embodiments, the general power domain 304 includes a user layer and an operating system. The user layer implements application layer post-processing, such as post-processing algorithms and video encoding encapsulation. The operating system includes other general-purpose controllers, a CPU, and DDR. The power consumption required by the general power domain 304 is greater than 300mW.

[0071] According to another aspect of the invention, Figure 10 This is a schematic diagram illustrating a device 400 for continuous recording according to an embodiment of the present invention. (Refer to...) Figure 10The electronic device 400 includes a memory 402 and at least one processor 404. The memory 402 is configured to store information associated with audio. The processor 404 is electrically coupled to the memory 402 and is configured to perform the steps of the method for continuous recording as described above.

[0072] In summary, according to the embodiments of this disclosure, the system is configured with a low-power power domain to control the system into sleep mode. The microphone in the low-power power domain continuously collects audio data, and the continuous recording controller in the low-power power domain performs abnormal sound detection on the audio data to obtain the detection result. If the detection result is that an abnormal sound is detected, the system is woken up and continuous recording is performed using the low-power power domain. After the system enters sleep mode, the low-power power domain continues to work and can also perform abnormal sound detection on the audio data collected by continuous recording, thereby realizing continuous recording in low-power mode and meeting the needs of battery-powered products for long-term use. In addition, during abnormal sound detection, a preliminary signal inspection is performed first. If an abrupt signal is detected, an artificial intelligence model is used for abnormal sound detection, achieving hierarchical detection. The preliminary signal inspection can be implemented using a low-complexity algorithm, filtering out most relatively quiet states and avoiding repeated triggering of the relatively power-intensive artificial intelligence model. This reduces system power consumption while ensuring detection accuracy. Furthermore, the artificial intelligence model includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected in sequence. The fully connected layer is used to fuse the audio features extracted by the convolutional neural network and the recurrent neural network, thereby mapping the probability of different types of audio, resulting in more accurate detection results.

[0073] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for continuous recording, characterized by, include: Configure the system's low-power power domain to control the system to enter sleep mode; Audio data is continuously collected by the microphone in the low-power power domain; The continuous recording controller in the low-power power domain performs abnormal sound detection on the audio data and obtains the detection result; as well as If the detection result indicates that an abnormal sound has been detected, the system is then activated.

2. The method of claim 1, wherein, Configuring the system's low-power power domain to control the system to enter sleep mode includes: Enable the modules in the low-power power domain and configure the audio data format required for recording; The acquired audio signal is converted into audio data according to the audio data format; The audio data is transferred to non-volatile memory via direct memory access; Perform signal processing on the data in the non-volatile memory; and If the above process works normally, the system will be controlled to enter hibernation mode.

3. The method of claim 1, wherein, The continuous acquisition of audio data by the microphone in the low-power power domain includes: The low-power analog-to-digital converter in the low-power power domain converts the audio signal acquired by the microphone into audio data with a pulse code modulation format, and the low-power analog-to-digital converter has a low-precision state. The audio data in the pulse code modulation format is transmitted to static random access memory via the low-power digital audio interface and low-power direct memory access of the low-power power domain; and The low-power microcontroller in the low-power power domain performs signal processing on the audio data in the static random access memory to obtain processed audio data. The continuous recording controller in the low-power power domain performs abnormal sound detection on the audio data, and the detection result is obtained by performing abnormal sound detection on the processed audio data by the continuous recording controller in the low-power power domain, and obtaining the detection result.

4. The method of claim 3, wherein, The low-power microcontroller in the low-power power domain performs signal processing on the audio data in the static random access memory to obtain the processed audio data, which includes: If the microphone is a single microphone, the low-power microcontroller in the low-power power domain preprocesses the audio data in the static random access memory by DC filtering to obtain filtered data. The low-power microcontroller performs noise reduction processing on the filtered data to obtain noise-reduced data. The low-power microcontroller uses automatic gain control technology to adjust the amplitude of the noise-reduced data, resulting in amplitude-adjusted data; and The low-power microcontroller uses an equalizer to adjust the frequency response of the amplitude-adjusted data to obtain processed audio data.

5. The method of claim 3, wherein, The low-power microcontroller in the low-power power domain performs signal processing on the audio data in the static random access memory to obtain the processed audio data, which includes: If the microphone is a multi-microphone, the low-power microcontroller in the low-power power domain preprocesses the audio data in the static random access memory by DC filtering to obtain filtered data. The low-power microcontroller performs beamforming processing on the filtered data to obtain beamformed data. The low-power microcontroller performs noise reduction processing on the beamforming data to obtain noise-reduced data. The low-power microcontroller uses automatic gain control technology to adjust the amplitude of the noise-reduced data, resulting in amplitude-adjusted data; and The low-power microcontroller uses an equalizer to adjust the frequency response of the amplitude-adjusted data to obtain processed audio data.

6. The method of claim 3, wherein, The continuous recording controller in the low-power power domain performs abnormal sound detection on the processed audio data, and the detection results include: The continuous recording controller in the low-power power domain performs initial signal detection on the processed audio data to obtain the initial detection result; and If the initial detection result indicates that an abrupt signal has been detected, an artificial intelligence model is used to perform abnormal sound detection on the abrupt signal to obtain a detection result. The artificial intelligence model includes a convolutional neural network, a recurrent neural network, and a fully connected layer connected in sequence.

7. The method of claim 3, wherein, After waking up the system, the following is also included: The audio data and the detection results in the static random access memory are copied to the memory in the general power domain of the system; Record the status of the continuous recording controller; Obtain the detection results of abnormal sound events; If the abnormal sound event detection result indicates the presence of an abnormal sound event, then the recording frame count during the wake-up period is cleared, and the execution of the continuous recording controller is returned to its current state. If the abnormal sound event detection result is that there is no abnormal sound event, then determine whether the recording frame count during the wake-up period meets the sleep threshold. If satisfied, the execution control system will return to sleep mode; and If the condition is not met, the execution record of the continuous recording controller's status will be returned.

8. The method of claim 3, wherein, After waking up the system, the following is also included: Control the low-power analog-to-digital converter to switch from the low-precision state to the normal-precision state.

9. An apparatus for continuous recording, characterized in that, include: A low-power power domain is configured to continue operating after the system enters sleep mode, and includes: An audio acquisition and processing module is configured to continuously acquire audio signals and convert the audio signals into audio data with a predetermined format; as well as The continuous recording control module is configured to perform abnormal sound detection on the audio data to obtain a detection result. If the detection result indicates that an abnormal sound has been detected, the system is woken up.

10. An electronic device, characterized in that, include: The memory is configured to store information associated with audio. as well as At least one processor is electrically coupled to the memory and configured to perform the method according to any one of claims 1 to 8.

Images