Multi-channel audio processing method and device for ultra-low power consumption end side AI chip

Abstract

The invention relates to the technical field of multi-channel audio processing, and discloses a multi-channel audio processing method and device for an ultra-low power consumption end side AI chip, and the method comprises the steps: carrying out the ADC conversion of a multi-channel analog audio signal from a microphone array, and obtaining a multi-channel digital audio data stream; performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; executing variational Bayesian inference to obtain an enhanced voice signal matrix; implementing two-stage cascade processing to obtain a voice event processing result; and executing an adaptive federated learning algorithm among the distributed devices to obtain globally optimized model parameters, and deploying the globally optimized model parameters to an ultra-low power consumption end side AI chip to execute multi-channel audio processing. And a complex audio processing algorithm can be efficiently operated on an end-side AI chip with limited computing resources.

Description

Technical Field

[0001] The present invention relates to the field of multi-channel audio processing technology, and in particular to a multi-channel audio processing method and device for an ultra-low power consumption end-side AI chip. Background Art

[0002] With the rapid development of artificial intelligence technology, the demand for high-quality voice interaction on edge devices such as smart homes, wearable devices and IoT sensors is growing. As the core technology of these devices, multi-channel audio processing can realize functions such as noise reduction, echo cancellation, sound source localization and speech recognition, significantly improving user experience. However, edge devices usually face strict power consumption constraints, limited computing resources and complex and changeable noise environments, which poses a huge challenge to multi-channel audio processing technology.

[0003] The current mainstream multi-channel audio processing technology mainly relies on two types of methods: traditional signal processing algorithms and deep neural networks. Although traditional signal processing algorithms such as beamforming and adaptive filtering have low computational complexity, their performance is limited in complex acoustic environments; and although deep neural networks have significant processing effects, they are too computationally burdened and have a huge number of model parameters (millions to billions), which seriously limits their application on low-power end-side devices. In addition, the training of neural network models relies on a large amount of labeled data, and the privacy issues of voice data make large-scale data collection difficult. Summary of the invention

[0004] The present invention provides a multi-channel audio processing method and device for an ultra-low power end-side AI chip. The present invention realizes efficient collaborative learning between end-side devices, solves the problems of model updating and lifelong learning, and enables complex audio processing algorithms to run efficiently on end-side AI chips with limited computing resources.

[0005] In a first aspect, the present invention provides a multi-channel audio processing method of an ultra-low power consumption end-side AI chip, and the multi-channel audio processing method of the ultra-low power consumption end-side AI chip includes: Performing ADC conversion on the multi-channel analog audio signals from the microphone array to obtain a multi-channel digital audio data stream; Performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; Performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; Implementing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; According to the voice event processing results, an adaptive federated learning algorithm is executed among distributed devices to obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

[0006] In a second aspect, the present invention provides a multi-channel audio processing device of an ultra-low power consumption end-side AI chip, wherein the multi-channel audio processing device of the ultra-low power consumption end-side AI chip comprises: A conversion module, used for performing ADC conversion on the multi-channel analog audio signal from the microphone array to obtain a multi-channel digital audio data stream; A transformation processing module, used for performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; An inference module, used for performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; A cascade processing module, used for performing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; A deployment module is used to execute an adaptive federated learning algorithm among distributed devices according to the voice event processing results, obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

[0007] In the technical solution provided by the present invention, a two-stage cascade processing architecture is adopted. The lightweight adaptive Wiener filter with a power consumption of less than 1mW in the first stage runs continuously, and the compressed neural network with 8-bit quantization in the second stage is activated only when key events are detected, so that the average power consumption of the system is controlled below 2mW. Combined with the layered Dirichlet process noise modeling and the variational Bayesian inference algorithm, the non-independent and non-identical distribution characteristics of noise are effectively captured, and the audio processing quality in complex environments is improved. Through the teacher-student knowledge distillation framework and structured pruning, the server-level large model is compressed into a lightweight model on the end side, reducing the number of parameters by 98% while maintaining a recognition accuracy of more than 90%. The frequency domain is divided into 25 Mel-band shared filter coefficients to reduce the amount of calculation, and the storage requirements are optimized by resetting 95% of the weights to zero. The introduced adaptive federated learning algorithm dynamically adjusts the number of iterations according to the computing power of the end-side device, and 1-4-bit gradient quantization and difference coding reduce communication overhead. A block loading strategy is adopted to store the resident core blocks (accounting for 30% of the total parameters) in the on-chip SRAM. The compressed sparse row storage method reduces the storage space by 80%, enabling the system to adapt to different end-side hardware resource constraints and achieve a balance between power consumption, processing quality and resource adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0009] Figure 1 This is a schematic diagram of the steps of a multi-channel audio processing method of an ultra-low power consumption end-side AI chip in an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a multi-channel audio processing device of an ultra-low power consumption end-side AI chip in an embodiment of the present invention. DETAILED DESCRIPTION

[0010] Embodiments of the present invention provide a multi-channel audio processing method and device for an ultra-low power end-side AI chip. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0011] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the multi-channel audio processing method of the ultra-low power consumption end-side AI chip in the embodiment of the present invention includes: Step S1, performing ADC conversion on a multi-channel analog audio signal from a microphone array to obtain a multi-channel digital audio data stream; It is understandable that the execution subject of the present invention can be a multi-channel audio processing device of an ultra-low power consumption end-side AI chip, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0012] Specifically, the microphone array is arranged and the signal is collected. The microphones are arranged in a linear array so that the spacing between them is controlled within an appropriate range, thereby optimizing the spatial sampling characteristics and ensuring the coherence of the multi-channel signal. On this basis, the microphone array is used to simultaneously collect sound wave information from different directions to form an original audio analog signal containing multi-channel data, which can reflect the multi-source sound characteristics in the environment. The original audio analog signal is input into the ADC converter for analog-to-digital conversion. The ADC converter adopts a low-power architecture to reduce energy consumption during the conversion process. By setting a reasonable sampling rate and quantization accuracy, it is ensured that the converted digital signal can maintain the details of the original audio without increasing too much data storage and calculation burden. During the sampling process, the analog signal of each microphone channel is input into the ADC converter for synchronous analog-to-digital conversion to obtain the initial digital audio data. The initial digital audio data is processed by time domain framing. Since the audio signal is dynamically changing, and digital signal processing relies on short-time analysis, the data is segmented at fixed time intervals to make it short-term stable. A certain frame length and frame shift parameters are used to ensure that the computational efficiency of frequency domain conversion is improved while maintaining time resolution. After framing, the continuous audio signal is divided into multiple independent short-time segments, each of which is regarded as a relatively static signal. After framing, in order to reduce spectrum leakage and improve the quality of the transformed spectrum, the framed digital audio data is windowed. The Hamming window function is applied to weight the signal to make its boundary transition smoothly, thereby avoiding high-frequency interference caused by directly truncating the time domain signal. The windowed data can improve the spectrum resolution and reduce noise interference in the calculation process. The windowed audio data frame is input into the fast Fourier transform processing unit, and the time domain signal is converted into frequency domain representation through butterfly operation. The spectrum information of the audio signal is quickly extracted, including the amplitude and phase of different frequency components, and the audio data represented in the frequency domain is obtained. Preliminary energy filtering is performed on the audio data represented in the frequency domain to reduce data redundancy and reduce the complexity of subsequent calculations. The energy distribution of the frequency domain data is analyzed to identify the high noise area and the low information area. The signal in the lower frequency band contains a lot of environmental noise, while the signal with too high frequency is system noise or irrelevant components. Appropriate high-pass and low-pass filters are used to reduce these invalid signals. Through this step, the main voice information is effectively retained and a multi-channel digital audio data stream is obtained.

[0013] Step S2, performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; Specifically, the multi-channel digital audio data stream is reframed, and the frame length and overlap rate are readjusted to optimize the time-frequency resolution and computational complexity. By setting appropriate frame length and frame shift parameters, adjacent frames have a reasonable overlap, so as to enhance short-term stability in the time domain, while ensuring that the spectrum after frequency domain conversion has sufficient resolution. After completing the reframe processing, in order to reduce spectrum leakage and improve the effect of subsequent transformation, a windowing operation is applied to each frame of audio data. The reframed audio data is multiplied by a preset Hamming window function coefficient. The Hamming window is a commonly used weighted window function that can effectively reduce the spectral energy diffusion caused by the discontinuity of the frame end, thereby improving the accuracy of frequency domain transformation. The Hamming window is applied to the data frame of each microphone channel to make the boundary transition of the signal smoother and suppress the high-frequency interference caused by the truncation effect. The audio data after windowing is input into the fast Fourier transform operation unit to realize the signal conversion from the time domain to the frequency domain. In this process, the butterfly operation is performed using the fast Fourier transform algorithm to quickly calculate the spectrum amplitude and phase information of the input signal. The calculation process of fast Fourier transform can efficiently decompose the different frequency components in the signal and generate frequency domain representation, so that the original time domain waveform is converted into complex data containing frequency characteristics. After obtaining the spectrum amplitude and phase information, the spectrum of the multi-channel signal is further processed to extract useful information and reduce noise interference. Based on the extracted spectrum amplitude and phase information, a frequency domain adaptive filter is established to achieve optimized noise suppression. The adaptive filtering algorithm is used to dynamically calculate the optimal noise suppression filter parameters so that it can automatically adjust the filtering characteristics under different acoustic environments. The design of the filter takes into account the covariance characteristics of different channel signals to ensure that background noise can be effectively removed under multi-channel conditions while maintaining the integrity and clarity of the speech signal. By optimizing the filter parameters based on the minimum mean square error criterion, the filter can adaptively adjust the gain in different frequency regions to maximize the suppression of environmental noise. The calculated noise suppression filter parameters are applied to the frequency domain signals of each channel, and the signal filtering operation is performed through frequency domain multiplication to achieve preliminary noise reduction processing. The frequency domain data of each channel is processed by the filter, so that the high-energy noise component is effectively attenuated, while the main frequency components of the speech signal are retained. The frequency domain signal after preliminary noise reduction is reorganized into a feature matrix form. The frequency domain data is arranged according to a predefined matrix structure, where the rows of the matrix represent different frequency point indexes and the columns of the matrix correspond to different microphone channels. Each element in the matrix stores a frequency domain coefficient in the form of a complex number, which contains the amplitude and phase information of the signal, thereby representing the spectral characteristics of the multi-channel audio signal.

[0014] Step S3, performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; Specifically, the multi-channel frequency domain feature matrix is ​​reorganized to construct a frequency point observation signal matrix suitable for subsequent inference calculation. Since the multi-channel frequency domain feature matrix contains complex spectrum data of different channels and different frequency points, the data is rearranged according to the frequency point dimension so that each frequency point corresponds to an independent observation signal matrix, the rows of which represent different microphone channels, and the columns represent time frame sequences. The noise covariance matrix model is constructed for the frequency point observation signal matrix to reflect the noise correlation between different microphone channels. The noise covariance matrix is ​​defined, and its prior distribution is assumed to follow the inverse Wishart distribution to ensure the stability of the covariance matrix in numerical calculation, and to effectively capture the distribution characteristics of the noise components, so as to obtain the noise covariance prior model. Based on the noise covariance prior model, the hierarchical Dirichlet process is introduced to obtain a more flexible noise model. Through the hierarchical Dirichlet process, the noise distribution of different frequency points is non-parametrically modeled, so that the noise change pattern is adaptively adjusted through data without presetting a fixed number of noise categories. This method can effectively cope with a variety of complex noise scenarios and improve the robustness of the speech enhancement algorithm without increasing additional computational overhead. On the basis of noise modeling, according to the low-rank characteristics of the frequency observation signal matrix, a decomposition model of the speech signal is constructed. Since the speech signal has a strong low-rank structure in the frequency domain, that is, most of the speech energy is concentrated in a specific frequency mode, the low-rank plus sparse decomposition method is used to separate the speech signal from the background noise. In this model, the observation signal is decomposed into speech signal components and noise components, where the representation of the speech signal is estimated by the low-rank matrix decomposition method, and the noise part is modeled by the noise model constructed previously. In order to optimize the parameters of the speech signal decomposition model and enable it to be adaptively adjusted under different environmental conditions, the variational Bayesian inference algorithm is used to optimize the calculation by maximizing the lower bound of evidence. In this process, the variational inference method is used to estimate the posterior distribution of the speech signal and noise model, and the model parameters are optimized by an alternating iterative strategy. In each round of iteration, in order to ensure that the computational complexity is within a controllable range, the calculation process is reasonably constrained so that the computational complexity of each round of iteration does not exceed the preset target value, thereby ensuring efficient operation on the ultra-low power end-side AI chip. In the calculation process, in order to improve the convergence speed and numerical stability, the initial parameters of the variational distribution are reasonably set, and an adaptive update strategy based on statistical characteristics is adopted to ensure that the algorithm can reach a convergence state within a small number of iterations. After the variational inference process converges, the final enhanced speech signal matrix is ​​calculated based on the obtained model parameters. Since the speech enhancement calculation process involves a large number of exponential and logarithmic operations, in order to reduce the computational complexity and optimize power consumption, the lookup table method is used to accelerate the calculation of exponential and logarithmic functions.By pre-calculating the exponential and logarithmic function values ​​within different input ranges and directly querying the lookup table during the inference process, the overhead of floating-point operations is greatly reduced and the computational efficiency is improved. In order to ensure that the inference process terminates the calculation when a reasonable convergence state is reached, a threshold for the relative change of the lower bound of the evidence is set, and the iterative calculation is stopped when the relative change of the lower bound of the evidence is less than the threshold, so as to avoid unnecessary computational overhead and ensure the optimization of computational efficiency. The enhanced speech signal matrix is ​​obtained through the above steps, which effectively improves the quality of the speech signal and significantly reduces the impact of environmental noise.

[0015] Step S4, performing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; Specifically, an adaptive Wiener filter is constructed based on the enhanced speech signal matrix to realize the filter structure of the first-level processing. The design of the Wiener filter dynamically adjusts the filter weight according to the statistical characteristics of the enhanced speech signal, so that it can adaptively optimize the signal quality in different noise environments. In order to reduce the computational complexity and ensure the low-power operation of the end-side chip, the coefficient calculation of the filter is based on the estimated result of the signal-to-noise ratio, and the filter gain is dynamically adjusted in different frequency ranges to achieve the optimal signal enhancement effect. The filter structure of the first-level processing is optimized by frequency domain division, and the filter calculation is divided into several frequency band regions, and the same filter coefficients are shared in each frequency band, thereby reducing the number of independent calculations. In this optimization process, the frequency band is divided according to the Mel scale to ensure perceptual consistency, while enabling the low-power band filter to cover the key speech information without performing complex calculations on unnecessary frequency components. Based on the optimized low-power band filter, an acoustic feature monitoring module is constructed to continuously monitor the acoustic features of the input signal in low-power mode and provide a trigger mechanism for the second-level processing. In this module, characteristic parameters such as the energy envelope and zero-crossing rate of the audio signal are calculated, and a dynamic threshold setting method is used to adapt to different environmental noise levels, so that when a voice event is detected, the activation process of the second-level processing unit can be triggered in time. After the trigger mechanism of the second-level processing is established, when the characteristics of the input signal meet the trigger conditions, the system enters the voice event detection mode and activates the second-level processing module. At the same time, the Mel spectrum features are extracted from the enhanced voice signal matrix and sent to the deep learning model as the target input features for analysis. Mel spectrum features are a feature representation method for speech signal analysis, which can effectively capture the spectral structure of speech signals and retain key information in the denoised signal. In the feature extraction process, the enhanced speech signal is short-time Fourier transformed, and the power spectrum in different frequency bands is calculated, and then converted through the Mel filter bank to obtain feature input suitable for deep neural network processing. The feature extraction process is performed in low power mode to ensure that the second-level processing unit completes the calculation in the shortest time and minimizes the occupation of computing resources. The target input features are sent to the compressed neural network model to perform voice event classification. The neural network model uses structured pruning and 8-bit quantization technology to enable it to run efficiently on the end device while maintaining a high classification accuracy. The structure of the neural network includes an input layer, multiple hidden layers, and an output layer, where the parameters of the hidden layers are compressed to reduce the amount of calculation and optimize energy consumption. During the inference process, low-precision integer operations are used to replace traditional floating-point calculations, and the calculation of activation functions is accelerated by lookup tables, thereby improving the execution efficiency of the model. When the neural network completes the inference, it outputs the corresponding voice event classification results, including voice recognition categories, keyword detection, or specific voice commands.Decision fusion is performed on the results of the first-level processing and the results of the second-level processing to generate the voice event processing results. In the decision fusion process, the signal enhancement effect of the first-level Wiener filter and the confidence of the second-level neural network classification are comprehensively considered to determine the final voice event decision. At the same time, in order to ensure the low-power operation of the system, the different processing modules in the chip are dynamically managed so that the second-level processing unit is only activated when necessary and returns to deep sleep mode immediately after the calculation is completed to reduce power consumption. In the actual operation process, the operating status of the processing module is continuously monitored, and the power supply strategy is dynamically adjusted according to the task requirements, so as to maximize the battery life of the device while ensuring the computing performance. Through the above steps, the voice event processing results containing the enhanced voice signal and the voice event classification label are obtained.

[0016] Step S5: Execute an adaptive federated learning algorithm among distributed devices according to the voice event processing results, obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

[0017] Specifically, a teacher-student model framework is established to achieve knowledge transfer and compression between the powerful computing resources on the server side and the low power consumption requirements of the end-side devices. In this framework, the teacher model adopts a server-level large-scale Transformer architecture, including twelve attention heads and six Transformer layers, to use its powerful feature extraction capabilities to deeply model the voice data, while the student model adopts a streamlined end-side structure, including only two attention heads and two Transformer layers, to ensure that the computational overhead is low enough to enable it to run efficiently on the end-side device, and obtain the initial teacher-student model structure. After the initial model structure is constructed, in order to effectively perform knowledge distillation optimization, a hybrid loss function is defined to ensure that the student model can learn the deep feature representation of the teacher model as much as possible under the condition of limited computing power. The loss function consists of hard label cross entropy loss, soft label distillation loss and feature-level distillation loss. The hard label cross entropy loss is used to ensure the basic classification ability of the student model. The soft label distillation loss uses the output probability distribution of the teacher model to help the student model learn smoother category boundaries. The feature-level distillation loss minimizes the feature differences between the teacher model and the student model at different levels, so that the student model can gradually approach the expressiveness of the teacher model, thereby obtaining the knowledge distillation optimization target. Based on this optimization target, in order to improve the efficiency of distillation and enhance the expressiveness of the student model, a hierarchical distillation strategy is introduced to align the output information of different levels of the server-level teacher model with the end-side student model at different levels. In this process, the output of the attention mechanism of each layer of the teacher model and the output of the feedforward network are respectively aligned with the corresponding layers of the student model, and inter-layer knowledge transfer is performed by minimizing the L2 distance, so that the student model can not only learn global semantic information, but also capture structural features at different levels. In this process, by controlling the weight distribution of distillation, the student model can gradually adapt to the computing limitations of the end-side device while maintaining a high recognition accuracy, thereby establishing a stable multi-level knowledge transfer mechanism. Distillation training is performed on the multi-level knowledge transfer mechanism to ensure that the student model effectively learns the core knowledge of the teacher model. During the training process, real data from the end-side device is used for fine-tuning so that the student model adapts to the specific noise and signal characteristics of the end-side environment. Then, based on the variational Bayesian optimization method, the distillation loss weights are dynamically adjusted so that knowledge transfer at different levels can be carried out under the optimal trade-off state. After the distillation training is completed, the preliminary trained student model is obtained. Due to the strict limitations of the end-side computing resources, the preliminary trained student model is then subjected to sparse processing. During the sparse process, L1 regularization and sensitivity analysis methods are used to identify and retain the weight parameters that contribute most to the model performance, and the remaining weights are quantized to zero to reduce computational complexity.In this process, the weight matrix is ​​sparsely processed by the structured pruning method, so that the parameter scale of the model can be compressed to less than 2% of the original teacher model, while maintaining more than 90% recognition accuracy, and a sparse student model is obtained. In order to improve the computing efficiency of the model on the end-side device, the sparse student model is quantized by 8 bits to reduce the storage space and power consumption required for calculation. In the process of fixed-point quantization, the floating-point weights are mapped to a fixed quantization space by using logarithmic quantization and clustering methods, and the continuous values ​​are converted into a finite discrete value set by building a quantization index table to reduce the memory bandwidth requirement in the calculation. In this process, in order to maintain the stability of the quantized model during the inference process, the quantization-aware training method is used to simulate the quantization error during training so that the model can still maintain a high recognition accuracy after quantization. After quantization, the lightweight proxy model parameters suitable for the end-side device are obtained. The adaptive federated learning algorithm is executed on the quantized proxy model parameters among distributed devices. In the federated learning stage, the adaptive federated learning algorithm is executed on multiple end-side devices to integrate the data distribution information of different devices and optimize the global model parameters. In this process, each device updates the model based on its own local data and periodically sends the updated parameters to the central server for aggregation. During the aggregation process, an adaptive computing time strategy is adopted to dynamically adjust the number of iterations of local training according to the computing power of the device, so that devices with stronger computing power can perform more rounds of local optimization, while devices with weaker computing power perform fewer iterations to ensure the fairness and efficiency of global training. At the same time, in order to reduce the communication bandwidth requirements, gradient quantization and difference coding techniques are used in the parameter transmission process to only transmit the parameter update amount relative to the previous round of training, and Huffman coding is used to further compress the data to reduce communication overhead. In the training process of federated learning, a node clustering method based on parameter similarity matrix is ​​introduced to identify devices with similar feature distributions and share model update information within the same cluster to improve the generalization ability and convergence speed of the model. After the federated learning is completed, the globally optimized model parameters are obtained and deployed to the ultra-low power end-side AI chip to perform multi-channel audio processing tasks. During the deployment process, the model parameters are reformatted into a storage format suitable for the end-side computing architecture, and compressed sparse row storage is used to reduce storage requirements. A dynamic load adjustment mechanism is implemented on the end-side chip, so that different computing units can dynamically switch operating modes according to task requirements during the inference process to further optimize power consumption. In actual operation, the heterogeneous computing architecture of the end-side chip is used to assign different computing tasks to the most appropriate computing unit, such as assigning feature extraction tasks to the DSP, assigning deep learning inference tasks to the neural network accelerator, and leaving control logic and simple arithmetic operations to low-power microcontrollers, thereby achieving optimal computing efficiency and energy utilization.

[0018] The quantized proxy model parameters and compressed neural network models are organized into distributed device nodes to build a federated learning infrastructure. Multiple end-side devices are defined as independent working nodes, each of which performs model reasoning and training locally and periodically interacts with the central server to achieve decentralized model optimization. At the same time, in order to adapt to the computing power and communication bandwidth of different devices, a hierarchical federated learning strategy is adopted so that nodes with strong computing resources can take on more computing tasks, while resource-constrained devices only perform necessary local updates, thereby ensuring that the entire system operates efficiently in a low-power mode. After the federated learning infrastructure is established, local update calculations are performed on each working node to obtain the local update parameters of the node. Each device performs mini-batch gradient descent or other optimization methods based on its own collected multi-channel audio data to minimize the local loss function and update the local model parameters. Since end-side devices usually have limited computing power, lightweight optimization algorithms such as quantized stochastic gradient descent are used in the local update process to reduce computing overhead and storage requirements while maintaining the convergence of the model. In order to further improve the computational efficiency of federated learning and reduce the computational burden between devices, an adaptive computation time mechanism is introduced to dynamically adjust the number of iterations of each node after the local update parameter calculation is completed. Under this mechanism, different computing tasks are dynamically allocated according to the computing power and communication status of each device, so that devices with stronger computing power can perform more rounds of local updates, while devices with weaker computing power perform fewer iterations, thereby optimizing the computational load balance of the entire distributed system. Through the adaptive iteration adjustment mechanism, the end-side computing resources are utilized to the greatest extent, while avoiding system bottlenecks caused by differences in computing power, and improving the overall convergence speed of federated learning. After completing the adaptive iteration adjustment, the local gradients of each node are quantized and compressed to reduce the amount of data transmitted during the communication process. In this process, the gradient quantization technology is used to map the continuous gradient values ​​to a limited set of discrete values, and different quantization levels are allocated between different parameter groups through the optimal bit allocation method to minimize the amount of data transmitted while ensuring the accuracy of the model. In order to further reduce communication overhead, differential coding technology is used to transmit only the changes in model parameters relative to the previous round, and the variable-length coding algorithm is used to further compress the quantized gradient, thereby optimizing bandwidth utilization and reducing communication power consumption of the end-side device. After obtaining the compressed gradient information, a parameter similarity matrix is ​​constructed to identify the parameter update mode between different nodes, and node clustering is performed based on the matrix to form an efficient knowledge sharing mechanism. In this process, the cosine similarity between the parameter vectors updated by different nodes is calculated, and a clustering algorithm is constructed based on the similarity matrix to group devices with similar learning patterns into the same cluster to reduce redundant model update calculations.In order to adapt to the dynamically changing device distribution and data characteristics, a hierarchical clustering method is adopted to enable devices at different levels to share knowledge at different granularities, thereby optimizing the generalization ability of the model and improving its adaptability in different environments. After completing node clustering and obtaining the knowledge sharing results after clustering, global parameter aggregation is performed on the parameter server to integrate local updates from different devices and calculate the final global optimization model parameters. In this process, a weighted average method is used to weight the update parameters of each node according to the local data volume and computing contribution to ensure that the optimization direction of the model can maximize the representation of the distribution characteristics of all devices. In order to avoid the excessive influence of a single device on the model update, a dynamic learning rate adjustment mechanism is introduced to smooth the parameter updates between different devices, thereby improving the stability and convergence speed of the global model. After obtaining the globally optimized model parameters, the compressed neural network model is updated based on these optimized parameters, and the updated model is deployed to the ultra-low power end-side AI chip to perform multi-channel audio processing. During the deployment process, the optimized model parameters are converted into a storage format suitable for the end-side hardware, and a sparse matrix storage method is used to reduce storage space requirements. At the same time, during the model reasoning process, the dynamic power management strategy enables different computing modules to dynamically adjust their working modes according to the task load to further optimize power consumption. In the final end-side operation process, this method achieves efficient voice processing, and through the optimization of federated learning, the model can be adaptively adjusted under different environmental conditions, thereby ensuring its long-term stable voice processing capabilities.

[0019] In the embodiment of the present invention, a two-stage cascade processing architecture is adopted. The lightweight adaptive Wiener filter with a power consumption of less than 1mW in the first stage runs continuously, and the compressed neural network with 8-bit quantization in the second stage is activated only when a key event is detected, so that the average power consumption of the system is controlled below 2mW. Combined with the layered Dirichlet process noise modeling and the variational Bayesian inference algorithm, the non-independent and non-identical distribution characteristics of the noise are effectively captured, and the audio processing quality in complex environments is improved. Through the teacher-student knowledge distillation framework and structured pruning, the server-level large model is compressed into a lightweight model on the end side, reducing the number of parameters by 98% while maintaining a recognition accuracy of more than 90%. The frequency domain is divided into 25 Mel-band shared filter coefficients to reduce the amount of calculation, and the storage requirements are optimized by resetting 95% of the weights to zero. The introduced adaptive federated learning algorithm dynamically adjusts the number of iterations according to the computing power of the end-side device, and the 1-4-bit gradient quantization and difference coding reduce the communication overhead. A block loading strategy is adopted to store the resident core blocks (accounting for 30% of the total parameters) in the on-chip SRAM. The compressed sparse row storage method reduces the storage space by 80%, enabling the system to adapt to different end-side hardware resource constraints and achieve a balance between power consumption, processing quality and resource adaptability.

[0020] In a specific embodiment, the process of executing step S1 may specifically include the following steps: Collect multi-channel analog audio signals from a microphone array through a linear array to obtain an original audio analog signal; Inputting the original audio analog signal into an ADC converter for analog-to-digital conversion to obtain initial digital audio data, and performing time domain framing processing on the initial digital audio data to obtain framed digital audio data; Applying a Hamming window function to the framed digital audio data to perform windowing to obtain a windowed audio data frame; The audio data frame after windowing is input into the fast Fourier transform processing unit, and the time domain signal is converted into the frequency domain through butterfly operation to obtain the audio data represented in the frequency domain; A preliminary energy filtering is performed on the audio data represented in the frequency domain to obtain a multi-channel digital audio data stream.

[0021] Specifically, a microphone array is used to collect multi-channel analog audio signals from the sounds in the environment in a linear arrangement. Assume that the microphone array consists of The distance between each microphone is , set between 5cm and 15cm to ensure that the spatial sound information can be effectively captured while avoiding too low or too high coherence between microphones. Set the direction of the microphone array to axis, and assume that the position of a sound source in space is given by Given, where is the distance between the sound source and the center of the array, is the incident angle relative to the normal direction of the microphone array. Under this condition, there is a time delay in the signals received by different microphones, and its delay model is expressed as: in, Indicates The delay of the signal received by each microphone, is the speed of sound (about 343 m / s), The value range is 0 to The time delay relationship determines the relative phase offset of different microphone signals, and this phase information will be used for signal enhancement and direction estimation in the subsequent frequency domain processing. The multi-channel analog audio signal is input into the ADC (analog-to-digital converter) for analog-to-digital conversion. The ADC adopts a successive approximation register architecture. Compared with the traditional Σ-Δ modulator, it has higher conversion efficiency and lower power consumption, which is suitable for the low power consumption requirements of the end-side AI chip. Set the ADC sampling rate to kHz, and the quantization accuracy is 16 bits to ensure sufficient time resolution and dynamic range. After ADC processing, each microphone channel obtains a corresponding digital signal ,in Represents a discrete time step, and the data format is a 16-bit PCM-encoded integer value. Since the audio signal is time-varying, in order to perform subsequent frequency domain analysis and speech enhancement, the initial digital audio data is framed in the time domain. Set the length of each frame to sampling points (corresponding to a 25ms time window), the frame shift is sampling points (corresponding to a 10ms sliding window), thus forming a frame sequence with a 75% overlap rate. Overlapping processing can improve the stability of spectrum estimation while ensuring time resolution and reduce spectrum distortion in short-time signal analysis. After framing, in order to reduce the problem of spectrum leakage, the Hamming window function is applied to each frame of audio data for windowing operation to make the signal boundary transition smoother, thereby optimizing the subsequent Fourier transform results. The Hamming window function is defined as: in, is the weight factor of the window function, The value range is 0 to The windowed signal is expressed as: in, It is the original framed digital signal. is the signal after windowing. Through the weighting of the Hamming window, both ends of the frame are smoothed, thereby reducing the spectrum energy diffusion problem caused by direct truncation. The audio data frame after windowing is input into the fast Fourier transform processing unit to realize the signal conversion from time domain to frequency domain. In the fast Fourier transform calculation process, an efficient algorithm based on butterfly operation is used to convert The time domain signal of the point is mapped to the frequency domain to obtain the spectrum representation. The calculation formula of the fast Fourier transform is as follows: in, is the complex spectrum in the frequency domain, The value range is 0 to , its real part represents amplitude information, and its imaginary part represents phase information. After the fast Fourier transform calculation, the spectral information of the signal is used for subsequent speech enhancement, noise suppression and feature extraction. Perform preliminary energy filtering on the frequency domain data after the fast Fourier transform. Since the main energy of human speech is concentrated between 100Hz and 7000Hz, and the part below 100Hz is mainly environmental noise, and the part above 7000Hz usually contains high-frequency noise and non-speech components, in the energy filtering stage, a high-pass filter is applied to remove signals below 100Hz, and a low-pass filter is applied to remove signals above 7000Hz. In order to suppress narrowband noise and burst interference, the energy distribution of the spectrum is calculated, and a dynamic threshold is set to further filter out the low-energy frequency band. Specifically, each frequency point is calculated The power spectral density is: Calculate the average power of the entire spectrum And set the energy threshold : in, is the control coefficient, which is set between 0.1 and 0.3 to ensure effective noise suppression without losing key speech information. The frequency components of the speech signal are set to zero, thereby reducing unnecessary noise components and optimizing the intelligibility of the speech signal.

[0022] In a specific embodiment, the process of executing step S2 may specifically include the following steps: Re-framing the multi-channel digital audio data stream to obtain re-framed multi-channel audio data; The re-framed multi-channel audio data is multiplied by a preset Hamming window function coefficient, and a windowing operation is performed on each frame signal of each channel to obtain windowed audio data; Perform fast Fourier transform operation on the windowed audio data to obtain spectrum amplitude and phase information; A frequency domain adaptive filter is established according to the spectrum amplitude and phase information to obtain the noise suppression filter parameters; Apply the noise suppression filter parameters to the frequency domain signals of each channel, perform signal filtering operations through frequency domain multiplication operations, and obtain frequency domain signals after preliminary noise reduction; The frequency domain signal after preliminary noise reduction is reorganized into a feature matrix form, where the matrix rows represent frequency point indexes, the columns represent different microphone channels, and the matrix elements store frequency domain coefficients in complex form to obtain a multi-channel frequency domain feature matrix.

[0023] Specifically, the multi-channel digital audio data stream is reframed to optimize the time-frequency resolution and improve the accuracy of signal analysis. The length of each frame is set to , the frame shift is ,in controls the frequency resolution of the spectrum analysis, and Set the frame overlap rate to ensure the stability of short-term signals. Assume that the input audio signal is ,in Represents the microphone channel index, Represents the time sampling point. After framing, each frame signal is expressed as ,in Represents the frame index, For the Frames of audio data. Ensure that there is a certain degree of overlap between frames to reduce information loss caused by boundary effects and optimize subsequent frequency domain analysis. After completing the framing, in order to reduce spectrum leakage and improve the resolution of the signal, each frame of audio data is multiplied by the preset Hamming window function coefficient to implement the windowing operation. The role of the Hamming window is to smooth the signal boundary and reduce the spectrum diffusion caused by direct truncation. Set the window function As a Hamming window, its expression is: The windowed signal is expressed as: in, represents the short-time audio signal after windowing. This step ensures that the spectrum after subsequent transformation can maintain a high frequency resolution and reduce sidelobe interference. After the windowing process is completed, the signal is sent to the fast Fourier transform unit to convert the time domain signal into a frequency domain representation. The calculation formula is as follows: in, It is Frame in The spectral coefficients of each frequency point contain amplitude and phase information. Through fast Fourier transform, the waveform signal in the time domain is converted into a spectral representation in complex form, where the amplitude information describes the energy distribution of the signal at different frequencies, and the phase information contains the timing characteristics of the signal. After obtaining the spectral amplitude and phase information, in order to enhance the speech signal and suppress background noise, a frequency domain adaptive filter is established to calculate the noise suppression filter parameters. In this process, the spectral power of each microphone channel is calculated, and the characteristics of the ambient noise are estimated. Assume that the short-time power spectrum of the signal is expressed as: In order to estimate the noise power spectrum, the minimum mean square error estimation algorithm is used to construct a noise power estimation model: in, is a smoothing factor, which is between 0.9 and 0.99 to balance the dynamics and stability of the noise power estimate. Based on the noise power estimate, the signal-to-noise ratio (SNR) estimate is calculated: A Wiener filter is constructed to suppress noise, where the gain function of the Wiener filter is expressed as: The spectrum of the filtered signal is calculated as follows: in, Represents the frequency domain signal after preliminary noise reduction. This step can effectively reduce the impact of background noise while maintaining the clarity of the speech signal. After completing the signal filtering, in order to optimize the subsequent speech enhancement and feature extraction, the frequency domain signal after preliminary noise reduction is reorganized into a feature matrix form to adapt to the multi-channel signal processing architecture. Define the multi-channel frequency domain feature matrix , whose rows represent different frequency point indices and columns represent different microphone channels. The matrix elements store the frequency domain coefficients in complex form, that is, in, Recorded the The frequency point is The noise-reduced frequency domain signal on each microphone channel.

[0024] In a specific embodiment, the process of executing step S3 may specifically include the following steps: Reorganize the multi-channel frequency domain feature matrix to construct the frequency point observation signal matrix; The noise covariance matrix model is constructed for the frequency observation signal matrix to obtain the noise covariance prior model; Based on the noise covariance prior model, the hierarchical Dirichlet process is introduced to obtain the noise model; According to the low-rank characteristics of the frequency observation signal matrix, a speech signal decomposition model is constructed; The variational Bayesian inference algorithm is applied to the speech signal decomposition model and the noise model. By maximizing the lower bound of evidence, adopting an alternating iteration strategy and controlling the computational complexity of each iteration within the preset target value, the converged model parameters are obtained. The enhanced speech signal matrix is ​​calculated based on the converged model parameters, and the exponential and logarithmic function calculations are accelerated using a lookup table. The calculation is terminated when the relative change of the lower bound of the evidence is less than a preset threshold, and the enhanced speech signal matrix is ​​obtained.

[0025] Specifically, the multi-channel audio signal after Fourier transformation obtains a multi-channel frequency domain feature matrix , where each column corresponds to the frequency domain signal of a different microphone, and each row corresponds to the spectrum information of a different frequency point. In order to perform noise suppression and speech enhancement, these frequency domain features are reorganized to construct a frequency point observation signal matrix , whose rows represent different frequency point indexes, and columns represent the time-frequency observation signals of different microphones. The elements in the matrix store the complex spectral coefficients of each microphone channel at each frequency point. This reorganization process ensures that the data can adapt to subsequent noise modeling and speech decomposition tasks, so as to perform signal processing more effectively. The noise covariance matrix model is constructed for the frequency point observation signal matrix, and the noise characteristics of each frequency point are estimated by constructing the noise covariance matrix model. Noise covariance matrix Assume that the noise at each frequency point conforms to a certain probability distribution and that the noise is not completely independent between microphone channels. By introducing a priori model of noise covariance, assume that the noise matrix It follows the inverse Wishart distribution (IW distribution). This distribution can effectively capture the multi-channel correlation of noise and provide a reasonable prior for noise estimation. The parameters of the covariance matrix are optimized through Bayesian inference. Specifically, the scale matrix of the noise covariance matrix is ​​set and the degrees of freedom parameter , and estimated by prior information. Based on the noise covariance prior model, the hierarchical Dirichlet process is introduced to model the noise. The hierarchical Dirichlet process is a non-parametric Bayesian method that can automatically capture the diversity of noise between multiple frequency points and microphone channels, and allows the noise model to be adaptively adjusted in different frequency bands and spatial regions. In the hierarchical Dirichlet process, the noise distribution of each frequency point is represented by a mixture model, each mixture component is modeled by an inverse Wishart distribution, and the weight of each noise component is determined by the Dirichlet distribution. In this way, the model flexibly adapts to complex noise environments and can automatically infer the distribution of noise without explicitly specifying the type of noise. After the noise model is established, the speech signal decomposition model is constructed based on the low-rank characteristics of the frequency point observation signal matrix. Speech signals usually have low-rank characteristics, that is, most of the information is concentrated on a few fundamental frequencies and sparse spectral components. Based on this characteristic, a low-rank plus sparse decomposition model is used to transform the observation signal of each frequency point into a sparse spectral component. Decomposed into two parts: one is the low-rank speech signal subspace , and the other is the noise component . This decomposition model is expressed as: in, is a The matrix represents the spatial correlation matrix of the microphone channels, is the subspace dimension of the speech signal, set to 1 to 3; is a The matrix represents the representation of the speech signal in the frequency domain; is the noise component. Through this model, the observed signal is decomposed into effective speech components and noise components. After the model is built, variational Bayesian inference is performed to estimate the parameters of the speech signal decomposition model and the noise model. The variational Bayesian inference algorithm optimizes the model by maximizing the lower bound of evidence. In each iteration, the model parameters are updated to maximize the lower bound of evidence. This process adopts an alternating iterative strategy, in which each round of iteration includes the optimization of the speech signal decomposition model and the noise model. Specifically, by maximizing the expected value of the lower bound To estimate the posterior distribution of the model, the formula is as follows: in, is the joint likelihood function, which represents the data With model parameters the relationship between; is an approximate posterior distribution, which represents the system's prior knowledge of the model parameters. In each iteration, the probability distribution of the model is updated according to the current parameter estimate, and the optimal solution is obtained through the optimization algorithm. In order to speed up the calculation and reduce the computational burden, the computational complexity of each iteration is controlled within within, among which is the number of frequencies, is the number of microphones, is the number of time frames. After obtaining the converged model parameters, the enhanced speech signal matrix is ​​calculated based on these parameters. In order to improve computational efficiency and reduce power consumption, a lookup table method is used to accelerate the calculation of exponential and logarithmic functions to avoid expensive floating-point operations at each inference. When the relative change of the lower bound of the evidence is less than the preset threshold, the iterative calculation is stopped to avoid unnecessary computational consumption. Through the above steps, the enhanced speech signal matrix is ​​finally obtained.

[0026] In a specific embodiment, the process of executing step S4 may specifically include the following steps: An adaptive Wiener filter is constructed according to the enhanced speech signal matrix to obtain the filter structure of the first-level processing; Perform frequency domain division optimization on the filter structure of the first-stage processing to obtain a low-power frequency band filter, and build an acoustic feature monitoring module based on the low-power frequency band filter to obtain a trigger mechanism for the second-stage processing; Detect potential speech events according to the trigger mechanism. When the trigger condition is met, activate the second-level processing module and extract the Mel spectrum features from the enhanced speech signal matrix to obtain the target input features. Sending the target input features into the compressed neural network model to perform speech event classification and obtain the speech event classification result; Decision fusion is performed on the results of the first-level processing and the results of the second-level processing. At the same time, the power supply status of different processing modules in the chip is controlled so that the second-level processing unit returns to deep sleep mode after completing the processing, and the voice event processing results including the enhanced voice signal and the voice event classification label are obtained.

[0027] Specifically, an adaptive Wiener filter is constructed to complete the first level of signal processing. This filter optimizes the signal-to-noise ratio by dynamically adjusting the gain of the signal, thereby removing background noise and retaining the speech signal. When constructing the Wiener filter, the gain of the filter is calculated based on the signal-to-noise ratio (SNR) of each frequency point. SNR is the ratio of the power of the signal to the power of the noise, which can effectively describe the clarity of the signal. Set the first Gain at each frequency point for: in, It is The signal-to-noise ratio of a frequency point is expressed as ,in is the power of the signal, is the power of the noise. At each frequency point, the Wiener filter weights the signal through this gain function, eliminates unnecessary noise components, and obtains a signal after preliminary noise reduction. The first-stage filter structure is optimized by frequency domain division. By dividing the frequency domain into multiple bands, the filter coefficients in each band are shared, thereby reducing the number of filter parameters and reducing the computational complexity. The frequency domain is divided into several Mel bands, and the same filter coefficients are applied in each band to reduce the amount of calculation. Mel band division is based on the auditory characteristics of the human ear and can better simulate human perception of different frequencies. After setting the band division, the first The center frequency of the mel band is , and for the filter coefficients in each frequency band are all fixed, reducing unnecessary calculations and improving efficiency. Based on this low-power frequency band filter, an acoustic feature monitoring module is constructed to monitor the characteristics of the audio signal in real time. The function of the acoustic feature monitoring module is to monitor the energy envelope, zero-crossing rate and other characteristics of the audio signal, and to determine whether there is a potential voice event based on these characteristics. When certain features are detected to exceed the preset threshold, the trigger mechanism will activate the second-level processing module and enter the voice event detection and classification stage. For example, if the energy envelope of the audio signal is detected Exceeds a certain threshold and the zero crossing rate If the frequency of the audio signal changes within a certain range, it indicates that there may be a voice event. At this time, the system starts the second-level processing module through a trigger mechanism. After activating the second-level processing module, the Mel spectrum feature is extracted from the enhanced speech signal matrix. The Mel spectrum feature can effectively represent the frequency distribution of the speech signal and match the auditory characteristics of the human ear. The process of extracting the Mel spectrum feature first converts each frame of the audio signal into a frequency domain signal through a fast Fourier transform, and weights the frequency domain signal through a Mel filter bank to obtain a Mel spectrum. Set the Mel spectrum feature to ,in represents the time frame, Represents the frequency point, and the spectrum after Mel filter weighting is expressed as: in, is the weight of the Mel filter, It is The frame audio signal is The mel spectrum feature is used to represent the frequency domain of each frequency point. Through this process, the mel spectrum feature can effectively represent the spectral information of the speech signal, and at the same time has high robustness and adapts to different noise environments. The extracted mel spectrum feature is sent to the compressed neural network model for speech event classification. By optimizing the network structure and parameters, the compressed neural network model can run efficiently on low-power hardware and realize fast classification of speech events. In order to adapt the model to the computing power of the end-side device, the network structure adopts compression and quantization techniques, such as structured pruning and 8-bit quantization, so that the network model can maintain high classification accuracy while greatly reducing the amount of calculation and storage requirements. The output of the compressed neural network is the classification result of each input feature, such as determining whether the audio signal belongs to speech, noise or other ambient sound. The results of the first-level processing and the second-level processing are fused by decision, and the noise reduction effect of the first-level filter and the accuracy of the second-level neural network classification are combined to make the final speech event recognition decision. In the decision fusion process, the output of the two-level processing modules is combined by weighted averaging or other decision fusion methods to generate the final speech event processing result, including the enhanced speech signal and the speech event classification label. In order to ensure low-power operation, the power supply status of different processing modules is controlled. When there is no voice event, the second-level processing unit enters deep sleep mode to reduce unnecessary energy consumption; when a voice event is triggered, the second-level processing module is activated and performs real-time processing.

[0028] In a specific embodiment, the process of executing step S5 may specifically include the following steps: Input the speech event processing results into the teacher-student model framework, build a server-level teacher model with 12 attention heads and 6 Transformer layers, and a client-side student model with 2 attention heads and 2 Transformer layers, and obtain the initial teacher-student model structure. Define the hybrid loss function according to the initial teacher-student model structure and obtain the knowledge distillation optimization objective; Based on the knowledge distillation optimization goal, a hierarchical distillation strategy is introduced to align the features of the attention mechanism output and feedforward network output of each layer of the server-level teacher model with the corresponding layers of the client-side student model. Inter-layer knowledge transfer is performed by minimizing the L2 distance to obtain a multi-level knowledge transfer mechanism. Performing distillation training on the multi-level knowledge transfer mechanism to obtain a preliminarily trained student model, and performing sparsification processing on the preliminarily trained student model to obtain a sparse student model; Perform 8-bit fixed-point quantization on the sparse student model to obtain the quantized proxy model parameters; An adaptive federated learning algorithm is executed on the quantized proxy model parameters among distributed devices to obtain globally optimized model parameters and deploy them to the ultra-low power end-side AI chip to perform multi-channel audio processing.

[0029] Specifically, a model structure suitable for the application of end-side AI chips is designed. The teacher model is set as a server-level large-scale Transformer model, including 12 attention heads and 6 Transformer layers. The role of the teacher model is to make full use of the computing power of the server, learn more complex and profound speech feature representations, and be able to perform efficient speech event classification and recognition. At the same time, the design of the end-side student model needs to be more streamlined to adapt to the end-side AI chip with limited computing resources. The end-side student model is designed as a lightweight structure containing 2 attention heads and 2 Transformer layers. This structure can significantly reduce the amount of calculation and memory usage while maintaining high accuracy, making it suitable for efficient operation in an ultra-low power environment. Through the construction of this initial model structure, a teacher-student model framework is established. In order to ensure that the student model can effectively learn from the teacher model, a hybrid loss function is defined as the optimization target of knowledge distillation. The hybrid loss function consists of hard label cross entropy loss, soft label distillation loss, and feature-level distillation loss. The hard label cross entropy loss is used to ensure the performance of the student model on basic classification tasks. The soft label distillation loss uses the probability distribution of the teacher model output to help the student model better learn the decision boundary of the teacher model in complex tasks, while the feature-level distillation loss helps the student model learn the knowledge of the teacher model layer by layer by comparing the differences in the features of each layer between the teacher model and the student model. Through the weighted combination of these three losses, a comprehensive learning goal is provided for the student model, thereby optimizing the parameters of the student model so that it can be as close to the performance of the teacher model as possible with fewer computing resources. In order to improve the effect of knowledge distillation, a hierarchical distillation strategy is introduced to align the features of the attention mechanism output and feedforward network output of each layer of the teacher model with the corresponding layers of the student model on the end side. By minimizing the L2 distance, knowledge transfer is performed between layers, thereby realizing multi-level knowledge transfer. In the specific implementation, the teacher model is calculated. Output of the layer and student model Output of the layer The L2 distance between them is: in, and Respectively represent The output of the layer The value of the feature, is the size of the feature dimension. By minimizing this L2 distance, the student model gradually learns the knowledge of each layer of the teacher model and approaches the teacher model layer by layer. The hierarchical distillation strategy not only helps the fine transfer of knowledge, but also can better adapt to the complexity of features at different levels. After the multi-level knowledge transfer mechanism is built, the mechanism is distilled and trained. The distillation training process optimizes the losses in the mixed loss function and uses the guidance of the teacher model to make the student model gradually approach the performance of the teacher model in each round of training. To ensure the balance between computational efficiency and effect during training, batch gradient descent or Adam optimizer is used for training, and the learning rate is dynamically adjusted to make the training process fast and stable. After completing the initial training, the parameters of the student model often still contain redundant parts and cannot fully utilize the computing resources of the end-side chip. Therefore, sparse processing is performed to identify parameters with less impact on performance through L1 regularization or sensitivity analysis, and these parameters are set to zero, thereby reducing unnecessary computational burden. The sparse model retains the 20% weights that contribute the most to the model performance, and sets the rest to zero. Through this process, a sparse student model is obtained. The sparse student model is quantized to 8-bit fixed-point. The quantization process converts the model parameters of floating-point values ​​into 8-bit integer representations, thereby reducing the storage space of the model and the bandwidth required for calculation. The quantization process is performed through a clustering algorithm, which maps continuous floating weight values ​​to a discrete set, and accelerates this mapping process through a lookup table. For the quantized weights, an optimization algorithm is used to ensure that the quantized model can still maintain high accuracy while reducing computing resource consumption. After quantization, the quantized proxy model parameters are applied between distributed devices to execute the adaptive federated learning algorithm. Through this algorithm, each device (such as smartphones, sensors, etc.) performs model training locally and sends the locally updated model parameters to the central server for aggregation. The core of federated learning is to integrate the data training capabilities of multiple devices into a global optimization model without uploading the device data to the central server. In adaptive federated learning, the computing power and network bandwidth of each device are different, so the federated learning algorithm dynamically adjusts the number of local iterations and the frequency of gradient updates according to the performance of the device, so that each device can fully participate in the training of the global model according to its own conditions. When performing model aggregation, the local updates of all nodes are combined through weighted averaging to generate a globally optimized model parameter. The global model parameters are transmitted back to each device for the next round of learning and updating. Through multiple rounds of iteration and optimization, the globally optimal model parameters are obtained. The optimized global model parameters are deployed to the ultra-low power end-side AI chip to perform multi-channel audio processing tasks. Through the computing power of the end-side chip, the system processes the audio signals collected from multiple microphones in real time and completes the recognition and processing of voice events.

[0030] In a specific embodiment, the process of executing the adaptive federated learning algorithm on the quantized proxy model parameters among distributed devices to obtain the globally optimized model parameters and deploying them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing may specifically include the following steps: The quantized proxy model parameters and compressed neural network model are organized into distributed device nodes to obtain a federated learning infrastructure. Based on the federated learning infrastructure, local update calculations are performed on each working node to obtain node local update parameters; An adaptive calculation time mechanism is introduced into the node local update parameters to obtain an adaptive iteration number adjustment mechanism; Quantize and compress the node gradients of the adaptive iteration adjustment mechanism to obtain compressed gradient information; Construct a parameter similarity matrix for the compressed gradient information, perform node clustering based on the parameter similarity matrix, and obtain the knowledge sharing result after clustering; Based on the knowledge sharing results after clustering, global parameter aggregation is performed on the parameter server to obtain the globally optimized model parameters; The compressed neural network model is updated based on the globally optimized model parameters, and the updated model is deployed to the ultra-low power end-side AI chip to perform multi-channel audio processing.

[0031] Specifically, establish a reasonable device node organization among distributed devices. devices, each device With local dataset , and holds the current quantitative proxy model parameters The goal of the entire system is to optimize global model parameters through distributed computing without centralizing data. In order to meet the computing requirements of ultra-low power end-side Al chips, a hierarchical federated learning architecture is constructed, in which edge devices are responsible for local computing and central servers are responsible for aggregation optimization. This architecture ensures the scalability of the system and reduces computing and communication costs. Under this federated learning framework, each device performs model update calculations locally to generate node local update parameters. Each device With its local data Based on this, multiple rounds of gradient descent optimization are performed, and the parameter update formula is: in, Indicates the device In the The local model parameters of the wheel, is the learning rate, It is a gradient calculated based on local data. Through this process, each device independently optimizes its own model parameters without exchanging data with the server, while reducing the overhead of global calculation. Due to the different computing power, bandwidth and data distribution of different devices, directly using a fixed number of iterations will lead to uneven convergence speed of the global model. Therefore, an adaptive calculation time mechanism is introduced to dynamically adjust the number of iterations of each node. Set each device The computing power is The average computing power is , define the adaptive calculation time adjustment coefficient of the device as: This mechanism ensures that devices with stronger computing power perform more local iterations, while devices with weaker computing power perform fewer iterations, allowing the entire system to reach global convergence at a better speed. At the same time, in order to reduce communication overhead, each device only submits updates to the server when a certain model improvement is achieved, thereby reducing the communication frequency. After adjusting the computing time of the device, the gradient of the local update is quantized and compressed to reduce the data transmission overhead between devices. Assuming that the device The calculated gradient is , and perform low-bit quantization on it, so that the compressed gradient Only a small amount of storage space is used. Specifically, the gradient value is Discretize as follows: in, is the quantization function, is the number of bits that are dynamically adjusted, usually floating between 1 and 4 bits. For parameters with small changes, lower bit quantization is used, while for parameters with large gradient changes, higher bit quantization is used to ensure the accuracy of model updates. After obtaining the compressed gradient information, a parameter similarity matrix is ​​constructed to cluster devices based on gradient information. Assume that each device The updated gradient vector transmitted is , calculate the cosine similarity of gradients between devices: in, Indicates the device and equipment If Exceeding a certain threshold , then it is considered that the two devices can share update information and be divided into the same cluster. In this way, redundant calculations between devices are reduced and the overall efficiency of federated learning is improved. Based on the calculated similarity matrix, device clustering is performed on the server side, and global parameter aggregation is performed on the parameter server based on the knowledge sharing results after clustering. Assume that the server receives Clustered device gradients , the calculation formula for global parameter update is as follows: in, Yes Equipment The weight of the contribution is adjusted based on the amount of data or computing power of the device. Through this global parameter aggregation process, the model update information from different devices is efficiently integrated and the stability of the model is improved. After obtaining the globally optimized model parameters, these parameters are used to update the compressed neural network model to adapt to the computing power on different devices. Model pruning technology is used to remove redundant parameters to make the model more compact; the model is low-bit quantized to adapt it to the low-power computing power of the end-side device. The optimized model is deployed to the ultra-low power end-side AI chip to perform multi-channel audio processing tasks.

[0032] The above describes the multi-channel audio processing method of the ultra-low power consumption end-side AI chip in the embodiment of the present invention. The following describes the multi-channel audio processing device of the ultra-low power consumption end-side AI chip in the embodiment of the present invention. Figure 2 , an embodiment of a multi-channel audio processing device of an ultra-low power consumption end-side AI chip in an embodiment of the present invention includes: A conversion module, used for performing ADC conversion on the multi-channel analog audio signal from the microphone array to obtain a multi-channel digital audio data stream; A transformation processing module, used for performing frame segmentation and frequency domain transformation processing on a multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; An inference module, used for performing variational Bayesian inference based on a multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; A cascade processing module, used for performing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; The deployment module is used to execute an adaptive federated learning algorithm among distributed devices based on the results of voice event processing, obtain globally optimized model parameters, and deploy them to the ultra-low power end-side AI chip to perform multi-channel audio processing.

[0033] Through the cooperation of the above components, a two-stage cascade processing architecture is adopted. The lightweight adaptive Wiener filter with a power consumption of less than 1mW in the first stage runs continuously, and the compressed neural network with 8-bit quantization in the second stage is activated only when detecting key events, so that the average power consumption of the system is controlled below 2mW. Combined with the layered Dirichlet process noise modeling and variational Bayesian inference algorithm, the non-independent and non-identical distribution characteristics of noise are effectively captured, improving the audio processing quality in complex environments. Through the teacher-student knowledge distillation framework and structured pruning, the server-level large model is compressed into a lightweight model on the end side, reducing the number of parameters by 98% while maintaining a recognition accuracy of more than 90%. The frequency domain is divided into 25 Mel-band shared filter coefficients to reduce the amount of calculation, and the storage requirements are optimized by resetting 95% of the weights to zero. The introduced adaptive federated learning algorithm dynamically adjusts the number of iterations according to the computing power of the end-side device, and the 1-4-bit gradient quantization and difference coding reduce the communication overhead. A block loading strategy is adopted to store the resident core blocks (accounting for 30% of the total parameters) in the on-chip SRAM. The compressed sparse row storage method reduces the storage space by 80%, enabling the system to adapt to different end-side hardware resource constraints and achieve a balance between power consumption, processing quality and resource adaptability.

[0034] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0035] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0036] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-channel audio processing method for an ultra-low power consumption end-side AI chip, characterized in that: include: Performing ADC conversion on the multi-channel analog audio signals from the microphone array to obtain a multi-channel digital audio data stream; Performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; Performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; Implementing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; According to the voice event processing results, an adaptive federated learning algorithm is executed among distributed devices to obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

2. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 1, characterized in that: The method of performing ADC conversion on the multi-channel analog audio signal from the microphone array to obtain a multi-channel digital audio data stream includes: Collecting multi-channel analog audio signals from the microphone array through a linear array to obtain original audio analog signals; Inputting the original audio analog signal into an ADC converter for analog-to-digital conversion to obtain initial digital audio data, and performing time domain framing processing on the initial digital audio data to obtain framed digital audio data; Applying a Hamming window function to the framed digital audio data to perform windowing to obtain a windowed audio data frame; Input the windowed audio data frame into a fast Fourier transform processing unit, convert the time domain signal into the frequency domain through a butterfly operation, and obtain audio data represented in the frequency domain; A preliminary energy filtering is performed on the audio data represented in the frequency domain to obtain a multi-channel digital audio data stream.

3. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 1, characterized in that: The performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix includes: Re-framing the multi-channel digital audio data stream to obtain re-framed multi-channel audio data; Multiplying the re-framed multi-channel audio data by a preset Hamming window function coefficient, performing a windowing operation on each frame signal of each channel, and obtaining windowed audio data; Performing a fast Fourier transform operation on the windowed audio data to obtain spectrum amplitude and phase information; Establish a frequency domain adaptive filter according to the spectrum amplitude and phase information to obtain noise suppression filter parameters; Applying the noise suppression filter parameters to the frequency domain signals of each channel, performing a signal filtering operation through a frequency domain multiplication operation, and obtaining a frequency domain signal after preliminary noise reduction; The frequency domain signal after the preliminary noise reduction is reorganized into a feature matrix form, where the matrix rows represent frequency point indexes, the columns represent different microphone channels, and the matrix elements store frequency domain coefficients in complex form, thereby obtaining a multi-channel frequency domain feature matrix.

4. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 1, characterized in that: The performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix includes: Reorganizing the multi-channel frequency domain feature matrix to construct a frequency point observation signal matrix; Constructing a noise covariance matrix model for the frequency point observation signal matrix to obtain a noise covariance prior model; Based on the noise covariance prior model, a hierarchical Dirichlet process is introduced to obtain a noise model; Constructing a speech signal decomposition model according to the low-rank characteristic of the frequency point observation signal matrix; Execute a variational Bayesian inference algorithm on the speech signal decomposition model and the noise model, and obtain converged model parameters by maximizing the lower bound of evidence, adopting an alternating iteration strategy, and controlling the computational complexity of each round of iteration within a preset target value; An enhanced speech signal matrix is ​​calculated based on the converged model parameters, and the lookup table is used to accelerate the exponential and logarithmic function calculations. When the relative change of the lower bound of the evidence is less than a preset threshold, the calculation is terminated to obtain an enhanced speech signal matrix.

5. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 1, characterized in that: The two-stage cascade processing is performed according to the enhanced voice signal matrix to obtain a voice event processing result, including: Constructing an adaptive Wiener filter according to the enhanced speech signal matrix to obtain a filter structure for first-level processing; Perform frequency domain division optimization on the filter structure of the first-stage processing to obtain a low-power frequency band filter, and construct an acoustic feature monitoring module based on the low-power frequency band filter to obtain a trigger mechanism for the second-stage processing; Detecting potential speech events according to the trigger mechanism, activating the second-level processing module when the trigger condition is met, and extracting Mel spectrum features from the enhanced speech signal matrix to obtain target input features; Sending the target input features into a compressed neural network model to perform speech event classification to obtain a speech event classification result; Decision fusion is performed on the results of the first-level processing and the results of the second-level processing. At the same time, the power supply status of different processing modules in the chip is controlled so that the second-level processing unit returns to deep sleep mode after completing the processing, and the voice event processing results including the enhanced voice signal and the voice event classification label are obtained.

6. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 5, characterized in that: The method executes an adaptive federated learning algorithm among distributed devices according to the voice event processing result, obtains globally optimized model parameters and deploys them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing, including: Input the speech event processing result into the teacher-student model framework, establish a server-level teacher model including 12 attention heads and 6 Transformer layers and a terminal-side student model including 2 attention heads and 2 Transformer layers, and obtain an initial teacher-student model structure; Define a hybrid loss function according to the initial teacher-student model structure to obtain a knowledge distillation optimization objective; Based on the knowledge distillation optimization goal, a hierarchical distillation strategy is introduced to align the features of the attention mechanism output and feedforward network output of each layer of the server-level teacher model with the corresponding layers of the client-side student model, and perform inter-layer knowledge transfer by minimizing the L2 distance to obtain a multi-level knowledge transfer mechanism. Performing distillation training on the multi-level knowledge transfer mechanism to obtain a preliminarily trained student model, and performing sparsification processing on the preliminarily trained student model to obtain a sparse student model; Performing 8-bit fixed-point quantization on the sparse student model to obtain quantized proxy model parameters; An adaptive federated learning algorithm is executed on the quantized proxy model parameters among distributed devices to obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

7. The multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to claim 6, characterized in that: The method executes an adaptive federated learning algorithm on the quantized proxy model parameters among distributed devices to obtain globally optimized model parameters and deploys them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing, including: Organizing the quantized proxy model parameters and the compressed neural network model into distributed device nodes to obtain a federated learning infrastructure; Based on the federated learning infrastructure, local update calculations are performed on each working node to obtain node local update parameters; An adaptive calculation time mechanism is introduced into the local update parameters of the node to obtain an adaptive iteration number adjustment mechanism; Quantizing and compressing the node gradients of the adaptive iteration number adjustment mechanism to obtain compressed gradient information; constructing a parameter similarity matrix for the compressed gradient information, performing node clustering based on the parameter similarity matrix, and obtaining a knowledge sharing result after clustering; Based on the clustered knowledge sharing results, global parameter aggregation is performed on the parameter server to obtain globally optimized model parameters; The compressed neural network model is updated based on the globally optimized model parameters, and the updated model is deployed to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

8. A multi-channel audio processing device of an ultra-low power consumption end-side AI chip, characterized in that: Used to execute the multi-channel audio processing method of the ultra-low power consumption end-side AI chip according to any one of claims 1 to 7, the multi-channel audio processing device of the ultra-low power consumption end-side AI chip comprises: A conversion module, used for performing ADC conversion on the multi-channel analog audio signal from the microphone array to obtain a multi-channel digital audio data stream; A transformation processing module, used for performing frame segmentation and frequency domain transformation processing on the multi-channel digital audio data stream to obtain a multi-channel frequency domain feature matrix; An inference module, used for performing variational Bayesian inference based on the multi-channel frequency domain feature matrix to obtain an enhanced speech signal matrix; A cascade processing module, used for performing two-stage cascade processing according to the enhanced voice signal matrix to obtain a voice event processing result; A deployment module is used to execute an adaptive federated learning algorithm among distributed devices according to the voice event processing results, obtain globally optimized model parameters and deploy them to the ultra-low power consumption end-side AI chip to perform multi-channel audio processing.

Images