Brain-like low-power audio classification method and device suitable for edge device, equipment, medium and product

Abstract

The present disclosure relates to the field of computer technology and audio signal processing, and provides a brain-like low-power audio classification method and device suitable for edge devices, equipment, medium and product, which comprises: obtaining a target audio; using an audio decomposition module of an audio classification model to decompose the target audio to obtain a first sub-band and a second sub-band; using a feature extraction module of the audio classification model to extract features from the first sub-band and the second sub-band to obtain first features corresponding to the first sub-band and second features corresponding to the second sub-band; and using a classification module of the audio classification model to predict the sound classification of the target audio based on the first features and the second features. The present disclosure can solve the problem of redundant calculation caused by homogenization processing of audio, avoid the problem of being unable to distinguish high-frequency information and low-frequency information, and perform targeted feature extraction on different frequency components, thereby avoiding redundant calculation and improving the calculation efficiency of audio classification.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology and audio signal processing, and in particular to a neuromorphic low-power audio classification method, apparatus, device, medium and product suitable for edge devices. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and edge computing technologies, audio intelligent sensing is being used more and more widely in fields such as smart homes, industrial monitoring, and human-computer interaction. However, existing audio classification technologies face a fundamental contradiction between performance and computing resources.

[0003] Specifically, traditional audio classification methods employ a homogenization strategy for the input audio, using a uniform convolution kernel to process the entire spectrogram. This ignores the essential differences between different frequency components in the audio signal. For example, low-frequency information such as the fundamental frequency and formants can reflect the long-term steady-state characteristics of the sound, while high-frequency information such as consonant initiation and environmental noise encodes transient events. Applying a homogenization strategy to such low-frequency and high-frequency information leads to a large number of parameters being wasted on redundant calculations, resulting in low computational efficiency and making it difficult to deploy such classification methods on edge devices. Summary of the Invention

[0004] This disclosure provides a neuromorphic low-power audio classification method, apparatus, device, medium, and product suitable for edge devices, to at least solve the problem of redundant computation caused by homogenized audio processing in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of this disclosure, an audio classification method is provided, the method comprising: acquiring target audio; decomposing the target audio using an audio decomposition module of an audio classification model to obtain a first sub-band and a second sub-band, wherein the first sub-band contains a first frequency component, the second sub-band contains a second frequency component, and the sound frequency of the first frequency component is higher than the sound frequency of the second frequency component; extracting features from the first sub-band and the second sub-band using a feature extraction module of the audio classification model to obtain a first feature corresponding to the first sub-band and a second feature corresponding to the second sub-band; and predicting the sound classification of the target audio based on the first feature and the second feature using a classification module of the audio classification model.

[0006] Optionally, the audio decomposition module includes a prediction operator and an update operator, wherein the audio decomposition module decomposes the target audio into the first sub-band and the second sub-band in the following manner: In the time dimension, according to the parity of the index of the audio features of the target audio, a first-level decomposition is performed on the audio features of the target audio to obtain an even-numbered part and an odd-numbered part; using the prediction operator, a prediction operation is performed on the even-numbered part to obtain a prediction result; by subtracting the prediction result from the odd-numbered part, a first coefficient is obtained, wherein the first coefficient represents the detailed features of the audio; using the update operator, an update operation is performed on the first coefficient to obtain an update result; by adding the update result to the even-numbered part, a second coefficient is obtained, wherein the second coefficient represents the overall features of the audio; in the frequency dimension, according to the frequency components, a second-level decomposition is performed on the first coefficient and the second coefficient to obtain the first sub-band and the second sub-band.

[0007] Optionally, the feature extraction module includes a first path and a second path, wherein the feature extraction module obtains the first feature and the second feature by: using the first path to extract transient features from the first sub-band as the first feature; and using the second path to extract steady-state features from the second sub-band as the second feature.

[0008] Optionally, the first pathway includes multiple cascaded neuron modules, which are used to simulate auditory neurons. The feature extraction module obtains the first feature by: inputting the first sub-band into the first pathway; extracting the first feature from the first sub-band using the multiple neuron modules; wherein the output feature of the last neuron module among the multiple neuron modules is the first feature; and for each neuron module, the output feature is obtained by: updating the current membrane potential based on a preset leakage factor and the current input feature to obtain an updated membrane potential; wherein for the first neuron module, the current input feature is the first sub-band, and the leakage factor is used to control the degree to which each neuron module retains historical information; generating the output feature of the current neuron module in response to the updated membrane potential being greater than or equal to a firing threshold; and resetting the updated membrane potential based on the firing threshold and the pulse output to obtain a reset membrane potential; wherein the firing threshold is a learnable parameter.

[0009] Optionally, the audio classification model is trained as follows: A training sample set and sample labels corresponding to each training sample in the training sample set are obtained, wherein the training samples include audio data, and each sample label indicates the sound category of the corresponding training sample; the audio decomposition module of the audio classification model is used to decompose the training samples in the training sample set to obtain a first sample sub-band and a second sample sub-band, wherein the first sample sub-band contains a first sample frequency component, and the second sample sub-band contains a second sample frequency component, and the sound frequency of the first sample frequency component is higher than the sound frequency of the second sample frequency component; the feature extraction module of the audio classification model is used to extract features from the first sample sub-band and the second sample sub-band to obtain a first sample feature corresponding to the first sample sub-band and a second sample feature corresponding to the second sample sub-band; the classification module of the audio classification model is used to predict the sound classification result of each training sample based on the first sample feature and the second sample feature; and the audio classification model is trained by comparing the sound classification result with the sample label.

[0010] Optionally, the audio decomposition module includes a prediction operator and an update operator. The prediction operator includes a first convolutional layer and a first scaling parameter. The first scaling parameter is used to scale the convolution result of the first convolutional layer. The update operator includes a second convolutional layer and a second scaling parameter. The second scaling parameter is used to scale the convolution result of the second convolutional layer. Both the first scaling parameter and the second scaling parameter are learnable parameters.

[0011] Optionally, training the audio classification model by comparing the sound classification result and the sample label includes: obtaining a first loss by comparing the sound classification result and the sample label; training the audio classification model based on the first loss and a second loss, wherein the second loss includes at least one of reconstruction loss, sparsity loss, and path loss, wherein the reconstruction loss is obtained by comparing the training sample set and reconstruction features, the reconstruction features are obtained by reconstructing audio data based on the first sub-band and the second sub-band, wherein the sparsity loss is obtained based on the output features of each neuron module in a plurality of neuron modules used to extract the first feature, the neuron modules being used to simulate auditory neurons, and wherein the path loss is obtained based on the classification confidence of the first feature and the classification confidence of the second feature.

[0012] According to a second aspect of this disclosure, an audio classification apparatus is provided, the apparatus comprising: an acquisition unit configured to acquire target audio; a decomposition unit configured to decompose the target audio using an audio decomposition module of an audio classification model to obtain a first sub-band and a second sub-band, wherein the first sub-band contains a first frequency component, the second sub-band contains a second frequency component, and the sound frequency of the first frequency component is higher than the sound frequency of the second frequency component; an extraction unit configured to extract features from the first sub-band and the second sub-band using a feature extraction module of the audio classification model to obtain a first feature corresponding to the first sub-band and a second feature corresponding to the second sub-band; and a prediction unit configured to predict the sound classification of the target audio based on the first feature and the second feature using a classification module of the audio classification model.

[0013] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform the audio classification method according to this disclosure.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the audio classification method according to this disclosure.

[0015] According to a fifth aspect of this disclosure, a computer program product is provided, comprising computer-executable instructions that, when executed by at least one processor, implement the audio classification method according to this disclosure.

[0016] The technical solution provided in this disclosure brings at least the following beneficial effects:

[0017] According to the audio classification scheme disclosed herein, the input audio can be decomposed into sub-bands containing different frequency components, and features can be extracted separately for each sub-band to classify the input audio based on the extracted features. This avoids the problem of homogenizing the entire input audio and failing to distinguish between high-frequency and low-frequency information. Targeted feature extraction can be performed on different frequency components, thereby avoiding redundant computation and improving the computational efficiency of audio classification.

[0018] Furthermore, according to the audio classification scheme disclosed herein, by dividing the audio into sub-bands containing different frequency components, the audio information of different sub-bands is processed in a targeted manner. This is similar to how the human brain's auditory nervous system processes received audio, thereby achieving brain-like audio processing effects.

[0019] Furthermore, according to the audio classification scheme disclosed herein, by dividing the sub-bands and using a dual-stream structure to extract features from the sub-bands separately, the feature extraction requirements can be made more targeted. This allows for the use of a low-parameter model for sub-band feature extraction. Such a low parameter count enables the model to be computed with low power consumption, thus allowing it to be deployed on low-power edge devices. This makes the scheme more universal and applicable to a wider range of scenarios.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0022] Figure 1 This is a schematic flowchart of an audio classification method according to an exemplary embodiment of the present disclosure.

[0023] Figure 2 This is a schematic flowchart illustrating the steps of decomposing a first sub-band and a second sub-band in an audio classification method according to an exemplary embodiment of the present disclosure.

[0024] Figure 3 This is a schematic diagram of the architecture of an audio classification model according to an exemplary embodiment of the present disclosure.

[0025] Figure 4 This is a schematic diagram of an audio decomposition module according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 This is a schematic diagram of a neuron module according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 This is a schematic flowchart of a training method for an audio classification model according to an exemplary embodiment of the present disclosure.

[0028] Figure 7 This is a schematic diagram illustrating the relationship between accuracy and parameter quantity according to an exemplary embodiment of the present disclosure.

[0029] Figure 8 This is a schematic block diagram of an audio classification apparatus according to exemplary embodiments of the present disclosure. Detailed Implementation

[0030] In order to enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0031] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0033] As mentioned earlier, traditional audio classification schemes suffer from redundant computation due to the homogenization of audio data.

[0034] Specifically, traditional audio classification methods primarily rely on convolutional neural networks, recurrent neural networks, and residual neural networks. While high-performance models such as ResNetSE and ERes2Net can achieve classification accuracy exceeding 95%, their parameter count typically ranges from 5 to 8 million, resulting in model sizes of 25 to 30 megabytes, far exceeding the capacity of edge devices. Models based on Transformer architectures such as AST and HTS-AT offer even better performance, but their parameter counts reach 31 to 87 million, making them even less suitable for deployment on edge devices.

[0035] To reduce model complexity, various lightweight methods have been proposed. Lightweight architectures such as MobileNetV3 and ShuffleNetV2 can reduce computation through techniques such as depthwise separable convolutions, but their accuracy on audio classification tasks is typically below 90%. Model compression methods such as quantization and pruning can reduce model size, but usually result in a 2% to 5% loss in accuracy.

[0036] Furthermore, for the human-like auditory system, the cochlear basilar membrane achieves frequency decomposition through physical properties, and the ventral and dorsal flows of the auditory cortex can respectively process sound object recognition and spatial localization, and perform event-driven sparse computation through discrete pulses.

[0037] In view of the above problems, exemplary embodiments of this disclosure provide an audio classification method, an audio classification model training device, an audio classification device, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0038] In a first aspect of an exemplary embodiment of this disclosure, an audio classification method is provided.

[0039] The audio classification method according to exemplary embodiments of this disclosure can be applied to electronic devices, particularly resource-constrained edge computing devices (or edge devices), to achieve low-power audio classification. The electronic device can obtain audio classification results by executing this audio classification method on the acquired audio. Here, the electronic device may be, for example, a tablet computer, a laptop computer, a digital assistant, a wearable device, etc., and the edge device may refer to a terminal or embedded system located close to the data source and limited by computing power / memory / power consumption, such as, but not limited to, ARM Cortex-A series gateways, smart microphone modules, etc. As an example, a terminal or system that meets at least one of the following conditions can be considered an edge device: computing power ≤ 10 GFLOPS; memory ≤ 4GB; power consumption ≤ 5W.

[0040] However, the above implementation scenario is only an example scenario. The audio classification method according to the exemplary embodiments of this disclosure can also be applied to other application scenarios. For example, electronic devices (e.g., mobile phones, desktop computers, tablets, etc.) can request training models from servers via the network. The server can complete the request by executing the audio classification method according to the exemplary embodiments of this disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0041] The audio classification method according to the exemplary embodiments of this disclosure can avoid the problem of homogenizing the input audio as a whole and failing to distinguish between high-frequency and low-frequency information. It can perform targeted feature extraction on different frequency components, thereby avoiding redundant calculations and improving the computational efficiency of audio classification.

[0042] Examples of audio classification methods according to embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the audio classification method may include the following steps:

[0044] In step S110, the target audio can be obtained.

[0045] Here, the target audio can be audio of any format, length, and content. For example, the target audio can be captured via a microphone, telephone, recording device, or streaming media.

[0046] In step S120, the audio decomposition module of the audio classification model can be used to decompose the target audio to obtain the first sub-band and the second sub-band.

[0047] Here, the first sub-band may contain a first frequency component, and the second sub-band may contain a second frequency component. The sound frequency of the first frequency component may be higher than the sound frequency of the second frequency component.

[0048] As an example, the audio decomposition module may include a prediction operator and an update operator, both of which may include learnable parameters during training, as will be described in detail below.

[0049] In this example, such as Figure 2 As shown, the audio decomposition module can decompose the target audio into a first sub-band and a second sub-band in the following way:

[0050] In step S210, the audio features of the target audio can be decomposed in the time dimension according to the parity of the index of the audio features of the target audio, to obtain the even part and the odd part.

[0051] Here, as Figure 2 As shown, the audio features of the target audio can be, for example, but not limited to, Mel spectrogram features or features obtained by processing Mel spectrogram features.

[0052] Specifically, before decomposing the audio signal into subbands, preprocessing can be performed on the original input audio signal. For example, the audio signal can be resampled to a standard sampling rate of 16 kHz. This is because most of the key information in ambient sound and speech is concentrated in the frequency band below 8 kHz, so resampling to 16 kHz can avoid losing key information in subsequent audio processing. Furthermore, the audio length can be standardized to 3 seconds. For audio shorter than 3 seconds, zeros are padded at the end; for audio longer than 3 seconds, a 3-second segment can be extracted from the middle.

[0053] As an example, Mel spectrogram features can be extracted from the target audio as a time-frequency representation. Specifically, a fast Fourier transform window length of 1024 points and a jump length of 512 points (equivalent to a 50% window overlap) can be used. The number of Mel filter banks can be set to multiple (e.g., 80) frequency bands, thus covering the frequency range most sensitive to human hearing. The calculated power spectrum can then be logarithmically compressed to simulate the logarithmic perception characteristics of the human ear. Finally, the Mel spectrogram can be normalized to a target level such as -20 dB to obtain the audio features of the target audio.

[0054] In step S210, audio features can be input into the audio decomposition module of the audio classification model (e.g., Figure 2 As shown). As an example, such as Figure 4 As shown, the audio decomposition module can be a module that uses discrete wavelet transform for frequency decoupling, and this module can be based on learnable adaptive wavelet decomposition.

[0055] As an example, in step S210, the input audio features can be separated into even rows and odd rows according to the parity of the feature index, resulting in the above-mentioned even part and odd part.

[0056] For example, the even part can be represented as The odd-numbered part can be represented as ,in, For the input audio features, such as the log-Mel spectrum (or equivalent time-frequency features) tensor, This represents the intermediate tensor that is divided into even and odd parts according to the time dimension. It is a non-negative integer index (e.g., it can be a time step). Represents even numbers, This indicates an odd number. As an example, the parity index can represent the feature number or the position of the feature, with each feature corresponding to a different index.

[0057] In step S220, the prediction operator can be used to perform a prediction operation on the even-numbered part to obtain the prediction result.

[0058] As an example, the prediction operator may include a first convolutional layer and a first scaling parameter, which can be used to scale the convolution result of the even-numbered parts. As an example, the first scaling parameter can be a learnable parameter. Here, the prediction operator may, for example, be used to perform a local linear / nonlinear fit on the even-numbered parts (or even-numbered components) to predict (or estimate) the expected value of the odd-numbered components, wherein the fitting and prediction of the expected value can employ existing fitting and prediction methods.

[0059] For example, the prediction operator can be represented as: ,in, Indicates to of convolution, The representation of ReLU activation of The convolution is performed, where α is a learnable first scaling parameter. The output of the prediction operator can be the prediction result described above.

[0060] In step S230, the first coefficient can be obtained by subtracting the prediction result from the odd-numbered portion. Here, the first coefficient can characterize the detailed features of the sound.

[0061] As an example, the first coefficient can be taken as the first power of the high-frequency coefficient d of the first-level decomposition. For example, the first coefficient can be determined by the following equation (1):

[0062] (1)

[0063] in, Indicates the first coefficient. This represents the prediction operator.

[0064] In step S240, the update operator can be used to update the first coefficient to obtain the update result.

[0065] As an example, the update operator may include a second convolutional layer and a second scaling parameter, which is used to scale the convolution result of the first coefficients. Both the first and second scaling parameters are learnable parameters. Here, the update operator may, for example, take the prediction error as input and perform a low-pass update on the even-numbered parts (or even-numbered components) to preserve the overall contour, achieving a learnable (or improved format) update.

[0066] Here, both the prediction operator and the update operator can include learnable parameters to identify high and low frequencies of audio for the training sample set during training, thereby more accurately dividing the first subband and the second subband containing different frequency components.

[0067] As an example, the update operator can be represented as: ,in, Indicates to of Convolution, where β is a learnable second scaling parameter. Here, the output of the update operator can be used as the update result described above.

[0068] In step S250, the second coefficient can be obtained by adding the update result to the even-numbered part. Here, the second coefficient can characterize the overall features of the sound.

[0069] As an example, the second coefficient can be taken as the first power of the low-frequency coefficient s of the first level. For example, the second coefficient can be determined by the following equation (2):

[0070] (2)

[0071] in, Indicates the second coefficient. This indicates the update operator.

[0072] In step S260, the first coefficient and the second coefficient can be decomposed in the frequency dimension according to the frequency components to obtain the first subband and the second subband.

[0073] As an example, the above process of parity separation, prediction, and updating can be repeated on the second coefficients (or low-frequency coefficients) obtained from the first stage to obtain LL and LH subbands. The same operation is performed on the first coefficients (or high-frequency coefficients) obtained from the first stage to obtain HL and HH subbands. Here, the LL subband corresponds to the low-frequency components and contains the main steady-state structure information of the sound; the LH subband corresponds to the low-frequency components, the HL subband corresponds to the high-frequency components, and the HH subband corresponds to the high-frequency components; these three subbands contain high-frequency detail information. The first subband can include the LH, HL, and HH subbands; for example, the first subband can be obtained by splicing the LH, HL, and HH subbands along the channel dimension. The second subband can include the LL subband. Here, the LL subband indicates that the frequency change of the subband is from low to low, the LH subband indicates that the frequency change of the subband is from low to high, similarly, the HL subband indicates that the frequency change of the subband is from low to high, and the HH subband indicates that the frequency change of the subband is from high to high.

[0074] Specifically, the second coefficient can be separated into even-numbered rows and odd-numbered rows to obtain the even-numbered part. and odd part .

[0075] Similar to the above, we can determine the first coefficient in this decomposition. Second coefficient Similarly, the first and second coefficients in this decomposition can also be determined based on the prediction operator and the update operator.

[0076] For example, the first coefficient can be expressed as The second coefficient can be expressed as .

[0077] In this example, the decomposed subbands are represented as follows: , , , .here, and Represents variables in the frequency dimension; , and , These are even / odd samples along the frequency dimension, respectively. and These are the learnable prediction / update operators for the frequency dimension, respectively. , , , These represent the four sub-band coefficients of the second level (from the high-pass / low-pass results of the s-path or d-path); A non-negative integer index (frequency channel / filter band number); These represent four sub-bands: low-low, low-high, high-low, and high-high. Depend on Obtained through frequency low-pass update, Its high-pass residual at frequency; and Each by The frequency of low-pass update and high-pass residual are obtained.

[0078] In this way, the optimal frequency decomposition mode can be automatically learned through end-to-end training. In addition, the convolutional layer parameters and scaling parameters in the prediction and update operators are learnable and can be adaptively adjusted according to the specific audio classification task.

[0079] Return to reference Figure 1 In step S130, the feature extraction module of the audio classification model can be used to extract features from the first sub-band and the second sub-band to obtain the first feature corresponding to the first sub-band and the second feature corresponding to the second sub-band.

[0080] like Figure 2 As shown, the first sub-band and the second sub-band can be sent to different paths in the feature extraction module.

[0081] As an example, the feature extraction module may include a first path and a second path, wherein the feature extraction module obtains the first feature and the second feature by: using the first path to extract transient features from the first sub-band as the first feature; and using the second path to extract steady-state features from the second sub-band as the second feature.

[0082] Here, the first path can be used to extract features of the high-frequency subband, and the second path can be used to extract features of the low-frequency subband.

[0083] As an example, transient characteristics can refer to event information showing a significant change in energy / spectral envelope within 3 or fewer audio frames (e.g., ≈≤96 ms). A significant change in energy / spectral envelope can, for example, mean that the amount of change in energy / spectral envelope within 3 or fewer audio frames exceeds a first preset threshold. Here, transient characteristics can be obtained by the LIF response of a first path (e.g., a high-frequency pulse) to a first sub-band, and spectral flow can also be introduced as an aid.

[0084] Steady-state features can refer to slowly varying structural information over 6 or more audio frames (e.g., ≈≥192 ms). Slow variation can mean that the change in the structural information of the feature over 6 or more audio frames is less than a second preset threshold or the rate of change is less than a third preset threshold. Here, steady-state features can be extracted from the second sub-band by the second path (e.g., low-frequency residuals), and long-term relevant information can be captured using depth-separable residual blocks.

[0085] In one example, the first pathway may include multiple neuronal modules connected in series. These neuronal modules can be used to simulate auditory neurons, enabling the model to possess brain-like audio recognition capabilities. Figure 5 The dynamics of each neuron module in the state-leaking integral-triggered neuron model used in the first pathway are illustrated.

[0086] In this example, the feature extraction module can obtain the first feature by inputting the first sub-band into the first pathway and extracting the first feature from the first sub-band using multiple neuron modules, wherein the output feature of the last neuron module among the multiple neuron modules is the first feature.

[0087] Here, for each neuron module, the output features can be obtained as follows: based on a preset leakage factor and the current input features, the current membrane potential is updated to obtain the updated membrane potential; in response to the updated membrane potential being greater than or equal to the firing threshold, the output features of the current neuron module can be generated. In response to the updated membrane potential being less than the firing threshold, no features can be output. Here, the firing threshold is obtained as a learnable parameter in the model through backpropagation optimization.

[0088] In response to the updated membrane potential being greater than or equal to the firing threshold, the updated membrane potential can also be reset based on the firing threshold and the pulse output to obtain the reset membrane potential.

[0089] In the example above, for the first neuron module, the current input feature is the first subband, and the leakage factor can be used to control the degree to which each neuron module retains historical information. The threshold is a learnable parameter.

[0090] Specifically Figure 3A schematic diagram of the overall architecture of WaveSpikeNet, an audio classification model according to an embodiment of this disclosure, is shown. Figure 3 For example, the high-frequency subbands LH, HL, and HH, which serve as the first subband, can be concatenated along the channel dimension and input into the first pathway for transient feature extraction. As an example, the first pathway can be a high-frequency pulse pathway. This pathway can employ a simplified leaky integral-triggered neuron model. Here, the neuron model can be, for example, a discretized leaky integrate-and-fire (LIF) neuron model, which can maintain a membrane potential state variable to simulate the potential accumulation process of biological neurons.

[0091] For example, such as Figure 5 As shown, within each neuron module, at each time step, the leakage decay of the membrane potential mem can be calculated by multiplying the current membrane potential mem by the leakage factor. Here, the leakage factor can be set according to actual needs to meet the requirements. For example, the leakage factor can be 0.9, which means that 90% of the historical potential is retained at each time step. Then, the input of the current time step can be added to the membrane potential after leakage to obtain the updated membrane potential, which can be expressed as follows: ,in, This indicates the updated membrane potential. This indicates the membrane potential before the update. Indicates input current / characteristic.

[0092] like Figure 5 As shown, the updated membrane potential can be compared with the firing threshold, and the updated membrane potential can be used to determine the appropriate parameters. It can be calculated ,in, Represents the Heaviside step function. This represents the output characteristics of the neuron module. , This represents the issuance threshold, which is a learnable parameter. When This indicates that the membrane potential mem is greater than or equal to the firing threshold. At this time, the neuron module generates or emits spikes as output features, and the output spike value can be, for example, 1; when This indicates that when the membrane potential is less than the firing threshold, no pulse is generated, and the output of the neuron module is 0.

[0093] Each neuron module needs to reset its membrane potential after generating a pulse output, such as... Figure 5 As shown, the membrane potential mem can be subtracted from the emission threshold. The reset process, multiplied by the pulse output spikes, can be represented, for example, as follows: Specifically, within each neuron module, at each time step, if a pulse is generated, the membrane potential is subtracted from the threshold; if no pulse is generated, the membrane potential remains unchanged.

[0094] The first path may include multiple such Figure 5 As shown in the diagram, the output features of the previous neuron module can be input into the next neuron module, and so on, until the last neuron module is reached. The output features of the last neuron module can be used as the first feature extracted from the first subband.

[0095] In the embodiments of this disclosure, by using a simplified LIF neuron model, the core computational characteristics of biological neurons can be maintained, including leakage integral dynamics, threshold triggering mechanisms, and pulse reset processes. Since the first sub-band, being a high-frequency sub-band, inherently possesses sparse characteristics, the embodiments of this disclosure find that the thresholding mechanism of LIF neurons is more suitable for processing such sparse signals. By employing a spiking neural network, such as the LIF neuron model, it is beneficial to reduce the number of parameters in the model, which is inherently sparse. Therefore, by dividing the signal into sub-bands containing different frequency components, suitable algorithms or networks can be selectively used in audio classification based on the characteristics of each sub-band. For example, a spiking neural network with fewer parameters can be used for sparse signals, thus reducing the overall number of parameters in the model.

[0096] The above describes an example process for extracting the first feature from the first sub-band. The following describes an example process for extracting the second feature from the second sub-band.

[0097] like Figure 3 As shown, steady-state feature extraction can be performed using the second subband input of the low-frequency subband LL as a second path. This path can, for example, employ a depth-optimized residual block sequence, with each residual block used to efficiently process steady-state structure information.

[0098] Specifically, the second path can extract features in the following way: perform a first-point convolution operation on the input second sub-band to reduce the number of channels, and perform batch normalization and activation operations; perform a depthwise convolution operation on the dimensionality-reduced features, with each channel independently undergoing spatial convolution, and perform batch normalization and activation operations; perform a second-point convolution operation on the depthwise convolution features to restore the number of channels, and perform batch normalization; add the second sub-band to the features after restoring the number of channels through residual connections, and perform a final activation operation to obtain the second feature.

[0099] Specifically, the residual block of the second channel can first be processed by passing it through the first point convolutional layer to reduce the number of input channels. This dimensionality reduction operation can reduce the computational cost of subsequent depthwise convolutions. The dimensionality-reduced features can then be processed by batch normalization and the ReLU activation function to obtain the processed features.

[0100] Spatial features can be extracted from the processed features using deep convolutional layers, which can process each input channel independently using convolutional kernels. After passing through the deep convolutional layers, the output features can be batch normalized and ReLU activated again. Then, a second point convolutional layer can be used to restore the number of channels and perform batch normalization to obtain the final processed features. Finally, the original input features of the second channel can be added to the final processed features through residual connections and then subjected to final ReLU activation to obtain the second feature.

[0101] In the example of the second path described above, depth-separable residual blocks can significantly reduce the number of parameters while maintaining effective feature extraction capabilities. Since the second path contains multiple such residual blocks, steady-state structural features at different scales can be captured through progressive feature extraction and downsampling strategies.

[0102] Furthermore, as an example, such as Figure 3 As shown, during the process of extracting the first feature in the first channel and the second feature in the second channel, a cross-channel attention mechanism (or cross-attention mechanism) can be introduced between the first and second channels.

[0103] Specifically, the method may further include: performing the following operations on the second path: performing a linear transformation on the features of the second path to obtain a query vector, and performing a linear transformation on the features of the first path to obtain a key vector and a value vector; determining the dot product of the query vector and the key vector, scaling the dot product, and normalizing the scaled dot product using a normalization function such as the softmax function to obtain attention weights; performing a weighted summation of the attention weights and the value vectors to obtain the attention enhancement features of the second path; and adding the attention enhancement features of the second path to the original second path features using learnable gating parameters to obtain the enhanced features of the second path.

[0104] For example, in the above process, regarding the process of the second path receiving information from the first path, the features of the second path are transformed linearly to obtain a query vector, such as the query vector. It can be represented as The features of the first path are transformed into key vectors and value vectors through different linear transformations, for example, key vectors. It can be represented as value vector It can be represented as ,in, This is an intermediate feature of the second pathway. This is an intermediate feature of the first pathway. These are linear projection weights.

[0105] Then, the query vector can be calculated. With key vector Transposed matrix multiplication, using key vectors Scaling is done by the square root of the dimension to avoid excessively small gradients when the dimension is large. The scaled vector is then normalized using a normalization function such as softmax to obtain the attention weights. For example, attention weights... It can be represented as ,in, The dimension of the key vector.

[0106] Then, the attention fusion results (e.g., attention weights) can be used. Multiply by the learnable first gating parameter This is then added to the original second pathway feature to obtain the enhanced second pathway feature. For example, the enhanced second pathway feature... It can be represented as Here, the first gating parameter can be initialized to, for example, 0.1, and the optimal fusion ratio can be learned through training.

[0107] The above describes attention fusion from the first path to the second path. Symmetrically, attention fusion can also be performed from the second path to the first path. Here, the first path receives information from the second path and performs the exact same attention calculation process.

[0108] Specifically, symmetrically to the operation described above for the second path, the method may further include the following operations for the first path: performing a linear transformation on the features of the first path to obtain a query vector; performing a linear transformation on the features of the first path to obtain a key vector and a value vector; determining the dot product of the query vector and the key vector, scaling the dot product, and normalizing the scaled dot product using a normalization function such as the softmax function to obtain attention weights; performing a weighted summation of the attention weights and the value vectors to obtain the attention enhancement features of the first path; and adding the attention enhancement features of the first path to the original first path features using learnable gating parameters to obtain the enhanced features of the first path.

[0109] For example, similar to the operation for the second path, in the process of the first path receiving information from the second path, the features of the first path are transformed linearly to obtain a query vector, such as the query vector. It can be represented as The features of the second pathway are transformed into key vectors and value vectors through different linear transformations, for example, key vectors. It can be represented as value vector It can be represented as ,in, These are linear projection weights.

[0110] Then, the query vector can be calculated. With key vector Transposed matrix multiplication, using key vectors Scaling is done by the square root of the dimension to avoid excessively small gradients when the dimension is large. The scaled vector is then normalized using a normalization function such as softmax to obtain the attention weights. For example, attention weights... It can be represented as .

[0111] Then, the attention fusion results (e.g., attention weights) can be used. Multiply by the learnable second gating parameter This is then added to the original first pathway features to obtain enhanced first pathway features. For example, enhanced first pathway features... It can be represented as Here, the second gating parameter can be initialized to, for example, 0.1, and the optimal fusion ratio can be learned through training.

[0112] In the above manner, information interaction between two pathways can be achieved through a cross-path attention mechanism in the intermediate stage of feature extraction. This bidirectional attention mechanism allows low-frequency and high-frequency pathways to pass complementary information to each other, so that the two pathways can learn from each other's extracted features. The low-frequency pathway can use the transient event information captured by the high-frequency pathway to enhance its sensitivity to time boundaries, while the high-frequency pathway can use the steady-state context information provided by the low-frequency pathway to constrain the interpretation of transient features.

[0113] In step S140, the classification module of the audio classification model can be used to predict the sound classification of the target audio based on the first feature and the second feature.

[0114] As an example, before predicting the sound classification (or sound category) of the target audio, the first feature and the second feature can be fused to obtain a fused feature.

[0115] Specifically, after obtaining the first and second features in step S130, the multi-level attention fusion module of the audio classification module can, for example, effectively fuse the final features of the two pathways after deep feature extraction of their respective pathways and multiple cross-path interactions.

[0116] For example, the multi-level attention fusion module can fuse the first feature and the second feature to obtain the fused feature in the following way: concatenating the enhanced second path feature and the enhanced first path feature in the channel dimension; integrating the concatenated features through convolutional layers and batch normalization layers; and triggering neurons to perform nonlinear transformation on the integrated features through leakage integrals to obtain the fused feature.

[0117] Specifically, such as Figure 3 As shown, the multi-level attention fusion module can concatenate the final features of the first and second pathways along the channel dimension to obtain concatenated features. For example, the concatenated features can be represented as... ,in, For channel splicing, This indicates the final characteristic of the first pathway. This indicates the final characteristic of the second pathway.

[0118] Following the multi-level attention fusion module of the audio classification module, the convolutional layers of the audio classification module can be used to compress the number of channels of the concatenated features to obtain compressed features, such as compressed features. It can be represented as ,in; For batch normalization, This enables channel integration. The convolutional operation facilitates information exchange and fusion across channels.

[0119] Following the convolutional layers of the audio classification module, compressed features can also be processed through LIF neuron layers. A nonlinear transformation is performed to obtain the final output fused features, such as fused features. It can be represented as ,in, For the leaky integral-firing neuron layer, LIF neurons act as activation functions while maintaining bio-inspired properties consistent with the first pathway.

[0120] In step S140, classification decision processing can be performed on the fused features. For example, global average pooling can be performed on the fused features first to convert the two-dimensional feature map into a one-dimensional feature vector; then, the pooled features can be classified through a fully connected layer to output the score of each category; finally, the softmax function is used to normalize the scores of each category to obtain the probability distribution of each category, so as to predict the sound classification of the target audio based on the probability distribution.

[0121] The above describes an example process for classifying audio using an audio classification model. In embodiments of this disclosure, the audio classification model can be trained based on a process similar to the classification process described above.

[0122] As an example, such as Figure 6 As shown, the audio classification model can be trained in the following way:

[0123] In step S610, a training sample set and a sample label corresponding to each training sample in the training sample set can be obtained. The training samples include audio data, and each sample label marks the sound category of the corresponding training sample.

[0124] In step S620, the audio decomposition module of the audio classification model can be used to decompose the training samples in the training sample set to obtain a first sample sub-band and a second sample sub-band. The first sample sub-band contains a first sample frequency component, and the second sample sub-band contains a second sample frequency component. The sound frequency of the first sample frequency component is higher than the sound frequency of the second sample frequency component.

[0125] As described above, the audio decomposition module includes a prediction operator and an update operator. The prediction operator includes a first convolutional layer and a first scaling parameter. The first scaling parameter is used to scale the convolution result of the first convolutional layer. The update operator includes a second convolutional layer and a second scaling parameter. The second scaling parameter is used to scale the convolution result of the second convolutional layer. Both the first scaling parameter and the second scaling parameter are learnable parameters.

[0126] Here, the process of decomposing the training samples to obtain the first sample subband and the second sample subband is similar to the process of obtaining the first subband and the second subband mentioned above, so it will not be repeated here.

[0127] In step S630, the feature extraction module of the audio classification model can be used to extract features from the first sample sub-band and the second sample sub-band to obtain the first sample feature corresponding to the first sample sub-band and the second sample feature corresponding to the second sample sub-band.

[0128] Here, the process of extracting features from the first sample subband and the second sample subband to obtain the first sample features and the second sample features is similar to the process of obtaining the first features and the second features mentioned above, so it will not be repeated here.

[0129] In step S640, the classification module of the audio classification model can be used to predict the sound classification result of each training sample based on the features of the first sample and the features of the second sample.

[0130] Here, the process of predicting the sound classification result of each training sample is similar to the process of predicting the sound classification of the target audio above, so it will not be repeated here.

[0131] In step S650, the audio classification model can be trained by comparing the sound classification results with the sample labels.

[0132] Here, training the audio classification model can be done end-to-end.

[0133] In one example, the training loss can be determined based on the difference between the sound classification results and the sample labels, and the learnable parameters in the audio classification model can be adjusted based on the training loss to train the audio classification model.

[0134] Specifically, the training loss in this example can also be called the classification loss, for example, the standard cross-entropy loss function can be used. For example, the classification loss can be expressed by the following equation (3):

[0135] (3)

[0136] in, For classification loss (e.g., cross-entropy classification loss). For the sample In the One-hot encoding of the class's actual label. Samples predicted by the model In the Predicted probability of class and These represent the number of samples and the number of categories, respectively.

[0137] In another example, step S650 may include: obtaining a first loss by comparing the sound classification result with the sample labels; and training the audio classification model based on the first loss and the second loss.

[0138] Here, the second loss may include at least one of the reconstruction loss, sparsity loss, and path loss.

[0139] Specifically, in this example, the first loss can be the classification loss shown in the example above, for example, it can be determined based on the above equation (3).

[0140] As an example, the reconstruction loss can be obtained by comparing the training sample set and the reconstruction features, which are obtained by reconstructing audio data based on the first and second subbands.

[0141] Specifically, the first and second sub-bands (e.g., four sub-bands: LL, LH, HL, and HH) can be reconstructed using inverse wavelet transform to obtain the reconstructed input features. The mean square error between the reconstructed input features and the original input features can be calculated as the reconstruction loss in this example.

[0142] This reconstruction loss ensures that the decomposition learned by audio decomposition modules such as wavelet transform is reversible and can preserve the complete information of the original signal. By adding a reconstruction constraint term to the loss function, the wavelet transform maintains good reconstruction properties, meaning that the original input can be recovered almost without loss through the inverse wavelet transform.

[0143] For example, the reconstruction loss can be expressed by the following equation (4):

[0144] (4)

[0145] in, The reconstruction loss is used to constrain the reconstructed input features. Close to the original input features That is, near-perfect reconstruction. Let be the norm, here it is the L2 norm.

[0146] The sparse loss can be obtained based on the output features of each of the multiple neuron modules used to extract the first feature, which are used to simulate auditory neurons.

[0147] Here, the sparsity loss, also known as the sparsity regularization loss, can be obtained by statistically analyzing the average pulse firing rate of all LIF neurons in the high-frequency pulse pathway across all time steps. This sparsity loss encourages neurons to maintain sparse activation, meaning they do not generate pulses most of the time.

[0148] For example, the sparsity loss can be expressed by the following equation (5):

[0149] (5)

[0150] in, For sparsity loss, Let the high-frequency pulse path be denoted as the time step The pulse matrix, where T is the number of time steps.

[0151] The path loss can be obtained based on the classification confidence of the first feature and the classification confidence of the second feature.

[0152] Here, the pathway loss, also known as the pathway balance loss, is determined by calculating the variances of the classification confidence scores output by the first and second pathways respectively, and then calculating the absolute difference between the two variances. This pathway loss ensures that the contributions of the two pathways to the final classification decision are balanced.

[0153] For example, the pathway loss can be expressed by the following equation (6):

[0154] (6)

[0155] in, This is a path balancing term used to align the variances of the output confidence levels for the low-frequency and high-frequency paths. This is the classification confidence (or logits) vector for the first pathway. This represents the classification confidence (or logits) vector for the second path. As an example, classification confidence can refer to the probability vector obtained by applying the classifier's output to the softmax function. For example, it can be represented as ,in, The raw scores are unstandardized (logits).

[0156] One or more of the above-mentioned second losses can be combined with the first loss to establish the training loss. For example, when using the above three second losses, the total loss function can be expressed by the following equation (7):

[0157] (7)

[0158] in, For the total loss, , , The weights for each loss term can be obtained by tuning on the validation set. For example, the weights for the classification loss. The weights can be 1.0, which reconstructs the loss weights. The value can be 0.005, representing the weight of the sparsity loss. It can be 0.0005, the weight for balancing the loss. These hyperparameters can be 0.02, and can be determined on the validation set through grid search.

[0159] Based on the above total loss function, the entire WaveSpikeNet model can be trained through end-to-end joint optimization.

[0160] As an example, the training optimizer can be AdamW, and the initial learning rate can be set to... The weight decay coefficient can be set to A warmup cosine annealing learning rate scheduling strategy is adopted. In the first 8 training epochs, the learning rate linearly increases from zero to its maximum value, and in the subsequent 80 training epochs, the learning rate decays from its maximum value to its minimum value according to a cosine function. The batch size is set to 16. Label smoothing is used, with a smoothing coefficient of 0.1. Additionally, gradient clipping can be applied, where the maximum gradient norm can be set to 1.0.

[0161] In terms of data augmentation, various audio enhancement techniques can be employed to improve the robustness of the model. For example, velocity perturbation randomly changes the playback speed of the audio, with an application probability set to 20%. Another example is additive noise, which adds white noise to the audio with a signal-to-noise ratio between 15 and 30 dB, also with an application probability set to 20%. Yet another example is SpecAugment, which applies frequency masking to the spectrogram, blocking up to 30 frequency bins, and temporal masking, blocking up to 40 time frames. Finally, random time offset randomly shifts the spectrogram along the time axis by ±10% of the total length, with an application probability set to 30%.

[0162] Experiments have verified that the audio classification method according to embodiments of this disclosure can improve parameter efficiency. For example, on the UrbanSound8K dataset, it achieves 95.91% accuracy with 1.9 million parameters, compared to ResNetSE's 7.8 million parameters and 95.07% accuracy, representing a 75.6% reduction in parameters while improving accuracy by 0.84%.

[0163] Furthermore, the audio classification method according to embodiments of this disclosure is deployment-friendly. The model size can be only 7.3 megabytes, a 75.7% compression compared to ResNetSE's 30.1 megabytes. Additionally, the model training time can be only 1.8 hours, a 52.6% faster time compared to ResNetSE's 3.8 hours.

[0164] Furthermore, the audio classification method according to embodiments of this disclosure exhibits excellent generalization ability. For example, it achieves 70.0% on ESC-50, 60.58% on GTZAN, and 96.35% on Speech Commands v2, with all tests trained from scratch.

[0165] Furthermore, the audio classification method according to the embodiments of this disclosure achieves an organic combination of theory and engineering. By deeply drawing on the principles of the biological auditory system, it realizes a principle-driven architecture design and achieves brain-like audio classification effects.

[0166] Furthermore, the audio classification method according to the embodiments of this disclosure has good scalability and supports multiple scale variants from nano, tiny, base to large scales.

[0167] Several examples of audio classification methods using embodiments of this disclosure will be given below.

[0168] The first example shows a specific application instance of the WaveSpikeNet model on the UrbanSound8K dataset.

[0169] UrbanSound8K is a widely used dataset for classifying urban environmental sounds. It contains 8732 audio clips divided into 10 categories, including air conditioner sounds, car horn sounds, children playing sounds, dog barking sounds, drilling sounds, engine idling sounds, gunshots, jackhammer sounds, police sirens, and street music. The dataset is pre-divided into 10 folds for 10-fold cross-validation.

[0170] All audio data is preprocessed using an audio classification method according to embodiments of this disclosure. The original audio is resampled to 16 kHz, truncated or padded to a length of 3 seconds, and log-Mel spectrograms of 80 Mel frequency bands are extracted. A Base model is constructed with 24 base channels and a total of 1.9 million parameters.

[0171] The training configuration strictly follows the settings described in Example 1. The AdamW optimizer is used, with an initial learning rate... After the first 8 warm-up rounds, cosine annealing was performed, with a total of 80 training rounds. The batch size was 16, label smoothing was 0.1, and gradient clipping norm was 1.0. The loss function weights were: classification loss 1.0, reconstruction loss 0.005, sparsity loss 0.0005, and balance loss 0.02.

[0172] WaveSpikeNet achieves an average accuracy of 95.91% on UrbanSound8K. Compared to other methods in the same field, ResNetSE has 7.8 million parameters and an accuracy of 95.07%. ERes2Net has 6.6 million parameters and an accuracy of 95.00%. CAM++ has 7.2 million parameters and an accuracy of 93.97%. WaveSpikeNet achieves the highest accuracy of 95.91% with the fewest parameters (1.9 million), representing a 75.6% reduction in parameters and a 0.84% ​​improvement in accuracy compared to ResNetSE.

[0173] Furthermore, the trained model size is 7.3 megabytes, only 24.3% of ResNetSE's 30.1 megabytes. The test set inference time is 9.19 seconds, processing 874 test samples, with an average time of approximately 10.5 milliseconds per sample, meeting real-time processing requirements. These advantages make the audio classification model (hereinafter also referred to as WaveSpikeNet) according to embodiments of this disclosure particularly suitable for deployment on resource-constrained edge devices.

[0174] In addition, such as Figure 7As shown, by plotting the distribution of different models in the parameter and accuracy space, it can be seen that WaveSpikeNet is located at the Pareto optimal front. Traditional high-performance models such as ResNetSE, ERes2Net, and CAM++ are located in the high-parameter, high-accuracy region. Traditional lightweight models such as EfficientAT, MobileNetV3, and ShuffleNetV2 are located in the low-parameter, low-accuracy region. The WaveSpikeNet model constructed according to the method of embodiments of this disclosure exclusively occupies the low-parameter, high-accuracy region, avoiding the limitations of traditional trade-offs. Figure 7 In this context, the number of parameters is in the millions, and a log scale is used.

[0175] In the second example, the effectiveness of each key component in the architecture is verified through ablation experiments on the system. Here, the ablation experiments are conducted on the UrbanSound8K dataset, keeping other conditions constant, and each component is removed or replaced sequentially.

[0176] Specifically, we first evaluated the contribution of the dual-path design. Changing the model to a single-path architecture, removing frequency decoupling and heterogeneous processing, resulted in an accuracy drop to 93.68%, a decrease of 2.23 percentage points compared to the complete model. This demonstrates the importance of dual-path heterogeneous processing.

[0177] Furthermore, the audio decomposition module employing learnable wavelet transform was evaluated. Specifically, replacing the learnable discrete wavelet transform with a fixed Haar wavelet transform resulted in an accuracy of 94.12%, a decrease of 1.79 percentage points compared to the full model. This demonstrates that task-adaptive frequency decomposition does indeed outperform fixed wavelet basis functions. Completely removing the wavelet transform module resulted in an accuracy decrease to 94.35%, a drop of 1.56 percentage points.

[0178] Furthermore, the effect of the spiking mechanism in the first pathway was evaluated. Replacing LIF neurons in the first pathway, which is a high-frequency pathway, with the standard ReLU activation function resulted in an accuracy of 94.89%, a decrease of 1.02 percentage points. Although the individual contribution of the spiking mechanism was relatively small, combined with the analysis of the activation patterns of the first pathway, it was found that LIF neurons achieved significant sparse activation.

[0179] Furthermore, the effect of multi-level fusion was evaluated. Removing the cross-pathway attention interactions in the middle layers and retaining only the final fusion resulted in a decrease in accuracy to 94.78%, a drop of 1.13 percentage points. This indicates that information exchange between the two pathways is beneficial during processing.

[0180] The ablation experiments described above demonstrate a synergistic effect among the components, which together achieve the superior performance of the WaveSpikeNet audio classification model according to embodiments of this disclosure.

[0181] In the third example, the performance and efficiency tradeoffs of different scale variants were also evaluated. A series of variant models, ranging from Nano, Tiny, Base to Large scales, were built by adjusting the base channel width.

[0182] The Nano-scale variant uses 16 as the base number of channels, has 0.8 million parameters, and achieves an accuracy of 93.12%. Although the accuracy is lower than the Base variant, its extremely small model size makes it particularly suitable for scenarios with extremely limited resources.

[0183] The Tiny scale variant uses 20 as the base number of channels, has 1.2 million parameters, and achieves an accuracy of 94.68%. The Tiny variant strikes a good balance between parameter count and performance.

[0184] The Base-scale variant uses 24 as the base number of channels, has 1.9 million parameters, and achieves 95.91% accuracy. The Base variant achieves an optimal balance across all metrics.

[0185] The Large variant uses 32 as the base number of channels, has 3.4 million parameters, and achieves an accuracy of 96.34%. The Large variant offers a 0.43 percentage point improvement over the Base variant, but with a 79% increase in parameters, demonstrating diminishing marginal returns.

[0186] The comparison revealed that the Base variant is located near the inflection point of the performance efficiency curve, achieving the best trade-off between performance and efficiency, and is the recommended standard configuration.

[0187] Furthermore, as can be seen from the above, the number of parameters of the audio classification model according to the embodiments of this disclosure can be in the range of 0.8 million to 3.4 million (or scale), with a basic scale of only 1.9 million parameters. Compared with the number of parameters of conventional audio classification models, it can be reduced by about 2 / 3, which makes the model suitable for low-power edge devices.

[0188] In the fourth example, in addition to the datasets mentioned above, generalization validation was performed on other datasets to verify the generalization ability of WaveSpikeNet in different audio domains.

[0189] Specifically, on the ESC-50 ambient sound classification dataset, which contains 50 classes of ambient sounds and a total of 2000 samples, WaveSpikeNet achieved an accuracy of 70.0%, a significant improvement over ResNetSE's 63.94%.

[0190] On the GTZAN music genre classification dataset, which contains 10 music styles and 1000 samples, WaveSpikeNet achieved an accuracy of 60.58%, which is close to CAM++'s 61.61%.

[0191] On the Speech Commands v2 speech recognition dataset, which contains 35 classes of speech commands, WaveSpikeNet achieved an accuracy of 96.35%, approaching the performance of large-scale models.

[0192] The experimental results above demonstrate that WaveSpikeNet has good generalization ability, performing excellently not only in urban environmental sound classification but also in other audio classification tasks.

[0193] The fifth example demonstrates the deployment of WaveSpikeNet on a real-world edge device. Here, the target edge device is an ARM-based embedded system with an ARM Cortex-A76 core processor and 4 gigabytes of system memory.

[0194] The trained model was optimized before deployment. First, model quantization was performed, converting the 32-bit floating-point weights to 8-bit integers. The quantized model size decreased from 7.3 megabytes to 1.8 megabytes. The accuracy of the quantized model was evaluated on the test set, achieving 95.73%, a decrease of only 0.18 percentage points.

[0195] Then, operator fusion optimization is performed, fusing the batch normalization layer into the convolutional layer to reduce memory accesses. Convolutional layers and activation functions are also fused to reduce the storage of intermediate results.

[0196] The optimized model can then be converted to a format suitable for the target platform. Using ONNX as an intermediate representation, the model is exported to ONNX format and then converted to TensorFlow Lite format, which is specifically optimized for mobile and embedded devices.

[0197] Furthermore, according to embodiments of this disclosure, an inference engine and an audio processing application can also be integrated on a target device, such as an edge device. The application may include, for example, an audio acquisition module, a preprocessing module, an inference module, and a post-processing module to implement real-time audio stream processing according to the audio classification method of embodiments of this disclosure.

[0198] The performance tests described above show that the audio classification model according to the embodiments of this disclosure, deployed on an edge device, has a single inference latency of approximately 45 milliseconds, which meets the requirements of real-time processing. It has a runtime memory footprint of approximately 38 megabytes and a power consumption of approximately 2.8 watts, making it suitable for long-term continuous operation and battery-powered scenarios.

[0199] Overall, the audio classification model and method according to embodiments of this disclosure can simulate the biological mechanisms of the human auditory system. Experimental results show that the audio classification model achieves good classification performance with very few parameters, significantly improving parameter efficiency. Furthermore, the practicality of this audio classification method has been verified in multiple datasets and real-world deployment scenarios, providing an efficient and feasible solution for audio intelligence applications on edge devices.

[0200] In a second aspect of exemplary embodiments of this disclosure, an audio classification apparatus is provided, such as... Figure 8 As shown, the audio classification device includes an acquisition unit 810, a decomposition unit 820, an extraction unit 830, and a prediction unit 840.

[0201] The acquisition unit 810 is configured to acquire the target audio.

[0202] The decomposition unit 820 is configured to decompose the target audio using an audio decomposition module of an audio classification model to obtain a first sub-band and a second sub-band, wherein the first sub-band contains a first frequency component and the second sub-band contains a second frequency component, and the sound frequency of the first frequency component is higher than the sound frequency of the second frequency component.

[0203] The extraction unit 830 is configured to use the feature extraction module of the audio classification model to extract features from the first sub-band and the second sub-band, thereby obtaining a first feature corresponding to the first sub-band and a second feature corresponding to the second sub-band.

[0204] The prediction unit 840 is configured to use the classification module of the audio classification model to predict the sound classification of the target audio based on the first feature and the second feature.

[0205] Optionally, the audio decomposition module includes a prediction operator and an update operator, wherein the audio decomposition module decomposes the target audio into a first sub-band and a second sub-band in the following manner: In the time dimension, the audio features of the target audio are decomposed into even-numbered and odd-numbered parts according to the parity of the index of the audio features; the prediction operator is used to perform a prediction operation on the even-numbered parts to obtain a prediction result; the prediction result is subtracted from the odd-numbered parts to obtain a first coefficient, wherein the first coefficient represents the detailed features of the audio; the update operator is used to update the first coefficient to obtain an update result; the update result is added to the even-numbered parts to obtain a second coefficient, wherein the second coefficient represents the overall features of the audio; in the frequency dimension, the first coefficient and the second coefficient are decomposed into a second sub-band according to the frequency components to obtain the first sub-band and the second sub-band.

[0206] Optionally, the feature extraction module includes a first path and a second path, wherein the feature extraction module obtains the first feature and the second feature in the following manner: using the first path, extracting transient features from the first sub-band as the first feature; and using the second path, extracting steady-state features from the second sub-band as the second feature.

[0207] Optionally, the first pathway includes multiple neuron modules connected in series, which are used to simulate auditory neurons. The feature extraction module obtains a first feature by: inputting a first sub-band into the first pathway; extracting the first feature from the first sub-band using multiple neuron modules; wherein the output feature of the last neuron module among the multiple neuron modules is the first feature; and for each neuron module, the output feature is obtained by: updating the current membrane potential based on a preset leakage factor and the current input feature to obtain an updated membrane potential; wherein for the first neuron module, the current input feature is the first sub-band, and the leakage factor is used to control the degree to which each neuron module retains historical information; and generating the output feature of the current neuron module in response to the updated membrane potential being greater than or equal to a firing threshold, and resetting the updated membrane potential based on the firing threshold and pulse output to obtain a reset membrane potential; wherein the firing threshold is a learnable parameter.

[0208] Optionally, the audio classification model is trained as follows: A training sample set and sample labels corresponding to each training sample in the training sample set are obtained, wherein the training samples include audio data, and each sample label indicates the sound category of the corresponding training sample; the audio decomposition module of the audio classification model is used to decompose the training samples in the training sample set to obtain a first sample sub-band and a second sample sub-band, wherein the first sample sub-band contains a first sample frequency component, the second sample sub-band contains a second sample frequency component, and the sound frequency of the first sample frequency component is higher than the sound frequency of the second sample frequency component; the feature extraction module of the audio classification model is used to extract features from the first sample sub-band and the second sample sub-band to obtain a first sample feature corresponding to the first sample sub-band and a second sample feature corresponding to the second sample sub-band; the classification module of the audio classification model is used to predict the sound classification result of each training sample based on the first sample feature and the second sample feature; and the audio classification model is trained by comparing the sound classification result with the sample label.

[0209] Optionally, the audio decomposition module includes a prediction operator and an update operator. The prediction operator includes a first convolutional layer and a first scaling parameter. The first scaling parameter is used to scale the convolution result of the first convolutional layer. The update operator includes a second convolutional layer and a second scaling parameter. The second scaling parameter is used to scale the convolution result of the second convolutional layer. Both the first scaling parameter and the second scaling parameter are learnable parameters.

[0210] Optionally, the audio classification model is trained by: obtaining a first loss by comparing the sound classification results and sample labels; training the audio classification model based on the first loss and a second loss, wherein the second loss includes at least one of reconstruction loss, sparsity loss, and path loss, wherein the reconstruction loss is obtained by comparing the training sample set and reconstruction features, the reconstruction features are obtained by reconstructing audio data based on the first sub-band and the second sub-band, wherein the sparsity loss is obtained based on the output features of each of the multiple neuron modules used to extract the first feature, the neuron modules being used to simulate auditory neurons, and wherein the path loss is obtained based on the classification confidence of the first feature and the classification confidence of the second feature.

[0211] It should be understood that the specific processing performed by the aforementioned device has been referenced. Figures 1 to 7 The above description has been detailed and will not be repeated here. Furthermore, it should be understood that the various units in the above-described apparatus can be implemented as hardware components and / or software components.

[0212] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform an audio classification method according to an embodiment of the present disclosure.

[0213] As an example, an electronic device does not necessarily have to be a single device; it can also be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. An electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a server, either locally or remotely (e.g., via wireless transmission).

[0214] In electronic devices, processors may include central processing units (CPUs), graphics processing units (GPUs), programmable logic devices, dedicated processor systems, microcontrollers, or microprocessors. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0215] The processor can execute instructions or code stored in memory, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0216] Memory can be integrated with the processor; for example, RAM or flash memory can be housed within an integrated circuit microprocessor. Alternatively, memory can comprise a separate device, such as an external disk drive, storage array, or other storage device that can be used by any database system. Memory and processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the memory.

[0217] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device can be interconnected via buses and / or networks.

[0218] In an exemplary embodiment, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the audio classification method as described in the exemplary embodiment above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, or a BD-R... LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0219] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer-executable instructions that, when executed by at least one processor, implement the audio classification method according to exemplary embodiments of the present disclosure.

[0220] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0221] Furthermore, it should be noted that although several examples of each step have been described above with reference to the specific accompanying drawings, it should be understood that the embodiments of this disclosure are not limited to the combinations given in the examples. The steps appearing in different drawings can be combined, and the execution order of each step can be changed, which will not be exhaustive here.

[0222] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio classification method, characterized in that, The audio classification method includes: Obtain the target audio; The target audio is decomposed using the audio decomposition module of the audio classification model to obtain a first sub-band and a second sub-band. The first sub-band contains a first frequency component, and the second sub-band contains a second frequency component. The sound frequency of the first frequency component is higher than the sound frequency of the second frequency component. Using the feature extraction module of the audio classification model, features are extracted from the first sub-band and the second sub-band to obtain a first feature corresponding to the first sub-band and a second feature corresponding to the second sub-band. Using the classification module of the audio classification model, based on the first feature and the second feature, the sound classification of the target audio is predicted. The audio decomposition module includes a prediction operator and an update operator, wherein the audio decomposition module obtains the first sub-band and the second sub-band from the target audio in the following manner: In the time dimension, the audio features of the target audio are decomposed into even and odd parts according to the parity of the index of the audio features of the target audio. Using the prediction operator, a prediction operation is performed on the even-numbered portion to obtain the prediction result; By subtracting the prediction result from the odd-numbered portion, a first coefficient is obtained, wherein the first coefficient characterizes the detailed features of the audio. The first coefficient is updated using the update operator to obtain the update result; By adding the updated result to the even-numbered portion, a second coefficient is obtained, wherein the second coefficient characterizes the overall features of the audio. In the frequency dimension, the first coefficient and the second coefficient are decomposed into a second-level decomposition according to their frequency components to obtain the first sub-band and the second sub-band. The prediction operator includes a first convolutional layer and a first scaling parameter, wherein the first scaling parameter is used to scale the convolution result of the first convolutional layer. The update operator includes a second convolutional layer and a second scaling parameter, wherein the second scaling parameter is used to scale the convolution result of the second convolutional layer. The audio classification model employs end-to-end training, and both the first and second scaling parameters are learnable parameters acquired during the training process of the audio classification model. The feature extraction module includes a first path and a second path, wherein the feature extraction module obtains the first feature and the second feature in the following manner: Using the first path, transient features are extracted from the first sub-band as the first feature, wherein the first path is implemented based on a spiking neural network; Using the second path, steady-state features are extracted from the second sub-band as the second feature, wherein the second path is implemented based on depth-separable residual blocks.

2. The audio classification method according to claim 1, characterized in that, The first pathway includes multiple neuronal modules connected in series, which are used to simulate auditory neurons. The feature extraction module obtains the first feature by inputting the first sub-band into the first pathway and extracting the first feature from the first sub-band using the plurality of neuron modules, wherein the output feature of the last neuron module among the plurality of neuron modules is the first feature. For each neuron module, the output features are obtained in the following way: Based on a preset leakage factor and current input features, the current membrane potential is updated to obtain an updated membrane potential. For the first neuron module, the current input feature is the first sub-band, and the leakage factor is used to control the degree to which each neuron module retains historical information. In response to the updated membrane potential being greater than or equal to the firing threshold, the output features of the current neuron module are generated, and based on the firing threshold and the output features, the updated membrane potential is reset to obtain a reset membrane potential, wherein the firing threshold is a learnable parameter.

3. The audio classification method according to claim 1, characterized in that, The audio classification model was trained in the following way: Obtain a training sample set and a sample label corresponding to each training sample in the training sample set, wherein the training samples include audio data, and each sample label marks the sound category of the corresponding training sample; Using the audio decomposition module of the audio classification model, the training samples in the training sample set are decomposed to obtain a first sample sub-band and a second sample sub-band. The first sample sub-band contains a first sample frequency component, and the second sample sub-band contains a second sample frequency component. The sound frequency of the first sample frequency component is higher than the sound frequency of the second sample frequency component. Using the feature extraction module of the audio classification model, feature extraction is performed on the first sample sub-band and the second sample sub-band to obtain the first sample feature corresponding to the first sample sub-band and the second sample feature corresponding to the second sample sub-band. Using the classification module of the audio classification model, the sound classification result of each training sample is predicted based on the first sample features and the second sample features; The audio classification model is trained by comparing the sound classification results with the sample labels.

4. The audio classification method according to claim 3, characterized in that, The step of training the audio classification model by comparing the sound classification results with the sample labels includes: The first loss is obtained by comparing the sound classification result with the sample label; The audio classification model is trained based on the first loss and the second loss. The second loss includes at least one of reconstruction loss, sparsity loss, and path loss. The reconstruction loss is obtained by comparing the training sample set and the reconstruction features, and the reconstruction features are obtained by reconstructing audio data based on the first sub-band and the second sub-band. The sparse loss is obtained based on the output features of each of a plurality of neuron modules used to extract the first feature, wherein the neuron modules are used to simulate auditory neurons. The path loss is obtained based on the classification confidence of the first feature and the classification confidence of the second feature.

5. An audio classification device, characterized in that, The audio classification device includes: The acquisition unit is configured to acquire the target audio. The decomposition unit is configured to decompose the target audio using an audio decomposition module of an audio classification model to obtain a first sub-band and a second sub-band, wherein the first sub-band contains a first frequency component and the second sub-band contains a second frequency component, and the sound frequency of the first frequency component is higher than the sound frequency of the second frequency component. The extraction unit is configured to use the feature extraction module of the audio classification model to extract features from the first sub-band and the second sub-band, thereby obtaining a first feature corresponding to the first sub-band and a second feature corresponding to the second sub-band. The prediction unit is configured to use the classification module of the audio classification model to predict the sound classification of the target audio based on the first feature and the second feature. The audio decomposition module includes a prediction operator and an update operator. The audio decomposition module obtains the first sub-band and the second sub-band from the target audio in the following manner: In the time dimension, the audio features of the target audio are decomposed at a first level according to the parity of their indexes to obtain even-numbered and odd-numbered parts; the prediction operator is used to predict the even-numbered parts to obtain a prediction result; the prediction result is subtracted from the odd-numbered parts to obtain a first coefficient, where the first coefficient represents the detailed features of the audio; the update operator is used to update the first coefficient to obtain an update result; the update result is added to the even-numbered parts to obtain a second coefficient, where the second coefficient represents the overall features of the audio; in the frequency dimension, the first coefficient and the second coefficient are decomposed at a second level according to their frequency components to obtain the first sub-band and the second sub-band. The prediction operator includes a first convolutional layer and a first scaling parameter, wherein the first scaling parameter is used to scale the convolution result of the first convolutional layer. The update operator includes a second convolutional layer and a second scaling parameter, wherein the second scaling parameter is used to scale the convolution result of the second convolutional layer. The audio classification model employs end-to-end training, and both the first and second scaling parameters are learnable parameters acquired during the training process of the audio classification model. The feature extraction module includes a first path and a second path, wherein the feature extraction module obtains the first feature and the second feature in the following manner: Using the first path, transient features are extracted from the first sub-band as the first feature, wherein the first path is implemented based on a spiking neural network; Using the second path, steady-state features are extracted from the second sub-band as the second feature, wherein the second path is implemented based on depth-separable residual blocks.

6. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions. Wherein, when the processor executes the processor, it causes the processor to perform the audio classification method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the audio classification method according to any one of claims 1 to 4.

8. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the audio classification method according to any one of claims 1 to 4.

Images