Method and device for multi-spectrogram lightweight acoustic scene classification facing hearing aid devices, equipment and storage medium

Abstract

This invention discloses a lightweight, multi-spectral acoustic scene classification method, apparatus, device, and storage medium for hearing aids. First, the audio signal of the hearing aid's environment is acquired and converted into Mel-spectral feature maps, constant Q-transform spectral feature maps, and first-order differential spectral feature maps. These are then stacked to form a multi-channel spectral feature map containing complementary time-frequency information, which serves as the model input. Subsequently, differential shallow feature extraction is performed using an asymmetric multi-stream module, and deep features are extracted on a low-power computing platform using a lightweight backbone network centered on a ghost bottleneck module and coordinate attention. A spectral morphology module, a bandgap attention module, and a channel rebalancing fusion module are introduced to enhance and dynamically weight the deep features in multiple dimensions. Finally, the classifier outputs the audio scene classification result. This invention can significantly reduce the number of model parameters and computational power consumption while improving the classification performance for complex acoustic scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent hearing aids technology, specifically to a lightweight multispectral acoustic scene classification method, apparatus, device, and storage medium for hearing aids. Background Technology

[0002] With the rapid development of hearing rehabilitation technology, cochlear implants and digital hearing aids have become core assistive devices for hearing-impaired individuals to rebuild their hearing. To balance speech clarity and listening comfort in different environments, modern high-end hearing aids typically have multiple pre-set signal processing strategies. However, accurately and in real-time determining the current acoustic environment is a prerequisite and the biggest bottleneck in ensuring the correct activation of these subsequent signal processing strategies. Traditional hearing aids often employ fixed or manually switched signal processing strategies, making it difficult to adapt to dynamically changing acoustic scenarios. For example, if a traffic scene cannot be accurately identified, the hearing aid may easily misinterpret its low-frequency noise as background noise from a normal indoor environment, thus failing to activate the powerful noise reduction mode, resulting in the user being unable to clearly see lip movements or hear speech in a noisy vehicle. Therefore, endowing hearing aids with high-precision "intelligent environmental perception" capabilities has become a key technological trend for improving the user experience for hearing-impaired individuals.

[0003] Currently, although deep learning technology has made significant progress in the field of Acoustic Scene Classification (ASC), its application to portable medical devices such as cochlear implants or hearing aids still faces serious challenges, mainly in the following three aspects: a) The contradiction between high-performance computing requirements and miniaturized, low-power hardware: Existing high-performance acoustic classification models (such as those based on VGG-Net or ResNet architectures) have a huge number of parameters and extremely high computational complexity. Hearing aids, limited by size, are usually powered by miniature batteries and rely on low-power DSP (Digital Signal Processor) chips. Directly deploying traditional large models will lead to a decrease in device battery life or unacceptable audio processing latency, failing to meet the stringent requirements of all-day wear and real-time performance of hearing aids. b) Insufficient recognition of similar scene features: Most existing lightweight models rely solely on a single Mel spectrogram as input. Although the Mel spectrogram can simulate human hearing, its frequency resolution in the low-frequency region is limited, and its ability to capture temporal dynamic changes is weak. This often leads to misjudgments by hearing aids, such as mistaking traffic noise for ordinary indoor background noise, resulting in the failure to activate targeted traffic noise reduction strategies. c) Inefficiency of multi-feature fusion mechanisms: To improve recognition rates, some studies have attempted to introduce multiple acoustic features, but usually employ a simple channel splicing strategy. This strategy ignores the differences in the importance of different features in different scenarios, easily causing the mainstream features to suppress the effective expression of key auxiliary features during model learning, thus failing to achieve true feature synergy.

[0004] Therefore, designing an acoustic scene classification scheme specifically tailored for hearing aids, which can efficiently integrate multiple heterogeneous acoustic features while meeting the requirements of low power consumption and lightweight deployment, thereby achieving high-precision identification of complex life scenarios, providing a reliable decision basis for the adaptive adjustment of hearing aid parameters, and having important application value and engineering significance for promoting the development of the next generation of intelligent hearing aids. Summary of the Invention

[0005] To address the challenges of existing acoustic scene classification models being ill-suited for resource-constrained platforms of hearing aids, the insufficient ability of single features to discern complex acoustic scenes, and the inefficiency of multi-feature fusion mechanisms, this invention proposes a lightweight multispectral acoustic scene classification method, apparatus, and storage medium specifically tailored for hearing aids. This method, based on a feature reuse mechanism, aims to achieve high-precision acoustic environment perception with minimal computational cost, providing a reliable decision-making basis for switching signal processing strategies in hearing aids.

[0006] To achieve the above effects, the technical solution adopted by the present invention is as follows:

[0007] A lightweight multispectral acoustic scene classification method for hearing aids, characterized by the following steps:

[0008] S1: Acquire the ambient audio signal of the hearing aid front-end to be processed;

[0009] S2: Input the ambient audio signal from the hearing aid front end to the feature extraction unit and output deep features;

[0010] The feature extraction unit includes:

[0011] The heterogeneous acoustic feature extraction and stacking module is used to convert the ambient audio signal of the hearing aid front end into a multi-channel spectral feature map;

[0012] The asymmetric multi-stream input module connected to the acoustic feature extraction and stacking module is used to receive the multi-channel spectral feature map sent by the heterogeneous acoustic feature extraction and stacking module and perform feature differentiation processing.

[0013] A ghost bottleneck module connected to the asymmetric multi-stream input module, the ghost bottleneck module outputs the depth feature;

[0014] S3: Input the deep features into the feature fusion module. The feature fusion module dynamically adjusts and fuses the deep features, and outputs the fused features.

[0015] S4: Input the fused features into the classifier to obtain the final audio scene classification result. The audio scene classification result is configured as the basis for the environmental perception decision-making of the hearing aid device during signal processing.

[0016] This invention first acquires the ambient audio signal of the hearing aid front-end to be processed and converts it into Mel-frequency spectral feature maps, constant Q-transform spectral feature maps, and first-order differential spectral feature maps. These are then stacked to form a multi-channel spectral feature map containing complementary time-frequency information, which serves as the model input. Subsequently, differential shallow feature extraction is performed through an asymmetric multi-stream module, and deep features are extracted on a low-power computing platform using a lightweight backbone network centered on a ghost bottleneck module and coordinate attention. Based on this, a spectral morphology module, a frequency band attention module, and a channel rebalancing fusion module are introduced to enhance and dynamically weight the deep features in multiple dimensions. Finally, the classifier outputs the audio scene classification results, and an adaptive composite contrastive loss function is used for optimization training. This invention can significantly reduce the number of model parameters and computational power consumption while improving the classification performance of complex acoustic scenes, providing a reliable decision-making basis for environmental perception and signal processing strategy switching of hearing aids.

[0017] In step S1, the ambient audio signal of the hearing aid front-end to be processed is acquired, specifically including:

[0018] During operation, hearing aids collect real-time acoustic environment data to obtain ambient audio signals from the hearing aid front-end environment that need to be processed.

[0019] In step S2, each channel of the multi-channel spectral feature map is a time-frequency representation of a certain type, and the time-frequency representations of each channel are of different types.

[0020] In step S2, the ambient audio signal of the hearing aid front end is converted into a multi-channel spectral feature map, specifically including:

[0021] Mel spectrogram, constant-Q transform spectrogram, and first-difference Mel-Delta spectrogram are extracted in parallel from the ambient audio signal of the hearing aid front end and stacked together to form a three-channel feature map, which is the multi-channel spectral feature map.

[0022] In step S2, the asymmetric multi-stream input module includes three independent convolutional streams: a convolutional stream containing four convolutional layers for processing Mel spectra, a convolutional stream containing two convolutional layers for processing constant Q transform spectra, and a convolutional stream containing two convolutional layers for processing first-order difference Mel spectra.

[0023] In step S2, the ghost bottleneck module integrates a coordinate attention module. By aggregating features along two spatial directions respectively, the positional information is encoded into the channel attention, thereby capturing direction-aware and position-sensitive information in the feature map while maintaining lightweight design, thus improving the model's feature extraction capability.

[0024] The specific implementation principle of the aforementioned ghost bottleneck module is as follows:

[0025] First, a small number of standard convolutional kernels are applied to the input feature map to generate a portion of "endogenous feature maps";

[0026] Then, a series of computationally inexpensive linear transformations are performed on this part of the endogenous feature map to generate additional "ghost feature maps";

[0027] Finally, the two feature maps are concatenated along the channel dimension to form the final output.

[0028] In step S3, the feature fusion module adopts a lightweight channel rebalanced fusion module, and its calculation process is represented by the following formula:

[0029]

[0030] in, Input feature map; An importance evaluator, consisting of a neural network with fully connected layers, is used to evaluate the importance of channels. This indicates a global average pooling operation used to extract channel statistics. for Activation function; The channel importance weight represents the degree of contribution of each channel to the classification. These are learnable bias parameters used to adjust the weight distribution after inversion; To pass Normalized rebalancing weights; This is an element-wise multiplication operation; These are characteristics after rebalancing; This is the final output feature.

[0031] In step S3, the dynamic weight adjustment of the feature channels is optimized using a composite contrastive learning loss function. The composite contrastive learning loss function... Its specific form is:

[0032]

[0033] in, The total loss function represents the model optimization. This represents the cross-entropy loss with label smoothing. It improves the model's generalization ability by converting one-hot hard labels into probabilistic soft labels; The weight hyperparameters are dynamically adjusted to control the contribution of the contrast loss. For training rounds in model learning; It is a minimum value function; To compare the learning loss, it is used to optimize the distribution of the feature space; To compare the initial weight values ​​of the loss; The average similarity of all positive sample pairs; The average similarity of all negative sample pairs; For temperature hyperparameters, adjust the smoothness of the similarity distribution; This is a numerical stability constant, ensuring computational stability.

[0034] A sound scene classification apparatus for implementing the multispectral lightweight acoustic scene classification method for hearing aids includes:

[0035] A sound sensor is used to acquire ambient audio signals from the hearing aid front end to be processed.

[0036] The feature extraction unit is used to convert the ambient audio signal of the hearing aid front end into a multi-channel spectral feature map and perform feature differentiation processing, and then perform feature extraction to output deep features.

[0037] The feature fusion module is used to dynamically adjust and fuse deep features, and output the fused features.

[0038] A classifier is used to output the sound scene category based on the fused features.

[0039] A hearing aid device includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the multispectral lightweight acoustic scene classification method for hearing aid devices.

[0040] The hearing aid is a smart home device or a personal assistant device.

[0041] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multispectral lightweight acoustic scene classification method for hearing aids.

[0042] Specifically, a lightweight multispectral acoustic scene classification method based on feature reuse mechanism for hearing assistive devices includes the following steps:

[0043] S1: Constructing a multi-channel heterogeneous auditory feature input for hearing aids. Acquire the ambient audio signal of the hearing aid front-end to be processed, which corresponds to the real-time ambient sound picked up by the microphone array during actual operation of the hearing aid. Convert the audio signal into a Mel spectrogram, a constant Q-transform spectrogram, and a first-order difference spectrogram, and stack these three spectrogram features along the channel dimension to form a multi-channel spectral feature map containing complementary time-frequency information. In a preferred embodiment, to adapt to the front-end signal processing standards of hearing aids, preprocessing operations including front-end signal conditioning, generalization training dataset partitioning, functional category mapping, and time-frequency alignment and gain normalization of heterogeneous features are performed before feature extraction.

[0044] S2: Perform asymmetric multi-stream shallow feature extraction. Input the multi-channel spectral feature map obtained in step S1 into the asymmetric multi-stream input module. Allocate different computational depths according to the information density of different spectral maps to efficiently extract shallow textures and reduce computational overhead under the condition of limited resources on the edge side.

[0045] S3: Extract deep features using a lightweight network. The shallow feature map output from step S2 is fed into a lightweight backbone network with a ghost bottleneck module at its core and an integrated coordinate attention mechanism to extract high-dimensional deep features with minimal computational power, ensuring the real-time operation of the hearing aid device.

[0046] S4: Implement multi-dimensional acoustic feature enhancement. The deep features extracted in step S3 are enhanced in multiple dimensions, sequentially through the spectral morphology module and the frequency band attention module, to enhance the fine-grained resolution of the features and the expression of key frequency bands.

[0047] S5: Construct a channel rebalancing fusion module. A lightweight channel rebalancing fusion module is used to weight the features enhanced in step S4. This module adjusts the channel weights in real time according to changes in the acoustic environment, suppressing redundant interference and improving the model's robustness in dynamic scenes.

[0048] S6: Model Optimization and Decision Output. During the training phase, an adaptive composite contrastive loss function is used to optimize the feature space. During the inference phase, the audio scene classification results are output as the basis for environmental perception decision-making by the main control unit of the hearing aid device.

[0049] Compared with existing methods, the beneficial effects of the present invention are mainly reflected in:

[0050] 1. This invention proposes a lightweight multispectral acoustic scene classification method, device, and storage medium for hearing aids. At the feature level, by constructing a multi-channel spectral feature map including Mel spectrograms, constant Q-transform spectrograms, and differential spectrograms as input, and employing an asymmetric multi-stream architecture, differentiated extraction of different feature sources is achieved, effectively solving the feature confusion problem faced by hearing aids when processing similar acoustic scenes (such as low-frequency traffic noise and indoor background noise).

[0051] 2. In terms of network backbone design, a lightweight backbone network with ghost bottleneck module and coordinate attention as the core is introduced. This design greatly reduces the number of model parameters and floating-point operations, making it easier to deploy on low-power DSP chips with limited computing resources. This helps to improve the real-time performance of environmental perception processing while meeting the battery life design requirements of hearing assistive devices.

[0052] 3. In the feature enhancement and fusion stage, a multi-dimensional feature enhancement and fusion mechanism was designed. Combining the spectral morphology module, the frequency band attention module, and the lightweight channel rebalancing fusion module, it not only enhances the ability to capture weak acoustic details, but also dynamically adjusts the channel weights in real time according to the environment, maximizing the expression of key auditory features and improving the robustness of hearing aids in dynamic environments.

[0053] 4. In model optimization and decision-making applications, an adaptive composite contrastive loss function is used to guide model optimization. By enhancing intra-class compactness and inter-class separability, the classification accuracy is significantly improved; the final output audio scene classification results can accurately represent the current acoustic environment state, providing a reliable decision-making reference for the main control unit of the hearing aid device to perform intelligent environmental perception and operational status adjustment.

[0054] This invention achieves a superior balance between lightweight model and high performance. While ensuring classification accuracy, it effectively reduces model size and computational overhead, making it well-suited for hearing aids with limited power consumption and computing power, and thus has strong practical application value. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process in an embodiment of the present invention.

[0056] Figure 2 This is an example of a Mel spectrogram for a single audio track in an indoor scene.

[0057] Figure 3 This is an example of a constant Q transform spectrogram of a single audio track in an indoor scene.

[0058] Figure 4 This is an example of a first-order difference spectrogram of a single audio track in an indoor scene.

[0059] Figure 5 This is an example of a Mel spectrogram for a single audio track in an outdoor scene.

[0060] Figure 6 This is an example of a constant Q transform spectrogram of a single audio track in an outdoor scene.

[0061] Figure 7 This is an example of a first-order difference spectrogram of a single audio track in an outdoor scene.

[0062] Figure 8 This is an example of a Mel spectrogram of a single audio signal in a traffic scenario.

[0063] Figure 9 This is an example of a constant Q transform spectrum of a single audio signal in a traffic scenario.

[0064] Figure 10 This is an example of a first-order difference spectrogram of a single audio signal in a traffic scenario.

[0065] Figure 11 This is a schematic diagram of the multi-channel heterogeneous feature extraction and stacking process for hearing aids in an embodiment of the present invention.

[0066] Figure 12 This is a schematic diagram of the asymmetric multi-stream input module in an embodiment of the present invention.

[0067] Figure 13 This is a schematic diagram of the ghost bottleneck module structure integrating coordinate attention in an embodiment of the present invention.

[0068] Figure 14 This is a schematic diagram of the multi-dimensional feature enhancement and lightweight channel rebalancing fusion process in an embodiment of the present invention.

[0069] Figure 15 This is a receiver operating characteristic (ROC) curve on the test set in an embodiment of the present invention.

[0070] Figure 16 This is a visualization of the t-SNE distribution on the test set in an embodiment of the present invention.

[0071] Figure 17 This is a graph showing the change of the loss function (Loss) during the model training process in an embodiment of the present invention.

[0072] Figure 18 This is a graph showing the change in classification accuracy during model training in an embodiment of the present invention.

[0073] Figure 19 This is a schematic diagram of the classification confusion matrix on the test set in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, design concepts, and technical solutions of the embodiments of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings.

[0075] Reference Figure 1 This invention proposes a lightweight multispectral acoustic scene classification method, device, and storage medium for hearing aids. The method includes the following steps:

[0076] Step S1: Constructing a multi-channel heterogeneous auditory feature input for hearing assistive devices

[0077] Reference Figure 11 This step first acquires the ambient audio signal of the hearing aid front-end to be processed. This signal corresponds to the real-time acoustic environment picked up by a microphone array during actual operation of hearing-assistive devices (such as cochlear implants and digital hearing aids). In a preferred embodiment of the invention, to construct complementary input information, three core heterogeneous feature spectra are extracted in parallel from this signal:

[0078] 1. Mel spectrum characteristic map By simulating the auditory characteristics of the human ear, it effectively captures the timbre and energy distribution of sound.

[0079] 2. Constant Q-transform spectrum characteristics It exhibits logarithmic distribution characteristics on the frequency axis, has higher frequency resolution in the low-frequency region, and excels at capturing pitch-related harmonic structures.

[0080] 3. First-order difference spectrum characteristic diagram It is obtained by calculating the first-order difference of the Mel spectrum on the time axis and is used to capture the dynamic information of the sound signal as it changes over time, that is, the transient characteristics of the sound.

[0081] like Figures 2 to 10 The figures show examples of three heterogeneous feature spectra of a single audio signal under three core scenarios: "indoor," "outdoor," and "traffic." The figures clearly demonstrate that the three features exhibit different textures and energy distributions in different scenarios, proving that each carries distinctive complementary information.

[0082] To integrate this information, the three extracted single-channel feature spectra are stacked along the channel dimension to form a multi-channel spectral feature map containing complementary time-frequency information, i.e., the input tensor. The process is as follows: Figure 11 As shown, its mathematical expression is as follows:

[0083] in, For input tensors; Mel spectrum feature map; This is a spectral characteristic diagram of the constant Q-transform; This is a first-order difference spectrum feature map; The operation involves splicing along the channel dimension. This step preserves the multi-dimensional information in the original audio to the greatest extent possible, laying a solid foundation for subsequent processing.

[0084] Step S101: Robust preprocessing of audio signals

[0085] In a preferred embodiment, to adapt to the front-end signal processing standards of hearing aids and improve the robustness of environmental perception, the following preprocessing operations are performed before feature extraction:

[0086] Front-end signal conditioning involves pre-emphasis on the acquired raw audio signal to enhance high-frequency information; DC component removal to eliminate zero-point drift in the signal; amplitude normalization to unify the volume levels of different audio files; and fade-in / fade-out processing to reduce abrupt changes at the edges of audio segments. This step aims to eliminate non-stationary interference in the input signal, making it compatible with the signal input specifications of the low-power DSP chip in hearing aids, thus providing a more stable input for subsequent feature extraction.

[0087] Generalization Training Strategy and Functional Category Mapping: During the dataset construction phase, a cross-validation method based on recording location grouping is used to divide the training and validation sets, ensuring that audio samples from the same geographical location do not appear simultaneously in the training and validation processes. This strategy aims to force the model to learn the essential features of the acoustic environment rather than background noise from a specific location, thereby improving the generalization adaptability of hearing assistive devices in unknown and unfamiliar environments. Simultaneously, the subdivided acoustic scenes (e.g., 15 categories) in the original dataset are mapped and summarized into three core functional categories: "indoor, outdoor, and traffic," thus constructing a classification label system that conforms to the actual operating logic of hearing assistive devices.

[0088] Time-frequency alignment and gain normalization of heterogeneous features: The time frame number of multiple spectra is unified using methods such as linear interpolation, and the frequency dimension of the CQT spectra is aligned to ensure spatial alignment of all spectra features. Simultaneously, a strict data leakage-free normalization strategy is implemented, which normalizes real-time input audio frames using pre-calculated global mean and standard deviation from the training set. This operation aims to eliminate potential volume gain differences under different acquisition conditions (such as different microphone models and wearing positions), ensuring the realism of the model's generalization performance.

[0089] Step S102: Setting key parameters for audio feature extraction

[0090] In a preferred embodiment of the present invention, to ensure that the hearing aid device can obtain high-fidelity environmental acoustic information and provide standardized input for subsequent feature extraction, the key parameters for audio feature extraction and preprocessing are set as follows:

[0091] 1. Basic signal parameter configuration

[0092] Sampling Rate: 44100 Hz. This high sampling rate ensures high-fidelity audio signals and provides rich, original acoustic information for hearing aids, especially crucial for distinguishing subtle ambient sounds and noise textures.

[0093] Audio Duration: 10.0 seconds, to provide sufficient temporal context for the model to analyze long-term environmental features.

[0094] 2. Short-Time Fourier Transform (STFT) Parameter Configuration

[0095] FFT window size and length: both set to 2048.

[0096] Frame shift: Set to 512. A smaller frame shift improves temporal resolution and helps capture transient noise.

[0097] 3. Mel spectrogram (Mel) parameter configuration

[0098] Number of Mel filters: Set to 128. This parameter is used to simulate the frequency perception characteristics of the human cochlea to obtain frequency domain characteristics that conform to auditory perception.

[0099] Effective frequency range: Covering 20 Hz to 22050 Hz, fully covering the human audible range.

[0100] 4. Constant Q Transform (CQT) Parameter Configuration

[0101] Number of frequency ranges: set to 84, covering 7 octaves.

[0102] Frequency resolution: Each octave contains 12 semitones (i.e., standard musical resolution).

[0103] Minimum frequency: set to 32.70 Hz, corresponding to the C1 note.

[0104] 5. Scene Category Mapping Strategy

[0105] The final classification target categories are explicitly mapped to three core categories: Indoor, Outdoor, and Transportation. This mapping strategy directly corresponds to the most common basic signal processing procedures in hearing-aiding devices, making it easy for the classification results to be directly read by the device's main control unit and used as a basis for decision-making.

[0106] Step S2: Perform asymmetric multi-flow shallow feature extraction

[0107] In step S1, a multi-channel spectral feature map containing complementary information (i.e., the input feature tensor) is obtained. After that, preliminary feature extraction and fusion processing are performed to generate a shallow fused feature with higher information density, which is more suitable for subsequent backbone network processing. . Reference Figure 12 This step is implemented through an asymmetric multi-stream input module, and the specific processing includes the following sub-steps:

[0108] Step S201: Parallel asymmetric extraction of heterogeneous features

[0109] Using three-channel input (Mel spectrum feature map) Constant Q-transform spectrum characteristics First-order difference spectrum feature map The module addresses the inherent information differences between these three heterogeneous spectral channels by employing three independent convolutional streams with asymmetric computational resources. Perform parallel processing, that is , and The depth and width of each convolutional stream are customized based on the information density of the feature type it processes.

[0110] For Mel spectrum feature map Because it captures rich details of timbre, tone quality, and energy distribution in sound signals, it has the highest information density. Therefore, it is assigned a relatively deeper network stream. In this embodiment, the network stream consists of four convolutional layers, denoted as Conv(4). This design aims to extract deeper and more complex abstract texture features.

[0111] For CQT spectral feature map Sum of differential spectral features : It focuses on logarithmic frequency information related to musical pitch, while It captures the dynamic information of sound signals as they change over time. The structures of these two features are relatively... More explicit and concise. Therefore, they are assigned shallower network flows. and In this embodiment, both network streams are specifically composed of two convolutional layers, denoted as Conv(2). This asymmetric design ensures effective extraction of core information from each channel while greatly avoiding computational redundancy on low-power DSP chips, significantly improving the overall operating efficiency of the end-side model of the hearing aid device.

[0112] Step S202: Splicing and fusing multi-stream features

[0113] In step S201, after the three independent convolutional streams complete parallel processing of their respective input features, their outputs are efficiently concatenated and fused to generate shallow fused features. The fusion process can be represented by the following formula:

[0114]

[0115] in: , , These represent the output feature maps of the three parallel processing streams in step S201. The operation concatenates the output feature maps of the three branches along the channel dimension. This operation spatially aligns feature information from different sources and juxtaposes them in the same high-dimensional feature space, providing a foundation for subsequent cross-channel information interaction. The operation is implemented through a 1x1 convolutional layer (i.e., a channel adjustment layer), which is crucial to this step. It is essential for... The concatenated high-dimensional features undergo linear cross-channel information fusion and dimensionality reduction. The 1x1 convolution acts as an information aggregator here, performing linear cross-channel information fusion and dimensionality reduction on the concatenated high-dimensional features. It learns how to weight and combine feature channels from different spectra, thereby suppressing redundant information, enhancing complementary information, and ultimately generating a unified and refined shallow fusion feature. This is the final shallow fusion feature generated.

[0116] Through the aforementioned asymmetric parallel processing and subsequent fusion steps, this invention allocates the most suitable computational resources to different types of acoustic features, achieving refined processing of heterogeneous information. This method not only preserves the unique advantages of each spectrogram but also solves the information dilution or redundancy problems that may arise from traditional simple splicing methods through a dedicated fusion stage. It provides a high-quality, high-information-density input for the lightweight backbone network running on hearing-aiding devices, representing a crucial step in achieving high-precision, low-latency acoustic scene classification.

[0117] Step S3: Extract deep features using a lightweight network

[0118] Reference Figure 13 The shallow fusion features obtained in step S2 The data is fed into a lightweight backbone network designed specifically for low-power computing platforms for deep feature extraction. This network preferably adopts an architecture with a ghost bottleneck module at its core and integrates a coordinate attention module to improve feature extraction capabilities while ensuring extremely low model parameter count and computational power consumption, effectively solving the problem that existing complex models are difficult to deploy on edge chips of hearing assistive devices.

[0119] Step S301: Ghost Bottleneck Module

[0120] The core principle of this module is to divide the original standard convolution into two steps, generating the same number of feature maps with lower computational cost. The specific implementation is as follows:

[0121] The first step is to apply a small number of standard convolutional kernels (e.g., 1x1 convolutions) to the input features to generate a subset of "endogenous feature maps". This subset of feature maps contains the main features of the input information.

[0122] The second step involves performing a series of computationally inexpensive linear transformations (such as depthwise separable convolution) on these endogenous feature maps to generate additional "ghost feature maps." These ghost feature maps can be seen as an effective utilization and supplement to the information redundancy of the endogenous feature maps.

[0123] Finally, the endogenous feature map generated in the first step and the ghost feature map generated in the second step are concatenated along the channel dimension to form the final output feature map. Its output It can be represented as:

[0124]

[0125] in, For endogenous feature maps, For linear transformation, Ghost feature maps generated for linear transformations This results in the final output feature map. This feature reuse mechanism significantly reduces the number of model parameters and computational cost while maintaining receptive field and feature diversity, thereby lowering the power consumption of hearing aids. Furthermore, this module typically includes residual connections to add input and output features, mitigating the vanishing gradient problem in deep networks and improving training stability.

[0126] Step S302: Coordinate Attention Module

[0127] To further enhance the expressive power of feature maps, a coordinate attention module is integrated within the ghost bottleneck module of the backbone network. Traditional channel attention mechanisms typically compress global spatial information into a single channel descriptor through global pooling, ignoring positional information. The coordinate attention module, however, embeds positional information into channel attention, enabling the network to capture long-range dependencies in feature maps, thus more accurately locating key regions of acoustic events in the time-frequency spectrum. Its core computational process can be represented as follows:

[0128]

[0129] in, Input features; The output features are those enhanced by coordinate attention. For activation functions; and For perceptual feature vectors; and To process the convolution corresponding to the vector; This is an element-wise multiplication operation. The process specifically includes:

[0130] Coordinate information embedding: First, to avoid the loss of position information caused by two-dimensional global pooling, this module embeds the input features... One-dimensional global average pooling is performed along the height and width directions respectively to obtain two independent orientation-aware feature vectors containing precise position information. and These two vectors aggregate feature information in the vertical and horizontal spatial directions, respectively.

[0131] Attention generation: and After concatenation, the data is fed into a shared 1x1 convolution for transformation, followed by a non-linear activation function and batch normalization. Next, the transformed feature vector is split into two independent parts, each then passed through two separate 1x1 convolutions. and Process and utilize the Sigmoid activation function. Generate attention maps in the height and width directions.

[0132] Feature recalibration: Finally, the generated attention maps in two directions are compared with the original input features. Perform element-wise multiplication ( ), to obtain the final enhanced output features .

[0133] This process enables the model to calibrate features simultaneously from both spatial and channel dimensions, improving the model's feature extraction capabilities while maintaining a lightweight design, and ensuring that hearing aids can accurately capture the features of target sound sources in complex noise environments.

[0134] Step S303: Lightweight Backbone Network Construction

[0135] In a specific embodiment of the present invention, the lightweight backbone network directly feeds the shallow fusion features output from step S2 into a multi-stage processing architecture. This architecture comprises four sequentially stacked processing stages, presenting a hierarchical structure. It aims to progressively extract higher-order semantic features while continuously reducing the data size through downsampling operations to accommodate the limited memory and computing bandwidth of hearing-aiding devices. The detailed configuration of each stage is as follows:

[0136] Phase 1: Contains one ghost bottleneck module. This module has 6 input channels, 9 output channels, and a stride of 2, halving the spatial resolution of the feature map. This step aims to quickly compress the spatial dimension of the input features, reducing the computational overhead of subsequent layers.

[0137] Phase 2: Includes one parallel attention module. This module has 9 input channels and 12 output channels, and is mainly responsible for cross-channel information interaction and fusion without changing the spatial resolution of the feature map. This design helps to enhance the model's ability to perceive the correlation between features from different acoustic frequency bands while maintaining feature resolution.

[0138] Phase 3: Includes one ghost bottleneck module. This module has 12 input channels and 12 output channels, with a stride of 2, further halving the feature map spatial resolution.

[0139] Phase 4: Includes one ghost bottleneck module. This module has 12 input channels and 12 output channels, with a stride of 2, halving the feature map spatial resolution again.

[0140] After the four stages of processing described above, the spatial resolution of the feature map is reduced to 1 / 8 of the input resolution. This hierarchical structural design, by setting different module types, output channel numbers, and strides at different stages, allows the network to progressively reduce the spatial resolution of the feature map and extract hierarchical features ranging from low-level texture and edge information to high-level, more abstract semantic information. This not only provides rich and efficient feature representations for subsequent classification tasks but also keeps the computational load to an extremely low level, ensuring the algorithm's real-time performance on hearing-assisted devices.

[0141] Step S4: Implement multi-dimensional acoustic feature enhancement

[0142] Reference Figure 14 After extracting high-dimensional depth features from the backbone network, this invention sequentially applies multiple subsequent processing modules to enhance the depth features in multiple dimensions, thereby further improving the ability of hearing aids to capture subtle acoustic details.

[0143] Step S401: Spectromorphology Module

[0144] First, the deep features are fed into a spectral morphology module, which utilizes parameter-learnable morphological operations—erosion and dilation—based on depthwise separable convolutions to extract edge contours and fine-grained structural features from the spectra. This design is particularly well-suited to the application scenarios of hearing-aiding devices because different acoustic environments (such as indoor, outdoor, and traffic) exhibit distinctly different texture structures in the spectra. For example, traffic scenes typically have pronounced low-frequency harmonic textures, while indoor scenes present a relatively smooth background noise. Morphological operations can effectively enhance these discriminative structural features, thereby assisting the model in making more accurate discriminations.

[0145] The specific processing flow of this module is as follows: First, calculate the morphological gradient. This captures edge information and texture details from the spectrogram; then, learnable weight parameters are used ( , The contribution of these morphological features is adaptively balanced and fused with the original features. The computational process can be expressed as follows:

[0146]

[0147] in, The input is the deep feature. and These represent the expansion and corrosion operations performed on it, respectively. This is a morphological gradient used to extract edge features. , For learnable weight parameters optimized through backpropagation, This is the output feature resulting from the fusion of original and morphological features. The dilation operation emphasizes high-energy regions in the spectrum, while the erosion operation preserves stable acoustic features. Combining the two effectively enhances the structural information of the spectrum.

[0148] This design not only preserves the original features but also significantly enhances the ability to perceive structural information in the spectrum, enabling the model to adaptively adjust the importance of various features according to the characteristics of different acoustic scenarios, thereby greatly improving the ability of hearing aids to distinguish complex acoustic environments such as indoor, outdoor, and traffic environments.

[0149] Step S402: Bandwidth Attention Module

[0150] Subsequently, the features are input into the frequency band attention module, which divides the feature map into multiple frequency bands along the frequency dimension (corresponding to the channel dimension of the feature tensor) and learns independent attention weights for each frequency band, enabling the model to focus on the feature channel group that is most important for classifying the current scene.

[0151]

[0152] in, The input features are the deep features enhanced by S401. The total number of frequency band components obtained by segmenting the input features along the frequency dimension; for The i-th frequency band component obtained after segmentation along the frequency dimension; This is a global average pooling operation; For activation functions; , It consists of two fully connected layers used to transform the feature descriptors generated by global average pooling; For activation functions; The attention weights are for the i-th frequency band component. These are the weighted frequency band components; For splicing operations; The enhanced output features of the frequency band attention module integrate weighted information from each frequency band.

[0153] Input features First, it is uniformly divided into frequency dimensions. Each frequency band component For each frequency band component First, its spatial information is aggregated through a global average pooling operation to generate a compact feature descriptor. This descriptor is then passed through a layer consisting of two fully connected layers and The network, composed of activation functions, processes and ultimately passes through... The function calculates the attention weights for that frequency band. The weight values ​​are then normalized to the (0,1) interval. Finally, each frequency band component is multiplied by its corresponding attention weight to achieve dynamic importance weighting for different channel groups. All weighted frequency band components are then processed. The features are reassembled along the frequency dimension to form the enhanced output features. .

[0154] This design enables the model to adaptively identify and emphasize the channel groups that are most discriminative for classifying the current acoustic scene, while suppressing irrelevant or redundant channel information (such as interference noise in non-target frequency bands), thereby significantly improving the ability of hearing aids to recognize different complex acoustic scenes.

[0155] Step S5: Construct the channel rebalancing fusion module

[0156] Reference Figure 14 The enhanced features are then fed into a lightweight channel rebalancing fusion module. The core innovation of this module lies in its ability to dynamically adjust the weights of each channel based on real-time changes in the acoustic environment (e.g., switching from a quiet indoor environment to a noisy traffic environment). By suppressing channels carrying redundant information and enhancing channels containing key complementary information, this module achieves adaptive feature optimization. Its calculation process can be represented as follows:

[0157]

[0158] in, Input features; As an importance evaluator, it is a lightweight neural network composed of fully connected layers, used to evaluate the importance of each channel in the current frame in real time; This indicates a global average pooling operation used to extract channel statistics. for Activation function; The channel importance weight represents the degree of contribution of each channel to the classification. These are learnable bias parameters used to adjust the weight distribution after inversion; To pass Normalized rebalancing weights; These are characteristics after rebalancing; This is an element-wise multiplication operation; The final output is the fused features.

[0159] The module's workflow is as follows: First, channel importance weights are calculated using global average pooling and an importance evaluator. Then invert these weights and add a bias. Post-application Generate rebalancing weights This mechanism allows channels that might be ignored in traditional fusion but are crucial for specific scenarios (such as weak speech signals) to receive higher weights. The rebalanced weights are then multiplied by the original features to obtain... Finally, the residual join is used to... and Add to get the output .

[0160] This design retains key feature information while enhancing the expression of complementary features, enabling hearing aids to adaptively adjust feature importance according to different acoustic scenarios. This significantly improves the ability to recognize subtle acoustic differences (such as similar background noise), laying a solid foundation for subsequent accurate classification.

[0161] Step S6: Model Optimization and Decision Output

[0162] Step S601: Construction of the adaptive composite contrastive loss function

[0163] During the model training and fitting process, in order to guide the model to learn high-quality feature representations that are compact within classes and dispersed between classes, thereby meeting the needs of hearing assistive devices for fine differentiation of complex acoustic scenes (such as similar background noise), this invention selects a composite loss function. The model's optimization objective is used as the loss function, which consists of two weighted parts: one part is the standard classification loss. (i.e., cross-entropy loss with label smoothing), the other part is contrastive learning loss. The specific form of the composite loss function used in this invention is shown in the formula:

[0164]

[0165] in, The overall objective loss function for model optimization; This is the cross-entropy loss function with label smoothing. By converting hard labels (one-hot) to soft labels, the model's generalization ability is enhanced, effectively preventing overfitting. These are dynamically adjusted weight hyperparameters used to control the contribution of the contrast loss. To compare the learning loss, it is used to optimize the distribution of the feature space; To compare the initial weight values ​​of the loss; For training rounds; It is a minimum value function; The average similarity of all positive sample pairs; The average similarity of all negative sample pairs; For temperature hyperparameters, adjust the smoothness of the similarity distribution; This is a numerical stability constant, ensuring computational stability.

[0166] The specific implementation process of this composite loss function is as follows: First, for a feature map within a training batch, it is converted into a one-dimensional feature vector through global average pooling and then normalized using the L2 norm. Next, the cosine similarity between all pairs of samples within the batch is calculated, forming a similarity matrix. Based on the true class label of the samples, the average similarity of all positive sample pairs is calculated from this matrix. and the average similarity of all negative sample pairs .

[0167] Step S602: Adaptive Optimization Strategy and Model Update

[0168] Subsequently, the contrastive learning loss was calculated according to the formula. This loss function drives the model to learn the most discriminative acoustic features by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.

[0169] Meanwhile, adaptive weights The design is a key aspect of this invention. In the early stages of training (when the number of epochs is small)... A large value of makes the contrastive learning loss dominant, forcing the model to quickly learn and establish a feature space with large inter-class distances and small intra-class distances. This effect continues as training progresses (epochs increase). The value will decrease linearly and eventually stabilize at the initial weight. At 50%, the optimization objective will focus more on the standard classification loss. This allows for more refined classification boundary learning on the already well-constructed feature space.

[0170] Finally, the classification loss is added to the weighted contrastive learning loss to obtain the total loss. The model parameters are then updated through backpropagation. This composite loss design allows the model to have different focuses at different stages of training, ultimately learning more robust and discriminative feature representations, laying the algorithmic foundation for high-precision environmental recognition of hearing assistive devices in practical applications.

[0171] The acoustic scene classification method based on lightweight neural networks for hearing aids proposed in this invention has been fully validated through a series of experiments. Figures 15 to 19 show the comprehensive performance evaluation results of the proposed method.

[0172] like Figure 15 As shown, the performance of the model was analyzed in depth from the perspective of classification and discrimination ability. Figure 15This displays the Receiver Operating Characteristic (ROC) curve of the model on the 2017 test set. The ROC curve is a graph that measures the performance of a classification model across all classification thresholds, plotting the relationship between the True Positive Rate (TPR) and the False Positive Rate (FPR). The Area Under the Curve (AUC) is a numerical metric that measures the model's ability to distinguish between positive and negative samples; the closer the AUC value is to 1, the better the model's discriminative performance. Figure 15 As can be seen, the ROC curves of the model in this invention for the three categories of "indoor," "outdoor," and "transportation" are all very close to the upper left corner, with corresponding AUC values ​​reaching excellent levels of 0.982, 0.997, and 0.982, respectively. This indicates that the model has a strong ability to distinguish between different acoustic scenarios and can meet the needs of hearing aids to accurately switch hearing modes in complex environments.

[0173] Figure 16 This displays a visualization of the deep features extracted by the model in two-dimensional space using t-SNE (t-Distributed Stochastic Neighbor Embedding). t-SNE is a non-linear dimensionality reduction algorithm, particularly adept at mapping high-dimensional data to low-dimensional space for visualization, and effectively preserving the local structure of the data. As shown in the figure, the sample points in the test set form three well-defined and compact clusters based on their categories (indoor - circular markers, outdoor - square markers, traffic - triangular markers). In particular, the "outdoor" cluster is completely separated from the other two categories, while there is very little overlap between the "indoor" and "traffic" categories, which is consistent with... Figure 19 The results observed in the confusion matrix (i.e., a small number of traffic scenes were misclassified as indoor scenes) are consistent with the findings. The visualization results of t-SNE strongly demonstrate that the model proposed in this invention can learn deep feature representations with high separability, which is the fundamental reason why the model can achieve high-precision classification and provide reliable decision-making basis for hearing-aiding devices.

[0174] like Figure 17 and Figure 18 As shown, this demonstrates the dynamic changes of the model during the final training phase and the classification results on the test set. Figure 17 and Figure 18The graphs show the changes in the loss function and accuracy during model training. As can be seen, the accuracy curves on both the training and validation sets rise rapidly and reach a relatively stable level (close to 97%), while the loss function curve declines smoothly and converges to a lower value. There is no significant difference between the two curves (training and validation), indicating that the model learning process is stable, has good convergence, and does not exhibit obvious overfitting. This demonstrates the effectiveness and robustness of the model design under limited data conditions in hearing aids.

[0175] Figure 19 The confusion matrix of the model on the test set is shown. The confusion matrix is ​​used to evaluate the accuracy of the classification model; the values ​​on the diagonal represent the number of correctly classified samples, while the values ​​off-diagonal represent the number of misclassified samples. As can be seen from the figure, the model performs excellently across all categories, with the values ​​on the diagonal significantly higher than those off-diagonal. Specifically, it correctly classified 514 cases for "indoor," 520 cases for "outdoor," and 475 cases for "transportation."

[0176] The main confusion occurred when the "transportation" category was misclassified as "indoor" (50 cases). This may be because some enclosed transportation scenarios (such as stationary subway platforms) and indoor scenarios share certain similarities in acoustic features (especially low-frequency background noise). Despite the small number of misclassifications, the overall confusion matrix intuitively demonstrates that the model has a high classification accuracy, which can meet the accuracy requirements of hearing-aiding devices for environmental perception in most daily life scenarios.

[0177] In summary, through a comprehensive analysis of the training process, confusion matrix, ROC curve, and t-SNE feature distribution, it can be confirmed that the method proposed in this invention has superior performance in acoustic scene classification tasks. The model has good convergence, high accuracy, and powerful feature extraction and discrimination capabilities, providing a solid algorithmic foundation for the intelligent parameter adjustment of hearing aids.

[0178] It should be noted that the present invention is not limited to the specific embodiments and figures described above. After reading the specification, those skilled in the art can make various modifications, equivalent substitutions, or combinations without departing from the core ideas and technical solutions of the present invention, all of which should fall within the protection scope of the present invention.

[0179] The embodiments shown in this specification and accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of protection of this invention.

Claims

1. A multi-spectrogram lightweight acoustic scene classification method for hearing-aid oriented devices, characterized in that, Includes the following steps: S1: Acquire the ambient audio signal of the hearing aid front-end to be processed; S2: Input the ambient audio signal from the hearing aid front end to the feature extraction unit and output deep features; The feature extraction unit includes: The heterogeneous acoustic feature extraction and stacking module is used to convert the ambient audio signal of the hearing aid front end into a multi-channel spectral feature map; The asymmetric multi-stream input module connected to the acoustic feature extraction and stacking module is used to receive the multi-channel spectral feature map sent by the heterogeneous acoustic feature extraction and stacking module and perform feature differentiation processing. A ghost bottleneck module connected to the asymmetric multi-stream input module, the ghost bottleneck module outputs the depth feature; The asymmetric multi-stream input module includes three independent convolutional streams: a convolutional stream with four convolutional layers for processing Mel spectra, a convolutional stream with two convolutional layers for processing constant Q transform spectra, and a convolutional stream with two convolutional layers for processing first-order difference Mel spectra. The aforementioned ghost bottleneck module integrates a coordinate attention module; S3: Implement multi-dimensional acoustic feature enhancement by sequentially passing the deep features extracted in step S2 through the spectral morphology module and the frequency band attention module; The spectral morphology module utilizes parameter-learnable morphological operations based on depthwise separable convolution: erosion and dilation, to extract edge contours and fine-grained structural features of the spectrum. The frequency band attention module divides the feature map into multiple frequency bands along the frequency dimension and learns independent attention weights for each frequency band. S4: Input the enhanced deep features into the feature fusion module. The feature fusion module dynamically adjusts and fuses the enhanced deep features, and outputs the fused features. S5: Input the fused features into the classifier to obtain the final audio scene classification result. The audio scene classification result is configured as the basis for the environmental perception decision-making of the hearing aid device during signal processing.

2. The multi-painting lightweight acoustic scene classification method for hearing-aid-oriented devices according to claim 1, characterized in that, In step S1, the ambient audio signal of the hearing aid front-end to be processed is acquired, specifically including: During operation, hearing aids collect real-time acoustic environment data to obtain ambient audio signals from the hearing aid front-end environment that need to be processed.

3. The multi-painting lightweight acoustic scene classification method for hearing-aid-oriented devices according to claim 1, characterized in that, In step S2, each channel of the multi-channel spectral feature map is a time-frequency representation of a certain type, and the time-frequency representations of each channel are of different types.

4. The method of claim 1, wherein, In step S2, the ambient audio signal of the hearing aid front end is converted into a multi-channel spectral feature map, specifically including: The Mel spectrum, constant Q transform spectrum, and first-order difference Mel spectrum are extracted in parallel from the audio signal of the hearing aid front-end environment and stacked together to form a three-channel feature map, which is the multi-channel spectral feature map.

5. A sound scene classification device for implementing the multi-spectrogram lightweight acoustic scene classification method for hearing-aid oriented devices according to any one of claims 1 to 4, characterized in that, include: A sound sensor is used to acquire ambient audio signals from the hearing aid front end to be processed. The feature extraction unit is used to convert the ambient audio signal of the hearing aid front end into a multi-channel spectral feature map and perform feature differentiation processing, and then perform feature extraction to output deep features. The subsequent processing module is used to enhance the deep features in multiple dimensions; The feature fusion module is used to dynamically adjust and fuse the enhanced deep features, and output the fused features. A classifier is used to output the sound scene category based on the fused features.

6. A hearing assistive device, comprising a processor and a memory, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the multispectral lightweight acoustic scene classification method for hearing aids as described in any one of claims 1 to 4.

7. The hearing aid device according to claim 6, characterized in that, The hearing aid is a smart home device or a personal assistant device.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multispectral lightweight acoustic scene classification method for hearing aids as described in any one of claims 1 to 4.

Images