Audio processing integration method and device of hearing aid

Abstract

The invention provides an audio processing integration method and device of a hearing aid. The method comprises the following steps: acquiring an audio signal acquired by a microphone of the hearing aid in real time; processing the audio signal through a feature extraction network to obtain first audio feature data; performing frequency domain conversion, noise distribution prediction and weighted filtering processing on the first audio feature data based on a noise suppression network to obtain a first noise reduction audio feature; processing the first audio feature data by using an adaptive enhancement network, and obtaining a second enhanced audio feature by randomly selecting a frequency reference point and collecting and processing a feature point; fusing the first noise reduction audio feature and the second enhancement audio feature through a cross attention network to obtain a fused audio feature; and finally, processing the fused audio features by using an audio enhancement network to obtain an enhanced audio signal and outputting the enhanced audio signal. According to the invention, the voice definition and sound quality performance of the hearing aid in a complex environment can be improved, and the hearing experience of a hearing impaired user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing for hearing aids, and more particularly to an integrated method and device for audio processing for hearing aids. Background Art

[0002] In modern hearing aid technology, audio processing is a key link in improving the auditory experience of hearing-impaired users. Existing hearing aid audio processing technologies mainly rely on traditional noise reduction algorithms and simple audio enhancement methods. Although these methods can improve speech clarity to a certain extent, their effectiveness is often limited in complex noise environments. Traditional noise reduction technologies are usually based on fixed filters or simple spectrum processing, which are difficult to adapt to diverse noise scenarios and individual hearing needs. At the same time, audio enhancement methods mostly use a single amplification strategy, which cannot effectively distinguish between speech and noise, resulting in reduced speech quality and poor user hearing experience.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the existing audio processing technology cannot effectively cope with the needs of speech extraction and enhancement in complex noise environments, and it is difficult to make adaptive adjustments based on individual differences of users, resulting in a significant reduction in the effectiveness of hearing aids in actual use, and unable to meet the needs of hearing-impaired users for a high-quality auditory experience. Summary of the invention

[0004] The present invention provides an audio processing integration method and device for a hearing aid.

[0005] In a first aspect of the present invention, there is provided an audio processing integration method for a hearing aid, comprising: Acquire the audio signal collected by the microphone deployed in the hearing aid in real time; Processing the audio signal based on a feature extraction network to obtain first audio feature data; The first audio feature data is processed based on a noise suppression network to obtain a first noise reduction audio feature, including: presetting a noise intensity distribution threshold based on the installation position of the microphone; converting the first audio feature data to the frequency domain; based on the noise intensity distribution threshold, using the noise suppression network to predict the noise distribution corresponding to the converted first audio feature data; weighting, filtering and minimum pooling aggregation processing are performed on the converted first audio feature data and the noise distribution to obtain the first noise reduction audio feature; The first audio feature data is processed based on an adaptive enhancement network to obtain a second enhanced audio feature, including: presetting an audio spectrum space, and converting each frequency point in the audio spectrum space to a frequency domain; randomly selecting a plurality of frequency reference points above the converted audio spectrum space, and projecting them to a time domain to obtain projection points; collecting feature points around the projection points in the first audio feature data; processing the feature points through an adaptive enhancement network to determine a feature value corresponding to each projection point; and determining a second enhanced audio feature based on the feature value; Based on the cross attention network, the first noise reduction audio feature and the second enhanced audio feature are fused to obtain the fused audio feature, including: applying linear transformation to the first noise reduction audio feature and the second enhanced audio feature respectively to obtain corresponding queries, keys and values, wherein the first noise reduction audio feature corresponds to the first query, the first key and the first value, and the second enhanced audio feature corresponds to the second query, the second key and the second value; calculating the first attention matrix based on the first key and the second query; calculating the second attention matrix based on the first query and the second key; processing the first value and the second value based on the first attention matrix and the second attention matrix to obtain the fused audio feature; The fused audio features are processed using an audio enhancement network to obtain an enhanced audio signal, wherein the audio enhancement network sequentially includes a plurality of filtering layers, an inverse filtering layer, an upsampling layer, and a frame-by-frame classification layer.

[0006] Further, the audio signal is processed based on a feature extraction network to obtain first audio feature data, including: using a WaveNet neural network to process the audio signal to obtain a multi-scale audio feature, wherein the multi-scale audio feature includes a plurality of audio feature data with different resolutions; The multi-scale audio features are sequentially fused according to the resolution from low to high, including: performing 1×1 convolution and upsampling processing on the audio feature data of the first resolution, and fusing the processing result with the audio feature data of the second resolution through the 1×1 convolution result; The first resolution is audio feature data with a lower resolution among the multi-scale audio features; the second resolution is audio feature data with a higher resolution among the multi-scale audio features; A convolutional neural network is used to aggregate the multi-scale audio feature fusion results to obtain first audio feature data.

[0007] Furthermore, the converted first audio feature data and noise distribution are weighted, filtered and minimum pooled to obtain a first noise reduction audio feature, including: weighting the converted first audio feature data and the noise distribution to obtain an adjusted audio feature; filtering the adjusted audio feature through a filter; dividing the filtered audio feature into multiple frequency bands, aggregating audio features belonging to the same frequency band, and performing minimum pooling on the aggregation results to obtain the first noise reduction audio feature.

[0008] Furthermore, the feature points are processed through an adaptive enhancement network to determine the eigenvalue corresponding to each projection point, including: processing the feature points through an adaptive enhancement network to determine the corresponding weight value; performing weighted summation on the feature points corresponding to each projection point to obtain the corresponding eigenvalue, and determining the position of the eigenvalue in the audio spectrum space.

[0009] Further, performing audio enhancement network processing on the fused audio features to obtain an enhanced audio signal includes: performing multi-layer filtering processing on the fused audio features to obtain first data; Performing inverse filter layer processing on the first data to obtain second data; Upsampling the second data to obtain third data having the same resolution as the fused audio feature; The third data is classified frame by frame to obtain an enhanced audio signal.

[0010] Furthermore, it also includes: constructing an audio processing model, wherein the audio processing model includes a feature extraction network, a noise suppression network, an adaptive enhancement network, a cross attention network and an audio enhancement network; Collecting multiple audio data and annotating the collected audio data, including: starting from a starting point of a speech segment in the audio data, extending along the speech segment in the audio data until a complete speech annotation is formed; The annotated audio data is input into the audio processing model for training to adjust the variable parameters in the audio processing model.

[0011] Furthermore, the audio processing model is optimized using the mean square error loss function.

[0012] Further, the first value and the second value are processed based on the first attention matrix and the second attention matrix to obtain a fused audio feature, including: Fusion of audio features The calculation formula is:

[0013] in, represents the second query, Indicates the first key, represents the first value, represents the first query, represents the second key, represents the second value, Represents the normalization function.

[0014] Furthermore, the expression of the mean square error loss function is:

[0015] in, represents the loss value, Represents the number of samples in the audio data, Representation sample The true value of Representation sample The predicted value of .

[0016] In a second aspect of the present invention, there is provided an audio processing integrated device for a hearing aid, comprising: An acquisition module acquires in real time the audio signal collected by the microphone deployed in the hearing aid; A feature extraction module processes the audio signal acquired by the acquisition module based on a feature extraction network to obtain first audio feature data; A first processing module processes the first audio feature data based on a noise suppression network to obtain a first noise reduction audio feature; A second processing module processes the first audio feature data based on an adaptive enhancement network to obtain a second enhanced audio feature; A feature fusion module, based on a cross attention network, fuses the first noise reduction audio feature output by the first processing module and the second enhanced audio feature output by the second processing module to obtain a fused audio feature; A third processing module processes the fused audio features output by the feature fusion module using an audio enhancement network to obtain an enhanced audio signal, wherein the audio enhancement network sequentially includes a plurality of filtering layers, an inverse filtering layer, an upsampling layer, and a frame-by-frame classification layer; The output module outputs the enhanced audio signal to the speaker of the hearing aid.

[0017] The above-mentioned embodiments of the present invention have at least the following beneficial effects: the hearing aid audio processing integrated method and device of the present invention can effectively reduce the interference of environmental noise on speech signals through multi-level network processing, while enhancing the clarity and intelligibility of speech signals. The feature extraction network can obtain multi-scale audio features, providing a rich information basis for subsequent processing; the noise suppression network accurately predicts and suppresses noise based on a preset noise intensity distribution threshold to ensure the purity of the speech signal; the adaptive enhancement network further improves the detail performance of the speech signal by randomly selecting frequency reference points and feature points for processing, thereby providing users with a clearer and more natural auditory experience in a complex noise environment.

[0018] In addition, the cross-attention network fuses the denoised and enhanced audio features to achieve more accurate feature extraction and fusion, while the audio enhancement network further optimizes the quality of the audio signal through multi-layer filtering, inverse filtering, upsampling, and frame-by-frame classification. This integrated approach can significantly improve the adaptability and stability of hearing aids in different scenarios, meet the needs of hearing-impaired users for high-quality audio processing, and provide new technical ideas and solutions for the development of hearing aid technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which: Figure 1 A schematic flow chart of an audio processing integration method for a hearing aid provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an audio processing integrated device for a hearing aid provided by an embodiment of the present invention; Figure 3 The schematic diagram schematically shows the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0021] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0022] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0023] Reference below Figure 1 , Figure 1 FIG. 1 is a flow chart of an audio processing integration method for a hearing aid provided by an embodiment of the present invention. Figure 1 As shown, a method 100 for integrating audio processing of a hearing aid includes: Step 101, acquiring in real time an audio signal collected by a microphone deployed in a hearing aid; Step 102, processing the audio signal based on a feature extraction network to obtain first audio feature data; Step 103, processing the first audio feature data based on the noise suppression network to obtain a first noise reduction audio feature, including: presetting a noise intensity distribution threshold based on the installation position of the microphone; converting the first audio feature data to the frequency domain; predicting the noise distribution corresponding to the converted first audio feature data using the noise suppression network based on the noise intensity distribution threshold; weighting, filtering and minimum pooling aggregation processing the converted first audio feature data and the noise distribution to obtain the first noise reduction audio feature; Step 104, processing the first audio feature data based on the adaptive enhancement network to obtain the second enhanced audio feature, including: presetting the audio spectrum space, and converting each frequency point in the audio spectrum space to the frequency domain; randomly selecting multiple frequency reference points above the converted audio spectrum space, and projecting them to the time domain to obtain projection points; collecting feature points around the projection points in the first audio feature data; processing the feature points through the adaptive enhancement network to determine the feature value corresponding to each projection point; and determining the second enhanced audio feature based on the feature value; Step 105, based on the cross attention network, fuse the first noise reduction audio feature and the second enhanced audio feature to obtain the fused audio feature, including: applying linear transformation to the first noise reduction audio feature and the second enhanced audio feature respectively to obtain corresponding queries, keys and values, wherein the first noise reduction audio feature corresponds to the first query, the first key and the first value, and the second enhanced audio feature corresponds to the second query, the second key and the second value; calculating the first attention matrix based on the first key and the second query; calculating the second attention matrix based on the first query and the second key; processing the first value and the second value based on the first attention matrix and the second attention matrix to obtain the fused audio feature; Step 106, using an audio enhancement network to process the fused audio features to obtain an enhanced audio signal, the audio enhancement network sequentially comprising a plurality of filtering layers, an inverse filtering layer, an upsampling layer and a frame-by-frame classification layer.

[0024] It should be noted that the present invention relates to an integrated audio processing method for a hearing aid, the core of which is to process the audio signal collected by the hearing aid microphone in real time through a series of network processing steps. First, the audio signal collected by the microphone deployed in the hearing aid is obtained in real time. The audio signal here refers to all sound information captured by the hearing aid microphone during use, including speech and environmental noise. Then, the audio signal is processed based on the feature extraction network to obtain the first audio feature data. The feature extraction network is a neural network structure based on deep learning, and its purpose is to extract useful feature information from the original audio signal for subsequent processing. After that, the first audio feature data is processed based on the noise suppression network to obtain the first noise reduction audio feature. The noise suppression network achieves effective noise suppression by presetting the noise intensity distribution threshold, combining frequency domain conversion and noise distribution prediction. At the same time, the adaptive enhancement network processes the first audio feature data to obtain the second enhanced audio feature. This process randomly selects frequency reference points and projects them into the time domain, and performs weighted processing on the feature points to enhance the details of the audio signal. Finally, the first noise reduction audio feature and the second enhanced audio feature are fused through the cross attention network to obtain the fused audio feature, and the fused audio feature is processed by the audio enhancement network to obtain the enhanced audio signal. The audio enhancement network includes multiple filtering layers, inverse filtering layers, upsampling layers, and frame-by-frame classification layers to further optimize the quality of the audio signal.

[0025] Specifically, the feature extraction network can adopt the WaveNet neural network structure, which can process audio signals and generate multi-scale audio features, which include audio feature data of different resolutions. The fusion process of multi-scale audio features is to perform 1×1 convolution and upsampling on the low-resolution audio feature data, and then fuse it with the high-resolution audio feature data. This fusion method can retain feature information of different scales and provide a more comprehensive feature basis for subsequent noise suppression and audio enhancement. In the noise suppression network, the preset noise intensity distribution threshold is pre-set according to the installation position and usage scenario of the microphone to distinguish between noise and speech signals. Frequency domain conversion is to convert audio feature data from time domain to frequency domain so as to more effectively predict and process noise distribution. In the adaptive enhancement network, the audio spectrum space refers to the distribution range of the audio signal in the frequency domain, and the randomly selected frequency reference points are used to determine the key frequency areas in the audio signal that need to be enhanced. By collecting feature points around these frequency points and performing weighted processing, the detail performance of the audio signal can be enhanced. The cross-attention network dynamically adjusts the weights of the fused audio features by calculating the attention matrix, thereby achieving more accurate feature fusion. The filtering layer, inverse filtering layer, upsampling layer and frame-by-frame classification layer in the audio enhancement network are used to further optimize the spectral characteristics, resolution and stability of the audio signal, and finally output a clear enhanced audio signal.

[0026] Preferably, in the feature extraction network, the fusion of multi-scale audio features can be achieved through multiple iterations of 1×1 convolution and upsampling operations to ensure smooth transition and information complementarity between features of different resolutions. In the noise suppression network, the noise intensity distribution threshold can be dynamically adjusted according to the actual usage scenario, such as increasing the threshold in a noisy environment to more effectively suppress noise. In the adaptive enhancement network, the selection of frequency reference points can be optimized based on the spectral characteristics of the audio signal, such as giving priority to the energy concentration area of ​​the speech signal. In addition, the attention matrix calculation in the cross-attention network can introduce a normalization function to enhance the stability and consistency of the fused audio features. In the audio enhancement network, the frame-by-frame classification layer can be optimized according to the timing characteristics of the audio signal, such as using a long short-term memory network (LSTM) structure to better handle the dynamic changes of the audio signal.

[0027] In some embodiments, the audio signal is processed based on a feature extraction network to obtain first audio feature data, including: using a WaveNet neural network to process the audio signal to obtain a multi-scale audio feature, wherein the multi-scale audio feature includes a plurality of audio feature data with different resolutions; The multi-scale audio features are sequentially fused according to the resolution from low to high, including: performing 1×1 convolution and upsampling processing on the audio feature data of the first resolution, and fusing the processing result with the audio feature data of the second resolution through the 1×1 convolution result; The first resolution is audio feature data with a lower resolution among the multi-scale audio features; the second resolution is audio feature data with a higher resolution among the multi-scale audio features; A convolutional neural network is used to aggregate the multi-scale audio feature fusion results to obtain first audio feature data.

[0028] It should be noted that the feature extraction network mentioned in the present invention is one of the core components of the audio processing method, and its function is to convert the audio signal collected by the microphone into data with multi-scale features. The multi-scale audio features here refer to the feature representation of the audio signal at different resolutions, and these features can capture the details and structural information in the audio signal. By processing the audio signal using the Wave Net neural network, multi-scale audio features can be obtained, which include audio feature data of different resolutions. Wave Net is a neural network architecture based on deep learning that can generate high-quality audio features and is suitable for audio processing tasks. Subsequently, by fusing the multi-scale audio features in order from low to high resolution, a richer and more complete audio feature representation can be obtained, providing a basis for subsequent noise suppression and audio enhancement.

[0029] Specifically, the Wave Net neural network can process audio signals frame by frame through its unique structure to generate audio feature data with different resolutions. In the multi-scale audio feature fusion process, the low-resolution audio feature data is first subjected to a 1×1 convolution operation to reduce the dimension of the feature and retain key information. Subsequently, the low-resolution feature is resized to be consistent with the high-resolution feature through upsampling, and then fused with the high-resolution feature. This fusion method can effectively retain the detailed information of the audio signal at different scales. For example, the low-resolution feature can capture the global structure of the audio signal, while the high-resolution feature can retain local details. During the fusion process, the parameters of the 1×1 convolution operation can be adjusted according to the characteristics of the audio signal to optimize the effect of feature fusion. In addition, the fused audio feature data can be further aggregated through a convolutional neural network to obtain a more compact and effective feature representation, providing high-quality input for subsequent audio processing steps.

[0030] Preferably, the number of layers and parameters of the Wave Net neural network can be adjusted according to the complexity of the audio signal. For example, in a high-noise environment, the number of network layers can be increased to improve the accuracy of feature extraction. In the process of multi-scale audio feature fusion, an attention mechanism can be introduced to enable the network to automatically learn the weight relationship between features of different resolutions, thereby more effectively fusing features. In addition, in addition to 1×1 convolution and upsampling operations, more complex feature fusion strategies such as residual connections or dense connections can be adopted to enhance information interaction between features. In the aggregation stage of the convolutional neural network, a deeper network structure can be used or batch normalization technology can be introduced to improve the stability and efficiency of feature extraction.

[0031] In some embodiments, the converted first audio feature data and noise distribution are weighted, filtered, and minimum pooled to obtain a first noise reduction audio feature, including: weighting the converted first audio feature data and the noise distribution to obtain an adjusted audio feature; filtering the adjusted audio feature through a filter; dividing the filtered audio feature into multiple frequency bands, aggregating audio features belonging to the same frequency band, and performing minimum pooling on the aggregation results to obtain the first noise reduction audio feature.

[0032] It should be noted that the steps of weighting, filtering and minimum pooling aggregation processing of the converted first audio feature data and noise distribution in the present invention are the key links to achieve the noise reduction effect. The weighted processing here refers to adjusting the weight of the audio feature according to the characteristics of the noise distribution and the audio feature data to reduce the impact of noise. The filtering process is to smooth the audio features through a specific filter to remove high-frequency noise components. The minimum pooling aggregation process is to divide the audio features into multiple frequency bands, aggregate the features in each frequency band, and take the minimum value to further suppress noise. Through this series of operations, the audio features after noise reduction can be effectively extracted, providing a purer signal basis for subsequent audio processing.

[0033] Specifically, weighted processing is to obtain the adjusted audio features by performing weighted operations on the converted first audio feature data and the noise distribution. The weights here can be dynamically adjusted according to the noise intensity distribution threshold to ensure that noise can be effectively suppressed in different noise environments. Filtering processing is to process the adjusted audio features by designing suitable filters, such as low-pass filters or band-pass filters, to remove noise components. Minimum pooling aggregation processing is to divide the filtered audio features according to frequency bands, for example, the audio features can be divided into low-frequency, medium-frequency and high-frequency bands, and then the features in each frequency band are aggregated and the minimum value is taken. This processing method can effectively reduce the impact of noise on the audio signal while retaining the key information of the speech signal. In practical applications, the parameters of the filter can be optimized according to the characteristics of the audio signal, such as adjusting the cutoff frequency of the filter to adapt to different noise environments.

[0034] Preferably, an adaptive weight adjustment mechanism can be introduced in the weighted processing, and the weight can be dynamically adjusted according to the noise intensity detected in real time, so as to improve the adaptability of the noise reduction effect. In the filtering process, a multi-stage filter structure can be adopted, for example, high-frequency noise is first removed by a low-pass filter, and then the spectral characteristics of the audio signal are further optimized by a band-pass filter. In addition, a threshold judgment mechanism can be introduced in the minimum pooling aggregation process. When the minimum value in the frequency band is lower than the preset threshold, it is considered that the frequency band mainly contains noise, and it can be directly set to zero, thereby further improving the noise reduction effect. At the same time, other pooling methods, such as average pooling or maximum pooling, can also be considered to select appropriate pooling strategies according to different application scenarios to achieve the best noise reduction effect.

[0035] In some embodiments, the feature points are processed through an adaptive enhancement network to determine the feature value corresponding to each projection point, including: processing the feature points through an adaptive enhancement network to determine the corresponding weight value; performing weighted summation on the feature points corresponding to each projection point to obtain the corresponding feature value, and determining the position of the feature value in the audio spectrum space.

[0036] It should be noted that the step of processing the feature points by an adaptive enhancement network in the present invention to determine the eigenvalue corresponding to each projection point is a key link in the audio enhancement process. The role of the adaptive enhancement network is to dynamically adjust the weight of the eigenvalue according to the distribution of feature points in the audio spectrum space, thereby enhancing the detail performance of the audio signal. The feature points here refer to representative frequency points in the audio spectrum space, and the projection points are the positions after these frequency points are projected into the time domain. By processing the feature points by an adaptive enhancement network, the eigenvalue corresponding to each projection point can be determined, and its position in the audio spectrum space can be further determined, thereby achieving the enhancement of the audio signal.

[0037] Specifically, the processing of feature points by the adaptive enhancement network includes the determination of weight values ​​and the weighted summation of eigenvalues. The determination of weight values ​​is to learn the spectral characteristics of the audio signal through the network and assign a weight to each feature point to reflect its importance in the audio signal. The weighted summation of eigenvalues ​​is to multiply the value of each feature point by its corresponding weight and then sum it to obtain the final eigenvalue of the projection point. This process can be achieved through the training of neural networks. The network automatically adjusts the weight value by learning a large amount of audio data to achieve the best enhancement effect. In the audio spectrum space, the selection of frequency reference points can be based on the energy distribution of the audio signal or a specific frequency range. For example, speech signals are usually concentrated in the low-frequency and medium-frequency regions, so the frequency points in these regions can be preferentially selected as reference points. In this way, the adaptive enhancement network can effectively enhance the details of the audio signal while avoiding excessive enhancement of noise.

[0038] Preferably, the weight values ​​in the adaptive enhancement network can be further optimized by introducing normalization processing to ensure that the distribution of weight values ​​is more reasonable and to avoid audio distortion caused by excessive or small weights. In the selection of feature points, a dynamic selection mechanism can be adopted to dynamically adjust the position and number of frequency reference points according to the real-time characteristics of the audio signal to better adapt to the audio enhancement needs in different scenarios. In addition, for the calculation of eigenvalues, nonlinear activation functions such as ReLU or Sigmoid can be introduced to enhance the nonlinear fitting ability of the network and further enhance the enhancement effect of the audio signal. At the same time, it is also possible to consider combining the adaptive enhancement network with other audio processing technologies, such as working in conjunction with a noise suppression network, to achieve more comprehensive audio optimization.

[0039] In some embodiments, performing audio enhancement network processing on the fused audio features to obtain an enhanced audio signal includes: performing multi-layer filtering processing on the fused audio features to obtain first data; Performing inverse filter layer processing on the first data to obtain second data; Upsampling the second data to obtain third data having the same resolution as the fused audio feature; The third data is classified frame by frame to obtain an enhanced audio signal.

[0040] It should be noted that the step of performing audio enhancement network processing on the fused audio features to obtain an enhanced audio signal in the present invention is the final output link of the entire audio processing flow. The role of the audio enhancement network is to further optimize the quality of the fused audio features through a series of processing operations to make it closer to the ideal audio signal. This process involves steps such as multi-layer filtering, inverse filtering, upsampling, and frame-by-frame classification, aiming to improve the clarity, stability, and naturalness of the audio signal, and ultimately output a high-quality enhanced audio signal to meet the hearing needs of hearing aid users in different environments.

[0041] Specifically, the multi-layer filtering process in the audio enhancement network optimizes the fused audio features layer by layer by designing filters with different parameters. The types of filters may include low-pass filters, high-pass filters, or band-pass filters, and their parameters such as cutoff frequency and filter order can be adjusted according to the characteristics of the audio signal. The inverse filtering layer removes possible filtering distortion and restores the original characteristics of the audio signal through inverse filtering operations. The upsampling operation is used to increase the resolution of the audio signal to be consistent with the input signal to ensure the integrity of the output signal. The frame-by-frame classification layer analyzes the processed audio signal frame by frame to determine whether each frame belongs to a speech signal, thereby further optimizing the clarity of the speech segment. In practical applications, the frame-by-frame classification layer can use deep learning models such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to improve the accuracy of classification.

[0042] Preferably, the filtering layer in the audio enhancement network can adopt a multi-stage filtering structure, for example, first removing high-frequency noise through a low-pass filter, and then optimizing the spectral range of the speech signal through a band-pass filter. The parameters of the inverse filtering layer can be dynamically adjusted according to the actual effect of the filtering layer to achieve optimal distortion compensation. In the upsampling process, an interpolation algorithm, such as linear interpolation or spline interpolation, can be introduced to improve the quality of the upsampled audio signal. For the frame-by-frame classification layer, an attention mechanism can be introduced to make the network pay more attention to the key frames of the speech signal, thereby further improving the speech enhancement effect. In addition, the audio enhancement network can also be combined with a user feedback mechanism to dynamically adjust the network parameters according to the user's hearing needs to achieve personalized audio enhancement effects.

[0043] In some embodiments, it further includes: constructing an audio processing model, wherein the audio processing model includes a feature extraction network, a noise suppression network, an adaptive enhancement network, a cross-attention network, and an audio enhancement network; Collecting multiple audio data and annotating the collected audio data, including: starting from a starting point of a speech segment in the audio data, extending along the speech segment in the audio data until a complete speech annotation is formed; The annotated audio data is input into the audio processing model for training to adjust the variable parameters in the audio processing model.

[0044] It should be noted that the process of constructing an audio processing model and training it in the present invention is a basic link for achieving high-quality audio processing. The audio processing model is a complex system comprising multiple sub-networks, including a feature extraction network, a noise suppression network, an adaptive enhancement network, a cross-attention network, and an audio enhancement network. These sub-networks work together to achieve noise reduction, enhancement, and optimization processing of audio signals. During the model training process, a large amount of audio data needs to be collected and annotated. The annotation process is based on the starting point of the speech segment in the audio data, extending along the speech segment until a complete speech annotation is formed. By inputting the annotated audio data into the model for training, the variable parameters in the model can be adjusted to optimize the audio processing effect.

[0045] Specifically, the construction of the audio processing model requires comprehensive consideration of the functions and relationships of each sub-network. The feature extraction network is responsible for extracting multi-scale features from the original audio signal, the noise suppression network is used to reduce background noise, and the adaptive enhancement network focuses on improving the details of the audio signal. The cross-attention network achieves more accurate feature extraction by fusing the output features of different networks, while the audio enhancement network is responsible for the final audio signal optimization. During the training process, the labeling of audio data is a key step, and the accuracy of the labeling directly affects the performance of the model. When labeling, the starting and ending points of the speech segment need to be accurately marked to ensure that the model can correctly distinguish between speech and noise. In addition, the loss functions used in the training process, such as the mean square error loss function, are used to measure the difference between the model output and the true value and guide the adjustment of the model parameters. The calculation formula of the mean square error loss function is: Loss = N1∑i = 1N(yi−y^i) 2 Where N is the number of samples, yi is the true value of the sample, and y^i is the predicted value of the model.

[0046] Preferably, in the construction of the audio processing model, a more complex network structure can be introduced, such as using a deeper Wave Net architecture in the feature extraction network to extract richer audio features. In the noise suppression network, multiple noise intensity distribution thresholds can be set according to different noise types to adapt to diverse noise environments. During the model training process, data enhancement techniques can be used, such as adding different types of noise or adjusting the signal-to-noise ratio of the audio signal, to improve the robustness of the model. In addition, the mean square error loss function can be combined with other loss functions, such as spectral distortion loss or perceptual loss, to further optimize the performance of the model. For labeled data, automated labeling tools can be introduced, combined with manual review, to improve labeling efficiency and accuracy.

[0047] In some embodiments, the audio processing model is optimized using a mean square error loss function.

[0048] It should be noted that the process of optimizing the audio processing model using the mean square error loss function in the present invention is a key link to ensure the performance of the model. The mean square error loss function is a commonly used optimization index used to measure the difference between the model output and the true value. By minimizing the loss function, the variable parameters in the audio processing model can be adjusted to optimize the performance of the model so that it can more accurately restore the speech signal and suppress noise when processing the audio signal. The calculation formula of the mean square error loss function is: Loss = N1∑i = 1N(yi−y^i) 2 Among them, N represents the number of samples in the audio data, yi represents the true value of the sample, and y^i represents the model's predicted value for the sample. By optimizing this loss function, it can be ensured that the audio processing model gradually approaches the ideal audio processing effect during the training process.

[0049] Specifically, the core of the mean square error loss function is to calculate the square difference between the model output and the true value, and average all samples. During the training process of the audio processing model, the model will adjust the weights and other variable parameters in the network according to the value of the loss function. For example, when the difference between the model output and the true value is large, the loss value will also increase accordingly, and the model will adjust the parameters through the back propagation algorithm to reduce this difference. In practical applications, the parameter setting of the mean square error loss function is relatively simple, mainly involving the determination of the number of samples N, which is usually consistent with the size of the training data set. In addition, the mean square error loss function is suitable for various audio processing tasks because it can directly reflect the error of the audio signal in the time domain or frequency domain, and is a general and effective optimization goal.

[0050] Preferably, in order to further improve the performance of the audio processing model, other auxiliary loss functions can be introduced on the basis of the mean square error loss function. For example, the spectral distortion loss function can be used to optimize the spectral characteristics of the audio signal to make it closer to the spectral distribution of the real speech; the perceptual loss function can optimize the audio quality from the perspective of auditory perception, so that the audio signal output by the model is more natural in subjective hearing. In addition, during the optimization process, a dynamic learning rate adjustment strategy, such as learning rate decay or an adaptive learning rate algorithm (such as the Adam optimizer), can be adopted to speed up the convergence of the model and improve the optimization effect. At the same time, in order to improve the generalization ability of the model, regularization techniques such as weight decay or Dropout can be introduced during the training process to prevent the model from overfitting.

[0051] In some embodiments, the first value and the second value are processed based on the first attention matrix and the second attention matrix to obtain a fused audio feature, including: Fusion of audio features The calculation formula is:

[0052] in, represents the second query, Indicates the first key, represents the first value, represents the first query, represents the second key, represents the second value, Represents the normalization function.

[0053] It should be noted that the calculation formula for the fused audio features mentioned in the present invention is a key part of realizing the cross-attention network function. The formula processes the query, key and value through a normalization function to realize the dynamic fusion of features. Among them, the query and key are used to calculate the attention matrix, and the value is used to store the feature information. In this way, the fused audio features can better retain the important information in the audio signal while suppressing noise interference. The role of the normalization function is to adjust the feature value to a suitable range to improve the stability and generalization ability of the model.

[0054] Specifically, the calculation formula for fusion audio features is: FusionFeature=Normalize(Q⋅KT)⋅V Among them, Q represents query, K represents key, and V represents value. The normalization function (Normalize) usually adopts the Softmax function or other similar normalization methods to normalize the attention score into a probability distribution. In the cross-attention network, the first noise reduction audio feature and the second enhanced audio feature correspond to different queries, keys, and values, respectively. By calculating the attention matrix between the first key and the second query, and the attention matrix between the first query and the second key, the effective fusion of the two features can be achieved. In practical applications, the calculation of the attention matrix can be achieved through matrix multiplication, and the parameters of the normalization function can be adjusted according to the characteristics of the audio signal to optimize the fusion effect.

[0055] Preferably, the normalization function can adopt a more complex multi-head attention mechanism to further improve the effect of feature fusion. The multi-head attention mechanism divides the query, key, and value into multiple heads, calculates the attention matrix separately, and then splices the results together, so that it can capture the features of different frequencies and time scales in the audio signal. In addition, the output of the normalization function can be combined with the residual connection to avoid the problem of gradient disappearance or explosion in the multi-layer network. When calculating the attention matrix, a scaling factor (such as dk1) can be introduced, where dk is the dimension of the key, to stabilize the propagation of the gradient. At the same time, in order to further optimize the fusion effect, an adaptive weight adjustment mechanism can be introduced to dynamically adjust the weights of the query, key, and value according to the real-time characteristics of the audio signal.

[0056] In some embodiments, the expression of the mean square error loss function is:

[0057] in, represents the loss value, Represents the number of samples in the audio data, Representation sample The true value of Representation sample The predicted value of .

[0058] It should be noted that the mean square error loss function mentioned in the present invention is a key tool for optimizing the audio processing model. The loss function quantifies the performance of the model by calculating the square difference between the model output and the true value and averaging all samples. During the training process of the audio processing model, the mean square error loss function can effectively measure the accuracy of audio signal processing and help the model adjust parameters to minimize errors, thereby achieving more accurate audio enhancement and noise reduction effects. By optimizing this loss function, the model can better restore speech signals in complex noise environments and enhance the user's auditory experience.

[0059] Specifically, the expression of the mean square error loss function is: Loss = N1∑i = 1N(yi−y^i) 2 Where N represents the number of samples in the audio data, yi is the true value of the sample, and y^i is the model's predicted value for the sample. In the training of audio processing models, the true value yi is usually a high-quality audio signal that has been manually annotated or preprocessed, and the predicted value y^i is the audio signal processed by the model. By calculating the square error of each sample and taking the average, the model can obtain a global error assessment. In practical applications, the smaller the value of the loss function, the better the performance of the model. In order to optimize the model, the weights and other parameters in the network are usually adjusted through the back-propagation algorithm to gradually reduce the loss value.

[0060] Preferably, in order to further improve the optimization effect of the model, other auxiliary loss functions can be introduced on the basis of the mean square error loss function. For example, the spectral distortion loss function can be used to optimize the spectral characteristics of the audio signal to make it closer to the spectral distribution of the real speech; the perceptual loss function can optimize the audio quality from the perspective of auditory perception, so that the audio signal output by the model is more natural in subjective hearing. In addition, the mean square error loss function can be weighted, and different weights can be assigned according to the importance of different frequency segments or time periods of the audio signal, so as to more accurately optimize the model performance. During the training process, dynamic learning rate adjustment strategies, such as learning rate decay or adaptive learning rate algorithms (such as Adam optimizer) can also be combined to speed up the convergence of the model and improve the optimization effect.

[0061] The above-mentioned embodiments of the present invention have the following beneficial effects: The hearing aid audio processing integrated method and device of the present invention can effectively improve the speech processing performance of the hearing aid in a complex noise environment through multi-level network processing. The feature extraction network can obtain multi-scale audio features, providing a rich information basis for subsequent processing; the noise suppression network can accurately predict and suppress noise based on a preset noise intensity distribution threshold to ensure the purity of the speech signal. The adaptive enhancement network can further enhance the detail performance of the speech signal by randomly selecting frequency reference points and feature points for processing, thereby providing users with a clearer and more natural auditory experience in a complex noise environment.

[0062] In addition, the cross-attention network can efficiently fuse the denoised and enhanced audio features, and dynamically adjust and optimize the features by calculating the attention matrix, further improving the quality of the audio signal. The multi-layer filtering, inverse filtering, upsampling, and frame-by-frame classification processing of the audio enhancement network can further optimize the resolution and stability of the audio signal, ensuring that the output enhanced audio signal is more natural and clear. Overall, the method and device can provide hearing aid users with a more personalized and accurate audio processing solution, while providing strong support for the intelligent development of hearing aid technology.

[0063] like Figure 2As shown, in some embodiments, an audio processing integrated device 200 for a hearing aid includes: An acquisition module 201 acquires in real time an audio signal collected by a microphone deployed in the hearing aid; The feature extraction module 202 processes the audio signal acquired by the acquisition module based on a feature extraction network to obtain first audio feature data; A first processing module 203 processes the first audio feature data based on a noise suppression network to obtain a first noise reduction audio feature; The second processing module 204 processes the first audio feature data based on an adaptive enhancement network to obtain a second enhanced audio feature; A feature fusion module 205, based on a cross attention network, fuses the first noise reduction audio feature output by the first processing module and the second enhanced audio feature output by the second processing module to obtain a fused audio feature; The third processing module 206 processes the fused audio features output by the feature fusion module using an audio enhancement network to obtain an enhanced audio signal, wherein the audio enhancement network sequentially includes a plurality of filtering layers, an inverse filtering layer, an upsampling layer, and a frame-by-frame classification layer; The output module 207 outputs the enhanced audio signal to the speaker of the hearing aid.

[0064] It is understandable that the modules described in the audio processing integrated device 200 of the hearing aid are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the audio processing integrated method for a hearing aid are also applicable to the audio processing integrated device 200 for a hearing aid and the modules contained therein, and will not be described in detail here.

[0065] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present invention. The electronic devices in some embodiments of the present invention may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0066] like Figure 3As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0067] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0068] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0069] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. An audio processing integration method for a hearing aid, characterized in that: include: Acquire the audio signal collected by the microphone deployed in the hearing aid in real time; Processing the audio signal based on a feature extraction network to obtain first audio feature data; The first audio feature data is processed based on a noise suppression network to obtain a first noise reduction audio feature, including: presetting a noise intensity distribution threshold based on the installation position of the microphone; converting the first audio feature data to the frequency domain; based on the noise intensity distribution threshold, using the noise suppression network to predict the noise distribution corresponding to the converted first audio feature data; weighting, filtering and minimum pooling aggregation processing are performed on the converted first audio feature data and the noise distribution to obtain the first noise reduction audio feature; The first audio feature data is processed based on an adaptive enhancement network to obtain a second enhanced audio feature, including: presetting an audio spectrum space, and converting each frequency point in the audio spectrum space to a frequency domain; randomly selecting a plurality of frequency reference points above the converted audio spectrum space, and projecting them to a time domain to obtain projection points; collecting feature points around the projection points in the first audio feature data; processing the feature points through an adaptive enhancement network to determine a feature value corresponding to each projection point; and determining a second enhanced audio feature based on the feature value; Based on the cross attention network, the first noise reduction audio feature and the second enhanced audio feature are fused to obtain the fused audio feature, including: applying linear transformation to the first noise reduction audio feature and the second enhanced audio feature respectively to obtain corresponding queries, keys and values, wherein the first noise reduction audio feature corresponds to the first query, the first key and the first value, and the second enhanced audio feature corresponds to the second query, the second key and the second value; calculating the first attention matrix based on the first key and the second query; calculating the second attention matrix based on the first query and the second key; processing the first value and the second value based on the first attention matrix and the second attention matrix to obtain the fused audio feature; The fused audio features are processed using an audio enhancement network to obtain an enhanced audio signal, wherein the audio enhancement network sequentially includes a plurality of filtering layers, an inverse filtering layer, an upsampling layer and a frame-by-frame classification layer.

2. The audio processing integration method of a hearing aid according to claim 1, characterized in that: Processing the audio signal based on a feature extraction network to obtain first audio feature data, including: using a WaveNet neural network to process the audio signal to obtain a multi-scale audio feature, wherein the multi-scale audio feature includes a plurality of audio feature data with different resolutions; The multi-scale audio features are sequentially fused according to the resolution from low to high, including: performing 1×1 convolution and upsampling processing on the audio feature data of the first resolution, and fusing the processing result with the audio feature data of the second resolution through the 1×1 convolution result; The first resolution is audio feature data with a lower resolution among the multi-scale audio features; the second resolution is audio feature data with a higher resolution among the multi-scale audio features; A convolutional neural network is used to aggregate the multi-scale audio feature fusion results to obtain first audio feature data.

3. The audio processing integration method of a hearing aid according to claim 1, characterized in that: The converted first audio feature data and noise distribution are weighted, filtered and minimum pooled to obtain a first noise reduction audio feature, including: weighting the converted first audio feature data and the noise distribution to obtain an adjusted audio feature; filtering the adjusted audio feature through a filter; dividing the filtered audio feature into multiple frequency bands, aggregating audio features belonging to the same frequency band, and performing minimum pooling on the aggregation results to obtain the first noise reduction audio feature.

4. The audio processing integration method of a hearing aid according to claim 1, characterized in that: The feature points are processed by an adaptive enhancement network to determine the feature value corresponding to each projection point, including: processing the feature points by an adaptive enhancement network to determine the corresponding weight value; performing weighted summation on the feature points corresponding to each projection point to obtain the corresponding feature value, and determining the position of the feature value in the audio spectrum space.

5. The audio processing integration method of a hearing aid according to claim 1, characterized in that: Performing audio enhancement network processing on the fused audio features to obtain an enhanced audio signal, including: performing multi-layer filtering processing on the fused audio features to obtain first data; Performing inverse filter layer processing on the first data to obtain second data; Upsampling the second data to obtain third data having the same resolution as the fused audio feature; The third data is classified frame by frame to obtain an enhanced audio signal.

6. The audio processing integration method of a hearing aid according to claim 1, characterized in that: Also includes: Constructing an audio processing model, wherein the audio processing model includes a feature extraction network, a noise suppression network, an adaptive enhancement network, a cross-attention network, and an audio enhancement network; Collecting multiple audio data and annotating the collected audio data, including: starting from a starting point of a speech segment in the audio data, extending along the speech segment in the audio data until a complete speech annotation is formed; The annotated audio data is input into the audio processing model for training to adjust the variable parameters in the audio processing model.

7. The audio processing integration method of a hearing aid according to claim 6, characterized in that: The audio processing model is optimized using the mean square error loss function.

8. The audio processing integration method of a hearing aid according to claim 3, characterized in that: The first value and the second value are processed based on the first attention matrix and the second attention matrix to obtain a fused audio feature, including: Fusion of audio features The calculation formula is: in, represents the second query, Indicates the first key, represents the first value, represents the first query, represents the second key, represents the second value, Represents the normalization function.

9. The audio processing integration method of a hearing aid according to claim 7, characterized in that: The expression of the mean square error loss function is: in, represents the loss value, Represents the number of samples in the audio data, Representation sample The true value of Representation sample The predicted value of .

10. A hearing aid device, characterized in that: A hearing aid audio processing integration method according to any one of claims 1 to 9 is implemented, wherein the hearing aid device comprises: an acquisition module for acquiring in real time an audio signal collected by a microphone deployed in the hearing aid; A feature extraction module processes the audio signal acquired by the acquisition module based on a feature extraction network to obtain first audio feature data; A first processing module processes the first audio feature data based on a noise suppression network to obtain a first noise reduction audio feature; A second processing module processes the first audio feature data based on an adaptive enhancement network to obtain a second enhanced audio feature; A feature fusion module, based on a cross attention network, fuses the first noise reduction audio feature output by the first processing module and the second enhanced audio feature output by the second processing module to obtain a fused audio feature; A third processing module processes the fused audio features output by the feature fusion module using an audio enhancement network to obtain an enhanced audio signal, wherein the audio enhancement network sequentially includes a plurality of filtering layers, an inverse filtering layer, an upsampling layer, and a frame-by-frame classification layer; The output module outputs the enhanced audio signal to the speaker of the hearing aid.

Images