An artificial intelligence-based hearing aid speech enhancement method, device and medium

Abstract

The application discloses an artificial intelligence-based hearing aid speech enhancement method and device and a medium, relates to the technical field of speech signal processing, and comprises the following steps: inputting standardized speech enhancement input features into a direction perception high-frequency speech enhancement model; extracting coding features through a shared coding unit in the direction perception high-frequency speech enhancement model; determining a target sound source direction through a direction information generation unit in the direction perception high-frequency speech enhancement model, converting the target sound source direction into direction-related information, combining the direction-related information with the coding features, and generating direction enhancement features; inputting the direction enhancement features into a high-frequency recovery unit of the direction perception high-frequency speech enhancement model, performing enhancement processing on a high-frequency band corresponding to the target sound source, obtaining high-frequency enhancement data, compensating the high-frequency enhancement data according to a wearer's audiogram, and obtaining individualized high-frequency enhancement frequency domain amplitude data. The application improves the wearer's recognition ability for target speech.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a method, device and medium for speech enhancement of hearing aids based on artificial intelligence. Background Technology

[0002] In the field of hearing aids today, speech enhancement technology for hearing aids is a core research direction. It aims to improve the speech perception ability of hearing-impaired users in complex environments through signal processing algorithms. Conventional methods are mainly based on digital signal processing technology, such as short-time Fourier transform for frequency domain analysis, beamforming to spatially selectively enhance speech in specific directions using microphone arrays, and spectral subtraction to suppress background noise. Based on traditional signal processing, these methods focus on generalized parameter adjustment and fixed-mode processing.

[0003] In existing technologies, traditional beamforming techniques typically rely on preset fixed beam patterns, which are difficult to dynamically adapt to changes in the location of the sound source or multiple interference scenarios, potentially leading to inaccurate target speech tracking or noise leakage. Furthermore, in terms of high-frequency enhancement strategies, conventional methods often use universal gain curves, which do not fully consider the differences in individual hearing loss, easily resulting in insufficient compensation or excessive amplification. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an artificial intelligence-based speech enhancement method for hearing aids to solve the problems of poor adaptability of beamforming technology and insufficient personalization of high-frequency enhancement.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an artificial intelligence-based method for enhancing speech in hearing aids, comprising,

[0008] A reference microphone is selected in the hearing aid, and the time-domain acoustic signals of all microphones are collected and preprocessed to generate standardized speech enhancement input features.

[0009] Standardized speech enhancement input features are input into a direction-aware high-frequency speech enhancement model. Encoded features are extracted through the shared coding unit in the direction-aware high-frequency speech enhancement model. The direction of the target sound source is determined through the direction information generation unit in the direction-aware high-frequency speech enhancement model. The direction of the target sound source is converted into direction-related information and then combined with the encoded features to generate direction enhancement features.

[0010] The directional enhancement features are input into the high-frequency recovery unit of the directional perception high-frequency speech enhancement model to enhance the high-frequency band corresponding to the target sound source, obtain high-frequency enhancement data, and compensate the high-frequency enhancement data according to the wearer's audiogram to obtain individualized high-frequency enhancement frequency domain amplitude data.

[0011] The individualized high-frequency enhanced frequency domain amplitude data is spliced ​​with the low-frequency frequency domain amplitude data of the reference microphone, and the complete frequency domain data is constructed using the frequency domain phase data of the reference microphone. The time domain speech signal is generated by inverse short-time Fourier transform and output to the hearing aid receiver.

[0012] As a preferred embodiment of the artificial intelligence-based speech enhancement method for hearing aids described in this invention, the steps of selecting a reference microphone in the hearing aid, acquiring the temporal acoustic signals of all microphones and performing preprocessing to generate standardized speech enhancement input features are as follows.

[0013] The reference microphone is selected by the internal configuration of the hearing aid, and the time-domain acoustic signals of all microphones are collected. The time-domain acoustic signals are processed by framing, window function and short-time Fourier transform to obtain the frequency domain amplitude information and frequency domain phase information of all microphones.

[0014] By performing spatial difference calculation on the frequency domain amplitude and phase information of all microphones, and extracting the low-frequency frequency domain amplitude and phase data of the reference microphone, standardized speech enhancement input features are generated.

[0015] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the steps of inputting standardized speech enhancement input features into a direction-aware high-frequency speech enhancement model and extracting coding features through the shared coding unit in the direction-aware high-frequency speech enhancement model are as follows:

[0016] Construct a direction-aware high-frequency speech enhancement model and input standardized speech enhancement input features into the direction-aware high-frequency speech enhancement model;

[0017] The shared coding unit of the direction-aware high-frequency speech enhancement model is used to perform time dimension processing, frequency dimension processing, and spatial difference fusion to generate coding features.

[0018] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the steps are as follows: determining the direction of the target sound source through the direction information generation unit in the direction-aware high-frequency speech enhancement model, converting the target sound source direction into direction-related information, and combining it with encoded features to generate direction enhancement features.

[0019] The directional information generation unit of the directional perception high-frequency speech enhancement model performs directional correlation feature analysis on the encoded features and uses a directional prediction algorithm to generate the direction of the target sound source.

[0020] The direction of the target sound source is converted into direction-related information by the direction feature extension method, and the direction-related information is then processed by dimensional expansion to generate direction-extended features.

[0021] The encoded features and the directional extension features are fused to generate directional enhancement features.

[0022] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the steps of inputting directional enhancement features into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model, enhancing the high-frequency band corresponding to the target sound source, and obtaining high-frequency enhancement data are as follows:

[0023] The directional enhancement features are input into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model to perform frequency band analysis and locate the high-frequency band feature region, thereby generating a high-frequency band feature set.

[0024] High-frequency enhancement processing is performed on the high-frequency band feature set to generate high-frequency enhanced data.

[0025] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the step of obtaining individualized high-frequency enhancement frequency domain amplitude data based on the wearer's audiogram compensation high-frequency enhancement data is as follows:

[0026] The wearer's audiogram is converted into a set of compensation parameters and combined with high-frequency enhancement data to generate a set of enhanced features after compensation;

[0027] The frequency dimension of the enhanced feature set after compensation is organized to generate individualized high-frequency enhanced frequency domain amplitude data.

[0028] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the steps of band splicing individualized high-frequency enhancement amplitude data with low-frequency amplitude data of a reference microphone, and constructing complete frequency domain data using frequency domain phase data of the reference microphone, are as follows:

[0029] Individualized high-frequency enhanced frequency domain amplitude data is spliced ​​with low-frequency frequency domain amplitude data of the reference microphone in the frequency dimension to generate complete frequency domain amplitude data;

[0030] The frequency domain phase data of the reference microphone is combined with the complete frequency domain amplitude data at each frequency point to generate complete frequency domain data.

[0031] As a preferred embodiment of the artificial intelligence-based hearing aid speech enhancement method of the present invention, the steps of generating a time-domain speech signal through inverse short-time Fourier transform and outputting it to the hearing aid receiver are as follows:

[0032] Perform an inverse short-time Fourier transform on the complete frequency domain data to generate a continuous time domain segment sequence;

[0033] A continuous time-domain composite signal is generated by splicing consecutive time-domain segments together in the time domain.

[0034] The continuous time-domain synthesized signal is processed into a time-domain speech signal and output to the hearing aid receiver.

[0035] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the artificial intelligence-based hearing aid speech enhancement method described in the first aspect of the present invention.

[0036] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the artificial intelligence-based hearing aid speech enhancement method described in the first aspect of the present invention.

[0037] The beneficial effects of this invention are as follows: By using a direction-aware high-frequency speech enhancement model, the target sound source is accurately located and high-frequency enhancement is performed, ensuring effective speech enhancement in complex sound source environments and improving the wearer's ability to recognize the target speech; by using individualized high-frequency enhancement frequency domain amplitude data, it is ensured that each wearer can perceive a clearer high-frequency signal in actual use, achieving precise compensation for the wearer's personalized hearing needs. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of an AI-based speech enhancement method for hearing aids.

[0040] Figure 2 This is a flowchart for multi-channel acoustic signal preprocessing.

[0041] Figure 3 The flowchart for processing the orientation-aware model.

[0042] Figure 4This is a flowchart for high-frequency recovery and compensation. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] Reference Figures 1-4 This is one embodiment of the present invention, which provides an artificial intelligence-based method for enhancing speech in hearing aids, comprising the following steps:

[0047] S1. Select a reference microphone in the hearing aid, collect the time-domain acoustic signals of all microphones and preprocess them to generate standardized speech enhancement input features.

[0048] The reference microphone is selected by the internal configuration of the hearing aid, and the time-domain acoustic signals of all microphones are collected. The time-domain acoustic signals are processed by framing, windowing function and short-time Fourier transform to obtain the frequency domain amplitude information and frequency domain phase information of all microphones.

[0049] Furthermore, microphones are placed at different positions on the hearing aid shell to perform hardware initialization of the hearing aid. The identification of all microphones is identified through the configuration parameters inside the hearing aid. By reading the microphone identification number and confirming the physical position of each microphone according to the microphone position table set at the factory, microphones located in stable positions on the hearing aid shell and having a relatively fixed acquisition direction are selected as reference microphone candidates. Reference microphones are screened by checking the startup status, noise floor level, and normal state of the electroacoustic path of the reference microphone candidates.

[0050] It should be noted that microphones are placed at different positions on the hearing aid shell, for example, three microphones are fixed at the front, outer, and top of the shell, respectively, to receive sound field information from different directions.

[0051] It should be noted that the reference microphone candidate is screened by checking the startup status, noise floor level, and normal electroacoustic path status of the candidate. Specifically, if any of the following conditions are present: abnormal noise, abnormally low signal amplitude, or not in a usable state, the current candidate reference microphone is excluded; when all candidate reference microphones are in a normal state, the reference microphone is determined according to the priority rules.

[0052] It should be noted that the priority rules are preset and sorted according to the microphone fixed position attributes, microphone pickup direction attributes, and microphone stability level in the hearing aid structure recorded in the microphone position table.

[0053] The preset sorting is determined by the hearing aid during the factory configuration phase and is used to perform deterministic priority selection when all candidate microphones are in normal condition.

[0054] Furthermore, time-domain acoustic signals are acquired from all microphones in the hearing aid. The internal sound acquisition circuit continuously samples the acoustic signals generated by all microphones after the hearing aid is activated, converting the continuous air pressure change information picked up by the microphones into time-domain acoustic signals. These time-domain acoustic signals are then segmented according to a preset frame length (e.g., the number of sampling points for a speech duration of 10 to 20 milliseconds). By sequentially dividing the continuous time-domain acoustic signal sequence into continuous segments, a set of segmented acoustic signal segments is generated. Each segment... The acoustic signal segments undergo window function processing. By applying a window function (e.g., Hamming window function) to each frame of the acoustic signal segment, smoothing is performed at the time boundary positions of the frame of the acoustic signal segment. Short-time Fourier transform processing is then performed on the frame of the acoustic signal segment. The short-time Fourier transform algorithm (e.g., fast Fourier transform algorithm) maps each frame of the acoustic signal segment from the time dimension to the frequency dimension. The short-time Fourier transform algorithm generates frequency domain amplitude information and frequency domain phase information by performing frequency domain calculations on each frame of the acoustic signal segment.

[0055] Among them, frame processing is used to convert continuous time-domain acoustic signals into independent short-time signal segments.

[0056] By performing spatial difference calculation on the frequency domain amplitude and phase information of all microphones, and extracting the low-frequency frequency domain amplitude and phase data of the reference microphone, standardized speech enhancement input features are generated.

[0057] Furthermore, the frequency domain amplitude and phase information of the microphones are spatially differentially processed using a spatial difference calculation method. The amplitude and phase differences from different microphones at each frequency point are calculated to obtain the difference information of the sound source location. For each pair of microphones, the amplitude and phase differences at the same frequency are calculated to obtain spatial difference features. The difference results of the frequency domain amplitude and phase information of all microphones are integrated to obtain a feature set of multi-channel spatial difference information. The low-frequency amplitude and phase data of the reference microphone are extracted from the spatial difference calculation results. All frequency domain data after spatial difference calculation and reference microphone data extraction are statistically analyzed to generate standardized speech enhancement input features.

[0058] Among them, spatial difference features are used to reflect the variation of sound sources in space.

[0059] S2. Input the standardized speech enhancement input features into the direction-aware high-frequency speech enhancement model, extract the coding features through the shared coding unit in the direction-aware high-frequency speech enhancement model, determine the direction of the target sound source through the direction information generation unit in the direction-aware high-frequency speech enhancement model, and then combine the target sound source direction with the coding features after converting the direction of the target sound source into direction-related information to generate direction enhancement features.

[0060] A direction-aware high-frequency speech enhancement model is constructed by inputting standardized speech enhancement input features into the direction-aware high-frequency speech enhancement model.

[0061] It should be noted that the structure of the direction-aware high-frequency speech enhancement model includes a shared coding unit, a direction information generation unit, a high-frequency recovery unit, and an output layer.

[0062] The shared coding unit includes a time-dimension processing unit, a frequency-dimension processing unit, and a spatial difference information fusion unit. The time-dimension processing unit extracts temporal features from the input signal and processes them using a convolutional neural network. The frequency-dimension processing unit processes frequency-related features, employing a frequency-domain convolutional network for spectral analysis to extract frequency-related information. The spatial difference information fusion unit performs spatial difference calculations on the multi-channel signals to extract spatial difference information between each microphone, which helps the direction-aware high-frequency speech enhancement model understand the spatial localization of the sound source. The output of the shared coding unit is a high-dimensional representation containing time, frequency, and spatial features.

[0063] The direction information generation unit includes a direction prediction unit, a self-attention mechanism, and a direction information extension unit. The direction prediction unit predicts the direction based on the encoded features and directly predicts the angle of the sound source in space through a regression model. The self-attention mechanism captures the spatial relationship between the input features. The direction information extension unit maps the direction information of the target sound source to a higher-dimensional and more spatially relevant representation. The output of the direction information generation unit is the predicted direction of the sound source, expressed in spatial coordinates (e.g., angle representation in polar coordinates).

[0064] The high-frequency recovery unit includes a high-frequency enhancement unit and a personalized compensation unit. The high-frequency enhancement unit enhances the high-frequency region in the signal through a frequency domain recovery network and adjusts and enhances the high-frequency components based on the directional information and coding features provided by the directional information generation unit. The personalized compensation unit compensates for the recovered high-frequency data based on the wearer's audiogram. The output of the high-frequency recovery unit is the enhanced high-frequency signal, which is finally synthesized with the low-frequency part to form a complete frequency domain signal.

[0065] The output layer combines the high-frequency recovery unit and the low-frequency signal, converts the frequency domain signal back to the time domain signal through the inverse short-time Fourier transform, and outputs the final speech signal through the receiver of the hearing aid. The output layer is used to convert the enhanced frequency domain information into sound that the user can hear, and ensures that the output signal is clear and matches the target of the input signal.

[0066] It should be noted that the training steps of the direction-aware high-frequency speech enhancement model are as follows: First, a training dataset is collected. Second, the data in the training dataset is preprocessed. Third, the direction information generation unit, the shared encoding unit, and the high-frequency recovery unit are trained. Fourth, the errors of all tasks are combined using a multi-task loss function, and the loss function is minimized using gradient descent (e.g., stochastic gradient descent). Fifth, the generalization ability of the direction-aware high-frequency speech enhancement model is improved using regularization methods (e.g., L2 regularization). Sixth, the model is periodically evaluated using a validation set. Seventh, the hyperparameters of the model are adjusted using grid search or Bayesian optimization methods to obtain optimal performance. Finally, the optimal training configuration is selected using K-fold cross-validation.

[0067] The training dataset includes multi-channel data, direction labels, and noise data. Each data sample in the multi-channel data must contain signals from several microphones, and each microphone signal has the same sampling rate and time synchronization. Each data sample includes the direction label of the target sound source (e.g., azimuth and elevation angle) as a supervision signal for training the direction information generation unit. The training dataset contains different types of background noise to ensure that the model can perform effective speech enhancement in noisy environments. Preprocessing of the data in the training dataset includes standardization, framing and window function processing, and feature extraction.

[0068] Training the direction information generation unit involves using regression methods to predict continuous angle values ​​and generate direction information. The training objective is to minimize the error between the predicted and true directions. Encoded features extracted from the shared coding unit are used as input, and the unit learns to predict the location of the target sound source through training, using mean squared error as the loss function. Training the shared coding unit involves inputting multi-channel audio signals and target direction labels together for end-to-end training. A convolutional neural network processes the time and frequency dimensions of the shared coding unit. The shared coding unit uses a multi-task learning approach for direction prediction and optimizes feature extraction through a high-frequency recovery task.

[0069] Training the high-frequency recovery unit refers to training it using supervised learning methods with data containing real target high-frequency signals. The mean square error method is used, and the goal is to minimize the error between the recovered signal and the real signal. The wearer's audiogram is used as input data for personalized compensation.

[0070] It should be noted that the training objective of the directional information generation unit is to predict the spatial direction of the target sound source through the input coded features; the training objective of the shared coding unit is to extract high-quality features from the multi-channel input signal; and the training objective of the high-frequency recovery unit is to recover the high-frequency part of the low-frequency signal and perform personalized compensation based on the wearer's audiogram.

[0071] It should be noted that the multi-task loss function of the direction-aware high-frequency speech enhancement model includes direction prediction loss, high-frequency recovery loss, and personalized compensation loss.

[0072] Personalized compensation loss refers to the loss calculated based on the wearer's audiogram.

[0073] The shared coding unit of the direction-aware high-frequency speech enhancement model is used to perform time dimension processing, frequency dimension processing, and spatial difference fusion to generate coding features.

[0074] Furthermore, through the shared coding unit of the direction-aware high-frequency speech enhancement model, the input standardized speech enhancement input features (including frequency domain amplitude information, frequency domain phase information, and spatial difference features) are processed. The time dimension processing unit extracts the temporal information of the input multi-channel signal, models the temporal features in the time domain signal using a convolutional neural network, and captures the time sequence dependencies contained in the signal. The features processed by the time dimension are then sent to the frequency dimension processing unit. The frequency dimension processing unit analyzes the input frequency domain amplitude information, performs frequency dimension processing using a frequency domain convolutional neural network, mines the features on the spectrum, and captures the detailed information of the frequency features. The features processed by the time and frequency dimensions are passed to the spatial difference fusion unit. In the spatial difference fusion unit, the signals from all microphones are fused to extract the spatial difference information. The amplitude and phase differences between different microphones are calculated in the spatial difference fusion unit to generate spatial difference features and obtain the spatial location information of the sound source. The features processed by time, frequency, and spatial difference are integrated into a high-dimensional coded feature.

[0075] The directional information generation unit of the directional perception high-frequency speech enhancement model performs directional correlation feature analysis on the encoded features and uses a directional prediction algorithm to generate the direction of the target sound source.

[0076] Furthermore, a self-attention mechanism is employed to perform direction-related feature analysis on the encoded features. The goal is to extract information related to the direction of the sound source from the encoded features, capture the spatial variation pattern of the target sound source, and understand the localization and direction features of the sound source by learning the spatial difference information collected by multiple microphones. A direction prediction algorithm is then used to generate the direction of the target sound source.

[0077] It should be noted that the direction prediction algorithm, specifically, is based on encoded features and uses a regression model to predict the azimuth and elevation angles of the target sound source by fitting the spatial location of the target sound source; the predicted direction of the target sound source is a direction-related feature.

[0078] The direction of the target sound source is converted into direction-related information by the direction feature extension method, and the direction-related information is then processed by dimensional expansion to generate direction-extended features.

[0079] Furthermore, a directional feature extension method is adopted to generate directional information by mapping the predicted azimuth and elevation angles to a specific spatial representation. For example, the direction of the target sound source is represented as a two-dimensional vector containing the specific location of the sound source in space. The direction of the target sound source is converted into directional related information, and the dimensionality of the directional related information is extended through a multi-layer network structure (such as a multi-layer perceptron). The directional related information is mapped to a higher-dimensional feature space to capture the spatial distribution features of the sound source and generate directional extended features.

[0080] It should be noted that dimensional expansion processing includes spatial feature expansion, frequency and time dimension combination, and dimensional mapping; spatial feature expansion refers to expanding orientation information from two dimensions to three dimensions; frequency and time dimension combination refers to combining orientation-related information with frequency and time dimension features, expanding it through a multilayer perceptron to form a higher-dimensional feature vector; dimensional mapping refers to mapping orientation-related information through a nonlinear mapping function (such as the ReLU function in nonlinear activation functions).

[0081] The encoded features and the directional extension features are fused to generate directional enhancement features.

[0082] Furthermore, the encoded features and directional extension features are standardized and normalized. The standardized encoded features and directional extension features are then concatenated, connecting the two feature vectors along the same dimension to generate a new composite feature vector. A feature fusion network is then used to extract higher-order features. The fused features are then subjected to a nonlinear transformation using a nonlinear activation function (e.g., ReLU) to capture more complex nonlinear relationships. Finally, the directional enhancement features are output through the feature fusion network.

[0083] It should be noted that the new composite feature vector contains time, frequency, space, and orientation information; the feature fusion network uses a deep neural network to map the spliced ​​features through a nonlinear activation function to extract higher-order features; the orientation enhancement feature is a joint representation that includes time domain, frequency domain, spatial difference information, and orientation information.

[0084] S3. Input the directional enhancement features into the high-frequency recovery unit of the directional perception high-frequency speech enhancement model, enhance the high-frequency band corresponding to the target sound source, obtain high-frequency enhancement data, compensate the high-frequency enhancement data according to the wearer's audiogram, and obtain individualized high-frequency enhancement frequency domain amplitude data.

[0085] The directional enhancement features are input into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model to perform frequency band analysis and locate the high-frequency band feature region, thereby generating a high-frequency band feature set.

[0086] Furthermore, the directional enhancement features are input into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model for frequency band analysis. Spectral analysis algorithms, such as Fourier transform-based spectrum calculation methods, are used to process each frequency band and identify the frequency band features corresponding to the high-frequency region. The directional-aware high-frequency speech enhancement model performs orientation analysis based on the directional information of the sound source provided by the directional enhancement features, matching the high-frequency components in the spectrum with the frequency band feature regions corresponding to the target sound source to generate a high-frequency band feature set.

[0087] It should be noted that the purpose of frequency band analysis is to perform frequency analysis on the input directional enhancement features to locate the high-frequency band feature region. Through frequency domain processing, the directional perception high-frequency speech enhancement model can identify the high-frequency components in the speech signal and determine the frequency band range where the high-frequency components are located. Frequency band analysis divides the frequency domain signal into different frequency bands and performs special enhancement processing on the high-frequency region to improve the clarity and intelligibility of the speech.

[0088] High-frequency enhancement processing is performed on the high-frequency band feature set to generate high-frequency enhanced data.

[0089] Furthermore, spectrum reconstruction and frequency domain filtering are performed on each high-frequency band feature in the high-frequency band feature set; frequency recovery algorithms (such as the minimum mean square error algorithm) are used to supplement the frequency components that are lost or affected by noise in the band; signal attenuation caused by noise and environmental interference is compensated by enhancing amplitude information; frequency feature adjustment, noise suppression and nonlinear enhancement are performed on the high-frequency band features to generate high-frequency enhanced data.

[0090] It should be noted that frequency characteristic adjustment of high-frequency band features includes adjusting the smoothness of the spectrum and the frequency response, and using nonlinear enhancement algorithms (such as the ReLU activation function) to nonlinearly adjust the frequency response.

[0091] It should be noted that noise suppression of high-frequency band features specifically involves predicting and modeling noise components in the high-frequency band using noise prediction methods (such as spectral subtraction), optimizing noise suppression using adaptive filters (such as Wiener filters), removing noise components in the high-frequency band, and preserving the high-frequency details of the speech signal.

[0092] It should be noted that nonlinear enhancement of high-frequency band characteristics specifically involves performing nonlinear transformation on the signal using a nonlinear activation function (such as the Sigmoid function) to improve the signal's expressiveness and clarity.

[0093] It should be noted that the goal of high-frequency enhancement processing is to enhance the high-frequency components in the speech signal, such as enhancing high-frequency details and speech clarity.

[0094] The wearer's audiogram is converted into a set of compensation parameters and combined with high-frequency enhancement data to generate a set of enhanced features after compensation.

[0095] Furthermore, based on the wearer's audiogram, the corresponding frequency band compensation gain is retrieved from a predefined gain compensation table to generate a set of compensation parameters. The set of compensation parameters is then combined with high-frequency enhancement data. By applying the compensation parameters to the frequency domain amplitude information of the high-frequency enhancement data, and using the gain value of each frequency band in the wearer's audiogram as a weighting coefficient, the amplitude of each frequency band is adjusted in a weighted manner to compensate for the signal attenuation caused by the wearer's hearing loss, thereby generating a set of enhanced features after compensation.

[0096] It should be noted that the predefined gain compensation table is set according to existing hearing loss standards and provides standard gain curves for each frequency band based on hearing test data of the wearer group. The standard gain curves represent the gain values ​​required for a specific frequency band at different levels of hearing loss. The wearer's audiogram shows the wearer's hearing loss at different frequencies. The gain compensation table provides the gain value for each frequency band according to different levels of hearing loss. By looking up the compensation gain within the corresponding frequency range, a set of compensation parameters is generated. The set of compensation parameters contains the gain coefficient for each frequency band to reflect the wearer's personalized hearing needs.

[0097] Specifically, missing frequency points in the compensation parameter set are filled using a linear interpolation method; the linear interpolation method calculates the compensation gain for the intermediate frequency band based on the known hearing loss and gain values ​​of the frequency band.

[0098] The frequency dimension of the enhanced feature set after compensation is organized to generate individualized high-frequency enhanced frequency domain amplitude data.

[0099] Furthermore, based on the enhanced feature set after compensation, the Z-Score normalization method is used to uniformly normalize the amplitude of each frequency band in the compensated high-frequency enhanced data; the amplitude in the frequency dimension is smoothed by a Gaussian smoothing method (e.g., two-dimensional Gaussian smoothing) to generate individualized high-frequency enhanced frequency domain amplitude data.

[0100] S4. The individualized high-frequency enhanced frequency domain amplitude data and the low-frequency frequency domain amplitude data of the reference microphone are spliced ​​together. The frequency domain phase data of the reference microphone is used to construct complete frequency domain data. The time domain speech signal is generated by inverse short-time Fourier transform and output to the hearing aid receiver.

[0101] The individualized high-frequency enhanced amplitude data is stitched together with the low-frequency amplitude data of the reference microphone in the frequency dimension to generate complete amplitude data in the frequency domain.

[0102] Furthermore, the individualized high-frequency enhanced frequency domain amplitude data is aligned with the low-frequency frequency domain amplitude data of the reference microphone on the frequency axis and merged in the frequency domain. Through frequency domain interpolation and smoothing, complete frequency domain amplitude data is generated.

[0103] The frequency domain phase data of the reference microphone is combined with the complete frequency domain amplitude data at each frequency point to generate complete frequency domain data.

[0104] Furthermore, by aligning the frequency domain phase data of the reference microphone with the complete frequency domain amplitude data, the complete frequency domain amplitude data and the frequency domain phase data of the reference microphone are combined at each frequency point using a complex frequency domain reconstruction formula to generate complete frequency domain data.

[0105] It should be noted that the complex frequency domain reconstruction formula is expressed as:

[0106] ;

[0107] in, Indicates frequency, Represents complete frequency domain data. Represents complete frequency domain amplitude data. This represents the frequency domain phase data of the reference microphone. Represents the imaginary unit. This represents phase data.

[0108] It should be noted that, It is obtained by referencing the frequency domain phase data of the microphone.

[0109] Perform an inverse short-time Fourier transform on the complete frequency domain data to generate a continuous time domain segment sequence.

[0110] Furthermore, each frequency band (amplitude and phase) in the complete frequency domain data is aligned with its corresponding time segment; the corresponding time domain signal is restored by performing an inverse Fourier transform on the amplitude and phase of each frequency point; each inverse short-time Fourier transform generates a time domain segment; all time domain segments are generated sequentially according to time order to form a continuous time domain signal sequence.

[0111] Among them, time-domain segments represent different time periods of the signal.

[0112] It should be noted that the inverse Fourier transform is performed on the amplitude and phase of each frequency point. Specifically, the complete frequency domain data is divided into time windows, and the inverse process of short-time Fourier transform is used to restore each frequency band (amplitude and phase) to the time domain signal.

[0113] A continuous time-domain composite signal is generated by splicing consecutive time-domain segments together.

[0114] Furthermore, by employing an overlapping windowing method (such as a Hamming window), continuous time-domain segment sequences are spliced ​​together on the time axis to generate a continuous time-domain synthesized signal.

[0115] The continuous time-domain synthesized signal is processed into a time-domain speech signal and output to the hearing aid receiver.

[0116] Furthermore, the continuous time-domain synthesized signal is cleaned by low-pass filtering to remove residual high-frequency noise; the amplitude of the time-domain speech signal is adjusted to generate a time-domain speech signal, which is then sent to the receiver of the hearing aid for sound output.

[0117] It should be noted that adjusting the amplitude of the time-domain speech signal means adjusting it according to the wearer's hearing requirements and the device's operating range to ensure that the signal volume is appropriate.

[0118] This embodiment also provides a computer device applicable to the artificial intelligence-based hearing aid speech enhancement method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the artificial intelligence-based hearing aid speech enhancement method proposed in the above embodiment.

[0119] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0120] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the artificial intelligence-based hearing aid speech enhancement method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0121] In summary, this invention, through a direction-aware high-frequency speech enhancement model, accurately locates the target sound source and enhances it at high frequencies, ensuring effective speech enhancement in complex sound source environments and improving the wearer's ability to recognize the target speech; through individualized high-frequency enhancement frequency domain amplitude data, it ensures that each wearer can perceive clearer high-frequency signals in actual use, achieving precise compensation for the wearer's personalized hearing needs.

[0122] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for enhancing speech in hearing aids based on artificial intelligence, characterized in that: include, A reference microphone is selected in the hearing aid, and the time-domain acoustic signals of all microphones are collected and preprocessed to generate standardized speech enhancement input features. Standardized speech enhancement input features are input into a direction-aware high-frequency speech enhancement model. Encoded features are extracted through the shared coding unit in the direction-aware high-frequency speech enhancement model. The direction of the target sound source is determined through the direction information generation unit in the direction-aware high-frequency speech enhancement model. The direction of the target sound source is converted into direction-related information and then combined with the encoded features to generate direction enhancement features. The directional enhancement features are input into the high-frequency recovery unit of the directional perception high-frequency speech enhancement model to enhance the high-frequency band corresponding to the target sound source, obtain high-frequency enhancement data, and compensate the high-frequency enhancement data according to the wearer's audiogram to obtain individualized high-frequency enhancement frequency domain amplitude data. The individualized high-frequency enhanced frequency domain amplitude data is spliced ​​with the low-frequency frequency domain amplitude data of the reference microphone, and the complete frequency domain data is constructed using the frequency domain phase data of the reference microphone. The time domain speech signal is generated by inverse short-time Fourier transform and output to the hearing aid receiver. The steps for inputting standardized speech enhancement input features into a direction-aware high-frequency speech enhancement model and extracting coding features through the shared coding unit in the direction-aware high-frequency speech enhancement model are as follows: Construct a direction-aware high-frequency speech enhancement model and input standardized speech enhancement input features into the direction-aware high-frequency speech enhancement model; High-dimensional coding features are generated by performing temporal dimension processing, frequency dimension processing, and spatial difference fusion on the shared coding unit of the direction-aware high-frequency speech enhancement model. The standardized speech enhancement input features include frequency domain amplitude information, frequency domain phase information, and spatial difference features; The time dimension processing includes extracting time-series information from the input multi-channel signal, modeling the time-series features in the time-domain signal using a convolutional neural network, and capturing the time-series dependencies contained in the signal. The frequency dimension processing includes analyzing the input frequency domain amplitude information, using a frequency domain convolutional neural network to perform frequency dimension processing, mining features on the spectrum and capturing detailed information of frequency features; The spatial differential fusion includes fusing the signals from all microphones, extracting spatial difference information, calculating the amplitude and phase differences between different microphones, generating spatial differential features, and obtaining the spatial location information of the sound source. The process involves determining the direction of the target sound source through the direction information generation unit in the direction-aware high-frequency speech enhancement model, converting the target sound source direction into direction-related information, and combining it with encoded features to generate direction enhancement features. The steps are as follows: The directional information generation unit of the directional perception high-frequency speech enhancement model performs directional correlation feature analysis on the encoded features and uses a directional prediction algorithm to generate the direction of the target sound source. The direction of the target sound source is converted into direction-related information by the direction feature extension method, and the direction-related information is then processed by dimensional expansion to generate direction-extended features. The encoded features and the directional extension features are fused to generate directional enhancement features; The direction-related feature analysis includes using a self-attention mechanism to perform direction-related feature analysis on the encoded features, extracting information related to the direction of the sound source from the encoded features, capturing the spatial variation pattern of the target sound source, and understanding the localization and directional features of the sound source by learning the spatial difference information collected by the microphone. The direction prediction algorithm includes predicting the azimuth and elevation angles of the target sound source by fitting the spatial location of the target sound source based on coded features and a regression model. The directional feature extension method includes mapping the predicted azimuth and elevation angles to a specific spatial representation; The dimensionality expansion processing includes converting the direction of the target sound source into direction-related information, performing dimensionality expansion processing on the direction-related information through a multi-layer network structure, mapping the direction-related information to a higher-dimensional feature space, and capturing the spatial distribution features of the sound source. The dimensionality expansion process also includes combining direction-related information with frequency and time dimension features, expanding it through a multilayer perceptron to form a higher-dimensional feature vector, and mapping the direction-related information through a nonlinear mapping function. The steps are as follows: inputting the directional enhancement features into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model to enhance the high-frequency band corresponding to the target sound source and obtain high-frequency enhancement data. The directional enhancement features are input into the high-frequency recovery unit of the directional-aware high-frequency speech enhancement model to perform frequency band analysis and locate the high-frequency band feature region, thereby generating a high-frequency band feature set. High-frequency enhancement processing is performed on the high-frequency band feature set to generate high-frequency enhanced data; The frequency band analysis includes using a spectrum analysis algorithm to process each frequency band and identify the frequency band characteristics corresponding to the high-frequency region; The location of the high-frequency band feature region includes directional analysis based on the directional information of the sound source provided by the directional enhancement feature, and matching the high-frequency components in the spectrum with the frequency band feature region corresponding to the target sound source; The high-frequency enhancement processing includes spectrum reconstruction and frequency domain filtering for each high-frequency band feature in the high-frequency band feature set, using a frequency recovery algorithm to supplement the frequency components that are lost or affected by noise in the band, compensating for signal attenuation caused by noise and environmental interference by enhancing amplitude information, and adjusting the frequency characteristics, suppressing noise, and enhancing the nonlinearity of the high-frequency band features.

2. The artificial intelligence-based speech enhancement method for hearing aids as described in claim 1, characterized in that: The steps for selecting a reference microphone in the hearing aid, acquiring the time-domain acoustic signals from all microphones, preprocessing them, and generating standardized speech enhancement input features are as follows. The reference microphone is selected by the internal configuration of the hearing aid, and the time-domain acoustic signals of all microphones are collected. The time-domain acoustic signals are processed by framing, window function and short-time Fourier transform to obtain the frequency domain amplitude information and frequency domain phase information of all microphones. By performing spatial difference calculation on the frequency domain amplitude and phase information of all microphones, and extracting the low-frequency frequency domain amplitude and phase data of the reference microphone, standardized speech enhancement input features are generated.

3. The artificial intelligence-based speech enhancement method for hearing aids as described in claim 2, characterized in that: The steps for obtaining individualized high-frequency enhancement amplitude data based on the wearer's audiogram compensation high-frequency enhancement data are as follows: The wearer's audiogram is converted into a set of compensation parameters and combined with high-frequency enhancement data to generate a set of enhanced features after compensation; The frequency dimension of the enhanced feature set after compensation is organized to generate individualized high-frequency enhanced frequency domain amplitude data.

4. The artificial intelligence-based speech enhancement method for hearing aids as described in claim 3, characterized in that: The steps for splicing individualized high-frequency enhanced amplitude data with low-frequency amplitude data from a reference microphone, and constructing complete frequency domain data using the frequency domain phase data from the reference microphone, are as follows: Individualized high-frequency enhanced frequency domain amplitude data is spliced ​​with low-frequency frequency domain amplitude data of the reference microphone in the frequency dimension to generate complete frequency domain amplitude data; The frequency domain phase data of the reference microphone is combined with the complete frequency domain amplitude data at each frequency point to generate complete frequency domain data.

5. The artificial intelligence-based speech enhancement method for hearing aids as described in claim 4, characterized in that: The steps for generating a time-domain speech signal through inverse short-time Fourier transform and outputting it to the hearing aid receiver are as follows: Perform an inverse short-time Fourier transform on the complete frequency domain data to generate a continuous time domain segment sequence; A continuous time-domain composite signal is generated by splicing consecutive time-domain segments together in the time domain. The continuous time-domain synthesized signal is processed into a time-domain speech signal and output to the hearing aid receiver.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the artificial intelligence-based hearing aid speech enhancement method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the artificial intelligence-based hearing aid speech enhancement method according to any one of claims 1 to 5.

Images