A method, system and hearing aid device for intelligibility assessment of a speech signal

Abstract

The application provides a speech intelligibility evaluation method and system and a hearing aid device. The method comprises the following steps: obtaining target domain speech data set of a technical field to which a speech signal to be evaluated belongs; retraining a trained original speech intelligibility evaluation model by using speech signals in the target domain speech data set, so as to fine-tune parameters of each target information processing layer in the trained original speech intelligibility evaluation model and obtain a final speech intelligibility evaluation model; and inputting the speech signal to be evaluated into the final speech intelligibility evaluation model to obtain an intelligibility evaluation score output by the final speech intelligibility evaluation model. The method and system provided by the application realize cross-domain generalization and efficient migration by migrating the trained speech intelligibility evaluation model to a target domain and adaptively fine-tuning the model, thereby improving the cross-domain generalization capability of the speech intelligibility evaluation model.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and more particularly to a method, system, and hearing aid device for assessing the intelligibility of speech signals. Background Technology

[0002] In existing technologies, speech intelligibility assessment mainly employs two methods: subjective assessment and objective assessment. Subjective assessment requires the participation of a large number of subjects and involves a lengthy testing process, thus resulting in high assessment costs. Objective assessment uses signal processing or algorithms to estimate speech intelligibility. However, the signal processing and auditory model methods used in objective assessment only demonstrate high prediction accuracy within the same scenario. In real-world applications, they exhibit poor generalization ability and lack the ability to adaptively adjust to individual differences, making them difficult to apply to cross-domain transfer scenarios with significant differences in data attributes.

[0003] Therefore, the existing technology needs further improvement. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a speech intelligibility assessment method, system and hearing aid device to solve the defect of poor generalization ability in the prior art when performing speech intelligibility assessment.

[0005] In a first aspect, the present invention provides a method for evaluating the intelligibility of speech signals, comprising: A target domain speech dataset is obtained, representing the technical field to which the speech signal to be evaluated belongs. Based on the technical field to which the speech signal to be evaluated belongs, the influence of different information processing layers in the trained original speech intelligibility evaluation model on the prediction performance within the target domain is determined. Specifically, Fisher information from transfer learning and a weighted calculation method based on parameter sensitivity analysis are used to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance within the target domain. The original speech intelligibility evaluation model is constructed based on an attention-based encoder-classification network structure. Based on the influence of different information processing layers on the prediction performance in the target domain, determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model. The trained original speech intelligibility evaluation model is retrained using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model, thereby obtaining the final speech intelligibility evaluation model. The speech signal to be evaluated is input into the final speech intelligibility evaluation model to obtain the intelligibility evaluation score output by the final speech intelligibility evaluation model.

[0006] The method of using Fisher information from transfer learning and weighted calculation based on parameter sensitivity analysis to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance in the target domain, and obtaining the influence of different information processing layers on the prediction performance in the target domain, includes the following steps: For each information processing layer in the original speech intelligibility evaluation model, perform layer masking finite difference perturbation based on the target domain calibration dataset, calculate the change in model prediction loss before and after the information processing layer is masked, and use it as the parameter sensitivity index of the information processing layer. For each information processing layer in the original speech intelligibility evaluation model, based on the source domain dataset or pre-trained dataset, the diagonal elements of the Fisher information matrix of each parameter in the information processing layer are calculated by backpropagation, and the diagonal elements of the Fisher information of all parameters in the layer are summed to obtain the Fisher information carrying capacity of the information processing layer. The sensitivity index of the parameters of the same information processing layer and the Fisher information carrying capacity are jointly normalized to obtain the influence of the prediction performance of the information processing layer in the target domain.

[0007] Optionally, the method for generating the trained original speech intelligibility evaluation model includes: The speech signals in the training dataset are divided into two channels, and the speech signals in the two channels are encoded respectively to obtain the first encoded speech signal and the second encoded speech signal. The first coded speech signal and the second coded speech signal are fused to obtain the training source fused coding features; The training source fusion encoded features are input into a preset neural network model to obtain the training source prediction intelligibility score output by the preset neural network. The loss function between the predicted intelligibility score of the training source and the actual intelligibility score of the speech signal in the training dataset is calculated, and the parameters of the preset neural network model are optimized by minimizing the loss function to obtain the trained original speech intelligibility evaluation model with optimized parameters.

[0008] Optionally, the step of dividing the speech signal of the training dataset into two channels and encoding the speech signals of the two channels respectively to obtain a first encoded speech signal and a second encoded speech signal includes: The dual-channel speech signals in the training dataset are preprocessed, and the preprocessed dual-channel speech signals are downsampled. The downsampled dual-channel speech signal is divided into a left channel and a right channel; The left and right channel signals are both input to the first encoder and the second encoder to obtain the left first encoder sub-signal and the right first encoder sub-signal output by the first encoder, and the left second encoder sub-signal and the right second encoder sub-signal output by the second encoder. The left first coded sub-signal and the left second coded sub-signal are fused to obtain the first coded speech signal; The right first encoded sub-signal and the right second encoded sub-signal are fused to obtain a second encoded speech signal; wherein the first encoder and the second encoder contain the same number of convolutional layers and different numbers of encoder layers, and the first encoder and the second encoder are trained based on different training methods to extract different speech features.

[0009] Optionally, the step of fusing features between the first coded speech signal and the second coded speech signal to obtain fused coded features includes: The first encoded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain the first fusion feature output by the fully connected layer. Furthermore, the second encoded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain the second fusion feature output by the fully connected layer; The first fusion feature and the second fusion feature are concatenated to obtain the concatenated fusion coding feature.

[0010] Optionally, each information processing layer in the preset neural network model includes: a multi-head attention layer, a fully connected layer, and a global average pooling layer; The step of inputting the fused encoded features into a preset neural network model to obtain the predicted intelligibility score output by the preset neural network includes: The fused encoded features are input into the multi-head attention layer to obtain the sharpness fusion features output by the multi-head attention layer; The sharpness fusion feature is input into the fully connected layer to obtain the frame-level intelligibility score output by the fully connected layer; The frame-level intelligibility score is input into the global average pooling layer to obtain the predicted intelligibility score output by the global average pooling layer.

[0011] Optionally, the step of training the original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model to obtain the final speech intelligibility evaluation model includes: Select a target number of sample speech signals with intelligibility score labels from the target domain speech dataset; The sample speech signal is divided into two channels, and the speech signals of the two channels are encoded respectively to obtain the third encoded speech signal and the fourth encoded speech signal. The third and fourth coded speech signals are fused to obtain the target domain fused coding features. The target domain fused encoding features are input into the trained original speech intelligibility evaluation model to obtain the target domain predicted intelligibility score output by the speech intelligibility evaluation model. The loss function between the predicted intelligibility score of the target source and the actual intelligibility score of the speech signal in the speech signal dataset is calculated. The parameters of each target information processing layer in the trained original speech intelligibility evaluation model are optimized by minimizing the loss function to obtain the final speech intelligibility evaluation model with optimized parameters.

[0012] Optionally, the speech signals in the target domain speech dataset are speech signal samples of the target individual; The steps of retraining the already trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the already trained original speech intelligibility evaluation model, and obtaining the final speech intelligibility evaluation model, include: The speech signal of the target individual is divided into two channels, and the speech signals of the two channels are encoded respectively to obtain the fifth encoded speech signal and the sixth encoded speech signal. The fifth and sixth coded speech signals are fused to obtain individual fused coding features. The individual fusion encoded features are input into the trained original speech intelligibility evaluation model to obtain the predicted intelligibility score corresponding to the target individual output by the trained original speech intelligibility evaluation model. The loss function between the predicted intelligibility score of the target individual and the actual intelligibility score of the speech signal in the speech signal dataset is calculated. The parameters of each target information processing layer in the trained original speech intelligibility assessment model are optimized by minimizing the loss function to obtain the final speech intelligibility assessment model for the corresponding target individual.

[0013] Secondly, this application provides a speech signal intelligibility evaluation system, comprising: The data acquisition and impact assessment module is used to acquire the target domain speech dataset to which the speech signal to be evaluated belongs, and to determine the impact of different information processing layers in the trained original speech intelligibility assessment model on the prediction performance in the target domain based on the technical field to which the speech signal to be evaluated belongs. Specifically, the impact of each information processing layer in the trained original speech intelligibility assessment model on the prediction performance in the target domain is calculated using Fisher information from transfer learning and a weighted calculation method based on parameter sensitivity analysis. The original speech intelligibility assessment model is constructed based on an attention-based encoder-classification network structure. The fine-tuning preparation module is used to determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model based on the influence of different modules on the prediction performance in the target domain. The parameter fine-tuning module is used to retrain the trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset, so as to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model and obtain the final speech intelligibility evaluation model. The intelligibility assessment module is used to input the speech signal to be assessed into the final speech intelligibility assessment model to obtain the intelligibility assessment score output by the final speech intelligibility assessment model.

[0014] Thirdly, this application provides a hearing aid device, which includes: a speech signal intelligibility assessment system as described above.

[0015] Beneficial effects: This invention provides a method, system, device, and storage medium for evaluating the intelligibility of speech signals. The method involves acquiring a target domain speech dataset belonging to the technical field of the speech signal to be evaluated; determining the influence of different information processing layers in a trained original speech intelligibility evaluation model on the prediction performance within the target domain based on the technical field of the speech signal; determining each target information processing layer in the trained original speech intelligibility evaluation model to be fine-tuned based on the influence of different information processing layers on the prediction performance within the target domain; retraining the trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model, resulting in a final speech intelligibility evaluation model; and inputting the speech signal to be evaluated into the final speech intelligibility evaluation model to obtain the intelligibility evaluation score output by the final speech intelligibility evaluation model. The method and system provided by this invention achieve feature distribution alignment and efficient transfer by transferring a trained original speech intelligibility evaluation model to the target domain and adaptively fine-tuning the parameters of the target information processing layer in the model. This enables prediction performance close to that of full training with only a limited number of samples. This mechanism effectively reduces the model's dependence on large-scale labeled data and improves cross-domain generalization ability. Attached Figure Description

[0016] Figure 1 This is a flowchart of the steps in the speech signal intelligibility assessment method provided by the present invention; Figure 2 This is a framework diagram of the feature fusion and processing module based on the speech basic model in this embodiment of the invention; Figure 3 This is a framework diagram of the trained speech intelligibility evaluation model in this embodiment of the invention; Figure 4 This is a flowchart of a few-sample cross-domain migration in an embodiment of the present invention; Figure 5 This is a flowchart of the real-time speech intelligibility assessment system for assistive hearing aid devices in an embodiment of the present invention; Figure 6 This is a block diagram illustrating the principle structure of the speech signal intelligibility assessment system described in this embodiment of the invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0019] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0020] In the development and optimization of hearing aid systems, speech intelligibility is one of the key indicators for measuring hearing improvement performance. Speech intelligibility refers to the proportion of words that a listener can correctly understand in speech, and it is an important parameter for evaluating speech audibility and clarity. Automatically and accurately predicting speech intelligibility is of great significance for improving the quality of life of hearing-impaired patients and the performance of hearing assistive devices.

[0021] Speech intelligibility assessment can currently be mainly divided into subjective assessment and objective assessment. Subjective assessment methods, namely hearing tests, are widely recognized as the gold standard. In practice, this involves a large number of subjects evaluating the intelligibility of the received speech signals. Therefore, subjective assessment requires the participation of a large number of subjects, is time-consuming and costly, and demands strict experimental conditions.

[0022] Therefore, to reduce testing time and costs, researchers have proposed various objective evaluation metrics based on signal processing and auditory models, such as the Speech Intelligibility Index (SII), Short-Time Objective Intelligibility (STOI), Improved Binaural Short-Time Objective Intelligibility (MBSTOI), and Hearing Aid Speech Perception Index (HASPI). However, these objective evaluation methods typically estimate intelligibility by comparing clean speech with distorted speech, relying on a reference signal and thus constituting an intrusive evaluation. In real-world scenarios, clean speech is often difficult to obtain, limiting the applicability of intrusive methods.

[0023] In recent years, non-invasive speech intelligibility assessment based on deep learning and data-driven approaches has become the mainstream method. This method trains a deep neural network model on a speech dataset with subjective intelligibility scores from test subjects, directly predicting intelligibility scores from the speech signal without requiring clean reference speech. Current mainstream methods consist of two parts: a pre-trained speech model and an intelligibility prediction network. The former is pre-trained on large-scale audio data to extract general and rich speech representations; the latter uses a deep neural network to model the extracted embedded features and outputs a speech intelligibility score, effectively improving the accuracy of intelligibility prediction and reducing dependence on experimental conditions.

[0024] However, existing non-invasive speech intelligibility assessment methods still have significant shortcomings. First, while models perform well on specific training datasets, their predictive performance declines significantly when faced with new subjects, hearing aid types, or acoustic environments, demonstrating poor generalization ability. Second, the effectiveness of speech intelligibility assessment for hearing-impaired patients is influenced by a variety of complex factors, such as the degree of individual hearing loss, hearing aid algorithms, and the listening environment. Existing speech intelligibility assessment models struggle to adaptively adjust to individual differences and lack personalized assessment mechanisms. Furthermore, although deep neural networks possess strong feature learning capabilities, their performance often declines significantly in cross-domain transfer scenarios with large differences in data attributes, making effective transfer difficult with a small number of new samples. Therefore, how to achieve effective transfer of assessment models between different datasets and individual conditions, improve the generalization ability and personalized assessment features of speech intelligibility prediction models in new scenarios, and reduce the need for subjective intelligibility scoring labels during model training has become a core problem that urgently needs to be solved in the field of speech intelligibility assessment.

[0025] To overcome the aforementioned shortcomings, this invention provides a speech intelligibility assessment method and system based on general speech representation fusion and few-shot learning, which improves the generalization performance of speech intelligibility assessment models under different subject conditions across datasets. In the target domain speech dataset, a small number of speech signals with intelligibility score labels are selected from each subject and merged to form a few-shot set. The remaining samples are used as the speech signals to be tested. The parameters of the intelligibility assessment model, already trained on the source dataset, are fine-tuned using the few-shot set. The fine-tuned model is then used to predict the intelligibility of the test speech signal, thus obtaining the intelligibility assessment result of the test speech signal.

[0026] In this invention, the original speech signal intelligibility assessment model is first trained, and then transferred to a new target domain speech dataset through a few-sample fine-tuning strategy. A small number of labeled samples are selected from each subject's data for fine-tuning of the model, so that the model can achieve prediction performance close to that of training with all available data in the target domain using only a small number of labeled samples. This reduces the dependence on large-scale labeled data and improves the stability and robustness of cross-dataset intelligibility prediction. Therefore, the final speech signal intelligibility assessment model can have high generalization ability and achieve better intelligibility assessment results.

[0027] The following description, in conjunction with the accompanying drawings, provides a more detailed explanation of the speech signal intelligibility assessment method, system, and hearing aid device provided in this embodiment.

[0028] Firstly, this application provides a method for evaluating the intelligibility of speech signals, such as... Figure 1 As shown, it includes: Step S1: Obtain the target domain speech dataset belonging to the technical field of the speech signal to be evaluated, and determine the influence of different information processing layers in the trained original speech intelligibility evaluation model on the prediction performance in the target domain according to the technical field of the speech signal to be evaluated. Specifically, the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance in the target domain is calculated using Fisher information from transfer learning and a weighted calculation method based on parameter sensitivity analysis; the original speech intelligibility evaluation model is constructed based on an attention-based encoder-classification network structure.

[0029] Once the speech signal to be evaluated is acquired, its technical field is determined based on its purpose, application scenario, source, or technical attributes. The purpose of the speech signal can be, for example, speech recognition, emotion analysis, voiceprint recognition, or in-vehicle voice commands. The application scenario refers to the possible applications of the speech signal. The source of the speech signal can be a user-generated message or a message from a machine. Technical attributes of the speech signal include sampling rate, language type, noise environment, and speaker characteristics. Based on this information, the technical field to which the speech signal to be evaluated belongs is determined. Such technical fields could include intelligent customer service voice, medical voice recording, or dialect speech recognition.

[0030] Once the technical field corresponding to the speech signal to be evaluated is determined, a speech signal dataset within that technical field is obtained. It is conceivable that the speech signal dataset for a specific technical field can be obtained from publicly available datasets on existing open platforms, or it can be created by collecting speech signals for that technical field using different acquisition methods and building a custom speech signal dataset for that technical field.

[0031] In one implementation, Fisher information from transfer learning and parameter sensitivity analysis are used to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance in the target domain, thus obtaining the influence of different information processing layers on the prediction performance in the target domain. Fisher information from transfer learning, specifically the Fisher information matrix, is the core tool for quantifying the sensitivity of parameters to model prediction. By calculating the impact of parameter perturbations on the output probability, key parameters in the source task are identified. Parameter sensitivity analysis optimizes model robustness by quantifying the impact of input perturbations on the output. In specific calculations, partial derivative methods or finite difference methods can be used to calculate the first derivative of the output with respect to the parameters, assessing the direct impact of small parameter changes on model performance. Alternatively, variance decomposition or Monte Carlo simulation can be used to quantify the impact of global parameter changes on the output, revealing the interactions between parameters.

[0032] In detail, the method of using Fisher information from transfer learning and weighted calculation based on parameter sensitivity analysis to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance in the target domain, and obtaining the influence of different information processing layers on the prediction performance in the target domain, includes the following steps: Step S11: For each information processing layer in the original speech intelligibility evaluation model, perform layer mask finite difference perturbation based on the target domain calibration dataset, calculate the change in model prediction loss before and after the information processing layer is masked, and use it as the parameter sensitivity index of the information processing layer.

[0033] The layer mask finite difference perturbation mentioned in this step includes at least one of the following: setting the output of the information processing layer to zero; replacing the parameters of the information processing layer with pre-trained initial values; replacing the output of the information processing layer with a learnable alternative vector; or skipping the residual connections of the information processing layer.

[0034] Step S12: For each information processing layer in the original speech intelligibility evaluation model, based on the source domain dataset or pre-trained dataset, calculate the diagonal elements of the Fisher information matrix of each parameter in the information processing layer through backpropagation, and sum the diagonal elements of the Fisher information of all parameters in the layer to obtain the Fisher information carrying capacity of the information processing layer.

[0035] Step S13: Perform joint normalization operation on the parameter sensitivity index of the same information processing layer and the Fisher information carrying capacity to obtain the influence degree of the prediction performance of the information processing layer in the target domain.

[0036] The joint normalization operation in this step includes: multiplying the parameter sensitivity index and the Fisher information carrying capacity after normalizing them respectively, as shown below: Λ_i = normalize(Δ_i) · normalize(F_i); Where Λ_i represents the influence of the i-th information processing layer on the prediction performance in the target domain, Δ_i represents the parameter sensitivity index of the i-th information processing layer, F_i represents the diagonal element of the Fisher information matrix in the i-th information processing layer, and normalize(·) is the normalization function.

[0037] Step S2: Based on the influence of different information processing layers on the prediction performance in the target domain, determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model.

[0038] Based on the influence levels of different information processing layers determined in step S1 above, the degree of influence of different information processing layers on the prediction of data within the target domain can be obtained. Based on the magnitude of the influence levels of different information processing layers, the target information processing layer that needs fine-tuning is determined.

[0039] Step S3: Use the speech signals in the target domain speech dataset to retrain the trained original speech intelligibility evaluation model, so as to fine-tune the parameters of different information processing layers in the trained original speech intelligibility evaluation model, and obtain the final speech intelligibility evaluation model.

[0040] Once the target domain speech dataset of the technical field to which the speech signal to be evaluated belongs is obtained, the original speech intelligibility evaluation model is retrained using the speech signals in the target domain speech dataset. The parameters in the original speech intelligibility evaluation model are fine-tuned so that the original speech intelligibility evaluation model with fine-tuned parameters can be adapted to the intelligibility evaluation of speech signals in the target domain speech dataset, thereby improving the generalization of the model evaluation.

[0041] Furthermore, the method for generating the trained original speech intelligibility evaluation model includes: Step S31: Divide the speech signal in the training dataset into two channels, and encode the speech signal in each channel to obtain the first encoded speech signal and the second encoded speech signal.

[0042] In this step, each speech signal in the training dataset is first divided into two channels, and the speech signals are encoded using two different encoders to obtain the first encoded speech signal and the second encoded speech signal corresponding to different speech features.

[0043] Further, the step of dividing the speech signal of the training dataset into two channels and encoding the speech signals of the two channels respectively to obtain the first encoded speech signal and the second encoded speech signal includes: Step S311: Preprocess the dual-channel speech signal in the training dataset and downsample the preprocessed dual-channel speech signal; divide the downsampled dual-channel speech signal into left channel and right channel.

[0044] In practice, the dual-channel speech signals of the training dataset are preprocessed. All signals are downsampled to 16kHz, and the length of the speech samples is standardized to 9s using zero padding. The signals are then divided into left and right channels.

[0045] Step S312: Input both the left channel and right channel signals to the first encoder and the second encoder to obtain the left first encoder sub-signal and the right first encoder sub-signal output by the first encoder, and the left second encoder sub-signal and the right second encoder sub-signal output by the second encoder.

[0046] like Figure 2 As shown, the left and right channel signals are input into the pre-trained speech baseline models WavLM-Large and Whisper-Large-v3, respectively, with frozen parameters. WavLM-Large is trained using a self-supervised denoising mask prediction method. The model structure includes one convolutional layer and 24 Transformer encoder layers, mainly capturing acoustic features such as energy changes, noise resistance, and speaker characteristics. Whisper-Large-v3 is trained on tasks such as speech recognition based on large-scale audio-text data. The model structure includes one convolutional layer and 32 Transformer encoder layers, mainly capturing global semantic information and speaker language features in the speech signal.

[0047] The output of the last Transformer encoder layer of the WavLM-Large and Whisper-Large-v3 models is used as the embedding representation, with feature dimensions of 1024 and 1280 respectively. After processing by the Adapter layer, the feature dimensions of both are reduced to 384.

[0048] ; ; ; ; in, , These are the left first encoder signal and the right first encoder signal output from the Transformer encoder layer of the Whisper-Large-v3 model, respectively. , These are the left second encoder signal and the right second encoder signal, respectively, output from the last Transformer encoder layer of the WavLM-Large model.

[0049] Step S313: Fuse the left first coded sub-signal and the left second coded sub-signal to obtain a first coded speech signal; fuse the right first coded sub-signal and the right second coded sub-signal to obtain a second coded speech signal; wherein the first encoder and the second encoder contain the same number of convolutional layers and different numbers of encoder layers, and the first encoder and the second encoder are trained based on different training methods to extract different speech features.

[0050] In one implementation, the single-channel Whisper embedding and WavLM embedding representations are concatenated along the feature dimension to generate a fused feature.

[0051] ; ; Step S32: Perform feature fusion on the first coded speech signal and the second coded speech signal to obtain the training source fused coding features.

[0052] Specifically, such as Figure 3 As shown, the step of fusing the features of the first coded speech signal and the second coded speech signal to obtain fused coded features includes: The first coded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain a first fusion feature output by the fully connected layer; and the second coded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain a second fusion feature output by the fully connected layer; the first fusion feature and the second fusion feature are concatenated to obtain a concatenated fused coded feature.

[0053] Bidirectional Long Short-Term Memory (Bi-LSTM) networks model temporal features, learn bidirectional temporal dependencies and combine them with contextual information, and introduce fully connected layers to reduce the feature dimension to 128 dimensions. ; ; in, , These represent the fused feature information after processing by Bi-LSTM and fully connected layers, respectively.

[0054] Subsequently, the information from the left and right channels after fusion and processing is integrated and concatenated along the feature dimension to obtain the fused representation output by the module: ; in, This represents the fusion coding feature that integrates information from the left and right channels of the signal.

[0055] Step S33: Input the training source fusion encoded features into a preset neural network model to obtain the training source prediction intelligibility score output by the preset neural network.

[0056] Once the above training source fusion encoding features are obtained, they are input into a preset neural network model to train the preset neural network model and obtain the final intelligibility evaluation model.

[0057] Specifically, after the preset network model outputs the predicted intelligibility score of the training source, the loss function between the predicted intelligibility score of the training source and the actual intelligibility score corresponding to the speech signal in the training dataset is calculated. The parameters of other modules except the feature fusion module are optimized by minimizing the loss function to obtain the trained original speech intelligibility evaluation model with optimized parameters.

[0058] In detail, combined Figure 3 As shown, the network structure of the preset neural network model further includes: a multi-head attention layer, a fully connected layer, and a global average pooling layer; The step of inputting the fused encoded features into a preset neural network model to obtain the predicted intelligibility score output by the preset neural network includes: The fused encoded features are input into the multi-head attention layer to obtain the sharpness fusion features output by the multi-head attention layer; The sharpness fusion feature is input into the fully connected layer to obtain the frame-level intelligibility score output by the fully connected layer; The frame-level intelligibility score is input into the global average pooling layer to obtain the predicted intelligibility score output by the global average pooling layer.

[0059] Using the Adam optimizer, with a learning rate of 1e-4, 50 training epochs, and 8 batches per training session, the weighted sum of the mean squared error loss and correlation error loss is calculated to optimize the model, as shown in the following formula: ; ; ; in, This represents the total number of training samples. and Represents the true score and the average true score. and This represents the predicted value score and the average predicted value score. It is the loss weighting coefficient. and These are the mean squared error loss and the correlation loss, respectively. It is the loss function after weighted summation.

[0060] Calculate the root mean square error and Pearson correlation coefficient of the true scores and model predicted scores of the sample data in the training dataset, and select the model with the best training performance in the domain as the trained original speech intelligibility evaluation model.

[0061] Once the trained original speech intelligibility evaluation model is obtained, it is transferred to the target application domain, making the trained original speech intelligibility evaluation model suitable for use in the target application domain.

[0062] Combination Figure 4 As shown, to make the trained original speech intelligibility assessment model applicable to the target application domain, the invention acquires a target domain speech dataset. From the assessment speech samples of each hearing-impaired patient in the target domain speech dataset, K samples with scoring labels (K=5,10,20,50,100) are randomly selected as a minority sample set. A fixed 50% of the remaining data is selected for testing. Due to the acoustic characteristics of the target domain data and the training data, differences exist in hearing-impaired subjects, speaker characteristics, hearing aid algorithms, and hearing scenarios. Directly using the transfer model would lead to a significant decrease in prediction accuracy. The system freezes the parameters of the feature fusion module and selectively fine-tunes the parameters of other modules in the model most relevant to intelligibility features, enabling the fine-tuned speech intelligibility assessment model to quickly adapt to the new cross-domain feature distribution.

[0063] To improve the accuracy of evaluation results while reducing data processing volume and increasing information processing efficiency during parameter fine-tuning, in practice, the parameters of all encoder layers in the WavLM-Large and Whisper-Large-v3 models, as well as the Adapter layer of the feature fusion module, can be frozen to maintain the general feature information processed by the speech intelligibility evaluation model. Subsequently, the parameters of other modules in the trained speech intelligibility evaluation model are fine-tuned. The preset neural network used to train the original speech intelligibility evaluation model includes a multi-head attention layer and a fully connected layer. This selective fine-tuning method allows the trained original speech intelligibility evaluation model to be finely adjusted only for differences in the target domain under a relatively small parameter scale, thereby avoiding overfitting under few-sample conditions. The specific fine-tuning steps are as follows: Preprocessed speech samples from a small dataset are input into the WavLM-Large and Whisper-Large-v3 models to extract features. These features are then fused by a fusion module to output a cross-domain speech representation. The fused features are subsequently input into an intelligibility prediction model, which minimizes... The loss is used to update the parameters of BSLTM, the fully connected layer, and the intelligibility prediction model. A learning rate scheduler is introduced, and the learning rate is reduced by a factor of 10 when the validation loss does not improve after 15 epochs. After training, the model parameters adapted with few samples are retained, and the model transfer performance is evaluated on the test set. By using a fine-tuning strategy with a target domain sample size of K, the model's prediction accuracy in the target domain is significantly improved compared to the direct transfer method. The intelligibility prediction performance under the few-sample condition of K=100 is close to that of training the model using all available data from hearing-impaired patients, reducing the need for labeled data while maintaining good performance on new datasets.

[0064] Step S4: Input the speech signal to be evaluated into the final speech intelligibility evaluation model to obtain the intelligibility evaluation score output by the final speech intelligibility evaluation model.

[0065] After fine-tuning the parameters of the trained original speech intelligibility evaluation model, the final speech intelligibility evaluation model is obtained. The speech signal to be evaluated is input into the final speech intelligibility evaluation model, and the intelligibility evaluation score output by the final speech intelligibility evaluation model is obtained.

[0066] In practice, the original speech intelligibility assessment model can be retrained using sample data from the target domain speech dataset. The final speech intelligibility assessment model obtained by fine-tuning the model parameters can be a speech intelligibility assessment model used across domains, or it can be a final speech intelligibility assessment model adapted to the target individual by training the original speech intelligibility assessment model using sample speech data of the target individual and fine-tuning the model parameters.

[0067] Specifically, the steps of retraining the already trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of the already trained original speech intelligibility evaluation model and obtain the final speech intelligibility evaluation model include: Step S411: Divide the speech signal in the target domain speech dataset into two channels, and encode the speech signal in each channel to obtain the third encoded speech signal and the fourth encoded speech signal. Step S412: Perform feature fusion on the third coded speech signal and the fourth coded speech signal to obtain the target domain fused coding features; Step S413: Input the target domain fused coding features into the trained original speech intelligibility evaluation model to obtain the target domain predicted intelligibility score output by the trained speech intelligibility evaluation model; Step S414: Calculate the loss function between the predicted intelligibility score of the target source and the actual intelligibility score corresponding to the speech signal in the speech signal dataset, and optimize the parameters of the trained original speech intelligibility evaluation model by minimizing the loss function to obtain the final speech intelligibility evaluation model after parameter optimization.

[0068] If a final speech intelligibility assessment model for an individual is obtained through training, then the speech signals in the speech signal dataset are speech signal samples of the target individual; the steps of retraining the already trained original speech intelligibility assessment model using the speech signals in the speech signal dataset to fine-tune the parameters of the already trained original speech intelligibility assessment model and obtain the final speech intelligibility assessment model include: Step S421: Divide the speech signal of the target individual's speech signal sample into two channels, and encode the speech signals of the two channels respectively to obtain the fifth encoded speech signal and the sixth encoded speech signal.

[0069] Step S422: Perform feature fusion on the fifth coded speech signal and the sixth coded speech signal to obtain individual fused coded features.

[0070] Step S423: Input the individual fusion encoded features into the trained original speech intelligibility evaluation model to obtain the predicted intelligibility score output by the trained original speech intelligibility evaluation model corresponding to the target individual.

[0071] Step S424: Calculate the loss function between the predicted intelligibility score of the target individual and the actual intelligibility score corresponding to the speech signal in the speech signal dataset, and optimize the parameters of the trained original speech intelligibility evaluation model by minimizing the loss function to obtain the final speech intelligibility evaluation model corresponding to the target individual after parameter optimization.

[0072] like Figure 5 As shown, when the method disclosed in this invention is applied to a hearing aid device, the hearing aid device acquires environmental speech signals in real time, enhances the received environmental speech signals using a speech enhancement module, and inputs the enhanced speech signals into a real-time speech intelligibility prediction system. The real-time speech intelligibility prediction system outputs the evaluated intelligibility score, which is then fed back to the speech enhancement module. The speech enhancement module adjusts the algorithm effect in real time based on the feedback speech intelligibility score, and the hearing aid outputs the processed speech signal.

[0073] This invention offers the following significant technical advantages: First, the system employs a non-invasive method for assessing hearing aid speech intelligibility, requiring no clean reference speech signal, thus having wider applicability in real-world scenarios. Second, compared to existing methods that generally rely on a single speech model, this invention effectively integrates WavLM representations (which encode speaker features and noise robustness features, referring to speech feature vectors extracted by the WavLM model) and Whisper representations (also known as Whisper features, referring to the sequence of feature vectors output by the Whisper model encoder after processing audio), which encode global semantic information and language characteristics. This allows the system to simultaneously utilize complementary information from both acoustic and semantic levels, improving the problem of limited assessment accuracy caused by insufficient information coverage in traditional features. Furthermore, based on few-shot learning technology, this invention requires only a small number of labeled speech samples from each subject in new datasets to quickly fine-tune the trained model, enabling the model to adapt to individual differences among subjects and the acoustic distribution of datasets with different attributes, thereby reducing the need for a large number of labeled samples.

[0074] This invention effectively solves the problem of poor generalization performance of existing non-invasive intelligibility assessment models when applied to speech datasets with different attributes, enabling personalized and accurate assessments for patients with different types of hearing loss. This technology provides more intelligent speech intelligibility feedback support for hearing-impaired patients, offers valuable reference for the development and personalized design of speech enhancement algorithms for hearing aids, and promotes the widespread application of personalized intelligent hearing devices.

[0075] Secondly, this application provides a speech signal intelligibility evaluation system, such as... Figure 6 As shown, it includes: The data acquisition and impact assessment module 610 is used to acquire the target domain speech dataset of the technical field to which the speech signal to be evaluated belongs, and to determine the impact of different information processing layers in the trained original speech intelligibility assessment model on the prediction performance in the target domain according to the technical field to which the speech signal to be evaluated belongs; its function is as described in step S1.

[0076] The fine-tuning preparation module 620 is used to determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model based on the influence of different modules on the prediction performance in the target domain; its function is as described in step S2.

[0077] The parameter fine-tuning module 630 is used to retrain the trained original speech intelligibility evaluation model using the speech signals in the target domain speech dataset, so as to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model and obtain the final speech intelligibility evaluation model; its function is as described in step S3.

[0078] The intelligibility assessment module 640 is used to input the speech signal to be assessed into the final speech intelligibility assessment model to obtain the intelligibility assessment score output by the final speech intelligibility assessment model, and its function is as described in step S4.

[0079] Thirdly, this application provides a hearing aid device, which includes: the intelligibility assessment system for the aforementioned speech signal.

[0080] Fourthly, this application provides a computer-readable storage medium for storing a computer program that causes a computer to perform the steps of the speech signal intelligibility assessment method.

[0081] This invention provides a method, system, device, and storage medium for evaluating the intelligibility of speech signals. The method involves acquiring a target domain speech dataset belonging to the technical field of the speech signal to be evaluated; retraining an existing speech intelligibility evaluation model using speech signals from the target domain speech dataset to selectively and efficiently fine-tune the parameters of the original trained speech intelligibility evaluation model, resulting in a final speech intelligibility evaluation model; inputting the speech signal to be evaluated into the final speech intelligibility evaluation model to obtain the intelligibility evaluation score output by the final speech intelligibility evaluation model. The method and system provided by this invention achieve feature distribution alignment and efficient transfer by transferring the trained speech intelligibility evaluation model to the target domain and adaptively fine-tuning the model, thereby achieving near-full-scale training prediction performance using only a limited number of samples. This mechanism effectively reduces the model's dependence on large-scale labeled data and improves cross-domain generalization ability.

[0082] Any process or method described in the flowcharts of this embodiment or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which the embodiments of this application pertain.

[0083] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0084] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0085] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0087] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for evaluating the intelligibility of speech signals, characterized in that, include: A target domain speech dataset is obtained, representing the technical field to which the speech signal to be evaluated belongs. Based on the technical field to which the speech signal to be evaluated belongs, the influence of different information processing layers in the trained original speech intelligibility evaluation model on the prediction performance within the target domain is determined. Specifically, Fisher information from transfer learning and a weighted calculation method based on parameter sensitivity analysis are used to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance within the target domain. The original speech intelligibility evaluation model is constructed based on an attention-based encoder-classification network structure. Based on the influence of different information processing layers on the prediction performance in the target domain, determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model. The trained original speech intelligibility evaluation model is retrained using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model, thereby obtaining the final speech intelligibility evaluation model. The speech signal to be evaluated is input into the final speech intelligibility evaluation model to obtain the intelligibility evaluation score output by the final speech intelligibility evaluation model.

2. The method for evaluating the intelligibility of speech signals according to claim 1, characterized in that, The method of using Fisher information from transfer learning and weighted calculation based on parameter sensitivity analysis to calculate the influence of each information processing layer in the trained original speech intelligibility evaluation model on the prediction performance in the target domain, and obtaining the influence of different information processing layers on the prediction performance in the target domain, includes the following steps: For each information processing layer in the original speech intelligibility evaluation model, perform layer masking finite difference perturbation based on the target domain calibration dataset, calculate the change in model prediction loss before and after the information processing layer is masked, and use it as the parameter sensitivity index of the information processing layer. For each information processing layer in the original speech intelligibility evaluation model, based on the source domain dataset or pre-trained dataset, the diagonal elements of the Fisher information matrix of each parameter in the information processing layer are calculated by backpropagation, and the diagonal elements of the Fisher information of all parameters in the layer are summed to obtain the Fisher information carrying capacity of the information processing layer. The sensitivity index of the parameters of the same information processing layer and the Fisher information carrying capacity are jointly normalized to obtain the influence of the prediction performance of the information processing layer in the target domain.

3. The method for evaluating the intelligibility of speech signals according to claim 1, characterized in that, The method for generating the trained original speech intelligibility evaluation model includes: The speech signals in the training dataset are divided into two channels, and the speech signals in the two channels are encoded respectively to obtain the first encoded speech signal and the second encoded speech signal. The first coded speech signal and the second coded speech signal are fused to obtain the training source fused coding features; The training source fusion encoded features are input into a preset neural network model to obtain the training source prediction intelligibility score output by the preset neural network. Calculate the loss function between the predicted intelligibility score of the training source and the actual intelligibility score corresponding to the speech signal in the training dataset, and optimize the parameters of the preset neural network model by minimizing the loss function to obtain the trained original speech intelligibility evaluation model with optimized parameters.

4. The method for evaluating the intelligibility of speech signals according to claim 3, characterized in that, The step of dividing the speech signal in the training dataset into two channels and encoding the speech signal in each channel to obtain a first encoded speech signal and a second encoded speech signal includes: The dual-channel speech signals in the training dataset are preprocessed, and the preprocessed dual-channel speech signals are downsampled. The downsampled dual-channel speech signal is divided into a left channel and a right channel; The left and right channel signals are both input to the first encoder and the second encoder to obtain the left first encoder sub-signal and the right first encoder sub-signal output by the first encoder, and the left second encoder sub-signal and the right second encoder sub-signal output by the second encoder. The left first coded sub-signal and the left second coded sub-signal are fused to obtain the first coded speech signal; The right first encoded sub-signal and the right second encoded sub-signal are fused to obtain a second encoded speech signal; wherein the first encoder and the second encoder contain the same number of convolutional layers and different numbers of encoder layers, and the first encoder and the second encoder are trained based on different training methods to extract different speech features.

5. The method for evaluating the intelligibility of speech signals according to claim 4, characterized in that, The step of fusing features between the first coded speech signal and the second coded speech signal to obtain fused coded features includes: The first encoded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain the first fusion feature output by the fully connected layer. Furthermore, the second encoded speech signal is sequentially input into a bidirectional long short-term memory network and a fully connected layer to obtain the second fusion feature output by the fully connected layer; The first fusion feature and the second fusion feature are concatenated to obtain the concatenated fusion coding feature.

6. The method for evaluating the intelligibility of speech signals according to claim 3, characterized in that, The information processing layers in the preset neural network model include: a multi-head attention layer, a fully connected layer, and a global average pooling layer; The step of inputting the fused encoded features into a preset neural network model to obtain the predicted intelligibility score output by the preset neural network includes: The fused encoded features are input into the multi-head attention layer to obtain the sharpness fusion features output by the multi-head attention layer; The sharpness fusion feature is input into the fully connected layer to obtain the frame-level intelligibility score output by the fully connected layer; The frame-level intelligibility score is input into the global average pooling layer to obtain the predicted intelligibility score output by the global average pooling layer.

7. The method for evaluating the intelligibility of speech signals according to claim 1, characterized in that, The step of retraining the already trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the already trained original speech intelligibility evaluation model, and obtaining the final speech intelligibility evaluation model, includes: Select a target number of sample speech signals with intelligibility score labels from the target domain speech dataset; The sample speech signal is divided into two channels, and the speech signals of the two channels are encoded respectively to obtain the third encoded speech signal and the fourth encoded speech signal. The third coded speech signal and the fourth coded speech signal are fused to obtain the target domain fused coding features; The target domain fused encoding features are input into the trained original speech intelligibility evaluation model to obtain the target domain predicted intelligibility score output by the trained original speech intelligibility evaluation model. The loss function between the predicted intelligibility score of the target source and the actual intelligibility score of the speech signal in the speech signal dataset is calculated. The parameters of each target information processing layer in the trained original speech intelligibility evaluation model are optimized by minimizing the loss function to obtain the final speech intelligibility evaluation model with optimized parameters.

8. The method for evaluating the intelligibility of speech signals according to claim 1, characterized in that, The speech signals in the target domain speech dataset are speech signal samples of the target individual; The steps of retraining the already trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset to fine-tune the parameters of each target information processing layer in the already trained original speech intelligibility evaluation model, and obtaining the final speech intelligibility evaluation model, include: The speech signal of the target individual is divided into two channels, and the speech signals of the two channels are encoded respectively to obtain the fifth encoded speech signal and the sixth encoded speech signal. The fifth coded speech signal and the sixth coded speech signal are fused to obtain individual fused coded features; The individual fusion encoded features are input into the trained original speech intelligibility evaluation model to obtain the predicted intelligibility score corresponding to the target individual output by the trained original speech intelligibility evaluation model; The loss function between the predicted intelligibility score of the target individual and the actual intelligibility score of the speech signal in the speech signal dataset is calculated. The parameters of each target information processing layer in the trained original speech intelligibility evaluation model are optimized by minimizing the loss function to obtain the final speech intelligibility evaluation model for the corresponding target individual.

9. A speech signal intelligibility assessment system, characterized in that, include: The data acquisition and impact assessment module is used to acquire the target domain speech dataset to which the speech signal to be evaluated belongs, and to determine the impact of different information processing layers in the trained original speech intelligibility assessment model on the prediction performance in the target domain based on the technical field to which the speech signal to be evaluated belongs. Specifically, the impact of each information processing layer in the trained original speech intelligibility assessment model on the prediction performance in the target domain is calculated using Fisher information from transfer learning and a weighted calculation method based on parameter sensitivity analysis. The original speech intelligibility assessment model is constructed based on an attention-based encoder-classification network structure. The fine-tuning preparation module is used to determine the target information processing layers to be fine-tuned in the trained original speech intelligibility evaluation model based on the influence of different modules on the prediction performance in the target domain. The parameter fine-tuning module is used to retrain the trained original speech intelligibility evaluation model using speech signals from the target domain speech dataset, so as to fine-tune the parameters of each target information processing layer in the trained original speech intelligibility evaluation model and obtain the final speech intelligibility evaluation model. The intelligibility assessment module is used to input the speech signal to be assessed into the final speech intelligibility assessment model to obtain the intelligibility assessment score output by the final speech intelligibility assessment model.

10. A hearing aid device, characterized in that, include: The speech signal intelligibility assessment system as described in claim 9.

Images