Non-intrusive voice score prediction method and system, and medium

Abstract

The invention discloses a non-intrusive voice score prediction method and system and a medium, and belongs to the technical field of audio signal processing. Aiming at the influence of a prescription formula on voice score prediction, a voice tag optimization strategy is fused, prediction deviation caused by prescription formula difference is effectively reduced, and the model prediction precision is further improved; the problem of influence of voice definition mapping on sensitivity is comprehensively considered, and voice envelope features are introduced, so that the voice definition score prediction precision is remarkably improved; compared with single-task learning, a multi-task learning framework is adopted, the prediction performance of each branch is improved through the synergistic effect between tasks, part of voice features in the model can be shared, computing resources are saved, the mutual promotion effect can be achieved, and the prediction precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal processing, and in particular to a non-invasive speech scoring prediction method, system and medium. Background Art

[0002] Everyone's auditory characteristics are different, and the problem of hearing loss is becoming increasingly prominent. Devices such as hearing aids, assistive listening headphones, and personal hearing amplifiers all perform loudness compensation on sound signals, and their effects directly affect the user's auditory experience. However, there is currently a lack of evaluation methods for the speech quality after personalized loudness compensation. Affected by many factors such as diverse prescription formulas, personalized user auditory characteristics, and speech environments, speech quality and clarity evaluation methods face many challenges.

[0003] Traditional subjective evaluation methods rely on users' scores for speech quality and clarity, which are not only time-consuming and laborious, but also have individual differences. Current objective speech evaluation methods are mostly designed for people with normal hearing, such as the commonly used objective speech evaluation metrics PESQ and POLQA. Since the speech quality and clarity after loudness compensation are easily interfered by many factors such as personalized hearing loss and speech environment, their objective evaluation methods are relatively more complex and challenging. According to the presence or absence of a reference signal, objective speech evaluation can be divided into invasive speech evaluation methods and non-invasive speech evaluation methods. For example, HASQI and HASPI require a reference signal. Although they have high accuracy, it is difficult to obtain a pure reference signal in the actual environment. In contrast, non-invasive speech evaluation does not require a reference signal and is suitable for real-time applications, so it has gradually become a research hotspot. In recent years, through its powerful non-linear modeling ability, deep learning technology has shown great potential in non-invasive speech evaluation methods, promoting the rapid development of this field.

[0004] The predicted values of HASPI in the existing speech evaluation prediction are relatively scattered, and a single prescription formula FIG6 is used in the simulation process. The influence of prescription formula differences on speech evaluation is ignored in the current research on non-invasive speech evaluation prediction methods for personalized compensation. In addition, scoring predictions are made separately for speech quality and clarity, wasting computing resources. It can be seen that there is still a large room for optimization in the above-mentioned existing non-invasive speech evaluation methods. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a non-invasive speech scoring prediction method, system and medium, which can solve the problem of insufficient prediction accuracy caused by the failure to fully consider the personalization of compensated speech and the influence of prescription formulas in the existing speech scoring prediction methods, can significantly improve the accuracy of the speech scoring prediction model, and at the same time optimize the utilization efficiency of computing resources.

[0006] To achieve the above object, the present invention is implemented by the following technical solutions:

[0007] On the one hand, the present invention provides a non-invasive voice scoring prediction method, including:

[0008] Obtain the voice to be scored and the audiogram of the hearing-impaired patient;

[0009] According to the audiogram and a preset prescription formula, perform multi-channel loudness compensation on the voice to be scored to obtain a compensated voice;

[0010] Perform frame segmentation on the compensated voice to obtain single-frame voices, and perform windowing processing on the single-frame voices to obtain windowed voices;

[0011] Perform short-time Fourier transform on the windowed voice to obtain the spectral amplitude of the windowed voice; within a preset range of speech linear frequencies, use a logarithmic filter bank to obtain the logarithmic time-frequency data of the windowed voice; extract the speech spectral envelope of the compensated voice at the center frequency of the windowed voice to obtain the first speech envelope feature;

[0012] Unfold the audiogram along the speech frequency to obtain the audiogram loss feature, and embed the audiogram loss feature into the spectral amplitude of the windowed voice and the logarithmic time-frequency data of the windowed voice respectively to obtain the first speech spectral feature and the speech time-frequency feature;

[0013] Input the first speech spectral feature and the first speech envelope feature into a pre-constructed feature extraction model, and output the second speech spectral feature and the second speech envelope feature;

[0014] Input the speech time-frequency feature into a pre-constructed label classification model, and output a classification label;

[0015] Input the second speech spectral feature, the second speech envelope feature, and the classification label into a pre-constructed voice scoring prediction model, and output the voice quality scoring prediction result and the speech clarity scoring prediction result.

[0016] Optionally, according to the audiogram and a preset prescription formula, performing multi-channel loudness compensation on the voice to be scored to obtain a compensated voice includes:

[0017] Randomly select an audiogram and a prescription formula from the audiogram and the preset prescription formula respectively, and calculate the input-output curve of each channel according to the hearing loss information in the audiogram and the prescription formula;

[0018] According to the input-output curve of each channel, calculate the gain of each channel, and apply the gain to the voice to be scored to obtain a compensated voice;

[0019] Among them, the audiogram includes flat audiogram, descending audiogram, ascending audiogram, steep descending audiogram, notch audiogram, and high-frequency steep descending audiogram; the preset prescription formulas include FIG6, DSLv5, NAL-R, and Berger.

[0020] Optionally, the windowed speech is expressed as:

[0021] ;

[0022] ;

[0023] Among them, represents the windowed speech; represents the single-frame speech; represents the Hamming window function; represents a constant; represents the sample point index of the single-frame speech; represents the frame length.

[0024] Optionally, performing a short-time Fourier transform on the windowed speech to obtain the spectral amplitude of the windowed speech includes:

[0025] ;

[0026] ;

[0027] Among them, represents the spectrum of the windowed speech; represents the windowed speech; represents the frequency index; represents the sample point index of the single-frame speech; represents the frame length; represents the imaginary part; represents the spectral amplitude of the windowed speech; represents the real part of the spectrum of the windowed speech; represents the imaginary part of the spectrum of the windowed speech.

[0028] Optionally, within the preset range of the speech linear frequency, obtaining the log time-frequency data of the windowed speech using a logarithmic filter bank includes:

[0029] Within the preset range of the speech linear frequency, converting the linear frequency of the windowed speech to logarithmic frequency, equally spacing the center frequencies of the windowed speech on the logarithmic frequency scale according to the number of logarithmic filters to obtain the logarithmic center frequencies, and mapping the logarithmic center frequencies back to the linear frequency to obtain the linear center frequencies;

[0030] Obtain the frequency response of the logarithmic filter according to the linear center frequency, and obtain the power spectrum of the windowed speech according to the spectral amplitude of the windowed speech; obtain the logarithmic time-frequency data of the windowed speech according to the frequency response of the logarithmic filter and the power spectrum of the windowed speech.

[0031] Optionally, within a preset range of the speech linear frequency, convert the linear frequency of the windowed speech to a logarithmic frequency, equally divide the center frequency of the windowed speech on the logarithmic frequency scale according to the number of logarithmic filters to obtain a logarithmic center frequency, and map the logarithmic center frequency back to the linear frequency to obtain a linear center frequency, including:

[0032] ;

[0033] ;

[0034] ;

[0035] Among them, represents the logarithmic frequency; and represent constants; represents the linear frequency of the windowed speech; represents the th logarithmic center frequency; represents the minimum logarithmic frequency corresponding to the preset range of the speech linear frequency; represents the maximum logarithmic frequency corresponding to the preset range of the speech linear frequency; represents the number of logarithmic filters; represents the th linear center frequency.

[0036] Optionally, obtain the frequency response of the logarithmic filter according to the linear center frequency, and obtain the power spectrum of the windowed speech according to the spectral amplitude of the windowed speech; obtain the logarithmic time-frequency data of the windowed speech according to the frequency response of the logarithmic filter and the power spectrum of the windowed speech, including:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] Among them, represents the frequency response of the th logarithmic filter; is the frequency index; represents the The lower boundary of the linear center frequency corresponding to a logarithmic filter; Denote the linear center frequency corresponding to the th logarithmic filter; The upper boundary of the linear center frequency corresponding to the th logarithmic filter; Denote the power spectrum of the windowed speech; Denote the spectral amplitude of the windowed speech; Denote the power value corresponding to the th logarithmic filter of the windowed speech; Denote the length of the short-time Fourier transform; Denote the power values of the logarithmic filter bank of the windowed speech; Denote the frame number index;

[0042] Concatenate the power values of the logarithmic filter bank of each frame of windowed speech to obtain the log time-frequency data of the windowed speech.

[0043] Optionally, the audiogram loss feature is expressed as:

[0044] ;

[0045] where Denote the audiogram loss feature; Denote the first audiogram loss feature; Denote the second audiogram loss feature; Denote the third audiogram loss feature; Denote the fourth audiogram loss feature; Denote the fifth audiogram loss feature; Denote the sixth audiogram loss feature; Denote the frequencies corresponding to each frequency point after short-time Fourier transform.

[0046] Optionally, the classification label is expressed as:

[0047] ;

[0048] where Denote the argmax activation function; Denote the th probability value corresponding to the classification label; Denote the th probability value corresponding to the classification label; Denote the number of classification labels; Denote the output dimension of the fully connected layer.

[0049] Optionally, the construction of the feature extraction model includes:

[0050] In the CNN part of the obtained original CNN-BiLSTM model, replace the ReLU activation function with the ELU activation function, and add an L after the ELU activation function 2 norm pooling function to obtain the constructed feature extraction model.

[0051] Optionally, the construction of the label classification model includes:

[0052] Add a second fully connected layer parallel to the first fully connected layer in the obtained original single-label classification model to obtain the constructed label classification model, and obtain the constructed label classification model.

[0053] Optionally, the voice score prediction model includes a voice quality score prediction branch and a voice clarity score prediction branch;

[0054] Both the voice quality score prediction branch and the voice clarity score prediction branch sequentially include a multi-head attention layer, a fully connected layer, and a global average pooling layer;

[0055] Input the second voice spectrum feature and the classification label into the constructed voice quality score prediction branch to output the voice quality score prediction result; input the second voice spectrum feature, the second voice envelope feature, and the classification label into the constructed voice clarity score prediction branch to output the voice clarity score prediction result.

[0056] Optionally, train the voice score prediction model using the mean absolute error loss function to obtain the trained voice score prediction model. The mean absolute error loss function is expressed as:

[0057] ;

[0058] where represents the mean absolute error loss function; represents the number of training samples; represents the voice quality score prediction result; represents the true voice quality score result; represents the voice clarity score prediction result; represents the true voice clarity score result.

[0059] In a second aspect, the present invention provides a non-intrusive voice score prediction system, including a processor and a storage medium;

[0060] The storage medium is used to store instructions;

[0061] The processor is used to operate according to the instructions to execute the method according to the first aspect.

[0062] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0063] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0064] In view of the influence of the prescription formula on the prediction of voice scores, the present invention integrates a voice label optimization strategy to effectively reduce the prediction deviation caused by differences in prescription formulas and further improve the prediction accuracy of the model. Considering the problem of the influence of voice clarity mapping on sensitivity, by introducing voice envelope features, a significant improvement in the prediction accuracy of voice clarity scores is achieved. Compared with single-task learning, the present invention adopts a multi-task learning framework, and through the synergistic effect between tasks, the prediction performance of each branch is improved. It can not only share some voice features in the model, save computing resources, but also play a mutually promoting role, which is beneficial to the improvement of prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure shows a schematic flowchart of the non-invasive voice score prediction method of the present invention in an embodiment;

[0066] Figure 2 The figure shows a schematic structural diagram of the feature extraction model of the present invention in an embodiment;

[0067] Figure 3 The figure shows a schematic structural diagram of the label classification model of the present invention in an embodiment;

[0068] Figure 4 The figure shows a schematic structural diagram of the score prediction model of the present invention in an embodiment;

[0069] Figure 5 The figure shows a schematic comparison diagram of the scatter mapping between the predicted result of the voice quality score and the true result of the voice quality score of the present invention in an embodiment;

[0070] Figure 6 The figure shows a schematic comparison diagram of the scatter mapping between the predicted result of the voice clarity score and the true result of the voice clarity score of the present invention in an embodiment;

[0071] Figure 7 The figure shows a schematic diagram of the fitting effect of the present invention, the unlabeled version, and other voice quality score prediction algorithms in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0072] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. Without conflict, the technical features in the embodiments of the present invention and the embodiments can be combined with each other.

[0073] The term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0074] Embodiment 1

[0075] As Figure 1 shown, this embodiment introduces a non-invasive voice scoring prediction method, aiming to get rid of the cumbersome scoring process and the cost of a large amount of manpower and material resources of the existing subjective compensation voice evaluation, and solve the problems of the existing non-invasive compensation voice scoring method.

[0076] The method includes the following steps:

[0077] Step 1: Obtain the voice to be scored and the audiogram of the hearing-impaired patient, that is:

[0078] Randomly obtain 16,000 voice data from the clean dataset in the DNS2020 database, 13,000 for training, 2,000 for evaluation during training, and 1,000 for testing after training; the voice sampling rate is 16,000 Hz and the duration is 10 s; randomly add white noise of -5 to 5 dB to the voice, and then perform noise reduction processing with Wiener filtering to simulate voice distortion to obtain the voice to be scored.

[0079] The audiogram of the hearing patient includes 6 different types of audiograms, specifically flat audiogram, descending audiogram, ascending audiogram, steep descending audiogram, notch audiogram, and high-frequency steep descending audiogram.

[0080] Step 2: Perform multi-channel loudness compensation on the voice to be scored according to the audiogram and a preset prescription formula to obtain a compensated voice, specifically:

[0081] Using a multi-channel loudness compensation algorithm combined with two non-linear prescription formulas FIG6 and DSLv5, and two linear prescription formulas NAL-R and Berger, randomly compensate under 6 different types of audiograms to obtain personalized compensated speech; among them, the multi-channel loudness compensation uses 16 channels, and the upper cut-off frequencies of each channel are 250Hz, 375Hz, 500Hz, 625Hz, 750Hz, 1000Hz, 1250Hz, 1625Hz, 2000Hz, 2375Hz, 2875Hz, 3500Hz, 4250Hz, 5125Hz, 6125Hz, 8000Hz respectively.

[0082] Among the 4 prescription formulas and 6 different types of audiograms, randomly select one audiogram and one prescription formula respectively, calculate the input-output curve corresponding to each channel according to the hearing loss information provided by the audiogram and the prescription formula; calculate the gain of each channel at different output sound pressure levels according to the input-output curve, and apply the gain to the scored speech to obtain personalized compensated speech.

[0083] In addition, the audiogram is a 1*6 array representing hearing loss, and each value is the minimum threshold for the hearing-impaired patient to hear a pure tone at the corresponding frequency; the corresponding frequencies are 250Hz, 500Hz, 1000Hz, 2000Hz, 4000Hz, 8000Hz respectively; each piece of speech records data during the processing of the gain module: two-dimensional speech true label, linearly recorded as 0, non-linearly recorded as 1, prescription formula label: FIG6 is recorded as 2, DSLv5 is recorded as 3, NAL-R is recorded as 4, Berger is recorded as 5. For example, if the compensation formula corresponding to the compensated speech is FIG6, record the corresponding label 1, 2; if the compensation formula is NAL-R, record the corresponding label 0, 4; in addition, the one-dimensional speech quality score true result is calculated using the HASQI true index and recorded as HASQItrue; the one-dimensional speech clarity score true result is calculated using the HASPI true index and recorded as HASPItrue.

[0084] Step three: Perform frame splitting on the compensated speech to obtain single-frame speech, and perform windowing on the single-frame speech to obtain windowed speech, specifically:

[0085] ;

[0086] ;

[0087] Among them, represents the windowed speech; represents the single-frame speech; represents the Hamming window function; represents a constant, which is set to 0.46 in this embodiment; Indicates the sample point index of a single-frame speech; Indicates the frame length.

[0088] Step 4: Perform a short-time Fourier transform on the windowed speech to obtain the spectral amplitude of the windowed speech, specifically:

[0089] Perform a short-time Fourier transform (STFT) on each frame of windowed speech. Without loss of generality, the frame length N can be taken as 512. The frame shift is half of the frame length, and calculate the spectral amplitude of the windowed speech;

[0090] Perform STFT on each frame of windowed speech to obtain the spectral amplitude of the windowed speech , expressed as:

[0091] ;

[0092] ;

[0093] Where, Indicates the spectrum of the windowed speech; Indicates the windowed speech; Indicates the frequency index; Indicates the sample point index of a single-frame speech; Indicates the frame length; Indicates the imaginary part; Indicates the spectral amplitude of the windowed speech; Indicates the real part of the spectrum of the windowed speech; Indicates the imaginary part of the spectrum of the windowed speech.

[0094] Step 5: In the preset range of the linear frequency of the speech, use a logarithmic filter bank to obtain the log time-frequency data of the windowed speech, specifically:

[0095] In the preset range of the linear frequency of the speech, convert the linear frequency of the windowed speech to logarithmic frequency. The linear frequency range of the speech is , , according to the Nyquist theorem, is half of the sampling rate, that is, 8000 Hz. Convert the linear frequency of the windowed speech to logarithmic frequency The conversion formula is:

[0096] ;

[0097] Where, A and B are constants, and their values are determined by experiments; the value range of the logarithmic frequency is obtained through The conversion formula can be obtained ; Where, Represents the minimum logarithmic frequency corresponding to the preset range of voice linear frequency; Represents the maximum logarithmic frequency corresponding to the preset range of voice linear frequency;

[0098] Within the range of logarithmic frequency , according to the number of logarithmic filters, equally divide the center frequencies of the windowed speech at the logarithmic frequency scale to obtain the logarithmic center frequencies. The th logarithmic center frequency is expressed as:

[0099] ;

[0100] Wherein, represents the number of logarithmic filters, which is set to 40; then map the logarithmic center frequencies back to the linear frequencies to obtain the linear center frequencies. The mapping formula for mapping the logarithmic center frequencies back to the linear frequencies is:

[0101] ;

[0102] Wherein, represents the th linear center frequency.

[0103] Step Six: Obtain the frequency response of the logarithmic filters according to the linear center frequencies, and obtain the power spectrum of the windowed speech according to the spectral amplitude of the windowed speech. Specifically:

[0104] The th logarithmic filter's frequency response is expressed as:

[0105] ;

[0106] Wherein, is the frequency index; represents the lower boundary of the linear center frequency corresponding to the th logarithmic filter; represents the linear center frequency corresponding to the th logarithmic filter; represents the upper boundary of the linear center frequency corresponding to the th logarithmic filter;

[0107] The power spectrum of the windowed speech is expressed as:

[0108] ;

[0109] The power spectrum of the windowed speech Multiply and accumulate the frequencies with each logarithmic filter in the logarithmic filter bank respectively and take the logarithm to obtain the output of the corresponding logarithmic filter bank.

[0110] The power value corresponding to the windowed speech in the th logarithmic filter is expressed as:

[0111] ;

[0112] where represents the length of the short-time Fourier transform;

[0113] Concatenate the power values corresponding to all logarithmic filters to obtain the power value corresponding to the logarithmic filter bank of the windowed speech , which is expressed as:

[0114] ;

[0115] where represents the frame number index;

[0116] Repeat the execution, concatenate the power values of the logarithmic filter bank of each frame of windowed speech to obtain the log time-frequency data of the windowed speech .

[0117] Step 7: Extract the speech spectral envelope of the compensated speech at the center frequency of the windowed speech to obtain the first speech envelope feature. Specifically: The speech envelope feature is modified based on the peripheral auditory model of the HASQI method. Remove the reference speech part in the model to allow it to be used in a non-invasive situation. Extract the speech spectral envelope of the compensated speech at the center frequency of 32 windowed speeches to obtain the first speech envelope feature .

[0118] Step 8: Unfold the audiogram along the speech frequency to obtain the audiogram loss feature, and embed the audiogram loss feature into the spectral amplitude of the windowed speech and the log time-frequency data of the windowed speech respectively to obtain the first speech spectral feature and the speech time-frequency feature. Specifically:

[0119] Unfold the audiogram frequency along the speech frequency direction; According to the following mapping rule, for a group of 1*256 audiogram loss features, a group of the audiogram loss features is expressed as:

[0120] ;

[0121] where represents the first audiogram loss feature; represents the second audiogram loss feature; represents the third audiogram loss feature; Represents the fourth audiogram loss feature; Represents the fifth audiogram loss feature; Represents the sixth audiogram loss feature; Represents the frequency corresponding to each frequency point after short-time Fourier transform.

[0122] Embed the audiogram loss feature into the spectral amplitude of the windowed speech and the log time-frequency data of the windowed speech respectively to obtain the first speech spectral feature and the speech time-frequency feature ; The data dimension size is [2, number of frames, number of features], where 2 is the number of channels, the first channel is the speech feature, and the second channel is the extended audiogram.

[0123] Step Nine: Input the first speech spectral feature and the first speech envelope feature into a pre-constructed feature extraction model, and output the second speech spectral feature and the second speech envelope feature, specifically:

[0124] The structure of the feature extraction model is as Figure 2 shown. In the CNN part of the obtained original CNN-BiLSTM model, replace the ReLU activation function with the ELU activation function, and add an L 2 norm pooling function after the ELU activation function to obtain the constructed feature extraction model.

[0125] The first speech spectral feature successively passes through a 2D convolutional layer (convolution kernel 3×3), a batch normalization layer, and an activation layer , L 2 norm pooling layer (pooling window 1×4, pooling stride 1×4); the subsequent four convolutional layers, two pooling layers, and a BiLSTM layer; where, after each convolution, uniform normalization and an activation function are performed; except for the number of channels, the other parameters of each convolutional and pooling layer are the same, and the second speech spectral feature is output.

[0126] Similarly, process the first speech envelope feature to output the second speech envelope feature 。

[0127] Using as the activation function is beneficial to smoothing small negative values during training and combines the advantages of linear and non-linear functions; the introduction of the L 2 norm pooling function is beneficial to balancing the retention of local features and dimensionality reduction.

[0128] The activation layer The function is expressed as:

[0129] ;

[0130] Among them, the default value is determined by experiments; represents learnable parameters.

[0131] L 2 The L-norm pooling function is expressed as:

[0132] ;

[0133] Among them, represents the width and height of the pooling window; represents the index corresponding to the pooled value; represents the corresponding pooling result.

[0134] The BiLSTM layer consists of a forward LSTM and a backward LSTM, which can learn the bidirectional dependencies in time series data and utilize the context information. Its formula is expressed as follows:

[0135] ;

[0136] Among them, represents the index of the current frame, represents the Sigmoid function, are the activation values of the input gate, forget gate, and output gate respectively; is the cell state, is the hidden state, is the hidden state of the previous time step; is the current input frame, are the weight matrices of the input gate, forget gate, cell state, and output gate respectively, is the bias vector of each gate; represents element-wise multiplication, the Hadamard product; is the hyperbolic tangent activation function, defined as follows:

[0137] .

[0138] Step ten: Input the speech time-frequency feature into the pre-constructed label classification model to output classification labels, specifically:

[0139] The structure of the label classification model is as Figure 3 shown. Add a second fully connected layer parallel to the first fully connected layer in the obtained original single-label classification model to obtain the constructed label classification model, and obtain the constructed label classification model.

[0140] The speech time-frequency feature It sequentially passes through a two-dimensional convolutional layer (convolution kernel 3×3), a batch normalization layer, an activation layer PReLU, and a two-dimensional max pooling layer (pooling window 2×2, default stride); there are two subsequent convolutional layers and two pooling layers; after each convolutional layer, there is uniform normalization and an activation function; for each convolutional and pooling layer, except for the number of channels, the other parameters are the same; there are two fully connected layers to map the corresponding scores, and then after passing through the Softmax layer, argmax outputs the classification label ; The double-label classification adopted by this part of the network structure is conducive to the promotion of learning between classification tasks compared with single-label classification, and improves the accuracy of label classification; among them, the PReLU function of the activation layer is defined as follows:

[0141] ;

[0142] Among them, is a learnable parameter that determines the slope of the negative value part and will be updated during each backpropagation; after passing through Softmax, the output of the fully connected layer is converted into a probability distribution. argmax selects the class index corresponding to the maximum value according to the probability output by Softmax, and the classification label is expressed as:

[0143] ;

[0144] Among them, represents the argmax activation function; represents the th probability value corresponding to the classification label; represents the th probability value corresponding to the classification label; represents the number of classification labels; represents the output dimension of the fully connected layer.

[0145] Step Eleven: Input the second speech spectral feature, the second speech envelope feature, and the classification label into a pre-constructed speech quality score prediction model to output a speech quality score prediction result and a speech clarity score prediction result. Specifically:

[0146] The structure of the speech score prediction model is as Figure 4 shown, including a speech quality score prediction branch and a speech clarity score prediction branch; both the speech quality score prediction branch and the speech clarity score prediction branch sequentially include a multi-head attention layer, a fully connected layer, and a global average pooling layer; for this part of the network structure, the multi-head attention mechanism is utilized to learn the influence of each frame of speech on the task prediction result, giving different attention weights to different frames, which is conducive to improving the accuracy of the prediction task; with the strategy of multi-task learning, the cooperation between tasks is beneficial to improving the prediction accuracy of each task.

[0147] Combine the second speech spectral feature with the classification label to form a combined feature and input it into the constructed speech quality score prediction branch to output the speech quality score prediction result HASQI pre ; Combine the second speech spectral feature , the second speech envelope feature with the classification label to form a combined feature and input it into the constructed speech intelligibility score prediction branch to output the speech intelligibility score prediction result HASPI pre .

[0148] The sum of the mean absolute errors between the predictions of the two branches and the corresponding true values is used as the loss of the overall network , and backpropagation is used to optimize the entire network; in addition, the Adam method of adaptive moment estimation is used for training optimization during the training of the model, with an initial learning rate of 0.0001, and the learning rate of each epoch is dynamically adjusted using cosine annealing. After training for 300 epochs, a compensation speech evaluation model with excellent performance is finally obtained. Among them, the mean absolute error loss function is expressed as:

[0149] ;

[0150] where represents the number of training samples, which is set to 16.

[0151] Example 2

[0152] This example introduces a specific test example of a non-invasive speech score prediction method Figure 5 and shows a scatter plot mapping comparison diagram between the speech quality score prediction result and the true speech quality score Figure 6 and a scatter plot mapping comparison diagram between the speech intelligibility score prediction result and the true speech intelligibility score. The results show that the speech quality and intelligibility evaluation prediction results of this example have a high linear correlation with the true scores, and the LCCs of the speech quality score and the intelligibility score reach 0.986 and 0.988 respectively. Figure 7 The fitting effects of the present invention, the unlabeled version, and other compensation speech quality score prediction algorithms are compared under three indicators: LCC, SRCC, and RMSE. The RMSE fitting accuracy of this example is improved by about 17% compared with the unlabeled version, and it is superior to other methods in all indicators, fully verifying the effectiveness and superiority of the method.

[0153] Example 3

[0154] This embodiment introduces a non-intrusive voice scoring prediction system, including a processor and a storage medium;

[0155] The storage medium is used to store instructions;

[0156] The processor is used to operate according to the instructions to execute the method according to Embodiment 1 or 2.

[0157] Embodiment 4

[0158] This embodiment introduces a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method according to Embodiment 1 or 2 is implemented.

[0159] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0161] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps of the process Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps of the functions specified in one block or a plurality of blocks.

[0163] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope of the present invention as protected by the claims. All of these are within the protection scope of the present invention.

Claims

1. A non-intrusive speech score prediction method, characterized in that: include: Obtain audiograms of patients with speech and hearing loss to be assessed; According to the audiogram and a preset prescription formula, multi-channel loudness compensation is performed on the speech to be scored to obtain compensated speech; Performing frame processing on the compensated speech to obtain a single-frame speech, and performing windowing processing on the single-frame speech to obtain a windowed speech; The windowed speech is subjected to short-time Fourier transform to obtain a frequency spectrum amplitude of the windowed speech; Using a logarithmic filter bank to obtain logarithmic time-frequency data of the windowed speech within a preset range of the speech linear frequency; extracting the speech spectrum envelope of the compensated speech at the center frequency of the windowed speech to obtain a first speech envelope feature; Expanding the audiogram along the speech frequency to obtain an audiogram loss feature, and embedding the audiogram loss feature into the spectrum amplitude of the windowed speech and the logarithmic time-frequency data of the windowed speech, respectively, to obtain a first speech spectrum feature and a speech time-frequency feature; Inputting the first speech spectrum feature and the first speech envelope feature into a pre-built feature extraction model, and outputting a second speech spectrum feature and a second speech envelope feature; Input the speech time-frequency features into a pre-built label classification model and output a classification label; The second speech spectrum feature, the second speech envelope feature and the classification label are input into a pre-built speech score prediction model, and a speech quality score prediction result and a speech clarity score prediction result are output.

2. The non-intrusive speech score prediction method according to claim 1, characterized in that: According to the audiogram and the preset prescription formula, multi-channel loudness compensation is performed on the speech to be scored to obtain compensated speech, including: Randomly selecting an audiogram and a prescription formula from the audiogram and the preset prescription formula, respectively, and calculating the input-output curve of each channel according to the hearing loss information in the audiogram and the prescription formula; Calculating the gain of each channel according to the input-output curve of each channel, and applying the gain to the speech to be scored to obtain compensated speech; The audiogram includes a flat audiogram, a falling audiogram, an ascending audiogram, a steeply falling audiogram, a notched audiogram and a high-frequency steeply falling audiogram; and the preset prescription formulas include FIG6, DSLv5, NAL-R and Berger.

3. The non-intrusive speech score prediction method according to claim 1, characterized in that: The windowed speech is represented as: ； ； in, Indicates windowed speech; Represents a single frame of speech; represents the Hamming window function; represents a constant; Indicates the sample point index of a single frame of speech; Indicates the frame length.

4. The non-intrusive speech score prediction method according to claim 1, characterized in that: The windowed speech is subjected to a short-time Fourier transform to obtain a spectrum amplitude of the windowed speech, comprising: ； ； in, represents the spectrum of the windowed speech; Indicates windowed speech; represents the frequency index; Indicates the sample point index of a single frame of speech; Indicates the frame length; represents the imaginary part; Indicates the spectrum amplitude of the windowed speech; Represents the real part of the spectrum of the windowed speech; Represents the imaginary part of the spectrum of the windowed speech.

5. The non-intrusive speech score prediction method according to claim 1, characterized in that: In the preset range of the speech linear frequency, the logarithmic time-frequency data of the windowed speech is obtained by using the logarithmic filter bank, including: Within a preset range of the linear frequency of the speech, the linear frequency of the windowed speech is converted into a logarithmic frequency, the center frequency of the windowed speech is equally divided on the logarithmic frequency scale according to the number of logarithmic filters to obtain the logarithmic center frequency, and the logarithmic center frequency is mapped back to the linear frequency to obtain the linear center frequency; The frequency response of the logarithmic filter is obtained according to the linear center frequency, and the power spectrum of the windowed speech is obtained according to the spectrum amplitude of the windowed speech; the logarithmic time-frequency data of the windowed speech is obtained according to the frequency response of the logarithmic filter and the power spectrum of the windowed speech.

6. The non-intrusive speech score prediction method according to claim 5, characterized in that: Within the preset range of the linear frequency of the speech, the linear frequency of the windowed speech is converted into a logarithmic frequency, and the center frequency of the windowed speech is equally divided on the logarithmic frequency scale according to the number of logarithmic filters to obtain the logarithmic center frequency, and the logarithmic center frequency is mapped back to the linear frequency to obtain the linear center frequency, including: ； ； ； in, represents logarithmic frequency; , represents a constant; Represents the linear frequency of the windowed speech; Indicates Logarithmic center frequency; Indicates the minimum value of the logarithmic frequency corresponding to the preset range of the speech linear frequency; Indicates the maximum value of the logarithmic frequency corresponding to the preset range of the linear frequency of the speech; represents the number of logarithmic filters; Indicates A linear center frequency.

7. The non-intrusive speech score prediction method according to claim 5, characterized in that: The method comprises: obtaining a frequency response of a logarithmic filter according to the linear center frequency, and obtaining a power spectrum of the windowed speech according to the spectrum amplitude of the windowed speech; and obtaining logarithmic time-frequency data of the windowed speech according to the frequency response of the logarithmic filter and the power spectrum of the windowed speech, including: ； ； ； ； in, Indicates The frequency response of a logarithmic filter; is the frequency index; Indicates The lower boundary of the linear center frequency corresponding to the logarithmic filter; Indicates The linear center frequency corresponding to the logarithmic filter; Indicates The upper boundary of the linear center frequency corresponding to the logarithmic filter; represents the power spectrum of the windowed speech; Indicates the spectrum amplitude of the windowed speech; Indicates that the windowed speech is in The power value corresponding to the logarithmic filter; represents the length of the short-time Fourier transform; The power value of the log filter bank representing the windowed speech; Indicates the frame number index; represents the number of logarithmic filters; The power values ​​of the logarithmic filter group of each frame of windowed speech are concatenated to obtain the logarithmic time-frequency data of the windowed speech.

8. The non-intrusive speech score prediction method according to claim 1, characterized in that: The audiogram loss characteristics are expressed as: ； in, Indicates audiogram loss characteristics; Indicates the first audiogram loss characteristics; Indicates the second audiogram loss characteristics; Indicates the third audiogram loss characteristic; Indicates the fourth audiogram loss characteristic; Indicates the fifth audiogram loss characteristic; Indicates the sixth audiogram loss feature; Indicates the frequency corresponding to each frequency point after short-time Fourier transform.

9. The non-intrusive speech score prediction method according to claim 1, characterized in that: The classification label is expressed as: ； in, represents the argmax activation function; Indicates The probability value corresponding to the classification label; Indicates The probability value corresponding to the classification label; Indicates the number of classification labels; Represents the output dimension of the fully connected layer.

10. The non-intrusive speech score prediction method according to claim 1, characterized in that: The construction of the feature extraction model includes: In the CNN part of the obtained original CNN-BiLSTM model, the ReLU activation function is replaced with the ELU activation function, and the L2 norm pooling function is added after the ELU activation function to obtain the constructed feature extraction model.

11. The non-intrusive speech score prediction method according to claim 1, characterized in that: The construction of the label classification model includes: A second fully connected layer is added to the obtained original single-label classification model in parallel with the first fully connected layer to obtain a constructed label classification model.

12. The non-intrusive speech score prediction method according to claim 1, characterized in that: The speech score prediction model includes a speech quality score prediction branch and a speech clarity score prediction branch; The speech quality score prediction branch and the speech clarity score prediction branch each include a multi-head attention layer, a fully connected layer, and a global average pooling layer in sequence; The second speech spectrum feature and classification label are input into the constructed speech quality score prediction branch, and the speech quality score prediction result is output; the second speech spectrum feature, the second speech envelope feature and the classification label are input into the constructed speech clarity score prediction branch, and the speech clarity score prediction result is output.

13. The non-intrusive speech score prediction method according to claim 1, characterized in that: The speech score prediction model is trained using the mean absolute error loss function to obtain a trained speech score prediction model. The mean absolute error loss function is expressed as: ； in, represents the mean absolute error loss function; Indicates the number of training samples; represents the prediction result of speech quality score; Indicates the actual result of voice quality rating; represents the speech intelligibility score prediction result; Indicates the actual result of speech intelligibility score.

14. A non-intrusive speech score prediction system, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images