Voice quality assessment method, device and storage medium

By extracting the spectrum information of noisy voice signals, using pre-trained speech quality evaluation network model, extracting channel attention and deep speech characteristics, combined with the bidirectional GRU module, the efficient and accurate speech quality evaluation is achieved, and the existing solution is solved Reliance on pure speech and model complexity issues in technology.

CN114694685BActive Publication Date: 2024-12-03BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210382372.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-12
Publication Date
2024-12-03
Estimated Expiration
2042-04-12

AI Technical Summary

Technical Problem

The existing speech quality evaluation network model requires a large amount of original pure speech during the training process, which is difficult to obtain, complex model structure, large parameters, slow training speed, and difficult optimization.

Method used

The pre-trained speech quality evaluation network model is adopted. By obtaining the spectrum information of the noisy speech signal, the channel attention characteristics and deep speech characteristics are extracted, the timing-related features are extracted using the bidirectional GRU module to predict the speech signal quality score, and the pure speech is reduced. simplifies the model structure.

Benefits of technology

Voice quality evaluation can be carried out without pure voice, improving evaluation accuracy and efficiency, reducing application difficulty, simplifying model structure, and improving training speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114694685B_ABST
    Figure CN114694685B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a speech quality assessment method, device and storage medium. The method comprises: obtaining spectrum information of multiple speech frames corresponding to a first speech signal; extracting channel attention feature information of the multiple speech frames using a pre-trained speech quality assessment network model for the spectrum information of the multiple speech frames, and extracting deep speech feature information of the multiple speech frames based on the channel attention feature information of the multiple speech frames; extracting time-series correlation feature information between the multiple speech frames based on the deep speech feature information; and predicting the quality score of the first speech signal according to the time-series correlation feature information between the multiple speech frames.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing, and in particular to a speech quality assessment method, device and storage medium. Background Art

[0002] The quality assessment of speech signals can assist in the verification of speech algorithms and speech systems, and is one of the main bases for judging the performance of speech systems and speech products. It is widely used in speech transmission, voice communication, speech enhancement, speech synthesis, speech recognition, and audio encoding and decoding.

[0003] With the rapid development of neural network technology, the speech quality assessment network model based on deep learning has a broad application market and development prospects as a research direction. Although the speech quality assessment network model in related technologies can realize objective quality assessment of speech signals, a large amount of original pure speech is required in the training process of the network model, and it is difficult to obtain training samples; in addition, the network model structure is relatively complex, the number of model parameters involved is large, the training speed is slow, and the model optimization is difficult. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a speech quality assessment method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for evaluating speech quality is provided, comprising:

[0006] Acquire frequency spectrum information of multiple speech frames corresponding to the first speech signal;

[0007] For the frequency spectrum information of the multiple speech frames, using a pre-trained speech quality assessment network model, extracting channel attention feature information of the multiple speech frames, and based on the channel attention feature information of the multiple speech frames, extracting deep speech feature information of the multiple speech frames;

[0008] Based on the deep speech feature information, extracting time-sequence related feature information between the multiple speech frames;

[0009] The quality score of the first speech signal is predicted based on the time sequence related feature information between the multiple speech frames.

[0010] Optionally, the speech quality assessment network model includes: an effective channel attention (ECA) module, a multi-level residual convolution module and a bidirectional gated recurrent unit (GRU) module;

[0011] The method further comprises: extracting channel attention feature information of the plurality of speech frames by using a pre-trained speech quality assessment network model for the spectrum information of the plurality of speech frames, and extracting deep speech feature information of the plurality of speech frames based on the channel attention feature information of the plurality of speech frames, including:

[0012] For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames;

[0013] Utilizing the multi-stage residual convolution module, extracting deep speech feature information based on the channel attention feature information;

[0014] The extracting the time sequence related feature information between the plurality of speech frames based on the deep speech feature information comprises:

[0015] The bidirectional GRU module is used to extract the time-series related feature information corresponding to the deep speech feature information.

[0016] Optionally, the multi-stage residual convolution module includes N residual convolution layers;

[0017] The residual convolution layer comprises: a cascaded first convolution layer and a second convolution layer; wherein the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit;

[0018] The N residual convolution layers are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0019] Optionally, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule;

[0020] The step of extracting the time-series-related feature information corresponding to the deep speech feature information by using the bidirectional GRU module includes:

[0021] Using the forward GRU submodule to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information;

[0022] Using the backward GRU submodule to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information;

[0023] Based on the forward time sequence feature information and the reverse time sequence feature information, the time sequence related feature information corresponding to the deep speech feature information is obtained.

[0024] Optionally, the speech quality assessment network model includes: a fully connected module and a global average pooling (GAP) module;

[0025] The predicting the quality score of the first speech signal according to the time sequence related feature information between the multiple speech frames includes:

[0026] Using the fully connected module, performing fully connected processing on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames;

[0027] The GAP module is used to perform global averaging processing on the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

[0028] Optionally, the acquiring frequency spectrum information of a plurality of speech frames corresponding to the first speech signal includes:

[0029] Preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal;

[0030] Perform time-frequency conversion processing on multiple speech frames of the first speech signal to obtain amplitude spectrum information of the multiple speech frames.

[0031] Optionally, the preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal includes:

[0032] Performing pre-emphasis processing on the first speech signal;

[0033] The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

[0034] Optionally, before acquiring frequency spectrum information of a plurality of speech frames corresponding to the first speech signal, the method includes:

[0035] Obtaining a training sample set of noisy speech and a quality score label of the training sample set;

[0036] Inputting the frequency spectrum information of speech frames corresponding to multiple noisy speech signals in the training sample set into the initial network model to be trained, and obtaining the prediction quality score output by the initial network model;

[0037] Determining a loss function value of the initial network model according to the predicted quality score and the quality score label;

[0038] Based on the loss function value, the parameters to be trained of the initial network model are adjusted to obtain the speech quality assessment network model.

[0039] Optionally, inputting the spectrum information of speech frames corresponding to the multiple noisy speech signals in the training sample set into the initial network model to be trained to obtain the prediction quality score output by the initial network model includes:

[0040] Determining a maximum number of frames of the multiple noisy speech signals based on frequency spectrum information of the multiple speech frames of the multiple noisy speech signals;

[0041] Determining whether the number of frames of the frequency spectrum information of the multiple noisy speech signals is the maximum number of frames;

[0042] If the number of frames of the frequency spectrum information of the multiple noisy speech signals is not the maximum number of frames, the frequency spectrum information is padded with zeros; the number of frames of the frequency spectrum information after the zero-padding process is the maximum number of frames;

[0043] The spectrum information after the zero-padding process is input into the initial network model to obtain a prediction quality score output by the initial network model.

[0044] According to a second aspect of an embodiment of the present disclosure, there is provided a speech quality assessment device, comprising:

[0045] An acquisition module, used to acquire frequency spectrum information of a plurality of speech frames corresponding to the first speech signal;

[0046] A first feature extraction module is used to extract channel attention feature information of the multiple speech frames using a pre-trained speech quality assessment network model for the spectrum information of the multiple speech frames, and extract deep speech feature information of the multiple speech frames based on the channel attention feature information of the multiple speech frames;

[0047] A second feature extraction module, configured to extract time-series related feature information between the plurality of speech frames based on the deep speech feature information;

[0048] A prediction module is used to predict the quality score of the first speech signal based on the time sequence related feature information between the multiple speech frames.

[0049] Optionally, the speech quality assessment network model includes: an effective channel attention ECA module, a multi-level residual convolution module and a bidirectional gated recurrent GRU module;

[0050] The first feature extraction module is used to:

[0051] For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames;

[0052] Utilizing the multi-stage residual convolution module, extracting deep speech feature information based on the channel attention feature information;

[0053] The extracting the time sequence related feature information between the plurality of speech frames based on the deep speech feature information comprises:

[0054] The bidirectional GRU module is used to extract the time-series related feature information corresponding to the deep speech feature information.

[0055] Optionally, the multi-stage residual convolution module includes N residual convolution layers;

[0056] The residual convolution layer comprises: a cascaded first convolution layer and a second convolution layer; wherein the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit;

[0057] The N residual convolution layers are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0058] Optionally, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule;

[0059] The second feature extraction module is used to:

[0060] Using the forward GRU submodule to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information;

[0061] Using the backward GRU submodule to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information;

[0062] Based on the forward time sequence feature information and the reverse time sequence feature information, the time sequence related feature information corresponding to the deep speech feature information is obtained.

[0063] Optionally, the speech quality assessment network model includes: a fully connected module and a global average pooling GAP module;

[0064] The prediction module is used to:

[0065] Using the fully connected module, performing fully connected processing on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames;

[0066] The GAP module is used to perform global averaging processing on the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

[0067] Optionally, the acquisition module is used to:

[0068] Preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal;

[0069] Perform time-frequency conversion processing on multiple speech frames of the first speech signal to obtain amplitude spectrum information of the multiple speech frames.

[0070] Optionally, the acquisition module is used to:

[0071] Performing pre-emphasis processing on the first speech signal;

[0072] The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

[0073] Optionally, the device comprises: a network training module, configured to:

[0074] Obtaining a training sample set of noisy speech and a quality score label of the training sample set;

[0075] Inputting the frequency spectrum information of speech frames corresponding to multiple noisy speech signals in the training sample set into the initial network model to be trained, and obtaining the prediction quality score output by the initial network model;

[0076] Determining a loss function value of the initial network model according to the predicted quality score and the quality score label;

[0077] Based on the loss function value, the parameters to be trained of the initial network model are adjusted to obtain the speech quality assessment network model.

[0078] Optionally, the network training module is used to:

[0079] Determining a maximum number of frames of the multiple noisy speech signals based on frequency spectrum information of the multiple speech frames of the multiple noisy speech signals;

[0080] Determining whether the number of frames of the frequency spectrum information of the multiple noisy speech signals is the maximum number of frames;

[0081] If the number of frames of the frequency spectrum information of the multiple noisy speech signals is not the maximum number of frames, the frequency spectrum information is padded with zeros; the number of frames of the frequency spectrum information after the zero-padding process is the maximum number of frames;

[0082] The spectrum information after the zero-padding process is input into the initial network model to obtain a prediction quality score output by the initial network model.

[0083] According to a third aspect of an embodiment of the present disclosure, there is provided a speech quality assessment device, including:

[0084] processor;

[0085] a memory for storing processor-executable instructions;

[0086] Wherein, the processor is configured to: when executing the executable instructions, implement the steps in the method described in the first aspect of the embodiment of the present disclosure.

[0087] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided. When the instructions in the storage medium are executed by a processor of a speech quality assessment device, the speech quality assessment device is enabled to perform the steps in the method described in the first aspect of the embodiment of the present disclosure.

[0088] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0089] The embodiment of the present disclosure obtains the spectrum information of a first noisy speech signal and a plurality of speech frames corresponding to the first speech signal, inputs the spectrum information of the plurality of speech frames corresponding to the first speech signal into a pre-trained speech quality assessment network model, and uses the speech quality assessment network model to extract channel attention feature information of the plurality of speech frames, thereby more effectively extracting more important deep speech feature information of the plurality of speech frames based on the channel attention feature information after channel attention enhancement;

[0090] In addition, based on the deep speech feature information of multiple speech frames, the timing-related feature information between the multiple speech frames is extracted, and the timing-related feature information that can reflect the changes in the deep speech feature information of the multiple speech frames is utilized. The mutual influence between the speech frames of different timings of the first speech signal is considered to more accurately predict the quality score of the first speech signal. There is no need to obtain a pure speech signal, thereby reducing the difficulty of applying speech quality assessment.

[0091] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0093] Figure 1 The present invention is a flowchart of a method for objectively evaluating speech quality without reference based on speech enhancement provided in the related art.

[0094] Figure 2 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 1 .

[0095] Figure 3 It is a schematic diagram of a network structure of a residual convolutional layer according to an exemplary embodiment.

[0096] Figure 4 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 2 .

[0097] Figure 5 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 3 .

[0098] Figure 6 The figure is a structural diagram of a speech quality assessment network model according to an exemplary embodiment.

[0099] Figure 7 The figure is a schematic diagram showing the structure of a speech quality assessment device according to an exemplary embodiment.

[0100] Figure 8 The present invention is a block diagram of a speech quality assessment device according to an exemplary embodiment. DETAILED DESCRIPTION

[0101] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0102] With the development of information and communication technology, intelligent voice technology based on voice signal transmission has become an important technical tool for information exchange and has gradually spread to various scenarios in life, such as VR games, karaoke, online teaching, and online meetings. The quality of voice signals has a direct impact on the communication of semantics, the judgment of sound, and the auditory perception. Whether it is the loss of useful information or the interference of redundant information, it will affect information exchange and voice signal processing.

[0103] In communication systems, packet loss and jitter often occur due to technical problems such as network instability, which affect the quality of voice in communication transmission. The voice in communication contains a lot of important information. For example, if the voice quality is poor, it will greatly reduce the communication quality and the user's auditory experience, resulting in invalid or inefficient communication. Accurately evaluating the voice quality can make information exchange smoother. Evaluating the quality of communication voice has become a key link in communication.

[0104] In the related technology, speech quality assessment is usually divided into: subjective assessment method and objective assessment method; among them, the subjective assessment method scores the audio by the human ear, and the result is more real and reliable. However, since the subjective assessment method uses humans as the assessment subject, its assessment results are easily affected by human subjective consciousness, and there is a large consumption of human and material resources, and the stability and repeatability of the assessment are not high enough.

[0105] Objective evaluation methods are voice quality evaluation methods implemented by computers, such as ANIQUE, E-Model, and P.563. Compared with subjective evaluation methods, objective evaluation methods have high repeatability, stable systems, can be adapted to a variety of environments, and have a faster evaluation speed.

[0106] For example, Figure 1 As shown, Figure 1 The present invention is a flowchart of a method for objectively evaluating speech quality without reference based on speech enhancement provided in the related art.

[0107] First, the speech to be tested is input into a trained speech enhancement model based on a deep belief network (DBN) to obtain an enhanced speech signal; then, the Mel-frequency cepstral coefficients of the speech signal before and after enhancement are extracted respectively to determine the coefficient difference; the coefficient difference is input into a deep learning (Back Propagation, BP) network model to obtain the final objective score.

[0108] However, this method requires the logarithmic power spectrum information of the original clean speech for training and scoring, but in many cases the original clean speech is not available; and the use of DBN network and BP network for training and scoring still has problems such as too many parameters to be trained, slow training speed and low evaluation correlation.

[0109] The present disclosure provides a method for evaluating speech quality. Figure 2 As shown, Figure 2 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 1 The method comprises:

[0110] Step S101, obtaining frequency spectrum information of a plurality of speech frames corresponding to a first speech signal;

[0111] Step S102, for the spectrum information of the multiple speech frames, using a pre-trained speech quality assessment network model, extracting channel attention feature information of the multiple speech frames, and based on the channel attention feature information of the multiple speech frames, extracting deep speech feature information of the multiple speech frames;

[0112] Step S103, extracting time sequence related feature information between the multiple speech frames based on the deep speech feature information;

[0113] Step S104: predicting a quality score of the first speech signal based on the timing-related feature information between the plurality of speech frames.

[0114] The speech quality assessment method involved in the embodiments of the present disclosure can be applied to electronic devices; here, the electronic devices include devices such as terminals or servers, and the terminal can be a mobile phone, a tablet computer, a laptop computer, etc.; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers.

[0115] In step S101, a first speech signal and spectrum information of a plurality of speech frames corresponding to the first speech signal are obtained.

[0116] The electronic device may be connected to an audio acquisition device to obtain a first noisy speech signal acquired by the audio acquisition device; and spectrum information of multiple speech frames of the first speech signal may be obtained.

[0117] It should be noted that in the related technologies, the evaluation and training process of the speech quality assessment network model usually requires the participation of pure original speech signals to ensure the evaluation accuracy of the network model. However, in many application scenarios, it is difficult or even impossible to obtain the original pure speech, making it impossible to train and evaluate the network model, which limits the application scenarios of the network model.

[0118] In the disclosed embodiment, a noisy first speech signal can be directly obtained, spectral information of multiple speech frames of the noisy first speech signal can be extracted, and the speech quality of the first speech signal can be evaluated based on the spectral information of the multiple speech frames of the first speech signal. There is no need for a pure speech signal to participate, thereby expanding the application scenarios of the speech quality evaluation network model.

[0119] In step S102, the frequency spectrum information of the multiple speech frames corresponding to the first speech signal is input into a pre-trained speech quality assessment network model, and the frequency spectrum information of the multiple speech frames is extracted using the speech quality assessment network model to obtain channel attention feature information of the multiple speech frames;

[0120] It should be noted that the speech quality assessment network model contains multiple convolution kernels, each of which corresponds to a feature channel. Based on the channel attention mechanism, the channel dependency between any two feature channels is learned, and based on the channel dependency, different feature information is adjusted, so that different feature information can be strengthened or suppressed to different degrees according to the importance of different feature channels.

[0121] After obtaining the channel attention feature information after strengthening the channel attention, the deep speech feature information of the multiple speech frames is extracted based on the channel attention feature information, thereby giving the speech quality assessment network model better performance in extracting features and stronger generalization ability.

[0122] In step S103, based on the deep speech feature information of the multiple speech frames, a speech quality assessment network model is used to extract the time-series related feature information between the multiple speech frames.

[0123] Here, the time sequence-related feature information is at least used to indicate the correlation between deep speech feature information of multiple speech frames at different time sequences.

[0124] The speech quality assessment network model may include a recurrent neural network module, and the recurrent neural network module may be used to extract timing-related feature information between the multiple speech frames.

[0125] It can be understood that recurrent neural networks are suitable for processing and predicting important events in time series; using the recurrent neural network module, the time sequence related feature information of multiple speech frames of different time sequences is extracted based on the deep speech feature information of the speech frames.

[0126] In step S104, based on the time-series correlation feature information between multiple speech frames of the first speech signal, a speech quality assessment network model is used to predict the quality score of the first speech signal.

[0127] Based on the time-series related feature information, the change of the deep speech feature information of the multiple speech frames can be determined, thereby effectively predicting the trend of the deep speech feature information of the multiple speech frames changing over time, and determining the speech quality of the first speech signal; improving the accuracy of the speech quality assessment of the first speech signal.

[0128] The embodiment of the present disclosure obtains the spectrum information of a first noisy speech signal and a plurality of speech frames corresponding to the first speech signal, inputs the spectrum information of the plurality of speech frames corresponding to the first speech signal into a pre-trained speech quality assessment network model, and uses the speech quality assessment network model to extract channel attention feature information of the plurality of speech frames, thereby more effectively extracting more important deep speech feature information of the plurality of speech frames based on the channel attention feature information after channel attention enhancement;

[0129] In addition, based on the deep speech feature information of multiple speech frames, the timing-related feature information between the multiple speech frames is extracted, and the timing-related feature information that can reflect the changes in the deep speech feature information of the multiple speech frames is utilized. The mutual influence between the speech frames of different timings of the first speech signal is considered to more accurately predict the quality score of the first speech signal. There is no need to obtain a pure speech signal, thereby reducing the difficulty of applying speech quality assessment.

[0130] Optionally, the speech quality assessment network model includes: an effective channel attention ECA module, a multi-level residual convolution module and a bidirectional gated recurrent GRU module;

[0131] The step S102 extracts channel attention feature information of the multiple speech frames using a pre-trained speech quality assessment network model for the spectrum information of the multiple speech frames, and extracts deep speech feature information of the multiple speech frames based on the channel attention feature information of the multiple speech frames, including:

[0132] For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames;

[0133] Utilizing the multi-stage residual convolution module, extracting deep speech feature information based on the channel attention feature information;

[0134] The step S103 of extracting the time sequence related feature information between the plurality of speech frames based on the deep speech feature information includes:

[0135] The bidirectional GRU module is used to extract the time-series related feature information corresponding to the deep speech feature information.

[0136] In the disclosed embodiment, the speech quality assessment network model includes: an ECA module, a multi-level residual convolution module and a bidirectional GRU module;

[0137] The spectral information of multiple speech frames corresponding to the first speech signal is input into the ECA module of the speech quality assessment network model. The ECA module extracts channel attention information of multiple channels of the multiple speech frames based on the spectral information of the multiple speech frames; and determines channel attention feature information of multiple channels of the multiple speech frames based on the channel attention information and the spectral information of the multiple speech frames.

[0138] Here, the channel attention information is at least used to indicate the importance of the multiple channels.

[0139] It should be noted that the ECA module can realize local cross-channel information interaction through one-dimensional convolution with adaptive kernel size; wherein the kernel size k represents the coverage of local cross-channel interaction, that is, by considering each channel and its k neighbors to capture local cross-channel interaction, model the correlation between different channels, and automatically obtain the importance of each channel through network learning, and finally assign different weight coefficients to each channel to strengthen important features and suppress unimportant features.

[0140] It can be understood that the channel attention information may include: weight coefficients of the multiple channels; the ECA module performs channel attention enhancement processing on the spectral information of the multiple speech frames based on the weight coefficients of the multiple channels to obtain channel attention feature information of multiple channels of multiple speech frames.

[0141] The channel attention feature information of multiple channels of multiple speech frames output by the ECA module is input into the multi-level residual convolution module, and the multi-level residual convolution module is used to perform deep feature extraction on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0142] The disclosed embodiment utilizes a multi-level residual convolution module to increase the network depth of the speech quality assessment network model and improve the assessment performance of the network model; and utilizes the multi-level residual convolution module to implement skip-layer connections within the network structure, thereby reducing the problem of gradient vanishing due to excessive network depth and effectively improving the performance of feature extraction of the speech quality assessment network model.

[0143] It should be noted that the performance of convolutional neural networks is strongly related to the depth of the network. The deeper the network structure, the better the recognition effect. However, in practice, when the depth of the convolutional neural network reaches a certain depth, the model performance will no longer improve, or even become worse. This phenomenon is called gradient vanishing. Therefore, multi-level residual convolution modules are used to implement skip-layer connections, so that in deep convolutional neural networks, the output of certain layers can be directly passed to the subsequent layers across the intermediate layers, thereby reducing the problem of gradient vanishing caused by the excessive depth of the network and effectively improving network performance.

[0144] The deep speech feature information of the multiple speech frames output by the multiple residual convolution modules is input into a bidirectional GRU module, and the time sequence related feature information of the multiple speech frames is extracted by using the bidirectional GRU module.

[0145] It should be noted that the transmission direction of the hidden layer in the GRU module is unidirectional from front to back, that is, the current state is only related to the previous content (i.e., the forward temporal correlation feature). However, in speech quality assessment, the current state often needs to be combined with contextual information to be more effective.

[0146] The basic idea of ​​the bidirectional GRU module is to superimpose two unidirectional GRU sub-modules up and down to obtain the previous information (i.e., forward temporal correlation features) and the following information (i.e., backward temporal correlation features) of the current state respectively; and obtain the complete context information of the current state through information fusion.

[0147] The output dimension of the bidirectional GRU module is twice that of the unidirectional GRU module, so the bidirectional GRU module has stronger expressive power than the unidirectional GRU module.

[0148] Optionally, the multi-stage residual convolution module includes N residual convolution layers; wherein the residual convolution layer includes: a cascaded first convolution layer and a second convolution layer; the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit;

[0149] The multi-level residual convolution module using the speech quality assessment network model extracts deep speech feature information based on the channel attention feature information, including:

[0150] The N residual convolution layers are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0151] In the disclosed embodiment, the multi-stage residual convolution module includes: N residual convolution layers; and the N residual convolution layers are cascaded.

[0152] The residual convolution layer comprises: a cascaded first convolution layer and a second convolution layer;

[0153] It can be understood that the output of the first convolutional layer will be used as the input of the second convolutional layer. There may be feature loss in this process. The residual unit is used to transfer the input of the first convolutional layer (i.e., the original feature information) to the output of the first convolutional layer, and the input of the first convolutional layer and the output of the first convolutional layer are used together as the input of the second convolutional layer, thereby achieving feature supplementation.

[0154] In some embodiments, Figure 3 As shown, Figure 3 1 is a schematic diagram of a network structure of a residual convolution layer according to an exemplary embodiment, wherein the first convolution layer includes: two depth-wise separable convolution layers, and the second convolution layer includes: one depth-wise separable convolution layer.

[0155] The output of the first convolutional layer is added to the input of the first convolutional layer by using a residual unit, and the sum of the output and the input of the first convolutional layer is input to the second convolutional layer.

[0156] Here, the depthwise separable convolutional layer consists of a depthwise convolutional layer and a 1×1 point convolutional layer; the convolution kernel size of the depthwise convolutional layer is 3×3, and the moving steps are (1,1), (1,2) and (1,3) respectively; the activation function of each convolutional layer uses ReLU.

[0157] The depthwise separable convolutional layer will perform a spatial convolution operation while keeping the channels independent, that is, perform convolution operations on multiple different channels respectively to obtain feature information of multiple different channels; then perform a depthwise convolution operation on the feature information of multiple different channels to stack the feature information of multiple different channels together again. It can be understood that the depthwise separable convolutional layer uses the depthwise convolutional layer to achieve spatial convolution while keeping the channels independent.

[0158] In the disclosed embodiment, the N residual convolution layers are utilized to determine the spectral information after enhanced attention based on the channel attention feature information; and for the spectral information after enhanced attention, deep separable convolution processing is performed through the deep separable convolution layer within the residual convolution layer to obtain the deep speech feature information of the multiple speech frames.

[0159] On the one hand, the depth-wise separable convolutional layer is used to map the correlation and spatial correlation between multiple channels separately, thereby reducing the complexity and training parameters, improving the computational efficiency, and ensuring the lightweight of the network model. On the other hand, the residual unit in the residual convolutional layer is used to pass the input of the first convolutional layer to the second convolutional layer, realizing the combination with direct mapping and improving the feature extraction capability.

[0160] Optionally, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule;

[0161] The step of extracting the time-series-related feature information corresponding to the deep speech feature information by using the bidirectional GRU module includes:

[0162] Using the forward GRU submodule to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information;

[0163] Using the backward GRU submodule to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information;

[0164] Based on the forward time sequence feature information and the reverse time sequence feature information, the time sequence related feature information corresponding to the deep speech feature information is obtained.

[0165] In the embodiment of the present disclosure, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule; it can be understood that at each moment, the input amount of the bidirectional GRU module is simultaneously provided to the forward GRU submodule and the backward GRU submodule; and the output amount of the bidirectional GRU module is jointly determined by the forward GRU submodule and the backward GRU submodule.

[0166] It should be noted that the GRU submodule consists of an update gate and a reset gate; the update gate is used to control the extent to which the state information of the previous moment is brought into the current state; the larger the value of the update gate, the more state information of the previous moment is brought in. It can be understood that the update gate helps to capture long-term dependencies in time series.

[0167] The reset gate is used to control how much information in the previous state is written into the current candidate set; the smaller the reset gate, the less information in the previous state is written; it can be understood that the reset gate helps capture short-term dependencies in time series.

[0168] The deep speech feature information output by the multi-level residual convolution module is input into the bidirectional GRU module, and the forward GRU submodule is used to perform forward feature extraction on the deep speech feature to obtain forward time series feature information;

[0169] It is understandable that the forward GRU submodule can be used to perform forward GRU calculations on the deep speech feature information to obtain the forward time sequence feature information; here, the forward time sequence feature information is used to indicate the contextual relationship of the speech frame.

[0170] Performing reverse feature extraction on the deep speech features using a backward GRU submodule to obtain reverse time series feature information;

[0171] It can be understood that the backward GRU submodule can be used to perform backward GRU calculation on the deep speech feature information to obtain the backward time sequence feature information; here, the backward time sequence feature information is used to indicate the context relationship of the speech frame.

[0172] The forward time sequence feature information and the reverse time sequence feature information are fused to obtain the time sequence related feature information of the deep speech feature information.

[0173] Optionally, the speech quality assessment network model includes: a fully connected module and a global average pooling GAP module;

[0174] The step S104 of predicting the quality score of the first speech signal according to the time sequence-related feature information between the plurality of speech frames includes:

[0175] Using the fully connected module, performing fully connected processing on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames;

[0176] The GAP module is used to perform global averaging processing on the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

[0177] In the disclosed embodiment, the speech quality assessment network model further includes: a fully connected module and a global average pooling GAP module.

[0178] After the bidirectional GRU module outputs the time-series related feature information, the fully connected module is used to perform full-connection processing on the time-series related feature information to obtain quality scores corresponding to the multiple speech frames.

[0179] The GAP module is used to perform pooling processing on the quality scores corresponding to the multiple speech frames to obtain the quality score of the first speech signal.

[0180] It can be understood that the GAP module can globally average the quality scores corresponding to multiple speech frames of the first speech signal to obtain and output the quality score of the first speech signal.

[0181] The disclosed embodiment performs full-connection processing on the timing-related feature information through a fully-connected module, adds a frame-level quality score evaluation, and determines the quality score of the speech signal based on the frame-level quality score, so that the evaluation process is smoother and the error between the predicted score and the actual score is detected.

[0182] Optionally, the acquiring frequency spectrum information of a plurality of speech frames corresponding to the first speech signal includes:

[0183] Preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal;

[0184] Perform time-frequency conversion processing on multiple speech frames of the first speech signal to obtain amplitude spectrum information of the multiple speech frames.

[0185] In an embodiment of the present disclosure, a noisy first speech signal may be acquired through an audio acquisition device, and the first speech signal may be preprocessed to obtain a plurality of speech frames of the first speech signal.

[0186] Here, the preprocessing includes at least: frame processing. It can be understood that the frame processing refers to dividing the speech signal to be evaluated into multiple speech frames of fixed size. Generally speaking, since the speech signal has short-term stability, the frame length of a speech frame can be set to the short-term stability period of the speech signal, that is, the frame length is set to 10ms to 30ms.

[0187] It should be noted that the speech signal is a non-stationary signal that varies with time, but has short-term stability. Therefore, the speech signal can be framed into short speech frames so that the speech signal exhibits some characteristics of a periodic function.

[0188] However, in order to ensure the smoothness of the connection between adjacent voice frames, that is, to eliminate the signal discontinuity that may be caused at both ends of each voice frame, an overlapping frame method can be adopted.

[0189] After obtaining a plurality of speech frames of the first speech signal, time-frequency conversion processing is performed on the plurality of speech frames of the first speech signal to obtain amplitude spectrum information of the plurality of speech frames.

[0190] Here, the time-frequency conversion process may be a Fourier transform, which converts the time-domain speech frames into frequency-domain speech frames by performing a Fourier transform on the multiple speech frames. The Fourier transform may include: discrete Fourier transform, fast Fourier transform, short-time Fourier transform, etc.; it may be selected according to the specific scenario, and the embodiments of the present disclosure are not limited here.

[0191] In the embodiment of the present disclosure, the amplitude spectrum information of the plurality of speech frames can be obtained by performing discrete Fourier transform on the plurality of speech frames of the first speech signal; the transformation formula is as follows:

[0192]

[0193] Among them, the X m (k) is the amplitude spectrum feature information of the mth speech frame; m (n) is the mth speech frame, and N is the frame length.

[0194] In some embodiments, the method further comprises:

[0195] The amplitude spectrum information of the multiple speech frames is filtered to obtain amplitude spectrum feature information of the multiple speech frames; and the amplitude spectrum feature information of the multiple speech frames is input into the speech quality assessment network model.

[0196] In the disclosed embodiment, after Fourier transform is performed on each speech frame of the first speech signal, filtering processing may be performed using a Mel filter to obtain amplitude spectrum feature information corresponding to each speech frame.

[0197] It should be noted that Mel filtering is to filter the spectrum information only with a Mel-scale triangular filter; it can be understood that by performing Mel filtering on the spectrum information of the speech frame, the spectrum information of the filtered speech frame is more consistent with the auditory characteristics of the human ear.

[0198] Optionally, the preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal includes:

[0199] Performing pre-emphasis processing on the first speech signal;

[0200] The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

[0201] In the embodiment of the present disclosure, the preprocessing includes: pre-emphasis processing and frame windowing processing.

[0202] The first voice signal is pre-emphasized to enhance the high-frequency signal in the first voice signal and compensate for the amplitude of the high-frequency part of the first voice signal.

[0203] It should be noted that since most of the energy of the speech signal is concentrated in the low-frequency part, the signal-to-noise ratio of the high-frequency part of the speech signal decreases; by pre-emphasis processing of the speech signal, the suppressed high-frequency part of the speech signal can be compensated, making the spectrum of the speech signal flat and removing the spectrum tilt.

[0204] In some embodiments, the first speech signal may be pre-emphasized by a high-pass filter, and the function expression of the high-pass filter is:

[0205] H(z)=1-μz -1 ;

[0206] Wherein, the z is the first speech signal, and the μ is the pre-emphasis coefficient, which generally ranges from 0.9 to 1.

[0207] By performing pre-emphasis processing on the first voice signal, the high frequency part of the first voice signal is highlighted, which is helpful to reduce the attenuation loss of the first voice signal.

[0208] A plurality of continuous speech frames are obtained by performing frame division and windowing processing on the first speech signal after the pre-emphasis processing.

[0209] Frame windowing can be achieved by weighting the speech signal with a movable fixed-length window function. In order to ensure the short-term nature of the processed speech signal, windowing can reduce the phenomenon of spectrum leakage of the speech signal, that is, except for the main lobe that should be there, there will be no side lobes that will be mistaken for false peaks, and the result will not have frequency components that did not exist originally.

[0210] In some embodiments, the speech signal after frame segmentation and windowing processing can be expressed as:

[0211] x m (n)=w(n)x(m+n),0≤n≤N-1;

[0212] Among them, the x m (n) is the mth speech frame, w(n) is the window function, x(n) is the speech signal, and N is the frame length.

[0213] It should be noted that, since the narrower the main lobe of the speech signal is, the smaller the side lobe is and the faster it decays, the better, windowing the speech signal can reduce spectrum energy leakage. The rectangular window commonly used in signal processing has a narrow main lobe, but a large side lobe and decays slowly, so the embodiment of the present disclosure can use a Hamming window with a slightly wider main lobe, but a smaller side lobe and a faster decay to perform frame windowing processing on the speech signal.

[0214] The formula for the Hamming window is as follows:

[0215]

[0216] The first speech signal is subjected to frame division and windowing processing by using a Hamming window, thereby facilitating obtaining a plurality of continuous speech frames, so as to eliminate the influence of signal discontinuity that may be caused at both ends of each speech frame.

[0217] Optionally, the speech quality assessment network model comprises: a rearrangement module, configured to perform dimensional transformation processing on input spectrum information and / or deep speech feature information;

[0218] The method of extracting channel attention feature information of each channel of the plurality of speech frames by using the ECA module for the spectrum information of the plurality of speech frames includes:

[0219] For the spectrum information of the multiple speech frames, the spectrum information is dimensionally transformed by the rearrangement module, and the spectrum information after the dimension transformation is input into the ECA module to extract the channel attention feature information of each channel of the multiple speech frames; wherein the dimension of the spectrum information of the multiple speech frames after the dimension transformation matches the input dimension of the ECA module;

[0220] The step of extracting the time-series-related feature information corresponding to the deep speech feature information by using the bidirectional GRU module includes:

[0221] The deep speech feature information is dimensionalized by using the rearrangement module, and the deep speech feature information after the dimensionalization is input into the bidirectional GRU module to extract the timing-related feature information corresponding to the deep speech feature information after the dimensionalization; wherein the dimension of the deep speech feature information after the dimensionalization matches the input dimension of the bidirectional GRU module.

[0222] In an embodiment of the present disclosure, the speech quality assessment network model may include two rearrangement modules; the rearrangement module may be applied before the ECA module and between the multi-stage residual convolution module and the bidirectional GRU module.

[0223] The rearrangement module is used to perform dimension transformation processing on the input spectrum information and / or deep speech feature information so that the output dimension of the previous module (i.e., the rearrangement module) matches the input dimension of the subsequent module (i.e., the ECA module and the bidirectional GRU module).

[0224] Optionally, before acquiring frequency spectrum information of a plurality of speech frames corresponding to the first speech signal, the method includes:

[0225] Obtaining a training sample set of noisy speech and a quality score label of the training sample set;

[0226] Inputting the frequency spectrum information of speech frames corresponding to multiple noisy speech signals in the training sample set into the initial network model to be trained, and obtaining the prediction quality score output by the initial network model;

[0227] Determining a loss function value of the initial network model according to the predicted quality score and the quality score label;

[0228] Based on the loss function value, the parameters to be trained of the initial network model are adjusted to obtain the speech quality assessment network model.

[0229] In the disclosed embodiment, the training sample set includes a plurality of noisy speech signals for adjusting the parameters to be trained in the initial network model; the quality score label includes at least: a subjective quality score corresponding to the noisy speech signal.

[0230] After obtaining the training sample set, the multiple noisy speech signals in the training sample set can be preprocessed and time-frequency converted to obtain spectral information of speech frames corresponding to the multiple noisy speech signals; the initial network model is trained in a fully supervised learning manner using the spectral information of speech frames corresponding to the multiple noisy speech signals and the quality score labels of the multiple noisy speech signals.

[0231] The spectral information of the speech frames corresponding to the multiple noisy speech signals can be input into the initial network model, the channel attention feature information can be extracted using the ECA module of the initial network model, and the deep speech feature information of the multiple speech frames of the noisy speech signal can be extracted based on the channel attention feature information using the multi-level residual convolution module of the initial network model; the deep speech feature information is input into the bidirectional GRU module, and the timing-related feature information between the multiple speech frames is extracted using the bidirectional GRU module; finally, the timing-related feature information is fully connected and average-pooled through the fully connected module and the GAP module to obtain the predicted quality scores of the multiple noisy speech signals.

[0232] According to the difference between the predicted quality scores of multiple noisy speech signals and the quality score labels of the multiple noisy speech signals, the loss function value of the initial network model is determined, the back propagation gradient is calculated according to the loss function value, the parameters to be trained in the initial network model are updated, and then the above steps are repeated until the number of iterations is reached or the loss function converges, so as to obtain a speech quality assessment network model.

[0233] It can be understood that the embodiment of the present disclosure uses a noisy speech signal with a quality score label including a subjective quality score to train a network model. The trained speech quality assessment network model performs speech quality assessment on the speech signal. The obtained predicted quality score has a high correlation with the subjective quality score, and the speech quality assessment performance is good.

[0234] Optionally, inputting the spectrum information of speech frames corresponding to the multiple noisy speech signals in the training sample set into the initial network model to be trained to obtain the prediction quality score output by the initial network model includes:

[0235] Determining a maximum number of frames of the multiple noisy speech signals based on frequency spectrum information of the multiple speech frames of the multiple noisy speech signals;

[0236] Determining whether the number of frames of the frequency spectrum information of the multiple noisy speech signals is the maximum number of frames;

[0237] If the number of frames of the frequency spectrum information of the multiple noisy speech signals is not the maximum number of frames, the frequency spectrum information is padded with zeros; the number of frames of the frequency spectrum information after the zero-padding process is the maximum number of frames;

[0238] The spectrum information after the zero-padding process is input into the initial network model to obtain a prediction quality score output by the initial network model.

[0239] In the embodiment of the present disclosure, during the training phase of the network model, batch training is performed on multiple noisy speech signals in a training sample set. Considering that there may be differences in the lengths of multiple noisy speech signals, in order to match the dimensions of the spectral information of the speech frames corresponding to the multiple noisy speech signals, a maximum number of frames can be determined from the multiple noisy speech signals. It is also determined whether the number of frames of the spectral information of the multiple noisy speech signals is the maximum number of frames. If there is a noisy speech signal whose number of frames of spectral information is less than the maximum number of frames, zero padding is performed on the spectral information of the noisy speech signal, so that the dimensions of the spectral information of the speech frames corresponding to the multiple noisy speech signals after the zero padding are the same.

[0240] The present disclosure also provides the following embodiments:

[0241] Figure 4 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 2 ,like Figure 4 As shown, the method comprises:

[0242] Step S201, obtaining a training sample set of noisy speech and a quality score label of the training sample set;

[0243] In this example, the training sample set includes a plurality of noisy speech signals, and each of the noisy speech signals corresponds to a quality score label.

[0244] Here, the quality score label is at least used to indicate the subjective quality average score corresponding to the noisy speech signal.

[0245] By obtaining a training sample set containing a plurality of noisy speech signals and quality score labels corresponding to the training sample set, the network model is trained in a fully supervised manner using the training sample set and the quality score labels.

[0246] Step S202, based on the spectrum information of speech frames corresponding to the multiple noisy speech signals in the training sample set, determine the maximum number of frames of the multiple noisy speech signals; determine whether the number of frames of the spectrum information of the multiple noisy speech signals is the maximum number of frames; if the number of frames of the spectrum information of the multiple noisy speech signals is not the maximum number of frames, perform zero padding on the spectrum information; the number of frames of the spectrum information after the zero padding is the maximum number of frames;

[0247] In this example, after obtaining the training sample set, it is necessary to perform batch preprocessing on the multiple noisy speech signals in the training sample set to obtain speech frames corresponding to the multiple noisy speech signals; and based on the speech frames corresponding to the multiple noisy speech signals, extract the spectral information of the speech frames corresponding to the multiple noisy speech signals.

[0248] Taking into account that there may be differences in the lengths of multiple noisy speech signals, when training the network model, batch training is performed on multiple noisy speech signals in the training sample set. Therefore, in order to unify the dimensions of the spectral information of the speech frames corresponding to the multiple noisy speech signals, the maximum number of frames can be determined from the multiple noisy speech signals in the same training batch; and the speech signals that do not meet the maximum number of frames in the multiple noisy speech signals are padded with zeros, so that the spectral information dimensions of the speech frames corresponding to the multiple noisy speech signals after the zero-padded processing are the same.

[0249] Step S203, inputting the spectrum information after the zero-padding process into the initial network model to obtain a prediction quality score output by the initial network model;

[0250] In this example, the frequency spectrum information of the multiple noisy speech signals after zero padding is input into the initial network model to obtain the predicted quality scores corresponding to the multiple noisy speech signals output by the initial network model.

[0251] Step S204, determining the loss function value of the initial network model according to the predicted quality score and the quality score label; adjusting the parameters to be trained of the initial network model based on the loss function value to obtain the speech quality assessment network model;

[0252] In this example, the loss function value of the initial network model can be determined based on the predicted quality scores of the multiple noisy speech signals output by the initial network model and the quality score labels corresponding to the multiple noisy speech signals; by judging whether the loss function value meets the stop training condition, if the loss function value does not meet the stop training condition, the parameters to be trained of the initial network model are optimized, and the above training steps are repeated until the loss function value meets the stop training condition, and a speech quality assessment network model can be obtained; subsequently, the trained speech quality assessment network model can be directly used to perform speech quality assessment on the speech signal.

[0253] Step S205, obtaining a first speech signal, preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal; performing time-frequency conversion processing on the plurality of speech frames of the first speech signal to obtain amplitude spectrum information of the plurality of speech frames;

[0254] In some embodiments, the preprocessing of the first speech signal to obtain a plurality of speech frames of the first speech signal includes:

[0255] Performing pre-emphasis processing on the first speech signal;

[0256] The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

[0257] Step S206, inputting the amplitude spectrum information of the multiple speech frames into a pre-trained speech quality assessment network model, and using the ECA module of the speech quality assessment network model to extract channel attention feature information of each channel of the multiple speech frames;

[0258] In this example, the ECA module completes cross-channel information interaction in a very lightweight manner, determines the importance of each feature channel, and obtains channel attention feature information after strengthening channel attention, so that the network model has better performance in extracting features and stronger generalization ability.

[0259] In some embodiments, the speech quality assessment network model also includes: a rearrangement module; located before the ECA module; used to perform dimensionality transformation processing on the input spectral information, and the dimension of the spectral information of the multiple speech frames after the dimensionality transformation processing matches the input dimension of the ECA module.

[0260] Step S207, using the multi-level residual convolution module of the speech quality assessment network model to extract deep speech feature information based on the channel attention feature information;

[0261] In this example, the multi-stage residual convolution module includes N residual convolution layers;

[0262] The residual convolution layer comprises: a cascaded first convolution layer and a second convolution layer; wherein the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit;

[0263] The N residual convolution layers are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0264] In this example, the residual convolution layer is used to perform deep separable convolution processing on the channel attention feature information, and the correlation and spatial correlation between feature channels are mapped separately, thereby reducing the computational complexity of the network model and making the network model more lightweight.

[0265] And considering the problem of gradient disappearance caused by the excessive network depth of the multi-level residual convolution module, the residual unit is used to pass the input of the first convolution layer in the residual convolution layer to the second convolution layer, achieving the result of direct mapping and improving the feature extraction ability of the network model.

[0266] Step S208, using the bidirectional GRU module of the speech quality assessment network model to extract the time-series related feature information corresponding to the deep speech feature information;

[0267] In this example, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule;

[0268] The forward GRU submodule can be used to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information; the backward GRU submodule can be used to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information; based on the forward time series feature information and the reverse time series feature information, the time series-related feature information corresponding to the deep speech feature information is obtained.

[0269] Here, the dropout parameters of the forward GRU submodule and the backward GRU submodule are both set to 0.35.

[0270] In this example, the bidirectional GRU module is formed by combining the forward GRU submodule and the backward GRU submodule, the forward GRU submodule is used to obtain the forward time sequence feature information (i.e., the above information), and the backward GRU submodule is used to obtain the reverse time sequence feature information (i.e., the following information); by fusing the forward time sequence feature information and the reverse time sequence feature information, the complete context information of the multiple speech frames is obtained.

[0271] It can be understood that the use of a bidirectional GRU module can more simply and effectively learn the bidirectional correlation at the time step and model the bidirectional time dependency of the perceived speech quality.

[0272] In some embodiments, the rearrangement module is also located between the multi-level residual convolution module and the bidirectional GRU module, and is used to perform dimensionality transformation processing on the input deep speech feature information, and the dimension of the deep speech feature information after the dimensionality transformation matches the input dimension of the bidirectional GRU module.

[0273] Step S209, using the fully connected module of the speech quality assessment network model, fully connected processing is performed on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames.

[0274] In this example, a quality score is determined on a frame-by-frame basis using a fully connected module to obtain a quality score for each speech frame in the first speech signal.

[0275] Step S210: Using the GAP module of the speech quality assessment network model, globally average the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

[0276] It should be noted that the GAP module is usually used to perform averaging processing on the elements on the entire feature map; in this example, the GAP module performs global averaging processing on the quality scores of multiple speech frames of the first speech signal to obtain the overall quality score of the first speech signal and output it.

[0277] It can be understood that the method of using a speech quality assessment network model to perform speech quality assessment can be divided into two stages: a network model training stage and a speech quality assessment stage; each stage requires preprocessing and time-frequency conversion of the noisy speech signal to obtain the required spectrum information.

[0278] like Figure 5 As shown, Figure 5 A flow chart of a method for evaluating speech quality according to an exemplary embodiment is shown. Figure 3 In the network model training phase, the network model is trained based on the training sample set and the corresponding quality score labels to obtain the model parameters of the network model that make the loss function converge; and based on the model parameters, the speech quality assessment network model in the speech quality assessment phase is determined, such as Figure 6 As shown, Figure 6 The figure is a structural diagram of a speech quality assessment network model according to an exemplary embodiment.

[0279] In the speech quality assessment stage, the noisy speech signal to be evaluated is input into the trained speech quality assessment network model to obtain the quality score output by the speech quality assessment network model.

[0280] The present disclosure also provides a speech quality assessment device. Figure 7 As shown, Figure 7 1 is a schematic diagram of a structure of a speech quality assessment device according to an exemplary embodiment. The device 100 comprises:

[0281] An acquisition module 101 is used to acquire frequency spectrum information of a plurality of speech frames corresponding to a first speech signal;

[0282] A first feature extraction module 102 is used to extract channel attention feature information of the multiple speech frames using a pre-trained speech quality assessment network model for the spectrum information of the multiple speech frames, and extract deep speech feature information of the multiple speech frames based on the channel attention feature information of the multiple speech frames;

[0283] A second feature extraction module 103 is used to extract time-sequence related feature information between the multiple speech frames based on the deep speech feature information;

[0284] The prediction module 104 is configured to predict the quality score of the first speech signal according to the time sequence related feature information between the plurality of speech frames.

[0285] Optionally, the speech quality assessment network model includes: an effective channel attention ECA module, a multi-level residual convolution module and a bidirectional gated recurrent GRU module;

[0286] The first feature extraction module 102 is used to:

[0287] For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames;

[0288] Utilizing the multi-stage residual convolution module, extracting deep speech feature information based on the channel attention feature information;

[0289] The extracting the time sequence related feature information between the plurality of speech frames based on the deep speech feature information comprises:

[0290] The bidirectional GRU module is used to extract the time-series related feature information corresponding to the deep speech feature information.

[0291] Optionally, the multi-stage residual convolution module includes N residual convolution layers;

[0292] The residual convolution layer comprises: a cascaded first convolution layer and a second convolution layer; wherein the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit;

[0293] The N residual convolution layers are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

[0294] Optionally, the bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule;

[0295] The second feature extraction module 103 is used to:

[0296] Using the forward GRU submodule to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information;

[0297] Using the backward GRU submodule to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information;

[0298] Based on the forward time sequence feature information and the reverse time sequence feature information, the time sequence related feature information corresponding to the deep speech feature information is obtained.

[0299] Optionally, the speech quality assessment network model includes: a fully connected module and a global average pooling GAP module;

[0300] The prediction module 104 is used to:

[0301] Using the fully connected module, performing fully connected processing on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames;

[0302] The GAP module is used to perform global averaging processing on the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

[0303] Optionally, the acquisition module 101 is used to:

[0304] Preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal;

[0305] Perform time-frequency conversion processing on multiple speech frames of the first speech signal to obtain amplitude spectrum information of the multiple speech frames.

[0306] Optionally, the acquisition module 101 is used to:

[0307] Performing pre-emphasis processing on the first speech signal;

[0308] The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

[0309] Optionally, the device includes: a network training module 105, which is used to:

[0310] Obtaining a training sample set of noisy speech and a quality score label of the training sample set;

[0311] Inputting the frequency spectrum information of speech frames corresponding to multiple noisy speech signals in the training sample set into the initial network model to be trained, and obtaining the prediction quality score output by the initial network model;

[0312] Determining a loss function value of the initial network model according to the predicted quality score and the quality score label;

[0313] Based on the loss function value, the parameters to be trained of the initial network model are adjusted to obtain the speech quality assessment network model.

[0314] Optionally, the network training module 105 is used to:

[0315] Determining a maximum number of frames of the multiple noisy speech signals based on frequency spectrum information of the multiple speech frames of the multiple noisy speech signals;

[0316] Determining whether the number of frames of the frequency spectrum information of the multiple noisy speech signals is the maximum number of frames;

[0317] If the number of frames of the frequency spectrum information of the multiple noisy speech signals is not the maximum number of frames, the frequency spectrum information is padded with zeros; the number of frames of the frequency spectrum information after the zero-padding process is the maximum number of frames;

[0318] The spectrum information after the zero-padding process is input into the initial network model to obtain a prediction quality score output by the initial network model.

[0319] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0320] Figure 8 2 is a block diagram of a speech quality assessment device according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a mobile computer, etc.

[0321] Reference Figure 8 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0322] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0323] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0324] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0325] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0326] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0327] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0328] The sensor assembly 214 includes one or more sensors for providing various aspects of status assessment for the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0329] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0330] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0331] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0332] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0333] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for evaluating speech quality, It is characterized in that The method comprises: Acquire frequency spectrum information of multiple speech frames corresponding to the first speech signal; For the frequency spectrum information of the multiple speech frames, using a pre-trained speech quality assessment network model, extracting channel attention feature information of the multiple speech frames, and based on the channel attention feature information of the multiple speech frames, extracting deep speech feature information of the multiple speech frames; Based on the deep speech feature information, extracting time-sequence related feature information between the multiple speech frames; Predicting a quality score of the first speech signal according to the timing-related feature information between the plurality of speech frames; The speech quality assessment network model includes: an effective channel attention ECA module and a multi-level residual convolution module; wherein the multi-level residual convolution module includes N residual convolution layers; the residual convolution layer includes: a cascaded first convolution layer and a second convolution layer; the first convolution layer includes two depth-separable convolution layers, and the second convolution layer includes one depth-separable convolution layer; the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit; The method further comprises: extracting channel attention feature information of the plurality of speech frames by using a pre-trained speech quality assessment network model for the spectrum information of the plurality of speech frames, and extracting deep speech feature information of the plurality of speech frames based on the channel attention feature information of the plurality of speech frames, including: For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames; The N residual convolution layers in the multi-stage residual convolution module are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

2. The method according to claim 1, It is characterized in that The speech quality assessment network model further includes: a bidirectional GRU module; The extracting the time sequence related feature information between the plurality of speech frames based on the deep speech feature information comprises: The bidirectional GRU module is used to extract the time-series related feature information corresponding to the deep speech feature information.

3. The method according to claim 2, It is characterized in that The bidirectional GRU module includes: a forward GRU submodule and a backward GRU submodule; The step of extracting the time-series-related feature information corresponding to the deep speech feature information by using the bidirectional GRU module includes: Using the forward GRU submodule to perform forward feature extraction on the deep speech feature information to obtain forward time series feature information; Using the backward GRU submodule to perform reverse feature extraction on the deep speech feature information to obtain reverse time series feature information; Based on the forward time sequence feature information and the reverse time sequence feature information, the time sequence related feature information corresponding to the deep speech feature information is obtained.

4. The method according to claim 1, It is characterized in that The speech quality assessment network model includes: a fully connected module and a global average pooling GAP module; The predicting the quality score of the first speech signal according to the time sequence related feature information between the multiple speech frames includes: Using the fully connected module, performing fully connected processing on the time sequence related feature information of multiple speech frames of the first speech signal to obtain quality scores of the multiple speech frames; The GAP module is used to perform global averaging processing on the quality scores of the multiple speech frames to obtain a quality score of the first speech signal.

5. The method according to claim 1, It is characterized in that The acquiring of frequency spectrum information of a plurality of speech frames corresponding to the first speech signal includes: Preprocessing the first speech signal to obtain a plurality of speech frames of the first speech signal; Perform time-frequency conversion processing on multiple speech frames of the first speech signal to obtain amplitude spectrum information of the multiple speech frames.

6. The method according to claim 5, It is characterized in that The preprocessing of the first speech signal to obtain a plurality of speech frames of the first speech signal includes: Performing pre-emphasis processing on the first speech signal; The first speech signal after the pre-emphasis processing is subjected to frame division and windowing processing to obtain a plurality of speech frames of the first speech signal.

7. The method according to claim 1, It is characterized in that Before acquiring frequency spectrum information of a plurality of speech frames corresponding to the first speech signal, the method includes: Obtaining a training sample set of noisy speech and a quality score label of the training sample set; Inputting the frequency spectrum information of speech frames corresponding to multiple noisy speech signals in the training sample set into the initial network model to be trained, and obtaining the prediction quality score output by the initial network model; Determining a loss function value of the initial network model according to the predicted quality score and the quality score label; Based on the loss function value, the parameters to be trained of the initial network model are adjusted to obtain the speech quality assessment network model.

8. The method according to claim 7, It is characterized in that The step of inputting the spectrum information of speech frames corresponding to the multiple noisy speech signals in the training sample set into the initial network model to be trained to obtain the prediction quality score output by the initial network model includes: Determining a maximum number of frames of the multiple noisy speech signals based on frequency spectrum information of the multiple speech frames of the multiple noisy speech signals; Determining whether the number of frames of the frequency spectrum information of the multiple noisy speech signals is the maximum number of frames; If the number of frames of the frequency spectrum information of the multiple noisy speech signals is not the maximum number of frames, the frequency spectrum information is padded with zeros; the number of frames of the frequency spectrum information after the zero-padding process is the maximum number of frames; The spectrum information after the zero-padding process is input into the initial network model to obtain a prediction quality score output by the initial network model.

9. A speech quality assessment device, It is characterized in that include: An acquisition module, used to acquire frequency spectrum information of a plurality of speech frames corresponding to the first speech signal; A first feature extraction module is used to extract channel attention feature information of the multiple speech frames using a pre-trained speech quality assessment network model for the spectrum information of the multiple speech frames, and extract deep speech feature information of the multiple speech frames based on the channel attention feature information of the multiple speech frames; A second feature extraction module, configured to extract time-series related feature information between the plurality of speech frames based on the deep speech feature information; A prediction module, configured to predict a quality score of the first speech signal based on the time sequence-related feature information between the plurality of speech frames; The speech quality assessment network model includes: an effective channel attention ECA module and a multi-level residual convolution module; wherein the multi-level residual convolution module includes N residual convolution layers; the residual convolution layer includes: a cascaded first convolution layer and a second convolution layer; the first convolution layer includes two depth-separable convolution layers, and the second convolution layer includes one depth-separable convolution layer; the input of the first convolution layer is transmitted to the output of the first convolution layer through a residual unit; The first feature extraction module is also used for: For the frequency spectrum information of the multiple speech frames, using the ECA module, extracting channel attention feature information of each channel of the multiple speech frames; The N residual convolution layers in the multi-stage residual convolution module are used to perform deep separable convolution processing on the channel attention feature information to obtain deep speech feature information of the multiple speech frames.

10. A speech quality assessment device, It is characterized in that include: processor; a memory for storing executable instructions; The processor is configured to implement the speech quality assessment method according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a speech quality assessment device, the speech quality assessment device is enabled to perform the speech quality assessment method according to any one of claims 1 to 8.