Teaching-aided speech analysis teaching audio evaluation method and system

Abstract

The invention discloses a teaching-aided speech analysis teaching audio evaluation method and system, and belongs to the technical field of speech processing. Teaching audio voice signals are segmented into a plurality of sub-segments through a sliding window, wavelet decomposition and reconstruction are carried out on each sub-segment, and low-frequency, medium-high-frequency and high-frequency components are obtained. On the basis of the time domain energy of the three components, consonant-vowel prominence, resonance intensity and voice massiveness are obtained, and a time domain feature vector is formed; obtaining low-frequency, medium-high-frequency and high-frequency energy dispersivity according to the energy dispersivity, and forming a frequency domain feature vector; and calculating a fluctuation coefficient through the time-frequency feature vectors of the adjacent sub-segments, and constructing time domain and frequency domain fluctuation vectors. And splicing the time-frequency feature vectors of the sub-segments into a matrix, extracting features, performing enhancement in combination with a fluctuation vector to obtain time-domain and frequency-domain enhanced features, inputting the time-domain and frequency-domain enhanced features into a dual-channel neural network, and finally outputting a teaching audio quality score. According to the invention, the teaching audio quality evaluation precision is improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to a speech analysis teaching audio evaluation method and system for assisting teaching. Background Technology

[0002] In the field of online education, instructional audio is the core carrier of knowledge transmission, and its quality directly affects learning outcomes. Unlike general audio, instructional audio contains a large amount of colloquial expressions and is susceptible to interference from factors such as environmental noise and differences in audio pickup equipment.

[0003] Currently, speech processing algorithms mostly assess audio quality by extracting basic features such as short-time energy and zero-crossing rate of speech signals. However, evaluating the quality of teaching audio requires a focus on key dimensions such as pronunciation clarity and expressiveness. These traditionally extracted features are difficult to accurately match the characteristics of teaching speech and cannot effectively capture important information. Furthermore, algorithms lack effective capture of the temporal dynamic changes in speech quality during explanations, resulting in an inability to reflect the true quality of teaching speech and a problem of low accuracy in teaching audio quality assessment. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the prior art, the present invention provides a speech analysis teaching audio evaluation method and system to assist teaching, which solves the problem of low accuracy in teaching audio quality evaluation in the prior art.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a speech analysis teaching audio evaluation method for auxiliary teaching, comprising the following steps:

[0006] The audio signal of the teaching audio is segmented using a sliding window to obtain multiple sub-segments. Wavelet decomposition and reconstruction are performed on each sub-segment to obtain low-frequency components, mid-high frequency components, and high-frequency components.

[0007] Based on the temporal energy of the three components, consonant-vowel prominence, resonance intensity, and speech weight are obtained to form a temporal feature vector;

[0008] Based on the energy dispersion of the three components, the low-frequency energy dispersion, mid-to-high-frequency energy dispersion, and high-frequency energy dispersion are obtained, forming a frequency domain feature vector;

[0009] Based on the time-domain and frequency-domain feature vectors of adjacent sub-segments, the fluctuation coefficients are calculated to form the time-domain fluctuation vector and the frequency-domain fluctuation vector.

[0010] The time-domain feature vectors of multiple segments are concatenated to obtain the time-domain feature matrix, and the frequency-domain feature vectors of multiple segments are concatenated to obtain the frequency-domain feature matrix.

[0011] Features are extracted from the time-domain feature matrix and the frequency-domain feature matrix. Feature enhancement is performed based on the time-domain fluctuation vector and the frequency-domain fluctuation vector to obtain time-domain enhanced features and frequency-domain enhanced features. These are then input into a dual-channel neural network to obtain a teaching audio quality score.

[0012] Furthermore, the process of obtaining low-frequency components, mid-to-high-frequency components, and high-frequency components includes:

[0013] Perform the first-level wavelet decomposition on each segment to obtain the first-level approximation coefficients and the first-level detail coefficients;

[0014] The first-level approximation coefficients are decomposed into second-level wavelet coefficients to obtain the second-level approximation coefficients and the second-level detail coefficients.

[0015] The low-frequency components are obtained by wavelet reconstruction of the second-level approximation coefficients.

[0016] The second-level detail coefficients are reconstructed using wavelet reconstruction to obtain the mid-to-high frequency components;

[0017] The high-frequency components are obtained by wavelet reconstruction of the first-level detail coefficients.

[0018] Furthermore, the process of constructing the time-domain feature vector includes:

[0019] The time-domain energy of the low-frequency component, mid-high frequency component, and high-frequency component is calculated respectively to obtain the low-frequency energy, mid-high frequency energy, and high-frequency energy.

[0020] Add the mid-to-high frequency energy and the high frequency energy, subtract the low frequency energy, and normalize to obtain the consonant-vowel prominence.

[0021] The ratio of low-frequency energy to mid-to-high-frequency energy is used as the resonance intensity;

[0022] The ratio of low-frequency energy to high-frequency energy is used as the depth of speech.

[0023] The consonant-vowel prominence, resonance intensity, and speech weight are used to construct a temporal feature vector.

[0024] Furthermore, the formula for consonant-vowel prominence is:

[0025] ,

[0026] Where θ is the consonant-vowel prominence, E H For high-frequency energy, E M For medium to high frequency energy, E L Low-frequency energy, is the coefficient in the denominator.

[0027] Furthermore, the formula for calculating the energy dispersion is:

[0028] ,

[0029] Where, γ i E represents the energy dispersion. i,k Let γ1 be the energy at the k-th frequency point in the spectrum, K be the number of frequency points in the spectrum, and k be the frequency point number. When i is 1, γ1 is the low-frequency energy dispersion; when i is 2, γ2 is the mid-to-high frequency energy dispersion; and when i is 3, γ3 is the high-frequency energy dispersion.

[0030] Furthermore, the process of constructing the time-domain fluctuation vector includes:

[0031] Taking each sub-segment as the center, extract N sub-segments to obtain a set of sub-segments, where N is a positive integer;

[0032] Extract the elements from the temporal feature vector of each segment in the segment set;

[0033] Calculate the standard deviation and mean for each element, and use the ratio of the standard deviation to the mean as the fluctuation value of the corresponding element;

[0034] The average value of the fluctuation values ​​corresponding to the three elements in the time-domain feature vector is taken to obtain the time-domain fluctuation coefficient of the central segment. The time-domain fluctuation coefficients of each segment are then used to construct the time-domain fluctuation vector.

[0035] Furthermore, the process of constructing the frequency domain fluctuation vector includes:

[0036] Taking each sub-segment as the center, extract N sub-segments to obtain a set of sub-segments, where N is a positive integer;

[0037] Extract the elements from the frequency domain feature vector of each segment in the segment set;

[0038] Calculate the standard deviation and mean for each element, and use the ratio of the standard deviation to the mean as the fluctuation value of the corresponding element;

[0039] The average value of the fluctuation values ​​corresponding to the three elements in the frequency domain feature vector is taken to obtain the frequency domain fluctuation coefficient of the central segment. The frequency domain fluctuation coefficients of each segment are then used to construct the frequency domain fluctuation vector.

[0040] Furthermore, the process of obtaining time-domain enhancement features and frequency-domain enhancement features includes:

[0041] Both the time-domain feature matrix and the frequency-domain feature matrix are processed through a 1×1 first convolutional layer to extract features, resulting in 3×M time-domain features and 3×M frequency-domain features, where 3 is the number of columns and M is the number of rows.

[0042] The 3×M time-domain features and 3×M frequency-domain features are both extracted from each row through a 3×1 second convolutional layer to obtain 1×M time-domain features and 1×M frequency-domain features.

[0043] Both the time-domain fluctuation vector and the frequency-domain fluctuation vector are processed through a 1×1 third convolutional layer to extract features, resulting in 1×M time-domain fluctuation features and 1×M frequency-domain fluctuation features.

[0044] Multiplying the 1×M time-domain features element-wise with the 1×M time-domain fluctuation features yields the time-domain enhancement features;

[0045] Multiplying the 1×M frequency domain features element-wise with the 1×M frequency domain fluctuation features yields the frequency domain enhancement features.

[0046] Furthermore, the dual-channel neural network includes: a first reshaping layer, a second reshaping layer, a first CNN network, a second CNN network, and a fully connected layer;

[0047] The input of the first reshaping layer is used to input temporal enhancement features;

[0048] The input of the second reshaping layer is used to input frequency domain enhancement features;

[0049] The input of the first CNN network is connected to the output of the first reshaping layer;

[0050] The input of the second CNN network is connected to the output of the second reshaping layer;

[0051] The input of the fully connected layer is connected to the output of the first CNN network and the output of the second CNN network, respectively, and its output serves as the output of the dual-channel neural network.

[0052] An audio evaluation system for speech analysis teaching that assists in instruction includes: a component extraction unit, a time-domain feature extraction unit, a frequency-domain feature extraction unit, a wave vector composition unit, a splicing unit, and a scoring unit;

[0053] The component extraction unit is used to segment the speech signal of the teaching audio using a sliding window to obtain multiple sub-segments. Each sub-segment is then decomposed and reconstructed to obtain low-frequency components, mid-to-high-frequency components, and high-frequency components.

[0054] The temporal feature extraction unit is used to obtain consonant-vowel prominence, resonance intensity, and speech weight based on the temporal energy of the three components, thus forming a temporal feature vector;

[0055] The frequency domain feature extraction unit is used to obtain low-frequency energy dispersion, mid-to-high frequency energy dispersion, and high-frequency energy dispersion based on the energy dispersion of the three components, and to form a frequency domain feature vector;

[0056] The fluctuation vector building unit is used to calculate the fluctuation coefficient based on the time-domain feature vector and frequency-domain feature vector of adjacent sub-segments, and to form the time-domain fluctuation vector and frequency-domain fluctuation vector.

[0057] The splicing unit is used to splice the time-domain feature vectors of multiple segments to obtain a time-domain feature matrix, and to splice the frequency-domain feature vectors of multiple segments to obtain a frequency-domain feature matrix.

[0058] The scoring unit is used to extract features from the time-domain feature matrix and the frequency-domain feature matrix, and to enhance the features based on the time-domain fluctuation vector and the frequency-domain fluctuation vector to obtain time-domain enhanced features and frequency-domain enhanced features. These features are then input into a dual-channel neural network to obtain a teaching audio quality score.

[0059] The beneficial effects of this invention are as follows:

[0060] 1. This invention uses wavelet decomposition to obtain low-frequency, mid-high-frequency, and high-frequency components, extracting temporal features such as consonant-vowel prominence (reflecting pronunciation clarity), resonance intensity (characterizing chest resonance quality), and speech weight (quantifying low-frequency dominance). This directly addresses key requirements in teaching scenarios such as "clear pronunciation" and "composed explanation." Furthermore, by combining the energy dispersion of each frequency band (e.g., consonants are more prominent when mid-high frequency dispersion is low), it characterizes the spectral distribution of speech in the frequency domain, overcoming the shortcomings of traditional short-time energy and zero-crossing rate features in capturing professional information about teaching speech.

[0061] 2. This invention constructs time-domain and frequency-domain fluctuation vectors by calculating the fluctuation coefficients of adjacent segments, effectively capturing the temporal dynamic changes in speech quality during teaching. This makes up for the shortcomings of traditional algorithms in capturing dynamic characteristics, allowing the evaluation results to better reflect the actual teaching speech quality.

[0062] 3. This invention first concatenates the time-domain and frequency-domain feature vectors of multiple sub-segments to obtain time-domain and frequency-domain feature matrices, respectively. Then, it enhances the features based on the time-domain and frequency-domain fluctuation vectors to obtain time-domain enhanced features and frequency-domain enhanced features. Finally, these are input into a dual-channel neural network to obtain a score. This invention, through multi-dimensional feature extraction, temporal dynamic capture, and the combination of feature enhancement and a dual-channel network, comprehensively and deeply mines the quality information of teaching audio, significantly improving the accuracy of teaching audio quality assessment and solving the problem of low assessment accuracy in existing technologies. Attached Figure Description

[0063] Figure 1 A flowchart for a speech analysis teaching audio assessment method that assists in instruction;

[0064] Figure 2 A flowchart for obtaining temporal enhancement features;

[0065] Figure 3 A flowchart for obtaining frequency domain enhancement features;

[0066] Figure 4 This is a schematic diagram of a dual-channel neural network. Detailed Implementation

[0067] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0068] Example 1, as Figure 1 As shown, a speech analysis teaching audio assessment method for assisting instruction includes the following steps:

[0069] The audio signal of the teaching audio is segmented using a sliding window to obtain multiple sub-segments. Wavelet decomposition and reconstruction are performed on each sub-segment to obtain low-frequency components, mid-high frequency components, and high-frequency components.

[0070] Based on the temporal energy of the three components, consonant-vowel prominence, resonance intensity, and speech weight are obtained to form a temporal feature vector;

[0071] Based on the energy dispersion of the three components, the low-frequency energy dispersion, mid-to-high-frequency energy dispersion, and high-frequency energy dispersion are obtained, forming a frequency domain feature vector;

[0072] Based on the time-domain and frequency-domain feature vectors of adjacent sub-segments, the fluctuation coefficients are calculated to form the time-domain fluctuation vector and the frequency-domain fluctuation vector.

[0073] The time-domain feature vectors of multiple segments are concatenated to obtain the time-domain feature matrix, and the frequency-domain feature vectors of multiple segments are concatenated to obtain the frequency-domain feature matrix.

[0074] Features are extracted from the time-domain feature matrix and the frequency-domain feature matrix. Feature enhancement is performed based on the time-domain fluctuation vector and the frequency-domain fluctuation vector to obtain time-domain enhanced features and frequency-domain enhanced features. These are then input into a dual-channel neural network to obtain a teaching audio quality score.

[0075] In this embodiment, the process of obtaining low-frequency components, mid-to-high-frequency components, and high-frequency components includes:

[0076] Perform the first-level wavelet decomposition on each segment to obtain the first-level approximation coefficients and the first-level detail coefficients;

[0077] The first-level approximation coefficients are decomposed into second-level wavelet coefficients to obtain the second-level approximation coefficients and the second-level detail coefficients.

[0078] The low-frequency components are obtained by wavelet reconstruction of the second-level approximation coefficients.

[0079] The second-level detail coefficients are reconstructed using wavelet reconstruction to obtain the mid-to-high frequency components;

[0080] The high-frequency components are obtained by wavelet reconstruction of the first-level detail coefficients.

[0081] In human speech, the duration of a complete syllable (including vowels and consonants) is typically 100ms to 300ms. However, teaching speech, emphasizing clarity, tends to have slightly longer syllables (average 200ms to 400ms). A window length of 200ms ≤ window length ≤ 500ms can fully encompass the acoustic information of at least one syllable, avoiding speech fragmentation caused by excessively short windows (e.g., <100ms). The sliding step size is set to half the window length.

[0082] In this embodiment, the wavelet basis function is selected as db4 (Daubechies4), the frequency range of the low-frequency components is ≤2kHz, the frequency range of the mid-to-high frequency components is 2kHz~4kHz, and the frequency range of the high-frequency components is ≥4kHz.

[0083] In this embodiment, the process of constructing the time-domain feature vector includes:

[0084] The time-domain energy of the low-frequency component, mid-high frequency component, and high-frequency component is calculated respectively to obtain the low-frequency energy, mid-high frequency energy, and high-frequency energy.

[0085] Add the mid-to-high frequency energy and the high frequency energy, subtract the low frequency energy, and normalize to obtain the consonant-vowel prominence.

[0086] The ratio of low-frequency energy to mid-to-high-frequency energy is used as the resonance intensity;

[0087] The ratio of low-frequency energy to high-frequency energy is used as the depth of speech.

[0088] The consonant-vowel prominence, resonance intensity, and speech weight are used to construct a temporal feature vector.

[0089] The time-domain energy is the sum of the squares of all amplitudes in the component signal.

[0090] In this embodiment, the formula for consonant-vowel prominence is:

[0091] ,

[0092] Where θ is the consonant-vowel prominence, E H For high-frequency energy, E M For medium to high frequency energy, E L Low-frequency energy, The denominator coefficient is a constant.

[0093] This invention uses consonant-vowel prominence to reflect the relative prominence of consonants and vowels in speech, which helps to judge the clarity and distinguishability of pronunciation; resonance intensity reflects the resonance characteristics of speech and is related to the fullness of the tone; and speech weight reflects the depth or weight of speech. These characteristics are crucial for the assessment of teaching speech (such as pronunciation accuracy and speech appeal).

[0094] Resonance intensity reflects the resonance effect of the vocal tract (oral cavity, nasal cavity, etc.) when the teacher pronounces words. Low-frequency components (100Hz~2kHz) mainly correspond to the fundamental frequency and low-order formants of vowels (such as F1, F2), and their energy strength directly reflects the fullness of vocal tract resonance. Mid-to-high frequency components (2kHz~4kHz) correspond to the higher harmonics and transitional sounds of vowels, with relatively stable energy. The ratio of these two components quantifies the balance between "low-frequency resonance energy" and "mid-to-high frequency transitional energy." Speech fullness directly reflects the balance between "low-frequency richness" and "high-frequency sharpness": the higher the ratio, the stronger the low-frequency energy, and the more "deep and resonant" the speech; the lower the ratio, the stronger the high-frequency energy, and the more "clear and thin" the speech.

[0095] Consonants (especially voiceless consonants such as "s" and "sh") are mainly distributed in high frequencies (≥4kHz) and mid-high frequencies (2kHz~4kHz), while vowels are mainly distributed in low frequencies (≤2kHz). The more pronounced the consonant is, the greater the proportion of mid-high frequencies → the larger the θ becomes; conversely, if a vowel is prolonged or the speech is unclear → low frequencies dominate → the θ decreases.

[0096] In this embodiment, a Fourier transform is performed on the low-frequency components to obtain the low-frequency component spectrum; a Fourier transform is performed on the mid-high frequency components to obtain the mid-high frequency component spectrum; and a Fourier transform is performed on the high-frequency components to obtain the high-frequency component spectrum. Energy dispersion is calculated for all three spectra: low-frequency, mid-high frequency, and high-frequency components. The formula for energy dispersion is as follows:

[0097] ,

[0098] Where, γ i E represents the energy dispersion. i,k Let γ be the energy at the k-th frequency point in the spectrum, K be the number of frequency points in the spectrum, k be the number of the frequency point, i = 1 corresponds to low frequency, γ1 is the low frequency energy dispersion, i = 2 corresponds to mid-high frequency, γ2 is the mid-high frequency energy dispersion, i = 3 corresponds to high frequency, and γ3 is the high frequency energy dispersion.

[0099] Low-frequency energy dispersion: can characterize the concentration of energy near low frequencies, indirectly reflecting the "stability" of the teacher's voice (e.g., low-frequency energy is more concentrated when giving a stable explanation); Mid-to-high frequency energy dispersion: strongly correlated with the "clarity" of the voice (when mid-to-high frequency energy is concentrated, details such as consonants are more prominent, and the pronunciation is clearer); High-frequency energy dispersion: related to the "infectiousness" of the voice (when high-frequency energy is concentrated, the intonation fluctuations or emphasized parts are more obvious, such as when asking questions, high-frequency energy may suddenly increase).

[0100] This invention, by extracting low-frequency energy dispersion, mid-to-high-frequency energy dispersion, and high-frequency energy dispersion respectively, can comprehensively characterize the energy distribution characteristics of teaching audio in different frequency bands, enhance the ability to distinguish factors such as pronunciation clarity and speech stability, improve the accuracy and robustness of the assessment, and make the quality assessment of teaching audio more objective and reliable.

[0101] In this embodiment, low-frequency energy dispersion, mid-to-high-frequency energy dispersion, and high-frequency energy dispersion are used as elements to construct the frequency domain feature vector. Consonant-vowel prominence, resonance intensity, and speech weight are used as elements to construct the time domain feature vector. Therefore, both the frequency domain feature vector and the time domain feature vector contain three elements.

[0102] The process of constructing the time-domain fluctuation vector and the frequency-domain fluctuation vector includes: for both the time-domain feature vector and the frequency-domain feature vector, N segments are extracted from each segment to form a set of segments; the elements of the feature vector corresponding to each segment in the set are extracted, and the ratio of the standard deviation to the mean of each element is calculated as the fluctuation value; the fluctuation values ​​corresponding to each element in the time-domain feature vector are averaged to obtain the time-domain fluctuation coefficient of the central segment, and the fluctuation values ​​corresponding to each element in the frequency-domain feature vector are averaged to obtain the frequency-domain fluctuation coefficient of the central segment. The time-domain fluctuation coefficients constitute the time-domain fluctuation vector, and the frequency-domain fluctuation coefficients constitute the frequency-domain fluctuation vector.

[0103] In this embodiment, the process of constructing the time-domain fluctuation vector includes:

[0104] Taking each sub-segment as the center, extract N sub-segments to obtain a set of sub-segments, where N is a positive integer;

[0105] Extract the elements from the temporal feature vector of each segment in the segment set;

[0106] Calculate the standard deviation and mean for each element, and use the ratio of the standard deviation to the mean as the fluctuation value of the corresponding element;

[0107] The average value of the fluctuation values ​​corresponding to the three elements in the time-domain feature vector is taken to obtain the time-domain fluctuation coefficient of the central segment. The time-domain fluctuation coefficients of each segment are then used to construct the time-domain fluctuation vector.

[0108] In this embodiment, N is set to 3.

[0109] This invention extracts multiple segments centered on each sub-segment to form a sub-segment set, which can fully cover the speech information surrounding the central sub-segment. After extracting the temporal feature vector elements of each sub-segment in the sub-segment set, the ratio of the standard deviation to the mean of the elements is calculated as the fluctuation value. Then, the average of the fluctuation values ​​corresponding to various elements in the temporal feature vector is taken to obtain the temporal fluctuation coefficient. This method can capture the fluctuation of teaching speech in the temporal domain in detail and accurately.

[0110] For example, the consonant-vowel prominence is extracted from the temporal feature vector of each segment in the segment set. The standard deviation and mean of the multiple consonant-vowel prominences corresponding to the segment set are calculated, and the ratio of the standard deviation to the mean is used as the fluctuation value of the corresponding consonant-vowel prominence.

[0111] In this embodiment, the process of constructing the frequency domain fluctuation vector includes:

[0112] Taking each sub-segment as the center, extract N sub-segments to obtain a set of sub-segments, where N is a positive integer;

[0113] Extract the elements from the frequency domain feature vector of each segment in the segment set;

[0114] Calculate the standard deviation and mean for each element, and use the ratio of the standard deviation to the mean as the fluctuation value of the corresponding element;

[0115] The average value of the fluctuation values ​​corresponding to the three elements in the frequency domain feature vector is taken to obtain the frequency domain fluctuation coefficient of the central segment. The frequency domain fluctuation coefficients of each segment are then used to construct the frequency domain fluctuation vector.

[0116] This invention extracts multiple segments centered on each sub-segment to form a sub-segment set, which can fully cover the speech information surrounding the central sub-segment. After extracting the frequency domain feature vector elements of each sub-segment in the sub-segment set, the ratio of the standard deviation to the mean of the elements is calculated as the fluctuation value. Then, the average of the fluctuation values ​​corresponding to various elements in the frequency domain feature vector is taken to obtain the frequency domain fluctuation coefficient. This method can capture the fluctuation of teaching speech in the frequency domain in detail and accurately.

[0117] This invention, by calculating low-frequency energy dispersion, mid-to-high-frequency energy dispersion, and high-frequency energy dispersion separately, can quantitatively reflect the energy concentration in three frequency ranges of teaching audio from the perspective of spectral distribution, thus characterizing the pronunciation. Furthermore, by calculating the frequency domain fluctuation coefficient, it can reflect the stability and variation patterns of spectral characteristics from a time perspective. Combining these two methods allows for a more accurate and comprehensive objective assessment of the pronunciation clarity and teaching quality of teaching audio.

[0118] like Figure 2 and Figure 3As shown, the process of obtaining time-domain enhancement features and frequency-domain enhancement features includes:

[0119] Both the time-domain feature matrix and the frequency-domain feature matrix are processed through a 1×1 first convolutional layer to extract features, resulting in 3×M time-domain features and 3×M frequency-domain features, where 3 is the number of columns and M is the number of rows.

[0120] The 3×M time-domain features and 3×M frequency-domain features are both extracted from each row through a 3×1 second convolutional layer to obtain 1×M time-domain features and 1×M frequency-domain features.

[0121] Both the time-domain fluctuation vector and the frequency-domain fluctuation vector are processed through a 1×1 third convolutional layer to extract features, resulting in 1×M time-domain fluctuation features and 1×M frequency-domain fluctuation features.

[0122] Multiplying the 1×M time-domain features element-wise with the 1×M time-domain fluctuation features yields the time-domain enhancement features;

[0123] Multiplying the 1×M frequency domain features element-wise with the 1×M frequency domain fluctuation features yields the frequency domain enhancement features.

[0124] The size of the time-domain feature matrix and the frequency-domain feature matrix is ​​3×M, where each row of the time-domain feature matrix is ​​a time-domain feature vector and each row of the frequency-domain feature matrix is ​​a frequency-domain feature vector, M is the number of rows, and M is equal to the number of sub-segments.

[0125] The 1×1 first convolutional layer can perform channel-level feature extraction and transformation on the time-domain and frequency-domain feature matrices without changing the feature dimension. The 3×1 second convolutional layer focuses on extracting features from each row, compressing the original 3×M feature matrix into a 1×M feature vector, thus achieving dimensionality reduction of features and aggregation of key information.

[0126] The temporal and frequency domain fluctuation vectors are extracted using a 1×1 third convolutional layer to obtain their corresponding fluctuation features. These features are then multiplied element-wise with the temporal and frequency domain features obtained from the first two convolutional layers. This method accurately integrates the fluctuation characteristics of speech (reflecting the dynamic changes in speech) into the temporal and frequency domain features. The enhanced features not only contain the original static feature information but also reflect the dynamic changes in speech over time, making the features more comprehensive and better reflecting the essential characteristics of teaching speech.

[0127] like Figure 4 As shown, the dual-channel neural network includes: a first reshaping layer, a second reshaping layer, a first CNN network, a second CNN network, and a fully connected layer;

[0128] The input of the first reshaping layer is used to input 1×M temporal enhancement features;

[0129] The input of the second reshaping layer is used to input 1×M frequency domain enhancement features;

[0130] The input of the first CNN network is connected to the output of the first reshaping layer;

[0131] The input of the second CNN network is connected to the output of the second reshaping layer;

[0132] The input of the fully connected layer is connected to the output of the first CNN network and the output of the second CNN network, respectively, and its output serves as the output of the dual-channel neural network.

[0133] The first reshape layer is used to convert the 1×M temporal enhancement features into a two-dimensional data form, and the second reshape layer is used to convert the 1×M frequency domain enhancement features into a two-dimensional data form.

[0134] This invention reshapes dimensions through a first reshaping layer and a second reshaping layer, making it easier to input into a CNN network for feature extraction. The fully connected layer receives the output features of the two CNN networks, combining the two aspects of features to improve the accuracy of teaching audio quality assessment.

[0135] Example 2: A speech analysis teaching audio evaluation system for teaching assistance, comprising: a component extraction unit, a time-domain feature extraction unit, a frequency-domain feature extraction unit, a wave vector construction unit, a splicing unit, and a scoring unit;

[0136] The component extraction unit is used to segment the speech signal of the teaching audio using a sliding window to obtain multiple sub-segments. Each sub-segment is then decomposed and reconstructed to obtain low-frequency components, mid-to-high-frequency components, and high-frequency components.

[0137] The temporal feature extraction unit is used to obtain consonant-vowel prominence, resonance intensity, and speech weight based on the temporal energy of the three components, thus forming a temporal feature vector;

[0138] The frequency domain feature extraction unit is used to obtain low-frequency energy dispersion, mid-to-high frequency energy dispersion, and high-frequency energy dispersion based on the energy dispersion of the three components, and to form a frequency domain feature vector;

[0139] The fluctuation vector building unit is used to calculate the fluctuation coefficient based on the time-domain feature vector and frequency-domain feature vector of adjacent sub-segments, and to form the time-domain fluctuation vector and frequency-domain fluctuation vector.

[0140] The splicing unit is used to splice the time-domain feature vectors of multiple segments to obtain a time-domain feature matrix, and to splice the frequency-domain feature vectors of multiple segments to obtain a frequency-domain feature matrix.

[0141] The scoring unit is used to extract features from the time-domain feature matrix and the frequency-domain feature matrix, and to enhance the features based on the time-domain fluctuation vector and the frequency-domain fluctuation vector to obtain time-domain enhanced features and frequency-domain enhanced features. These features are then input into a dual-channel neural network to obtain a teaching audio quality score.

[0142] The specific implementation methods of Example 2 and Example 1 are the same.

[0143] In this embodiment, Figure 2 , Figure 3 and Figure 4 The structure is used as a model, and the time-domain feature matrix, frequency-domain feature matrix, time-domain fluctuation vector and frequency-domain fluctuation vector are collected as input samples. The samples are manually labeled and scored to train the model.

[0144] This invention uses wavelet decomposition to obtain low-frequency, mid-high-frequency, and high-frequency components, and extracts temporal features such as consonant-vowel prominence (reflecting pronunciation clarity), resonance intensity (characterizing chest resonance quality), and speech weight (quantifying low-frequency dominance). This directly addresses key requirements in teaching scenarios such as "clear pronunciation" and "composed explanation." Furthermore, by combining energy dispersion across frequency bands (e.g., consonants are more prominent when mid-high frequency dispersion is low), it characterizes the spectral distribution of speech in the frequency domain, overcoming the shortcomings of traditional short-time energy and zero-crossing rate features in capturing professional information about teaching speech.

[0145] This invention constructs time-domain and frequency-domain fluctuation vectors by calculating the fluctuation coefficients of adjacent segments, effectively capturing the temporal dynamic changes in speech quality during teaching. This makes up for the shortcomings of traditional algorithms in capturing dynamic characteristics, allowing the evaluation results to better reflect the actual teaching speech quality.

[0146] This invention first concatenates the time-domain and frequency-domain feature vectors of multiple sub-segments to obtain time-domain and frequency-domain feature matrices, respectively. Then, it enhances the features based on the time-domain and frequency-domain fluctuation vectors to obtain enhanced time-domain and frequency-domain features. Finally, these enhanced features are input into a dual-channel neural network to obtain a score. This invention, through multi-dimensional feature extraction, temporal dynamic capture, and the combination of feature enhancement and a dual-channel network, comprehensively and deeply mines the quality information of teaching audio, significantly improving the accuracy of teaching audio quality assessment and solving the problem of low assessment accuracy in existing technologies.

[0147] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech analysis teaching audio evaluation method for assisting teaching, characterized by, The method comprises the following steps: The speech signal of the teaching audio is segmented by using a sliding window to obtain a plurality of subsegments, each subsegment is decomposed and reconstructed by using a wavelet to obtain a low-frequency component, a medium-high frequency component and a high-frequency component; According to the time-domain energy of the three components, a consonant-vowel prominence, a resonance intensity and a speech thickness are obtained to form a time-domain feature vector; According to the energy dispersion degrees of the three components, a low-frequency energy dispersion degree, a medium-high frequency energy dispersion degree and a high-frequency energy dispersion degree are obtained to form a frequency-domain feature vector; A fluctuation coefficient is calculated according to the time-domain feature vectors and the frequency-domain feature vectors of adjacent subsegments to form a time-domain fluctuation vector and a frequency-domain fluctuation vector; The time-domain feature vectors of the plurality of subsegments are spliced to obtain a time-domain feature matrix, and the frequency-domain feature vectors of the plurality of subsegments are spliced to obtain a frequency-domain feature matrix; Features are extracted from the time-domain feature matrix and the frequency-domain feature matrix, the features are enhanced based on the time-domain fluctuation vector and the frequency-domain fluctuation vector to obtain time-domain enhanced features and frequency-domain enhanced features, and the time-domain enhanced features and the frequency-domain enhanced features are input into a double-channel neural network to obtain a teaching audio quality score.

2. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The process of obtaining the low-frequency component, the medium-high frequency component and the high-frequency component comprises: Each subsegment is decomposed by using a first-layer wavelet to obtain first-layer approximation coefficients and first-layer detail coefficients; The first-layer approximation coefficients are decomposed by using a second-layer wavelet to obtain second-layer approximation coefficients and second-layer detail coefficients; The second-layer approximation coefficients are reconstructed by using a wavelet to obtain the low-frequency component; The second-layer detail coefficients are reconstructed by using a wavelet to obtain the medium-high frequency component; The first-layer detail coefficients are reconstructed by using a wavelet to obtain the high-frequency component.

3. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The process of forming the time-domain feature vector comprises: The time-domain energy of the low-frequency component, the medium-high frequency component and the high-frequency component is calculated respectively to obtain a low-frequency energy, a medium-high frequency energy and a high-frequency energy; The medium-high frequency energy and the high-frequency energy are added, the low-frequency energy is subtracted, and normalization is performed to obtain a consonant-vowel prominence; The ratio of the low-frequency energy to the medium-high frequency energy is taken as a resonance intensity; The ratio of the low-frequency energy to the high-frequency energy is taken as a speech thickness; The consonant-vowel prominence, the resonance intensity and the speech thickness form the time-domain feature vector.

4. The speech analysis teaching audio evaluation method for assistive teaching of claim 3, wherein, The formula of the consonant-vowel prominence is: ， where θ is the consonant-vowel prominence, E H is the high frequency energy, E M is the mid-high frequency energy, E L is the low frequency energy, is the denominator coefficient.

5. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The formula for calculating the energy dispersion degree is: ， wherein γ i is the energy dispersion, E i,k is the energy of the kth frequency point in the spectrum, K is the number of frequency points in the spectrum, k is the number of the frequency point, i is 1, γ1 is the low frequency energy dispersion, i is 2, γ2 is the medium-high frequency energy dispersion, i is 3, γ3 is the high frequency energy dispersion.

6. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The process of forming the time-domain fluctuation vector comprises: A plurality of subsegments are intercepted with each subsegment as a center to obtain a subsegment set, wherein N is a positive integer; Elements in the time-domain feature vector of each subsegment in the subsegment set are extracted; A standard deviation and a mean value of each element are calculated, and the ratio of the standard deviation to the mean value is taken as a fluctuation value of the corresponding element; The time-domain fluctuation coefficients of the subsegments are combined to form the time-domain fluctuation vector.

7. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The process of forming the frequency-domain fluctuation vector comprises: A plurality of subsegments are intercepted with each subsegment as a centre to obtain a subsegment set, wherein N is a positive integer; Elements in the frequency-domain feature vector of each subsegment in the subsegment set are extracted; A standard deviation and mean value of each element are calculated, and the ratio of the standard deviation to the mean value of the corresponding element is taken as a fluctuation value of the corresponding element; The fluctuation values corresponding to the three elements in the frequency domain feature vector are averaged to obtain the frequency domain fluctuation coefficient of the central sub-section, and the frequency domain fluctuation coefficients of the sub-sections are combined to form a frequency domain fluctuation vector.

8. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The process of obtaining the time domain enhancement feature and the frequency domain enhancement feature includes: The time domain feature matrix and the frequency domain feature matrix are extracted through a first convolution layer with a size of 1*1 to obtain a time domain feature with a size of 3*M and a frequency domain feature with a size of 3*M, wherein 3 is the number of columns and M is the number of rows; The time domain feature with a size of 3*M and the frequency domain feature with a size of 3*M are extracted through a second convolution layer with a size of 3*1 to obtain a time domain feature with a size of 1*M and a frequency domain feature with a size of 1*M; The time domain fluctuation vector and the frequency domain fluctuation vector are extracted through a third convolution layer with a size of 1*1 to obtain a time domain fluctuation feature with a size of 1*M and a frequency domain fluctuation feature with a size of 1*M; The time domain feature with a size of 1*M and the time domain fluctuation feature with a size of 1*M are multiplied element by element to obtain a time domain enhancement feature; The frequency domain feature with a size of 1*M and the frequency domain fluctuation feature with a size of 1*M are multiplied element by element to obtain a frequency domain enhancement feature.

9. The speech analysis teaching audio evaluation method for assistive teaching of claim 1, wherein, The dual-channel neural network includes a first reshaping layer, a second reshaping layer, a first CNN network, a second CNN network, and a fully connected layer; The input end of the first reshaping layer is configured to input the time domain enhancement feature; The input end of the second reshaping layer is configured to input the frequency domain enhancement feature; The input end of the first CNN network is connected to the output end of the first reshaping layer; The input end of the second CNN network is connected to the output end of the second reshaping layer; The input end of the fully connected layer is connected to the output end of the first CNN network and the output end of the second CNN network, and the output end thereof serves as the output end of the dual-channel neural network.

10. A speech analysis teaching audio evaluation system for assisting teaching, characterized by The speech analysis teaching audio evaluation method for assisting teaching according to any one of claims 1-9 includes a component extraction unit, a time domain feature extraction unit, a frequency domain feature extraction unit, a fluctuation vector construction unit, a splicing unit, and a scoring unit. The component extraction unit is configured to divide the speech signal of the teaching audio by using a sliding window to obtain a plurality of sub-sections, and perform wavelet decomposition and reconstruction on each sub-section to obtain low-frequency components, medium-high-frequency components, and high-frequency components. The time domain feature extraction unit is configured to obtain the voice-vowel prominence, the resonance intensity, and the speech thickness degree according to the time domain energy of the three components to construct a time domain feature vector. The frequency domain feature extraction unit is configured to obtain the low-frequency energy dispersion, the medium-high-frequency energy dispersion, and the high-frequency energy dispersion according to the energy dispersion of the three components to construct a frequency domain feature vector. The fluctuation vector construction unit is configured to calculate the fluctuation coefficient according to the time domain feature vector and the frequency domain feature vector of adjacent sub-sections to construct a time domain fluctuation vector and a frequency domain fluctuation vector. The splicing unit is configured to splice the time domain feature vectors of the plurality of sub-sections to obtain a time domain feature matrix, and splice the frequency domain feature vectors of the plurality of sub-sections to obtain a frequency domain feature matrix. The scoring unit is configured to extract features from the time domain feature matrix and the frequency domain feature matrix, perform feature enhancement based on the time domain fluctuation vector and the frequency domain fluctuation vector to obtain a time domain enhancement feature and a frequency domain enhancement feature, and input the features into a dual-channel neural network to obtain a teaching audio quality score.

Images