Chinese language pronunciation intonation evaluation method

Abstract

The invention discloses a Chinese language pronunciation intonation evaluation method, and belongs to the technical field of speech processing. The method comprises the following steps: firstly, dividing a Chinese language speech signal by adopting an autocorrelation method to obtain a plurality of sound state sets; eMD decomposition is carried out on short-time frames in the sound state set, and four characteristics of low-frequency gravity center, low-frequency energy, high-frequency gravity center and high-frequency energy are extracted; for a single sound state set, obtaining a sound state instability coefficient and constructing a sequence based on the four EMD features of each short-time frame, and meanwhile, respectively averaging each EMD feature; obtaining a change rate according to the EMD feature mean value difference of the adjacent sound state sets, constructing four change rate sequences, and subtracting each change rate sequence from the corresponding change rate storage sequence to obtain a change rate difference sequence; and finally, processing the four change rate difference sequences by using a pronunciation intonation evaluation neural network, and obtaining a pronunciation intonation score by combining the pronunciation intonation evaluation neural network with the sound state instability coefficient sequence. The precision of Chinese language pronunciation intonation evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular to a method for evaluating the pronunciation intonation of the Chinese language. Background Art

[0002] Traditional Chinese pronunciation assessment relies heavily on manual listening, a labor-intensive process that also suffers from significant subjective bias and inconsistent standards. With the advancement of speech signal processing technology, automated assessment methods have become a research hotspot. However, existing technologies still have significant limitations. Most methods analyze only a single speech feature (Mel-frequency cepstral coefficients (MFCCs)), making it difficult to fully capture the dynamics of Chinese pronunciation. Furthermore, they inadequately characterize transitions between speech states, making it difficult to accurately identify pronunciation deviations in continuous speech streams. Summary of the Invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for evaluating pronunciation and intonation of the Chinese language, which solves the problem of low accuracy in the prior art.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a method for evaluating the pronunciation intonation of the Chinese language, comprising the following steps:

[0005] The autocorrelation method is used to divide the Chinese language speech signal and obtain multiple sound state sets;

[0006] Perform EMD decomposition on each short-time frame in the sound state set and extract four EMD features of each short-time frame, where the four EMD features include: EMD low-frequency center of gravity, EMD low-frequency energy, EMD high-frequency center of gravity and EMD high-frequency energy;

[0007] For a sound state set, the sound state instability coefficient is obtained based on the four EMD features of each short time frame, and the sound state instability coefficient sequence is constructed;

[0008] For a sound state set, take the mean of each EMD feature of each short time frame;

[0009] According to the difference of the EMD feature mean values ​​of adjacent sound state sets, the change rate is obtained and four EMD feature change rate sequences are constructed;

[0010] Subtract each change rate sequence from the corresponding change rate storage sequence to obtain the change rate difference sequence;

[0011] The pronunciation intonation evaluation neural network is used to process the four EMD feature change rate difference sequences, and feature enhancement is performed based on the sound state instability coefficient sequence to obtain the pronunciation intonation score.

[0012] Furthermore, the process of obtaining the plurality of sound state sets includes: dividing the Chinese language speech signal according to the frame length and the frame shift to obtain a plurality of short time frames;

[0013] For each short time frame, the autocorrelation value is calculated based on the autocorrelation function;

[0014] When the autocorrelation values ​​of multiple consecutive short-time frames are greater than the autocorrelation threshold, and the peak position change values ​​of the autocorrelation values ​​of multiple consecutive short-time frames are less than the change threshold, the multiple short-time frames are arranged in chronological order to form a sound state set.

[0015] Furthermore, the formula for calculating the autocorrelation value is: , where R i (k) is the autocorrelation value of the i-th short-time frame when the delay is k, y i (n) is the amplitude of the nth sampling point in the i-th short time frame, y i (n+k) is the amplitude of the n+kth sampling point in the i-th short-time frame, the value range of k is 0~K / 2, K is the total number of sampling points in a single short-time frame, and i, n, and k are positive integers.

[0016] Furthermore, the method for obtaining the peak position change value includes: selecting the maximum value from multiple autocorrelation values ​​corresponding to each short-time frame, recording the delay number corresponding to the maximum value as the peak number, finding the corresponding maximum peak number and minimum peak number in multiple consecutive short-time frames, subtracting the minimum peak number from the maximum peak number to obtain the peak position deviation value, and taking the ratio of the peak position deviation value to the maximum peak number as the peak position change value.

[0017] Furthermore, the process of extracting the four EMD features of each short-time frame includes: performing EMD decomposition on each short-time frame in the sound state set to obtain multiple intrinsic mode functions;

[0018] Perform Fourier transform on each eigenmode function to obtain the eigenmode function spectrum;

[0019] Divide the multiple intrinsic mode function spectra corresponding to each short time frame into a low-frequency subset and a high-frequency subset;

[0020] According to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset, the EMD low-frequency center of gravity and the EMD low-frequency energy are obtained;

[0021] According to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset, the EMD high-frequency center of gravity and the EMD high-frequency energy are obtained.

[0022] Furthermore, the process of dividing the multiple intrinsic mode function spectra corresponding to each short time frame into low-frequency subsets and high-frequency subsets includes: dividing the first intrinsic mode function spectrum and the second intrinsic mode function spectrum into the high-frequency subset; dividing the remaining intrinsic mode function spectra into the low-frequency subset;

[0023] The process of obtaining the EMD low-frequency centroid includes: obtaining the EMD low-frequency centroid according to the fundamental frequency and the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset;

[0024] The process of obtaining the EMD low-frequency energy includes: obtaining the EMD low-frequency energy according to the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset;

[0025] The process of obtaining the EMD high-frequency centroid includes: obtaining the EMD high-frequency centroid according to the fundamental frequency and the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset;

[0026] The process of obtaining the EMD high-frequency energy includes: obtaining the EMD high-frequency energy according to the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset.

[0027] Furthermore, the process of constructing the sound state instability coefficient sequence includes:

[0028] In a sound state set, the same EMD features of adjacent short time frames are subtracted to obtain the EMD feature increment;

[0029] In the same sound state set, the fluctuation coefficient is obtained for each EMD feature increment. The four fluctuation coefficients are added and normalized to obtain the sound state instability coefficient.

[0030] Arrange the various sound state instability coefficients in chronological order to construct a sound state instability coefficient sequence.

[0031] Furthermore, the process of constructing the EMD characteristic change rate sequence includes:

[0032] The EMD feature mean of the sound state set at time t+1 is subtracted from the EMD feature mean of the sound state set at time t to obtain the EMD feature mean difference. The ratio of the EMD feature mean difference to the EMD feature mean of the sound state set at time t is used as the EMD feature change rate at time t. The EMD feature change rates are arranged in chronological order to obtain the EMD feature change rate sequence, where t is the time number.

[0033] When the EMD feature mean is the mean of the EMD low-frequency centroid, the EMD feature change rate is the EMD low-frequency centroid change rate. The EMD low-frequency centroid change rates are arranged in chronological order to obtain the EMD low-frequency centroid change rate sequence; when the EMD feature mean is the EMD low-frequency energy mean, the EMD feature change rate is the EMD low-frequency energy change rate. The EMD low-frequency energy change rate is arranged in chronological order to obtain the EMD low-frequency energy change rate sequence; when the EMD feature mean is the mean of the EMD high-frequency centroid, the EMD feature change rate is the EMD high-frequency centroid change rate. The EMD high-frequency centroid change rate is arranged in chronological order to obtain the EMD high-frequency centroid change rate sequence; when the EMD feature mean is the EMD high-frequency energy mean, the EMD feature change rate is the EMD high-frequency energy change rate. The EMD high-frequency energy change rate is arranged in chronological order to obtain the EMD high-frequency energy change rate sequence.

[0034] Furthermore, the pronunciation intonation assessment neural network includes: a sequence feature fusion unit, an enhancement unit, a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, and a fully connected layer;

[0035] The input end of the sequence feature fusion unit serves as the input end of the pronunciation intonation evaluation neural network, and its output end is connected to the first input end of the enhancement unit; the second input end of the enhancement unit is used to input the sound state instability coefficient sequence, and its output end is connected to the input end of the first one-dimensional convolutional layer; the output end of the first one-dimensional convolutional layer is connected to the input end of the second one-dimensional convolutional layer; the output end of the second one-dimensional convolutional layer is connected to the input end of the fully connected layer; and the output end of the fully connected layer serves as the output end of the pronunciation intonation evaluation neural network.

[0036] Furthermore, the sequence feature fusion unit includes: a third one-dimensional convolution layer, a fourth one-dimensional convolution layer, a fifth one-dimensional convolution layer, a sixth one-dimensional convolution layer and an adder A1;

[0037] The third one-dimensional convolution layer is used to process the EMD low-frequency center of gravity change rate difference sequence; the fourth one-dimensional convolution layer is used to process the EMD low-frequency energy change rate difference sequence; the fifth one-dimensional convolution layer is used to process the EMD high-frequency center of gravity change rate difference sequence; the sixth one-dimensional convolution layer is used to process the EMD high-frequency energy change rate difference sequence;

[0038] The input end of adder A1 is connected to the output end of the third one-dimensional convolutional layer, the output end of the fourth one-dimensional convolutional layer, the output end of the fifth one-dimensional convolutional layer, and the output end of the sixth one-dimensional convolutional layer, and its output end serves as the output end of the sequence feature fusion unit;

[0039] The enhancement unit is used to multiply the audio instability coefficient sequence by the output of the sequence feature fusion unit element by element.

[0040] The beneficial effects of the present invention are:

[0041] 1. Chinese pronunciation (especially vowels and voiced consonants) exhibits significant periodicity (e.g., the vibration patterns of the vocal cords in vowels). Therefore, the autocorrelation method is used to extract sustained, stable pronunciation phases as a set of sound states. EMD decomposition is then performed on each short time frame within this set of sound states to extract four EMD features: low-frequency centroid, low-frequency energy, high-frequency centroid, and high-frequency energy. This overcomes the limitations of existing technologies that rely solely on a single feature (such as Mel-frequency cepstral coefficients). These four features can more comprehensively reflect subtle changes in pronunciation by analyzing the energy distribution and centroid position of different frequency bands.

[0042] 2. On the one hand, the present invention can effectively quantify the pronunciation stability within a single voice state set by constructing a sequence of voice state instability coefficients, and capture unstable phenomena such as trembling and offset that may occur during the pronunciation process; on the other hand, the change rate sequence calculated based on the mean of the EMD features of adjacent voice state sets characterizes the characteristic changes in the voice state transition stage, which makes up for the defect of the existing technology in insufficient characterization of the voice state transition in continuous speech streams.

[0043] 3. The present invention processes four EMD feature change rate difference sequences through a pronunciation intonation assessment neural network, and performs feature enhancement in combination with a sound state instability coefficient sequence. This not only avoids the subjective bias of manual evaluation, but also highlights the impact of abnormal pronunciation features through feature enhancement, significantly improving the objectivity and accuracy of intonation assessment, and can more accurately reflect the learner's pronunciation level. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for evaluating pronunciation and intonation of the Chinese language;

[0045] Figure 2 Schematic diagram of the neural network structure for pronunciation intonation assessment;

[0046] Figure 3 Schematic diagram of the structure of the sequence feature fusion unit. DETAILED DESCRIPTION

[0047] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0048] like Figure 1 As shown, a method for evaluating the pronunciation intonation of the Chinese language comprises the following steps:

[0049] The autocorrelation method is used to divide the Chinese language speech signal and obtain multiple sound state sets;

[0050] Perform EMD decomposition on each short-time frame in the sound state set and extract four EMD features of each short-time frame, where the four EMD features include: EMD low-frequency center of gravity, EMD low-frequency energy, EMD high-frequency center of gravity and EMD high-frequency energy;

[0051] For a sound state set, the sound state instability coefficient is obtained based on the four EMD features of each short time frame, and the sound state instability coefficient sequence is constructed;

[0052] For a sound state set, take the mean of each EMD feature of each short time frame;

[0053] According to the difference of the EMD feature mean values ​​of adjacent sound state sets, the change rate is obtained and four EMD feature change rate sequences are constructed;

[0054] Subtract each change rate sequence from the corresponding change rate storage sequence to obtain the change rate difference sequence;

[0055] The pronunciation intonation evaluation neural network is used to process the four EMD feature change rate difference sequences, and feature enhancement is performed based on the sound state instability coefficient sequence to obtain the pronunciation intonation score.

[0056] In this embodiment, the process of obtaining the plurality of sound state sets includes: dividing the Chinese language speech signal according to the frame length and the frame shift to obtain a plurality of short time frames;

[0057] For each short time frame, the autocorrelation value is calculated based on the autocorrelation function;

[0058] When the autocorrelation values ​​of multiple consecutive short-time frames are greater than the autocorrelation threshold, and the peak position change values ​​of the autocorrelation values ​​of multiple consecutive short-time frames are less than the change threshold, the multiple consecutive short-time frames are arranged in chronological order to form a sound state set.

[0059] In this embodiment, the formula for calculating the autocorrelation value is: , where R i (k) is the autocorrelation value of the i-th short-time frame when the delay is k, y i (n) is the amplitude of the nth sampling point in the i-th short time frame, y i (n+k) is the amplitude of the n+kth sampling point in the i-th short-time frame, the value range of k is 0~K / 2, K is the total number of sampling points in a single short-time frame, and i, n, and k are positive integers.

[0060] For Chinese speech, vowels and voiced consonants have stable periodicity (such as the vibration of the vocal cords of "ā"). The formula of the autocorrelation value can accurately extract this periodicity. When k corresponds to the fundamental pitch period, the autocorrelation value will have a significant peak.

[0061] In this embodiment, taking a simple Chinese speech "nǐ hǎo" as an example, the sampling rate is 16 kHz (i.e., 16,000 sampling points per second), the duration is about 0.8 seconds, and the total number of corresponding sampling points is 12,800 (16,000×0.8). "nǐ hǎo" contains two syllables: "nǐ" and "hǎo", and the waveform shows obvious "voiced - transition - voiced" characteristics.

[0062] Frame length: Take 20 ms (corresponding to 320 sampling points, 16,000×0.02); Frame shift: Take 10 ms (160 sampling points) to ensure frame - to - frame overlap and avoid information loss. The 0.8 - second signal of "nǐ hǎo" will be divided into approximately 79 frames ((12,800 - 320) / 160 + 1≈79), and there is overlap between frames.

[0063] The value range of k is half of the frame length, that is, 0~160.

[0064] In this embodiment, the autocorrelation threshold is the mean of the autocorrelation values of all short - time frames.

[0065] For example, if there is an autocorrelation value greater than the autocorrelation threshold in the first short - time frame, an autocorrelation value greater than the autocorrelation threshold in the second short - time frame, and an autocorrelation value greater than the autocorrelation threshold in the third short - time frame, according to the positions corresponding to the maximum autocorrelation values in the first short - time frame, the positions corresponding to the maximum autocorrelation values in the second short - time frame, and the positions corresponding to the maximum autocorrelation values in the third short - time frame, obtain the peak position change value. When the peak position change value is less than the change threshold, the first short - time frame, the second short - time frame, and the third short - time frame are grouped into one phonetic state set.

[0066] In this embodiment, the method for obtaining the peak position change value includes: selecting the maximum value from multiple autocorrelation values corresponding to each short - time frame, recording the delay number corresponding to the maximum value as the peak number, finding the maximum peak number and the minimum peak number corresponding to consecutive multiple short - time frames, subtracting the minimum peak number from the maximum peak number to obtain the peak position deviation value, and taking the ratio of the peak position deviation value to the maximum peak number as the peak position change value.

[0067] The present invention screens out consecutive multiple short - time frames that meet the conditions according to the deviation between the maximum peak number and the minimum peak number in consecutive multiple short - time frames, so that the consecutive multiple short - time frames belong to the same phonetic state. In Chinese language, the pronunciation process of the same phonetic state (such as the vowel "ā" of "mā") has physical continuity - the fundamental pitch period of vocal cord vibration (corresponding to the autocorrelation peak position) does not change suddenly, but shows a smooth transition or regular change.

[0068] The delay amount k corresponding to the voice state period needs to fall within the voice state range of human speech. Due to the different pronunciation frequencies of men and women, for adult males, the effective range of k is within 107~160, and for adult females, the effective range of k is within 53~160. Therefore, in this embodiment, when the voice signal is emitted by a female, the maximum autocorrelation value can be found in the range of k∈(53~160), and when the voice signal is emitted by a male, the maximum autocorrelation value can be found in the range of k∈(107~160).

[0069] In this embodiment, the formula for obtaining the peak position deviation value is: , where γ is the peak position deviation value, k pv,max k is the maximum peak number found from the corresponding peak numbers of multiple consecutive short time frames, pv,min The minimum peak number found from the corresponding peak numbers in multiple consecutive short time frames.

[0070] In this embodiment, the change threshold may be set to 20%.

[0071] In this embodiment, the process of extracting four EMD features of each short time frame includes: performing EMD decomposition on each short time frame in the sound state set to obtain multiple intrinsic mode functions;

[0072] Perform Fourier transform on each eigenmode function to obtain the eigenmode function spectrum;

[0073] Divide the multiple intrinsic mode function spectra corresponding to each short time frame into a low-frequency subset and a high-frequency subset;

[0074] According to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset, the EMD low-frequency center of gravity and the EMD low-frequency energy are obtained;

[0075] According to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset, the EMD high-frequency center of gravity and the EMD high-frequency energy are obtained.

[0076] In this embodiment, the process of dividing the multiple intrinsic mode function spectra corresponding to each short time frame into low-frequency subsets and high-frequency subsets includes: dividing the first intrinsic mode function spectrum and the second intrinsic mode function spectrum into the high-frequency subset; dividing the remaining intrinsic mode function spectra into the low-frequency subset;

[0077] The process of obtaining the EMD low-frequency centroid includes: obtaining the EMD low-frequency centroid according to the fundamental frequency and the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset;

[0078] The process of obtaining the EMD low-frequency energy includes: obtaining the EMD low-frequency energy according to the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset;

[0079] The process of obtaining the high-frequency centroid of EMD includes: obtaining the high-frequency centroid of EMD according to the fundamental frequency and the corresponding amplitude of the fundamental frequency of the spectrum of each intrinsic mode function in the high-frequency subset;

[0080] The process of obtaining the high-frequency energy of EMD includes: obtaining the high-frequency energy of EMD according to the amplitude corresponding to the fundamental frequency of the spectrum of each intrinsic mode function in the high-frequency subset.

[0081] Speech signals (especially the tone changes and phonetic transitions of the Chinese language) have significant nonlinearity and non-stationarity (such as the continuously rising fundamental frequency of the "rising tone" and the sudden change in energy at the connection of "initial consonant-final vowel"). Traditional Fourier transform is difficult to characterize such dynamic changes, while EMD decomposition can adaptively decompose short-time frame signals into multiple intrinsic mode functions (IMFs), and each IMF corresponds to frequency components of different scales (separated layer by layer from high frequency to low frequency): high-frequency IMFs (the 1st and 2nd) capture fast-changing details (such as the frictional noise of consonants and the impact characteristics of plosives); low-frequency IMFs (the remaining part) reflect slow-changing trends (such as the fundamental period of vowels and the overall trend of tones).

[0082] The high-frequency subset focuses on the frictional noise and explosive components of initial consonants (especially voiceless consonants such as "sh, x, k"), and these components have high frequencies and short durations, which exactly match the high-frequency characteristics of the first 2 IMFs. By extracting the high-frequency centroid (reflecting the concentrated area of frictional frequencies) and high-frequency energy (reflecting the frictional intensity), the pronunciation differences of easily confused initial consonants such as "shī - shī" and "kě - kě" can be accurately distinguished.

[0083] Low-frequency subset (the remaining IMFs): corresponding to the vocal cord vibrations of final vowels (such as "a, i, ü") and voiced consonants (such as "m, n, l"), and their energy is concentrated in the low-frequency band and shows regular changes with tones. The low-frequency centroid is directly related to the fundamental frequency (pitch height), and the low-frequency energy reflects the fullness of vowels.

[0084] The fundamental frequency is the frequency corresponding to the maximum amplitude in the spectrum of the intrinsic mode function.

[0085] In this embodiment, the acquisition formulas for both the EMD low-frequency centroid and the EMD high-frequency centroid are: , where C is the centroid, f j is the fundamental frequency of the spectrum of the j-th intrinsic mode function in the subset, y j is the amplitude corresponding to the fundamental frequency of the spectrum of the j-th intrinsic mode function in the subset, L is the number of spectra of the intrinsic mode functions in the subset, and j is a positive integer.

[0086] The specific acquisition processes for both the EMD low-frequency energy and the EMD high-frequency energy are: adding the amplitudes corresponding to the fundamental frequencies of the spectra of each intrinsic mode function in the subset to obtain the energy.

[0087] In this embodiment, the process of constructing the sound state instability coefficient sequence includes:

[0088] In a sound state set, the same EMD features of adjacent short time frames are subtracted to obtain the EMD feature increment;

[0089] In the same sound state set, the fluctuation coefficient is obtained for each EMD feature increment. The four fluctuation coefficients are added and normalized to obtain the sound state instability coefficient.

[0090] Arrange the various sound state instability coefficients in chronological order to construct a sound state instability coefficient sequence.

[0091] The present invention captures the feature increments at adjacent moments through the difference of the same EMD features in adjacent frames, calculates the fluctuation coefficient of the increment (reflecting the severity of the incremental change), adds the fluctuation coefficients of the four features and normalizes them to evaluate the instability of the sound state in the sound state set. The smaller the coefficient, the smoother the pronunciation.

[0092] In this embodiment, the fluctuation coefficient is the variance or standard deviation of each EMD feature increment.

[0093] In a sound state set, the EMD low-frequency centroids of adjacent short-time frames are subtracted to obtain the EMD low-frequency centroid increment, the EMD low-frequency energies of adjacent short-time frames are subtracted to obtain the EMD low-frequency energy increment, the EMD high-frequency centroids of adjacent short-time frames are subtracted to obtain the EMD high-frequency centroid increment, and the EMD high-frequency energies of adjacent short-time frames are subtracted to obtain the EMD high-frequency energy increment;

[0094] In the same sound state set, the fluctuation coefficient is obtained for each EMD low-frequency center of gravity increment, the fluctuation coefficient is obtained for each EMD low-frequency energy increment, the fluctuation coefficient is obtained for each EMD high-frequency center of gravity increment, and the fluctuation coefficient is obtained for each EMD high-frequency energy increment. The four fluctuation coefficients are added and normalized to obtain the sound state instability coefficient.

[0095] In this embodiment, the process of constructing the EMD characteristic change rate sequence includes:

[0096] The EMD feature mean of the sound state set at time t+1 is subtracted from the EMD feature mean of the sound state set at time t to obtain the EMD feature mean difference. The ratio of the EMD feature mean difference to the EMD feature mean of the sound state set at time t is used as the EMD feature change rate at time t. The EMD feature change rates are arranged in chronological order to obtain the EMD feature change rate sequence, where t is the time number.

[0097] When the EMD feature mean is the mean of the EMD low-frequency centroid, the EMD feature change rate is the EMD low-frequency centroid change rate. The EMD low-frequency centroid change rates are arranged in chronological order to obtain the EMD low-frequency centroid change rate sequence; when the EMD feature mean is the EMD low-frequency energy mean, the EMD feature change rate is the EMD low-frequency energy change rate. The EMD low-frequency energy change rate is arranged in chronological order to obtain the EMD low-frequency energy change rate sequence; when the EMD feature mean is the mean of the EMD high-frequency centroid, the EMD feature change rate is the EMD high-frequency centroid change rate. The EMD high-frequency centroid change rate is arranged in chronological order to obtain the EMD high-frequency centroid change rate sequence; when the EMD feature mean is the EMD high-frequency energy mean, the EMD feature change rate is the EMD high-frequency energy change rate. The EMD high-frequency energy change rate is arranged in chronological order to obtain the EMD high-frequency energy change rate sequence.

[0098] The present invention uses the ratio of the difference in the EMD feature mean of the sound state set at adjacent moments to the mean at the previous moment as the rate of change. It eliminates the interference of the absolute value differences of different sound state features (such as the difference in the low-frequency center of gravity baselines of different syllables, and the problem of different pronunciation frequencies and amplitudes of different people) in the form of "relative values", and can reflect the migration of feature changes.

[0099] The rate of change storage sequence includes: EMD low-frequency center of gravity rate of change storage sequence, EMD low-frequency energy rate of change storage sequence, EMD high-frequency center of gravity rate of change storage sequence and EMD high-frequency energy rate of change storage sequence, which are rate of change sequences of four EMD features corresponding to pre-stored standard speech signals.

[0100] like Figure 2 As shown, the pronunciation intonation assessment neural network includes: a sequence feature fusion unit, an enhancement unit, a first one-dimensional convolutional layer, a second one-dimensional convolutional layer and a fully connected layer;

[0101] The input end of the sequence feature fusion unit serves as the input end of the pronunciation intonation evaluation neural network, and its output end is connected to the first input end of the enhancement unit; the second input end of the enhancement unit is used to input the sound state instability coefficient sequence, and its output end is connected to the input end of the first one-dimensional convolutional layer; the output end of the first one-dimensional convolutional layer is connected to the input end of the second one-dimensional convolutional layer; the output end of the second one-dimensional convolutional layer is connected to the input end of the fully connected layer; and the output end of the fully connected layer serves as the output end of the pronunciation intonation evaluation neural network.

[0102] In this embodiment, the convolution kernel sizes of the first one-dimensional convolution layer and the second one-dimensional convolution layer are 1×3.

[0103] The present invention fuses the features of four sequences through a sequence feature fusion unit, and then introduces a sound state instability coefficient sequence (reflecting the degree of fluctuation within the same sound state) through an enhancement unit. By multiplying the sound state instability coefficient with each sound state fusion feature, the abnormal features are highlighted and the accuracy of pronunciation intonation assessment is improved.

[0104] like Figure 3 As shown, the sequence feature fusion unit includes: a third one-dimensional convolution layer, a fourth one-dimensional convolution layer, a fifth one-dimensional convolution layer, a sixth one-dimensional convolution layer and an adder A1;

[0105] The third one-dimensional convolution layer is used to process the EMD low-frequency center of gravity change rate difference sequence; the fourth one-dimensional convolution layer is used to process the EMD low-frequency energy change rate difference sequence; the fifth one-dimensional convolution layer is used to process the EMD high-frequency center of gravity change rate difference sequence; the sixth one-dimensional convolution layer is used to process the EMD high-frequency energy change rate difference sequence;

[0106] The input end of adder A1 is connected to the output end of the third one-dimensional convolutional layer, the output end of the fourth one-dimensional convolutional layer, the output end of the fifth one-dimensional convolutional layer, and the output end of the sixth one-dimensional convolutional layer, and its output end serves as the output end of the sequence feature fusion unit;

[0107] The enhancement unit is used to multiply the audio instability coefficient sequence by the output of the sequence feature fusion unit element by element.

[0108] In this embodiment, the convolution kernel sizes of the third one-dimensional convolution layer, the fourth one-dimensional convolution layer, the fifth one-dimensional convolution layer, and the sixth one-dimensional convolution layer are 1×1.

[0109] The present invention processes each sequence through a 1×1 one-dimensional convolution layer to extract features from each element, adds the four-dimensional features belonging to the same moment to obtain a fused feature, and multiplies the fused feature with the sound instability coefficient to enhance the abnormal features at the same position.

[0110] In the present invention, the EMD feature change rate sequence is obtained by subtracting the EMD feature means of adjacent sound state sets. Therefore, the EMD feature change rate sequence is one element shorter than the sound state instability coefficient sequence. 0 is used to fill the end of the EMD feature change rate sequence so that the length of the EMD feature change rate sequence and the sound state instability coefficient sequence are consistent.

[0111] Chinese pronunciation (especially vowels and voiced consonants) exhibits significant periodicity (e.g., the vibration patterns of the vocal cords). Therefore, the autocorrelation method is used to extract sustained, stable pronunciation phases as a set of sound states. EMD decomposition is then performed on each short-time frame within this set of sound states to extract four EMD features: low-frequency centroid, low-frequency energy, high-frequency centroid, and high-frequency energy. This overcomes the limitations of existing technologies that rely solely on a single feature (such as the Mel-frequency cepstral coefficient). These four features can more comprehensively reflect subtle changes in pronunciation by analyzing the energy distribution and centroid position of different frequency bands.

[0112] On the one hand, the present invention can effectively quantify the pronunciation stability within a single voice state set by constructing a sequence of voice state instability coefficients, and capture unstable phenomena such as tremors and offsets that may occur during the pronunciation process; on the other hand, the change rate sequence calculated based on the mean of the EMD features of adjacent voice state sets characterizes the characteristic changes in the voice state transition stage, which makes up for the defect of the existing technology in insufficient characterization of the voice state transition in continuous speech streams.

[0113] The present invention processes four EMD feature change rate difference sequences through a pronunciation intonation assessment neural network, and performs feature enhancement in combination with a sound state instability coefficient sequence. This not only avoids the subjective bias of manual evaluation, but also highlights the impact of abnormal pronunciation features through feature enhancement, significantly improving the objectivity and accuracy of intonation assessment, and can more accurately reflect the learner's pronunciation level.

[0114] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the pronunciation intonation of the Chinese language, characterized in that: The following steps are involved: The autocorrelation method is used to divide the Chinese language speech signal and obtain multiple sound state sets; Perform EMD decomposition on each short-time frame in the sound state set and extract four EMD features of each short-time frame, where the four EMD features include: EMD low-frequency center of gravity, EMD low-frequency energy, EMD high-frequency center of gravity and EMD high-frequency energy; For a sound state set, the sound state instability coefficient is obtained based on the four EMD features of each short time frame, and the sound state instability coefficient sequence is constructed; For a sound state set, take the mean of each EMD feature of each short time frame; According to the difference of the EMD feature mean values ​​of adjacent sound state sets, the change rate is obtained and four EMD feature change rate sequences are constructed; Subtract each change rate sequence from the corresponding change rate storage sequence to obtain the change rate difference sequence; The pronunciation intonation evaluation neural network is used to process the four EMD feature change rate difference sequences, and feature enhancement is performed based on the sound state instability coefficient sequence to obtain the pronunciation intonation score.

2. The Chinese language pronunciation intonation evaluation method according to claim 1, wherein: The process of obtaining the plurality of sound state sets includes: dividing the Chinese language speech signal according to the frame length and the frame shift to obtain a plurality of short time frames; For each short time frame, the autocorrelation value is calculated based on the autocorrelation function; When the autocorrelation values ​​of multiple consecutive short-time frames are greater than the autocorrelation threshold, and the peak position change values ​​of the autocorrelation values ​​of multiple consecutive short-time frames are less than the change threshold, the multiple short-time frames are arranged in chronological order to form a sound state set.

3. The Chinese language pronunciation intonation evaluation method according to claim 2, characterized in that: The formula for calculating the autocorrelation value is: , where R i (k) is the autocorrelation value of the i-th short-time frame when the delay is k, y i (n) is the amplitude of the nth sampling point in the i-th short time frame, y i (n+k) is the amplitude of the n+kth sampling point in the i-th short-time frame, the value range of k is 0~K / 2, K is the total number of sampling points in a single short-time frame, and i, n, and k are positive integers.

4. The Chinese language pronunciation intonation evaluation method according to claim 3, wherein: The method for obtaining the peak position change value includes: selecting the maximum value from multiple autocorrelation values ​​corresponding to each short-time frame, recording the delay number corresponding to the maximum value as the peak number, finding the corresponding maximum peak number and minimum peak number in multiple consecutive short-time frames, subtracting the minimum peak number from the maximum peak number to obtain the peak position deviation value, and taking the ratio of the peak position deviation value to the maximum peak number as the peak position change value.

5. The Chinese language pronunciation intonation evaluation method according to claim 1, wherein: The process of extracting the four EMD features of each short-time frame includes: performing EMD decomposition on each short-time frame in the sound state set to obtain multiple intrinsic mode functions; Perform Fourier transform on each eigenmode function to obtain the eigenmode function spectrum; Divide the multiple intrinsic mode function spectra corresponding to each short time frame into a low-frequency subset and a high-frequency subset; According to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset, the EMD low-frequency center of gravity and the EMD low-frequency energy are obtained; According to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset, the EMD high-frequency center of gravity and the EMD high-frequency energy are obtained.

6. The Chinese language pronunciation intonation evaluation method according to claim 5, characterized in that: The process of dividing the multiple intrinsic mode function spectra corresponding to each short time frame into low frequency subsets and high frequency subsets includes: dividing the first intrinsic mode function spectrum and the second intrinsic mode function spectrum into the high frequency subset; dividing the remaining intrinsic mode function spectra into the low frequency subset; The process of obtaining the EMD low-frequency centroid includes: obtaining the EMD low-frequency centroid according to the fundamental frequency and the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset; The process of obtaining the EMD low-frequency energy includes: obtaining the EMD low-frequency energy according to the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the low-frequency subset; The process of obtaining the EMD high-frequency centroid includes: obtaining the EMD high-frequency centroid according to the fundamental frequency and the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset; The process of obtaining the EMD high-frequency energy includes: obtaining the EMD high-frequency energy according to the amplitude corresponding to the fundamental frequency of each intrinsic mode function spectrum in the high-frequency subset.

7. The Chinese language pronunciation intonation evaluation method according to claim 1, wherein: The process of constructing the sequence of acoustic instability coefficients includes: In a sound state set, the same EMD features of adjacent short time frames are subtracted to obtain the EMD feature increment; In the same sound state set, the fluctuation coefficient is obtained for each EMD feature increment. The four fluctuation coefficients are added and normalized to obtain the sound state instability coefficient. Arrange the various sound state instability coefficients in chronological order to construct a sound state instability coefficient sequence.

8. The Chinese language pronunciation intonation evaluation method according to claim 1, wherein: The process of constructing the EMD characteristic change rate series includes: The EMD feature mean of the sound state set at time t+1 is subtracted from the EMD feature mean of the sound state set at time t to obtain the EMD feature mean difference. The ratio of the EMD feature mean difference to the EMD feature mean of the sound state set at time t is used as the EMD feature change rate at time t. The EMD feature change rates are arranged in chronological order to obtain the EMD feature change rate sequence, where t is the time number. When the EMD feature mean is the mean of the EMD low-frequency centroid, the EMD feature change rate is the EMD low-frequency centroid change rate. The EMD low-frequency centroid change rates are arranged in chronological order to obtain the EMD low-frequency centroid change rate sequence; when the EMD feature mean is the EMD low-frequency energy mean, the EMD feature change rate is the EMD low-frequency energy change rate. The EMD low-frequency energy change rate is arranged in chronological order to obtain the EMD low-frequency energy change rate sequence; when the EMD feature mean is the mean of the EMD high-frequency centroid, the EMD feature change rate is the EMD high-frequency centroid change rate. The EMD high-frequency centroid change rate is arranged in chronological order to obtain the EMD high-frequency centroid change rate sequence; when the EMD feature mean is the EMD high-frequency energy mean, the EMD feature change rate is the EMD high-frequency energy change rate. The EMD high-frequency energy change rate is arranged in chronological order to obtain the EMD high-frequency energy change rate sequence.

9. The Chinese language pronunciation intonation evaluation method according to claim 1, wherein: The pronunciation intonation assessment neural network includes: a sequence feature fusion unit, an enhancement unit, a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, and a fully connected layer; The input end of the sequence feature fusion unit serves as the input end of the pronunciation intonation evaluation neural network, and its output end is connected to the first input end of the enhancement unit; the second input end of the enhancement unit is used to input the sound state instability coefficient sequence, and its output end is connected to the input end of the first one-dimensional convolutional layer; the output end of the first one-dimensional convolutional layer is connected to the input end of the second one-dimensional convolutional layer; the output end of the second one-dimensional convolutional layer is connected to the input end of the fully connected layer; and the output end of the fully connected layer serves as the output end of the pronunciation intonation evaluation neural network.

10. The Chinese language pronunciation intonation evaluation method according to claim 9, characterized in that: The sequence feature fusion unit includes: a third one-dimensional convolution layer, a fourth one-dimensional convolution layer, a fifth one-dimensional convolution layer, a sixth one-dimensional convolution layer and an adder A1; The third one-dimensional convolution layer is used to process the EMD low-frequency center of gravity change rate difference sequence; the fourth one-dimensional convolution layer is used to process the EMD low-frequency energy change rate difference sequence; the fifth one-dimensional convolution layer is used to process the EMD high-frequency center of gravity change rate difference sequence; the sixth one-dimensional convolution layer is used to process the EMD high-frequency energy change rate difference sequence; The input end of adder A1 is connected to the output end of the third one-dimensional convolutional layer, the output end of the fourth one-dimensional convolutional layer, the output end of the fifth one-dimensional convolutional layer, and the output end of the sixth one-dimensional convolutional layer, and its output end serves as the output end of the sequence feature fusion unit; The enhancement unit is used to multiply the audio instability coefficient sequence by the output of the sequence feature fusion unit element by element.

Images