Intelligent voice-based pte oral question and answer evaluation method and system

Abstract

The application relates to the technical field of speech processing, and particularly discloses a PTE oral language question and answer evaluation method and system based on intelligent speech, which comprises the following steps: acquiring an original speech signal and extracting an acoustic feature sequence; calculating an instantaneous speech speed through phase space reconstruction and a recursive rate matrix, generating a time length regularization factor when the value exceeds a threshold; determining a fractional Fourier transform order according to the factor, calculating an instantaneous frequency deviation weighted cosine distance matrix, and dynamically programming backtracking to obtain a regularized feature sequence; performing multiple fractal detrended fluctuation analysis to locate a singular fluctuation candidate segment, determining an ambiguous phoneme segment in combination with a recursive graph certainty percentage and a laminar flow percentage, and extracting adjacent stable segment phoneme labels as reconstruction conditions; performing phase reconstruction on a clear feature sequence, calculating a phase mutual correlation peak value and a spectral envelope Riemann geodesic distance, obtaining pronunciation confidence, and weighting and summing fluency parameters to obtain an evaluation score; and the application solves the score distortion problem caused by time length mismatch and unclear read-through boundaries under an abnormal speech speed.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to a PTE oral question-and-answer assessment method and system based on intelligent speech. Background Technology

[0002] Currently, PTE speaking assessment methods based on intelligent speech primarily rely on a combination of automatic speech recognition technology and acoustic feature analysis. Existing technologies typically first extract acoustic features such as Mel-frequency cepstral coefficients using speech signal processing techniques. Then, a forced alignment algorithm based on a Hidden Markov Model is used to align the candidate's speech with a standard phoneme sequence along the time axis. Finally, the likelihood ratio score between the acoustic features and the standard template is calculated to evaluate pronunciation accuracy, and a comprehensive score is derived by combining parameters such as speech rate and pauses.

[0003] The existing technology has the following shortcomings:

[0004] When test takers speak at an abnormally fast pace due to factors such as nervousness, the speech signal exhibits nonlinear compression of phoneme duration in the time domain and blurred acoustic boundaries due to increased coarticulation in the frequency domain. These two factors couple to form a double degradation effect. Existing forced alignment techniques rely on linear duration constraints and steady-state acoustic models, which cannot effectively normalize the nonlinear duration distortion under extreme speaking speeds and lack a mechanism for identifying and repairing the blurred acoustic boundaries caused by connected speech. This leads to boundary misalignment or phoneme insertion / deletion errors in the blurred segments during forced alignment. More seriously, because the speech energy spectrum tends to smooth in fast speech, likelihood ratio scoring based solely on energy features cannot distinguish between "efficient connected speech" and "unclear elision," causing a systematic bias in the scoring system's quantification of pronunciation accuracy, ultimately resulting in distorted oral assessment results. This problem has remained unresolved for a long time because traditional methods treat duration normalization and acoustic boundary restoration as two independent processes, lacking a coupled processing mechanism. Summary of the Invention

[0005] The purpose of this invention is to provide a PTE oral question-and-answer assessment method and system based on intelligent voice, so as to solve the problems mentioned above.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] The PTE speaking test assessment method based on intelligent voice includes the following steps:

[0008] S1: Obtain the original speech signal of the candidate's answer to the PTE Speaking task, perform noise reduction and endpoint detection on the original speech signal, and extract a continuous acoustic feature sequence.

[0009] S2: Analyze the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds the preset speech rate threshold, generate a normalization control parameter to indicate the duration of the abnormal speech rate, and output the acoustic feature sequence in association with the normalization control parameter.

[0010] S3: Based on the regularization control parameters, the acoustic feature sequence is input into the dynamic time regularization algorithm. By calculating the globally optimal path between it and the standard reference phoneme template, the acoustic feature sequence is nonlinearly time-scaling and mapping is performed, and the regularized feature sequence after time regularization is output.

[0011] S4: Identify fuzzy phoneme segments in regular feature sequences whose acoustic boundaries are blurred due to liaison. Input the fuzzy phoneme segments into a pre-trained conditional variational autoencoder network, reconstruct features using the corresponding standard phoneme labels as conditions, and output a clear feature sequence after replacing the fuzzy phoneme segments.

[0012] S5: Force alignment of the clear feature sequence with the standard reference phoneme template, determine the boundary of each phoneme, and calculate the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. Combine the pronunciation confidence and fluency parameters to generate the final spoken language assessment score.

[0013] As a further aspect of the present invention: S1 specifically includes:

[0014] The original speech signal is subjected to variational mode decomposition to obtain multiple intrinsic mode function components;

[0015] Singular value decomposition is performed on each intrinsic mode function component. The denoised speech signal is obtained by retaining the main singular values ​​and reconstructing them by inverse transformation.

[0016] The Hilbert-Huang spectrum of the denoised speech signal is calculated, and the start and end points of the speech are detected based on the instantaneous energy fluctuation rate of the Hilbert-Huang spectrum to extract the effective speech segment.

[0017] An adaptive weighted gamma-pass filter is applied to the effective speech segments, and the gamma-pass cepstral coefficients of each frame are extracted as the acoustic feature sequence.

[0018] As a further aspect of the present invention: S2 specifically includes:

[0019] Phase space reconstruction is performed on the acoustic feature sequence to obtain a high-dimensional phase space trajectory;

[0020] Calculate the recurrence rate matrix of the high-dimensional phase space trajectory, and extract the average length of the diagonal structure in the recurrence rate matrix as the instantaneous speech rate value;

[0021] The instantaneous speech rate value is compared with the preset speech rate threshold. When the instantaneous speech rate value exceeds the preset speech rate threshold, the duration normalization factor is calculated based on the amount of excess.

[0022] The duration warping factor and acoustic feature sequence are encapsulated to form associated data pairs and then output.

[0023] As a further aspect of the present invention: the phase space reconstruction of the acoustic feature sequence to obtain a high-dimensional phase space trajectory specifically includes:

[0024] Continuous wavelet transform is performed on the acoustic feature sequence to obtain the time-frequency energy distribution of the acoustic feature sequence;

[0025] The instantaneous dominant frequency and instantaneous bandwidth values ​​at each moment are extracted from the time-frequency energy distribution to form an instantaneous feature pair sequence;

[0026] Phase space reconstruction is performed on the instantaneous feature pair sequence, and the instantaneous feature pair at each moment is combined with the feature pairs at several moments before and after the corresponding moment to form the high-dimensional embedding point at the corresponding moment.

[0027] Arrange the high-dimensional embedding points at all times in chronological order to obtain the high-dimensional phase space trajectory.

[0028] As a further aspect of the present invention: S3 specifically includes:

[0029] Based on the duration normalization factor in the normalization control parameters, a fractional Fourier transform is performed on the acoustic feature sequence to obtain the rotation domain feature sequence.

[0030] Calculate the cosine distance matrix between the rotated domain feature sequence and the standard reference phoneme template in the fractional Fourier domain, and use it as a measure of local cost;

[0031] Dynamic programming backtracking is performed on the cosine distance matrix to search for the globally optimal path from the starting point to the ending point, and to obtain the mapping relationship between the acoustic features of each frame and the template frame.

[0032] Based on the mapping relationship, the original acoustic feature sequence is interpolated and resampled to obtain a regular feature sequence that is aligned with the duration of the standard reference phoneme template.

[0033] As a further aspect of the present invention: the calculation of the cosine distance matrix between the rotated domain feature sequence and the standard reference phoneme template in the fractional Fourier domain, as a local cost metric, specifically includes:

[0034] The amplitude of the rotation domain feature sequence is normalized in the fractional Fourier domain to obtain the amplitude-normalized rotation domain feature sequence.

[0035] The amplitude-normalized rotation domain feature sequence and the standard reference phoneme template are subjected to instantaneous frequency trajectory extraction to obtain the feature instantaneous frequency curve and the template instantaneous frequency curve.

[0036] Calculate the local frequency deviation between the feature instantaneous frequency curve and the template instantaneous frequency curve at the same rotation order, and use the reciprocal of the corresponding frequency deviation as the weight to weight the original amplitude in the fractional Fourier domain.

[0037] Calculate the inner product between the weighted rotated domain feature sequence and the standard reference phoneme template, invert the inner product value and use it as the corresponding element in the local cost matrix, and obtain the complete cosine distance matrix after traversing all frame pairs.

[0038] As a further aspect of the present invention: S4 specifically includes:

[0039] Multifractal detrending fluctuation analysis was performed on the regular characteristic sequence to calculate the generalized Hearst exponent sequence of the regular characteristic sequence.

[0040] Based on the fluctuation inflection point of the generalized Hearst exponent sequence, locate the candidate segment where the acoustic feature exhibits singular fluctuations;

[0041] Recursive graph analysis is performed on the regular feature subsequences corresponding to the candidate segment positions to calculate the deterministic percentage and laminar flow percentage of the recursive graph;

[0042] When the certainty percentage is lower than the preset first threshold and the laminar flow percentage is higher than the preset second threshold, the corresponding candidate segment is determined to be a fuzzy phoneme segment, and the standard phoneme labels corresponding to the stable phoneme segments adjacent to the corresponding fuzzy phoneme segment are extracted as reconstruction conditions.

[0043] As a further aspect of the present invention: S5 specifically includes:

[0044] Phase reconstruction is performed on each phoneme segment in the clear feature sequence, and the instantaneous phase and group delay features of each frame in the corresponding phoneme segment are extracted.

[0045] The instantaneous phase and group delay feature are aligned by affine transformation, and the phase cross-correlation peak between the corresponding phoneme segment and the corresponding standard reference phoneme template is calculated.

[0046] Based on the magnitude of the phase cross-correlation peak and the similarity of the amplitude spectrum envelope within the corresponding phoneme segment, the initial confidence value of the corresponding phoneme segment is obtained through weighted fusion.

[0047] After traversing all phoneme segments to obtain the initial confidence value sequence, the initial confidence value sequence is subjected to context smoothing to output the final pronunciation confidence sequence.

[0048] As a further aspect of the present invention: the step of obtaining the initial confidence value of the corresponding phoneme segment by weighted fusion based on the magnitude of the phase cross-correlation peak and the amplitude spectrum envelope similarity within the corresponding phoneme segment specifically includes:

[0049] A complex cepstral transform is performed on the amplitude spectrum envelope within the phoneme segment, and the low-order complex cepstral coefficients of the amplitude spectrum envelope are extracted as spectral feature vectors.

[0050] Calculate the geodesic distance on the Riemannian manifold between the spectral feature vector and the spectral feature vector of the corresponding standard reference phoneme template, and convert the corresponding distance value into amplitude spectrum envelope similarity;

[0051] The phase cross-correlation peaks are nonlinearly normalized to obtain the phase similarity. The phase similarity and the amplitude spectrum envelope similarity are then combined in a convex combination operation in the Banach space to obtain the fusion confidence.

[0052] The fused confidence score is input into the trust function mapping layer based on evidence theory. Through confidence allocation and synthesis rules, the initial confidence score of the corresponding phoneme segment is output.

[0053] The PTE speaking assessment system based on intelligent voice includes:

[0054] The acoustic feature extraction module acquires the raw speech signal of the candidate's answer to the PTE speaking task, performs noise reduction and endpoint detection on the raw speech signal, and extracts a continuous acoustic feature sequence.

[0055] The speech rate analysis and normalization parameter generation module analyzes the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds the preset speech rate threshold, it generates normalization control parameters to indicate the duration of abnormal speech rate and outputs the acoustic feature sequence in association with the normalization control parameters.

[0056] The nonlinear duration warping module, based on the warping control parameters, inputs the acoustic feature sequence into the dynamic time warping algorithm. By calculating the globally optimal path between the acoustic feature sequence and the standard reference phoneme template, it performs nonlinear duration scaling mapping on the acoustic feature sequence and outputs the duration-warped feature sequence.

[0057] The fuzzy phoneme segment recognition and reconstruction module identifies fuzzy phoneme segments in a regular feature sequence whose acoustic boundaries are blurred due to liaison. The fuzzy phoneme segments are input into a pre-trained conditional variational autoencoder network, and feature reconstruction is performed using the corresponding standard phoneme labels as conditions. The output is a clear feature sequence after replacing the fuzzy phoneme segments.

[0058] The forced alignment and confidence scoring module forces the clear feature sequence to align with the standard reference phoneme template, determines the boundary of each phoneme, and calculates the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. The final oral assessment score is generated by combining the pronunciation confidence and fluency parameters.

[0059] The beneficial effects of this invention are:

[0060] (1) This invention accurately calculates instantaneous speech rate through phase space reconstruction and recursion rate analysis, and performs nonlinear duration scaling mapping in the fractional Fourier domain based on the duration regularization factor, thus solving the alignment failure problem caused by duration mismatch in traditional forced alignment under abnormal speech rates. Combined with the cosine distance matrix weighted by instantaneous frequency deviation, it effectively distinguishes between efficient and fluent expression and unclear elision, avoiding misjudgment of pronunciation accuracy due to excessive speech rate, and improving the robustness and scoring accuracy of the evaluation system under extreme speech rate conditions.

[0061] (2) This invention uses multifractal detrended fluctuation analysis and recursive graph analysis to accurately locate phoneme segments with blurred acoustic boundaries caused by connected speech, and reconstructs features based on adjacent stable phoneme labels to restore the acoustic boundary information of the blurred segments. In the confidence calculation, the instantaneous phase, group delay features and Riemann geodesic distance of the amplitude spectrum envelope are integrated, and evidence theory is introduced for contextual confidence synthesis, so that the pronunciation quality assessment not only depends on the energy spectrum, but also takes into account the fine time-frequency structure, effectively solving the boundary confusion problem caused by connected speech. Attached Figure Description

[0062] The invention will now be further described with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart of the method of the present invention;

[0064] Figure 2 This is a system block diagram of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Please see Figure 1 As shown, this invention is a PTE oral question-and-answer assessment method based on intelligent voice, including the following steps:

[0067] S1: Obtain the original speech signal of the candidate's answer to the PTE Speaking task, perform noise reduction and endpoint detection on the original speech signal, and extract a continuous acoustic feature sequence.

[0068] S2: Analyze the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds the preset speech rate threshold, generate a normalization control parameter to indicate the duration of the abnormal speech rate, and output the acoustic feature sequence in association with the normalization control parameter.

[0069] S3: Based on the regularization control parameters, the acoustic feature sequence is input into the dynamic time regularization algorithm. By calculating the globally optimal path between it and the standard reference phoneme template, the acoustic feature sequence is nonlinearly time-scaling and mapping is performed, and the regularized feature sequence after time regularization is output.

[0070] S4: Identify fuzzy phoneme segments in regular feature sequences whose acoustic boundaries are blurred due to liaison. Input the fuzzy phoneme segments into a pre-trained conditional variational autoencoder network, reconstruct features using the corresponding standard phoneme labels as conditions, and output a clear feature sequence after replacing the fuzzy phoneme segments.

[0071] S5: Force alignment of the clear feature sequence with the standard reference phoneme template, determine the boundary of each phoneme, and calculate the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. Combine the pronunciation confidence and fluency parameters to generate the final spoken language assessment score.

[0072] In S1, the raw audio signal of the candidate's response to the PTE Speaking task is acquired. Noise reduction and endpoint detection are performed on the raw audio signal to extract a continuous acoustic feature sequence, specifically including:

[0073] First, the raw audio signal of the candidate's response to the PTE Speaking task is acquired. The specific acquisition process is as follows: the audio is acquired using a high-fidelity microphone at a sampling rate of 16 kHz, with a quantization precision of 16 bits, and stored as raw audio data in pulse-code modulation format. Then, this raw audio signal is input into a variational mode decomposition algorithm for processing. In this algorithm, a penalty factor of 2000 is set, and the preset number of mode decompositions is 5. Through iterative solution of the variational problem, the raw audio signal is adaptively decomposed into 5 finite-bandwidth intrinsic mode function components around their respective center frequencies. The iteration termination condition is set to the relative change in the norm of each mode component obtained from two consecutive iterations being less than 10 to the power of -6.

[0074] The five intrinsic mode function (IMF) components obtained in the above steps are used to construct trajectory matrices, and singular value decomposition (SVD) is performed on each trajectory matrix. After decomposition, a sequence of singular values ​​is obtained. For each IMF component, the top 85% of the singular values ​​in its sequence are retained, and the remaining singular values ​​are set to zero. Inverse transform reconstruction is performed using the retained main singular values ​​to obtain the reconstructed IMF components. All reconstructed IMF components are then superimposed to obtain the denoised speech signal.

[0075] The denoised speech signal is subjected to a Hilbert-Huang transform to obtain its Hilbert amplitude spectrum. Based on the Hilbert amplitude spectrum, the instantaneous energy curve is obtained by integrating along the time axis. Then, the instantaneous energy fluctuation rate of this instantaneous energy curve is calculated, defined as the difference between the instantaneous energy value at the current moment and the instantaneous energy value at the previous moment divided by the instantaneous energy value at the current moment. Speech start and end point detection thresholds are set: the start point threshold is set to 3 times the average instantaneous energy value of the background noise segment, and the end point threshold is set to the same as the start point threshold. Scanning begins from the start of the speech signal; when the instantaneous energy fluctuation rate continuously exceeds the start point threshold for 30 milliseconds, that moment is marked as the speech start point. Scanning in reverse from the end of the speech signal, when the instantaneous energy fluctuation rate continuously exceeds the end point threshold for 30 milliseconds, that moment is marked as the speech end point. The signal segment between the speech start and end points is extracted to obtain the effective speech segment.

[0076] The effective speech segments are segmented into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. Each frame of speech signal is adaptively weighted and filtered using a filter bank consisting of 50 gamma-pass filters. The center frequency of this filter bank is uniformly distributed within an equivalent rectangular bandwidth scale from 80 Hz to 8000 Hz. The adaptive weighting coefficients are dynamically adjusted based on the ratio of the short-time energy of each frame to the global average energy. For each frame output after filtering, the logarithmic energy is taken, and a discrete cosine transform is performed to obtain the static gamma-pass cepstral coefficients for each frame. Then, the first and second differences of the static gamma-pass cepstral coefficients are calculated, and the static parameters, first-order difference parameters, and second-order difference parameters are concatenated to form a 39-dimensional acoustic feature vector for each frame. The 39-dimensional acoustic feature vectors of all frames are arranged in chronological order to obtain a continuous acoustic feature sequence.

[0077] In S2, the instantaneous speech rate value of the acoustic feature sequence is analyzed. When the instantaneous speech rate value exceeds a preset speech rate threshold, a normalization control parameter is generated to indicate the duration of the abnormal speech rate. The acoustic feature sequence is then correlated with the normalization control parameter and output, specifically including:

[0078] Phase space reconstruction is performed on the acoustic feature sequence to obtain a high-dimensional phase space trajectory that reflects the dynamic characteristics of speech. Specifically, a continuous wavelet transform is first performed on the acoustic feature sequence. The complex Morlet wavelet is selected as the mother wavelet, with a center frequency of 1 Hz and a bandwidth parameter of 2 Hz. The scale range of the wavelet transform is set to 1 to 64, with a scale step of 0.5. Through continuous wavelet transform, the complex wavelet coefficients of the acoustic feature sequence at each scale are obtained, and the square of their modulus is the time-frequency energy distribution of the acoustic feature sequence.

[0079] From the time-frequency energy distribution, the instantaneous dominant frequency and instantaneous bandwidth are extracted for each moment. For a given moment, the maximum value of the wavelet coefficient modulus at all scales is searched, and the scale corresponding to this maximum value is converted into a frequency value, which is the instantaneous dominant frequency. Simultaneously, with this instantaneous dominant frequency as the center, the width of the frequency range corresponding to when the wavelet coefficient modulus drops to half of its maximum value is calculated, and this width value is the instantaneous bandwidth. The instantaneous dominant frequency and instantaneous bandwidth values ​​at each moment are combined to form an instantaneous feature pair sequence.

[0080] The instantaneous feature pair sequence is reconstructed in phase space. The delay time is calculated using the mutual information method, with three time points defined. The embedding dimension is calculated using the pseudo-nearest neighbor method, with an embedding dimension of 8. For the i-th time point in the instantaneous feature pair sequence, the instantaneous feature pair at that time is combined with the instantaneous feature pairs at the 3rd, 6th, and 9th time points before, and the 3rd, 6th, and 9th time points after, forming a 16-dimensional high-dimensional embedding point. This process is repeated for all time points, and the high-dimensional embedding points are arranged chronologically to obtain the high-dimensional phase space trajectory.

[0081] Calculate the recurrence rate matrix of the high-dimensional phase space trajectory. First, calculate the Euclidean distance between any two high-dimensional embedding points in the high-dimensional phase space trajectory. Set the recurrence radius to 0.15 times the standard deviation of the Euclidean distance sequence. For any two times i and j, if the distance between the high-dimensional embedding points corresponding to these two times is less than the recurrence radius, set the element in the i-th row and j-th column of the recurrence rate matrix to 1; otherwise, set it to 0. After traversing all times, obtain the recurrence rate matrix composed of 0 and 1 elements.

[0082] Extract the average length of the diagonal structure in the recurrence rate matrix. Scan all diagonals parallel to the main diagonal of the recurrence rate matrix and record the length of the line segment with consecutive elements of 1 on each diagonal. Ignore isolated points with a length of 1, and statistically calculate the arithmetic mean of the lengths of all line segments with a length greater than or equal to 2. Use this average as the instantaneous speech rate value. The larger this value, the stronger the periodicity of the speech signal trajectory in phase space, corresponding to more syllables per unit time, i.e., a faster speech rate.

[0083] The preset speech rate threshold is set to 7 syllables per second. The calculated instantaneous speech rate value is compared with the preset speech rate threshold. When the instantaneous speech rate value exceeds the preset speech rate threshold, the excess is calculated, which is the difference between the instantaneous speech rate value and the preset speech rate threshold. A duration warping factor is calculated based on this excess. Specifically, the difference is divided by the preset speech rate threshold to obtain a relative excess ratio. This relative excess ratio is then multiplied by 0.3 to obtain a duration compression coefficient. Finally, this duration compression coefficient is subtracted from 1 to obtain the duration warping factor. This duration warping factor is a value between 0.7 and 1.0, used to indicate the proportion of compression applied to the original acoustic feature sequence in subsequent nonlinear duration warping operations.

[0084] The calculated duration warping factor is encapsulated with the original acoustic feature sequence to form a correlated data pair. This correlated data pair consists of two parts: the duration warping factor and the original acoustic feature sequence. The correlated data pair is then output to the next step.

[0085] In S3, based on the warping control parameters, the acoustic feature sequence is input into the dynamic time warping algorithm. By calculating the globally optimal path between the sequence and the standard reference phoneme template, a nonlinear time-scaling mapping is performed on the acoustic feature sequence, outputting a time-warped feature sequence, specifically including:

[0086] First, the associated data pair output from step S2 is received. This associated data pair includes a duration warping factor and the original acoustic feature sequence. The duration warping factor is a value between 0.7 and 1.0, used to indicate the compression ratio of the acoustic feature sequence; the acoustic feature sequence consists of a 39-dimensional feature vector of T frames, denoted as... The frame rate is 100 frames per second. Simultaneously, a pre-stored standard reference phoneme template is acquired, which consists of 39-dimensional feature vectors from S frames, denoted as... , where S corresponds to the reference duration of the standard pronunciation.

[0087] Based on the duration warping factor, a fractional Fourier transform is performed on the original acoustic feature sequence. First, the transform order is determined according to the duration warping factor. Specifically, the smaller the duration warping factor value, the greater the degree of compression required, and the closer the transform order is to 1. In this embodiment, when the duration warping factor is 1.0, the transform order is set to 0; when the duration warping factor is 0.7, the transform order is set to 0.5; for duration warping factors between 0.7 and 1.0, the transform order is determined by linear interpolation. A fractional Fourier transform is then performed on the 39-dimensional feature vector of each frame in the acoustic feature sequence, transforming the temporal features to the fractional Fourier domain. After the transform, the 39-dimensional feature vector of each frame becomes a 39-dimensional rotation domain feature vector. The rotation domain feature vectors of all frames are arranged in chronological order, forming a rotation domain feature sequence, denoted as . The superscript 'a' indicates the corresponding transformation order. Simultaneously, a fractional Fourier transform of the same order is performed on the standard reference phoneme template to obtain the rotation domain standard template, denoted as . .

[0088] Calculate the cosine distance matrix between the feature sequence of the rotated domain and the standard template of the rotated domain, which serves as the local cost metric in subsequent dynamic programming. This process specifically includes the following sub-steps:

[0089] The first step is to perform fractional Fourier domain amplitude normalization on the rotation domain feature sequence. For each frame feature vector in the rotation domain feature sequence... Calculate its L2 norm, which is the square root of the sum of the squares of each dimension of the vector. Then, divide each element of the vector by its L2 norm to obtain the amplitude-normalized rotation domain feature vector. Perform the same operation on all frames to obtain the amplitude-normalized rotation domain feature sequence, denoted as . Perform the same normalization operation on the standard template of the rotated domain to obtain the normalized standard template of the rotated domain. .

[0090] The second step involves extracting the instantaneous frequency trajectories from the amplitude-normalized rotational domain feature sequence and the standard rotational domain template. For the normalized rotational domain feature sequence, the phase spectrum of each frame's feature vector in the fractional Fourier domain is calculated. Then, the partial derivative of the phase spectrum along the frequency axis is taken to obtain the corresponding instantaneous frequency value for that frame, which is a 39-dimensional instantaneous frequency vector. The instantaneous frequency vectors of all frames are arranged in chronological order to obtain the characteristic instantaneous frequency curve, denoted as... , where f t Let represent the 39-dimensional instantaneous frequency vector of the t-th frame. Similarly, extract the template instantaneous frequency curve from the standard template in the rotated domain, denoted as . ,in This represents the 39-dimensional instantaneous frequency vector of the s-th frame.

[0091] The third step involves calculating the local frequency deviation between the feature instantaneous frequency curve and the template instantaneous frequency curve at the same rotation order, and then using the reciprocal of this frequency deviation as a weight to weight the original amplitudes in the fractional Fourier domain. Specifically, for the t-th frame of the feature sequence and the s-th frame of the template, the frequency deviation between the two frames is first calculated. This deviation is defined as the square root of the sum of the squares of the differences between the corresponding dimensions of the instantaneous frequency vectors of the two frames, i.e., the Euclidean distance, denoted as . Then, calculate the reciprocal of the frequency deviation, i.e. , as weighting coefficients. Next, the normalized rotation domain eigenvectors are... With the normalized rotation domain template vector Each element of the dimension is multiplied by the weighting coefficient. The weighted feature vector is obtained. and weighted template vector The physical significance of this step is that when the instantaneous frequency characteristics of two frames differ greatly, the frequency deviation is large and its reciprocal is small, thus reducing the contribution of these two frames to the subsequent similarity calculation; conversely, when the instantaneous frequency characteristics are similar, the frequency deviation is small and its reciprocal is large, thus enhancing the contribution of these two frames.

[0092] The fourth step involves calculating the inner product between the weighted rotated domain feature sequence and the standard reference phoneme template, and then inverting this inner product value to use the corresponding element in the local cost matrix. For the t-th frame of the feature sequence and the s-th frame of the template, the weighted feature vector is calculated. with the weighted template vector The inner product is the sum of the products of the corresponding dimensions of the two vectors. Since both vectors have been normalized in magnitude and have the same weighting coefficients, this inner product is actually equivalent to the weighted cosine similarity. Then, inverting the inner product value yields... This yields the local cost metric in row t and column s. The above calculation is performed for all t from 1 to T and s from 1 to S, resulting in a T-row, S-column cosine distance matrix after traversing all frame pairs. This matrix is ​​denoted as... Each element in this matrix represents the local cost required to map the t-th frame of the feature sequence to the s-th frame of the template; the smaller the cost, the better the match between the two frames.

[0093] Perform dynamic programming backtracking on the cosine distance matrix C to search for the globally optimal path from the starting point (1,1) to the ending point (T,S). The recursive process of dynamic programming is as follows: First, initialize the cumulative cost matrix D, let... Then, traverse each position (t, s) in the matrix in ascending row and column order, and calculate the minimum cumulative cost to reach that position. This minimum cumulative cost is the sum of the cumulative costs of the three possible previous positions (t-1, s), (t, s-1), and (t-1, s-1) plus the current local cost. The minimum value in the sequence determines the path. Simultaneously, the direction of the previous position chosen for each location is recorded, forming a path backtracking pointer. After traversal, starting from the endpoint (T,S), the path is traced back to the starting point (1,1) according to the backtracking pointer, resulting in a path composed of several (t,s) coordinate points. This path is the globally optimal path. Each coordinate point (t,s) on this path represents that the t-th frame in the original acoustic feature sequence should be mapped to the s-th frame in the standard reference phoneme template. Thus, the mapping relationship between each frame of acoustic features and the template frame is obtained.

[0094] Based on the mapping relationship, the original acoustic feature sequence is interpolated and resampled to obtain a regularized feature sequence aligned with the duration of the standard reference phoneme template. Specifically, for the s-th frame in the standard reference phoneme template, all feature sequence frame indices mapped to this template frame in the global optimal path are found, and the set of these frame indices is denoted as Is = {t | (t, s) ∈ optimal path}. If the set Is contains multiple frame indices, cubic spline interpolation is performed on the original feature vectors corresponding to these frames to obtain the regularized feature vector of the s-th frame; if the set Is contains only one frame index, the original feature vector corresponding to that frame is directly taken as the regularized feature vector of the s-th frame; if the set Is is empty, linear interpolation is performed based on the regularized feature vectors corresponding to the adjacent template frames to obtain the regularized feature vector of the s-th frame. Perform the above operation on all s from 1 to S to obtain a regularized feature sequence consisting of 39-dimensional feature vectors of S frames. This sequence has the same duration S as the standard reference phoneme template, thus completing the nonlinear duration scaling mapping of the original acoustic feature sequence and outputting the duration-regularized feature sequence.

[0095] In S4, blurred phoneme segments with indistinct acoustic boundaries due to liaison are identified in the regular feature sequence. These blurred phoneme segments are input into a pre-trained conditional variational autoencoder network, and feature reconstruction is performed using the corresponding standard phoneme labels as conditions. The output is a clear feature sequence after replacing the blurred phoneme segments, specifically including:

[0096] First, the duration-normalized feature sequence output from step S3 is received. This normalized feature sequence consists of S frames of 39-dimensional feature vectors, with each frame corresponding to a phoneme position in the standard reference phoneme template. The frame rate is 100 frames per second, and the total duration is strictly aligned with the standard reference phoneme template.

[0097] Multifractal detrended fluctuation analysis is performed on the regular feature sequence to calculate the generalized Hearst exponent sequence of the regular feature sequence. The specific implementation process is as follows: First, each dimension of the regular feature sequence is processed separately. For a feature sequence of a certain dimension with length S, it is divided into non-overlapping sub-intervals of length L, where the value of L is set to range from 16 frames to 256 frames, increasing in powers of 2. For each sub-interval, the local trend is fitted using the least squares method, and then the detrended fluctuation function is calculated. The above process is repeated by changing the division scale L to obtain the fluctuation function values ​​at different scales. For different fractal orders q, the value of q is set to range from -5 to 5, with a step size of 1, and the weighted average q-order fluctuation function is calculated. Finally, the relationship between the q-order fluctuation function and the scale L is fitted in logarithmic coordinates, and its slope is the generalized Hearst exponent of that dimension of feature at a given q value. The generalized Hearst exponents (GHES) for each of the 39 features are calculated separately, and the arithmetic mean of the GHES for each dimension under the same q value is taken to obtain the average GHES for that q value. The above calculation is repeated by changing the q value to obtain a set of average GHES sequences with q as the independent variable. This sequence reflects the long-range correlation characteristics of the regular feature sequences under different fluctuation amplitudes.

[0098] Based on the fluctuation inflection points of the generalized Hearst exponent sequence, candidate segments where singular fluctuations occur in the acoustic features are located. Specifically, the generalized Hearst exponent sequence is plotted as a curve with q values ​​on the x-axis and exponent values ​​on the y-axis. The first derivative of this curve is calculated, which is the difference between the exponent values ​​corresponding to adjacent q values. Points where the sign of the first derivative changes, i.e., from positive to negative or from negative to positive, are identified as fluctuation inflection points. The q value corresponding to each fluctuation inflection point is recorded. Then, the original regular feature sequence is divided into multiple candidate segments in the time domain, based on the following criteria: for each fluctuation inflection point corresponding to a q value, the time position of the fluctuation function in each sub-interval under that q value is checked. Specifically, during the calculation of the q-order fluctuation function, the sum of squared residuals after detrending in each sub-interval is recorded. When the sum of squared residuals exceeds twice the mean of the sum of squared residuals of all sub-intervals, the time corresponding to that sub-interval is marked as a singular fluctuation time. All marked singular fluctuation times are arranged in chronological order, and the time intervals formed by consecutively occurring singular fluctuation times are determined as candidate segment positions. Each candidate segment position corresponds to a regular feature subsequence, which may contain regions with blurred acoustic boundaries due to continuous reading.

[0099] A recursive graph analysis is performed on the regular feature subsequences corresponding to the candidate segment positions. First, the regular feature subsequences within the candidate segment are extracted, assuming they contain N frames of 39-dimensional feature vectors. The Euclidean distance between any two frames in this subsequence is calculated, resulting in an N x N distance matrix. The recursion radius is set to 0.2 times the standard deviation of all non-zero elements in this distance matrix. For the element in the i-th row and j-th column of the matrix, if its distance value is less than the recursion radius, the position in the i-th row and j-th column of the recursive graph is marked as 1, indicating a recursive state; otherwise, it is marked as 0. After traversing all i and j, a recursive graph consisting of 0 and 1 elements is obtained.

[0100] Calculate the deterministic percentage and laminar flow percentage of the recursive graph. The deterministic percentage is calculated as follows: count the number of diagonal structures in the recursive graph consisting of consecutive 1s with a length greater than or equal to 2, calculate the total number of 1s contained in these diagonal structures, divide by the total number of 1s in the entire recursive graph, and then multiply by 100% to obtain the deterministic percentage. The laminar flow percentage is calculated as follows: count the number of vertical or horizontal line structures in the recursive graph consisting of consecutive 1s with a length greater than or equal to 2, calculate the total number of 1s contained in these line structures, divide by the total number of 1s in the entire recursive graph, and then multiply by 100% to obtain the laminar flow percentage.

[0101] A preset first threshold is set to 60%, and a preset second threshold is set to 15%. The calculated percentage of certainty is compared with the preset first threshold, and the calculated percentage of laminar flow is compared with the preset second threshold. When the percentage of certainty is less than 60% and the percentage of laminar flow is greater than 15%, the candidate segment is determined to be an ambiguous phoneme segment with blurred acoustic boundaries due to liaison.

[0102] For candidate segments identified as fuzzy phoneme segments, the standard phoneme labels corresponding to the adjacent stable phoneme segments are extracted as reconstruction conditions. Specifically, based on the start and end frame numbers of the fuzzy phoneme segment in the regularized feature sequence, the phoneme position corresponding to it in the standard reference phoneme template is located. Since the regularized feature sequence and the standard reference phoneme template are strictly aligned, the corresponding phoneme label can be directly determined based on the frame number. The phoneme label corresponding to the frame preceding the fuzzy phoneme segment is taken as the forward condition, and the phoneme label corresponding to the frame following the fuzzy phoneme segment is taken as the backward condition. If the fuzzy phoneme segment is located at the beginning of the sequence, the forward condition is empty, and only the backward condition is used; if it is located at the end of the sequence, the backward condition is empty, and only the forward condition is used. The regularized feature subsequence corresponding to the fuzzy phoneme segment, along with the forward and backward conditions, are output to the next step for subsequent feature reconstruction processing.

[0103] In S5, the clear feature sequence is forcibly aligned with the standard reference phoneme template to determine the boundary of each phoneme. The pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features is calculated. The pronunciation confidence and fluency parameters are then combined to generate the final spoken language assessment score, specifically including:

[0104] First, the clear feature sequence output from step S4 is received. This clear feature sequence consists of S frames of 39-dimensional feature vectors, each frame corresponding to a phoneme position in the standard reference phoneme template, and the original blurred phoneme segments have been replaced by feature reconstruction. Simultaneously, the pre-stored standard reference phoneme template and its corresponding phoneme boundary annotation information are acquired.

[0105] Phase reconstruction is performed on each phoneme segment in the clear feature sequence to extract the instantaneous phase and group delay features of each frame within the corresponding phoneme segment. The specific implementation process is as follows: For each phoneme segment in the clear feature sequence, assume that the phoneme segment contains M frames of 39-dimensional feature vectors. For each frame feature vector, its complex spectrum is first obtained through short-time Fourier transform. The instantaneous phase of each frequency component, i.e., the principal argument value corresponding to that frequency component, is calculated from the real and imaginary parts of the complex spectrum, ranging from -π to π. The instantaneous phases of all frequency components are arranged in frequency order to obtain the instantaneous phase spectrum of the frame, which is a 39-dimensional instantaneous phase vector. Simultaneously, the group delay of each frequency component is calculated, defined as the negative derivative of the instantaneous phase with respect to frequency, approximated by the instantaneous phase difference between adjacent frequency points. The group delays of all frequency components are arranged in frequency order to obtain the group delay spectrum of the frame, which is also a 39-dimensional group delay vector. The instantaneous phase vector and the group delay vector are concatenated to form a 78-dimensional phase feature vector. Repeat the above operation for all M frames within the phoneme segment to obtain the phase feature sequence corresponding to the phoneme segment, which contains M 78-dimensional phase feature vectors. Perform the same phase reconstruction operation on the corresponding phoneme segment in the standard reference phoneme template to obtain the standard phase feature sequence.

[0106] The phase feature sequence is aligned with the standard phase feature sequence through an affine transformation, and the peak value of the aligned phase cross-correlation is calculated. Specifically, using the standard phase feature sequence as a reference, an affine transformation is performed on the phase feature sequence of the current phoneme segment, including translation and scaling operations. The translation parameter is used to adjust the relative position of the two sequences on the time axis, and the scaling parameter is used to adjust the scaling ratio of the time axis. The optimal translation and scaling parameters are searched to minimize the phase difference between the two sequences. For each set of candidate parameters, the cross-correlation coefficient between the transformed current phase feature sequence and the standard phase feature sequence is calculated in each frequency dimension, and the average cross-correlation coefficients of all frequency dimensions are taken to obtain the average cross-correlation coefficient for that set of parameters. All candidate parameter combinations are traversed, and the parameter combination that reaches the maximum value of the average cross-correlation coefficient is recorded. This maximum value is the peak value of the phase cross-correlation. The closer this peak value is to 1, the more consistent the phase features of the current phoneme segment with the phase features of the standard template are in the time-frequency structure, and the higher the pronunciation accuracy.

[0107] Based on the magnitude of the phase cross-correlation peak and the amplitude spectrum envelope similarity within the corresponding phoneme segment, the initial confidence value of the corresponding phoneme segment is obtained through weighted fusion. This process includes the following sub-steps:

[0108] The first step involves performing a complex cepstral transform on the amplitude spectrum envelope within the phoneme segment, extracting the low-order complex cepstral coefficients of the envelope as the spectral feature vector. Specifically, for each frame within the phoneme segment, its amplitude spectrum is first calculated, and then the natural logarithm of the amplitude spectrum is taken to obtain the logarithmic amplitude spectrum. An inverse Fourier transform is then performed on the logarithmic amplitude spectrum to obtain a complex cepstral sequence. The first few coefficients in the complex cepstral sequence reflect the overall shape of the spectral envelope, while the latter few coefficients reflect the fine structure of the spectrum. In this embodiment, the first 13 coefficients of the complex cepstral sequence are taken as the low-order complex cepstral coefficients of the frame, forming a 13-dimensional spectral feature vector. The above operation is repeated for all M frames within the phoneme segment, and the spectral feature vectors of all frames are averaged frame-by-frame to obtain a 13-dimensional average spectral feature vector, which serves as the spectral feature vector for the phoneme segment. The same operation is performed on the corresponding phoneme segment in the standard reference phoneme template to obtain the standard spectral feature vector.

[0109] The second step is to calculate the geodesic distance on the Riemannian manifold between the spectral feature vector and the spectral feature vector of the corresponding standard reference phoneme template. Since the complex cepstral space containing the spectral feature vector has a Riemannian manifold structure, the distance between two vectors cannot be simply measured using Euclidean distance. In this embodiment, the two spectral feature vectors are first mapped to symmetric positive definite matrices. Specifically, each 13-dimensional vector is outer-producted with its transpose to obtain a 13x13 symmetric positive definite matrix. Then, the Riemann geodesic distance between these two symmetric positive definite matrices is calculated, defined as the Frobenius norm of the difference between the logarithms of the two matrices. The calculated geodesic distance value is converted to amplitude spectral envelope similarity using a negative exponential function. The conversion method is as follows: calculation is performed with the natural constant e as the base and the negative geodesic distance as the exponent. The result is the amplitude spectral envelope similarity, ranging from 0 to 1. The closer the value is to 1, the more similar the spectral envelope shapes.

[0110] The third step involves nonlinearly normalizing the phase cross-correlation peak values ​​to obtain the phase similarity. The phase cross-correlation peak values ​​are used as input and mapped using a sigmoid function. Specifically, the calculation is performed with the natural constant e as the base and five times the absolute value of the difference between the negative phase cross-correlation peak value and 0.5 as the exponent. The result is the phase similarity, ranging from 0 to 1. This process ensures that when the phase cross-correlation peak value is close to 1, the phase similarity approaches 1; and when the phase cross-correlation peak value is close to 0, the phase similarity approaches 0.

[0111] The fourth step involves performing a convex combination operation on the phase similarity and the amplitude spectrum envelope similarity in Banach space to obtain a fusion confidence score. Specifically, a first weighting coefficient is set to 0.6, and a second weighting coefficient is set to 0.4. The phase similarity is multiplied by the first weighting coefficient to obtain a first product; the amplitude spectrum envelope similarity is multiplied by the second weighting coefficient to obtain a second product; the first product and the second product are added together to obtain the fusion confidence score. This fusion confidence score is a value between 0 and 1, comprehensively reflecting the consistency between the phase characteristics and the spectral envelope characteristics.

[0112] The fifth step involves inputting the fused confidence score into a trust function mapping layer based on evidence theory. Through confidence allocation and synthesis rules, the initial confidence score of the phoneme segment is output. First, an identification framework is established, containing two propositions: accurate pronunciation and inaccurate pronunciation. For each input fused confidence score, its confidence allocation for the accurate pronunciation proposition, its confidence allocation for the inaccurate pronunciation proposition, and its uncertain confidence allocation for the entire identification framework are calculated. In this embodiment, the confidence allocation rule is as follows: the fused confidence score itself is used as the confidence score for the accurate pronunciation proposition; the difference between 1 and the fused confidence score, multiplied by 0.8, is used as the confidence score for the inaccurate pronunciation proposition; and 1 minus the sum of the two is used as the uncertain confidence score. Then, the confidence allocation of the current phoneme segment is fused with the confidence allocation of the previous phoneme segment according to the Dempster synthesis rule. In the fused confidence allocation, the confidence value for the accurate pronunciation proposition is the initial confidence score of the phoneme segment. This initial value, after correction by contextual information, is more stable and reliable.

[0113] Traverse all phoneme segments, and repeat the above-mentioned phase reconstruction, phase cross-correlation calculation, amplitude spectrum envelope similarity calculation, weighted fusion, and evidence theory mapping process for each phoneme segment to obtain the initial confidence value of each phoneme segment. Arrange the initial confidence values ​​of all phoneme segments in chronological order to form the initial confidence value sequence.

[0114] The original confidence score sequence is then subjected to context smoothing. Specifically, a five-point cubic moving average filter is used: for the i-th initial value in the sequence, the two preceding values, the i-th value itself, and the two following values ​​are taken, for a total of five initial values. If the i-th value is at a sequence boundary, the number of initial values ​​within the valid range is taken. The weighted average of these initial values ​​is calculated. The weighting coefficients are determined by fitting a cubic polynomial, with the center point having the highest weight and the weights of the points on both sides decreasing sequentially. After smoothing, the final pronunciation confidence score sequence is obtained, where each value corresponds to the final pronunciation confidence score of a phoneme segment.

[0115] Obtain pre-calculated fluency parameters, including indicators such as speech rate, average pause duration, and pause frequency. Take the arithmetic mean of the confidence scores for each phoneme segment in the pronunciation confidence sequence to obtain the average pronunciation confidence score. Then, weight the average pronunciation confidence score and the fluency parameters, with the weighting coefficients as follows: average pronunciation confidence score accounts for 70%, and fluency parameters account for 30%. The calculated result is the final speaking assessment score, which serves as the candidate's evaluation result in the PTE speaking task.

[0116] Please see Figure 2 As shown, the PTE oral question-and-answer assessment system based on intelligent voice includes:

[0117] The acoustic feature extraction module acquires the raw speech signal of the candidate's answer to the PTE speaking task, performs noise reduction and endpoint detection on the raw speech signal, and extracts a continuous acoustic feature sequence.

[0118] The speech rate analysis and normalization parameter generation module analyzes the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds the preset speech rate threshold, it generates normalization control parameters to indicate the duration of abnormal speech rate and outputs the acoustic feature sequence in association with the normalization control parameters.

[0119] The nonlinear duration warping module, based on the warping control parameters, inputs the acoustic feature sequence into the dynamic time warping algorithm. By calculating the globally optimal path between the acoustic feature sequence and the standard reference phoneme template, it performs nonlinear duration scaling mapping on the acoustic feature sequence and outputs the duration-warped feature sequence.

[0120] The fuzzy phoneme segment recognition and reconstruction module identifies fuzzy phoneme segments in a regular feature sequence whose acoustic boundaries are blurred due to liaison. The fuzzy phoneme segments are input into a pre-trained conditional variational autoencoder network, and feature reconstruction is performed using the corresponding standard phoneme labels as conditions. The output is a clear feature sequence after replacing the fuzzy phoneme segments.

[0121] The forced alignment and confidence scoring module forces the clear feature sequence to align with the standard reference phoneme template, determines the boundary of each phoneme, and calculates the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. The final oral assessment score is generated by combining the pronunciation confidence and fluency parameters.

[0122] The working principle of this invention is as follows: The original speech signal of the examinee's answer is acquired, and noise reduction is performed through variational mode decomposition and singular value decomposition, endpoint detection based on Hilbert-Huang spectrum, and adaptive weighted gamma-pass filtering to extract a continuous acoustic feature sequence. This sequence is then reconstructed in phase space to obtain a high-dimensional phase space trajectory. The instantaneous speech rate is calculated using the diagonal length of the recursion rate matrix. When the speech rate exceeds a preset threshold, a duration normalization factor is generated based on the excess and associated with the feature sequence. The fractional Fourier transform order is determined based on this factor, and the feature sequence and standard reference phoneme template are transformed to the rotation domain. Finally, by calculating the weighted cosine distance matrix of the instantaneous frequency deviation and performing dynamic programming backtracking, the globally optimal mapping relationship is obtained, followed by interpolation and resampling. The system outputs a regularized feature sequence after duration regularization. Multifractal detrended fluctuation analysis is performed on the regularized feature sequence to locate candidate segments of singular fluctuations. The percentage of certainty and laminar flow in the recursive graph are combined to determine ambiguous phoneme segments, and phoneme labels of adjacent stable segments are extracted as reconstruction conditions. Finally, phase reconstruction is performed on the clear feature sequence to extract instantaneous phase and group delay. After affine transformation alignment, the peak value of phase cross-correlation is calculated. Simultaneously, the complex cepstral coefficients of the amplitude spectrum envelope are extracted to calculate the Riemann geodesic distance to obtain spectral envelope similarity. These two values ​​are convexly combined in Banach space and mapped using evidence theory to obtain the initial confidence values ​​of each phoneme segment. After smoothing, these values ​​are weighted and summed with the fluency parameter to generate the final spoken language assessment score.

[0123] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A PTE oral question-and-answer assessment method based on intelligent voice, characterized in that, Includes the following steps: S1: Obtain the original speech signal of the candidate's answer to the PTE Speaking task, perform noise reduction and endpoint detection on the original speech signal, and extract a continuous acoustic feature sequence. S2: Analyze the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds a preset speech rate threshold, generate a normalization control parameter to indicate the duration of the abnormal speech rate, and output the acoustic feature sequence in association with the normalization control parameter. Specifically, this includes: Phase space reconstruction is performed on the acoustic feature sequence to obtain a high-dimensional phase space trajectory; Calculate the recurrence rate matrix of the high-dimensional phase space trajectory, and extract the average length of the diagonal structure in the recurrence rate matrix as the instantaneous speech rate value; The instantaneous speech rate value is compared with the preset speech rate threshold. When the instantaneous speech rate value exceeds the preset speech rate threshold, the duration normalization factor is calculated based on the amount of excess. The duration warping factor and acoustic feature sequence are encapsulated to form associated data pairs and output. The phase space reconstruction of the acoustic feature sequence to obtain a high-dimensional phase space trajectory specifically includes: Continuous wavelet transform is performed on the acoustic feature sequence to obtain the time-frequency energy distribution of the acoustic feature sequence; The instantaneous dominant frequency and instantaneous bandwidth values ​​at each moment are extracted from the time-frequency energy distribution to form an instantaneous feature pair sequence; Phase space reconstruction is performed on the instantaneous feature pair sequence, and the instantaneous feature pair at each moment is combined with the feature pairs at several moments before and after the corresponding moment to form the high-dimensional embedding point at the corresponding moment. Arrange the high-dimensional embedding points of all moments in chronological order to obtain the high-dimensional phase space trajectory; S3: Based on the regularization control parameters, the acoustic feature sequence is input into the dynamic time regularization algorithm. By calculating the globally optimal path between it and the standard reference phoneme template, the acoustic feature sequence is nonlinearly time-scaling and mapping is performed, and the regularized feature sequence after time regularization is output. S4: Identify fuzzy phoneme segments in regular feature sequences whose acoustic boundaries are blurred due to liaison. Input the fuzzy phoneme segments into a pre-trained conditional variational autoencoder network, reconstruct features using the corresponding standard phoneme labels as conditions, and output a clear feature sequence after replacing the fuzzy phoneme segments. S5: Force alignment of the clear feature sequence with the standard reference phoneme template, determine the boundary of each phoneme, and calculate the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. Combine the pronunciation confidence and fluency parameters to generate the final spoken language assessment score.

2. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 1, characterized in that, S1 specifically includes: The original speech signal is subjected to variational mode decomposition to obtain multiple intrinsic mode function components; Singular value decomposition is performed on each intrinsic mode function component. The denoised speech signal is obtained by retaining the main singular values ​​and performing inverse transform reconstruction. The Hilbert-Huang spectrum of the denoised speech signal is calculated, and the start and end points of the speech are detected based on the instantaneous energy fluctuation rate of the Hilbert-Huang spectrum to extract the effective speech segment. An adaptive weighted gamma-pass filter is applied to the effective speech segments, and the gamma-pass cepstral coefficients of each frame are extracted as the acoustic feature sequence.

3. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 1, characterized in that, S3 specifically includes: Based on the duration normalization factor in the normalization control parameters, a fractional Fourier transform is performed on the acoustic feature sequence to obtain the rotation domain feature sequence. Calculate the cosine distance matrix between the rotated domain feature sequence and the standard reference phoneme template in the fractional Fourier domain, and use it as a measure of local cost; Dynamic programming backtracking is performed on the cosine distance matrix to search for the globally optimal path from the starting point to the ending point, and to obtain the mapping relationship between the acoustic features of each frame and the template frame. Based on the mapping relationship, the original acoustic feature sequence is interpolated and resampled to obtain a regular feature sequence that is aligned with the duration of the standard reference phoneme template.

4. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 3, characterized in that, The calculation of the cosine distance matrix between the rotated domain feature sequence and the standard reference phoneme template in the fractional Fourier domain, as a local cost metric, specifically includes: The amplitude of the rotation domain feature sequence is normalized in the fractional Fourier domain to obtain the amplitude-normalized rotation domain feature sequence. The amplitude-normalized rotation domain feature sequence and the standard reference phoneme template are subjected to instantaneous frequency trajectory extraction to obtain the feature instantaneous frequency curve and the template instantaneous frequency curve. Calculate the local frequency deviation between the feature instantaneous frequency curve and the template instantaneous frequency curve at the same rotation order, and use the reciprocal of the corresponding frequency deviation as the weight to weight the original amplitude in the fractional Fourier domain. Calculate the inner product between the weighted rotated domain feature sequence and the standard reference phoneme template, invert the inner product value and use it as the corresponding element in the local cost matrix, and obtain the complete cosine distance matrix after traversing all frame pairs.

5. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 1, characterized in that, S4 specifically includes: Multifractal detrending fluctuation analysis was performed on the regular characteristic sequence to calculate the generalized Hearst exponent sequence of the regular characteristic sequence. Based on the fluctuation inflection point of the generalized Hearst exponent sequence, locate the candidate segment where the acoustic feature exhibits singular fluctuations; Recursive graph analysis is performed on the regular feature subsequences corresponding to the candidate segment positions to calculate the deterministic percentage and laminar flow percentage of the recursive graph; When the certainty percentage is lower than the preset first threshold and the laminar flow percentage is higher than the preset second threshold, the corresponding candidate segment is determined to be a fuzzy phoneme segment, and the standard phoneme labels corresponding to the stable phoneme segments adjacent to the corresponding fuzzy phoneme segment are extracted as reconstruction conditions.

6. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 1, characterized in that, S5 specifically includes: Phase reconstruction is performed on each phoneme segment in the clear feature sequence, and the instantaneous phase and group delay features of each frame in the corresponding phoneme segment are extracted. The instantaneous phase and group delay feature are aligned by affine transformation, and the phase cross-correlation peak between the corresponding phoneme segment and the corresponding standard reference phoneme template is calculated. Based on the magnitude of the phase cross-correlation peak and the similarity of the amplitude spectrum envelope within the corresponding phoneme segment, the initial confidence value of the corresponding phoneme segment is obtained through weighted fusion. After traversing all phoneme segments to obtain the initial confidence value sequence, the initial confidence value sequence is subjected to context smoothing to output the final pronunciation confidence sequence.

7. The PTE oral question-and-answer assessment method based on intelligent voice according to claim 6, characterized in that, The initial confidence value of the corresponding phoneme segment is obtained by weighted fusion based on the magnitude of the phase cross-correlation peak and the amplitude spectrum envelope similarity within the corresponding phoneme segment. Specifically, this includes: A complex cepstral transform is performed on the amplitude spectrum envelope within the phoneme segment, and the low-order complex cepstral coefficients of the amplitude spectrum envelope are extracted as spectral feature vectors. Calculate the geodesic distance on the Riemannian manifold between the spectral feature vector and the spectral feature vector of the corresponding standard reference phoneme template, and convert the corresponding distance value into amplitude spectrum envelope similarity; The phase cross-correlation peaks are nonlinearly normalized to obtain the phase similarity. The phase similarity and the amplitude spectrum envelope similarity are then combined in a convex combination operation in the Banach space to obtain the fusion confidence. The fused confidence score is input into the trust function mapping layer based on evidence theory. Through confidence allocation and synthesis rules, the initial confidence score of the corresponding phoneme segment is output.

8. A PTE oral question-and-answer assessment system based on intelligent voice, characterized in that: The method for performing the PTE oral question-and-answer assessment based on intelligent voice as described in any one of claims 1-7 includes: The acoustic feature extraction module acquires the raw speech signal of the candidate's answer to the PTE speaking task, performs noise reduction and endpoint detection on the raw speech signal, and extracts a continuous acoustic feature sequence. The speech rate analysis and normalization parameter generation module analyzes the instantaneous speech rate value of the acoustic feature sequence. When the instantaneous speech rate value exceeds a preset speech rate threshold, it generates normalization control parameters to indicate the duration of the abnormal speech rate and outputs the acoustic feature sequence in association with the normalization control parameters. Specifically, this includes: Phase space reconstruction is performed on the acoustic feature sequence to obtain a high-dimensional phase space trajectory; Calculate the recurrence rate matrix of the high-dimensional phase space trajectory, and extract the average length of the diagonal structure in the recurrence rate matrix as the instantaneous speech rate value; The instantaneous speech rate value is compared with the preset speech rate threshold. When the instantaneous speech rate value exceeds the preset speech rate threshold, the duration normalization factor is calculated based on the amount of excess. The duration warping factor and acoustic feature sequence are encapsulated to form associated data pairs and output. The phase space reconstruction of the acoustic feature sequence to obtain a high-dimensional phase space trajectory specifically includes: Continuous wavelet transform is performed on the acoustic feature sequence to obtain the time-frequency energy distribution of the acoustic feature sequence; The instantaneous dominant frequency and instantaneous bandwidth values ​​at each moment are extracted from the time-frequency energy distribution to form an instantaneous feature pair sequence; Phase space reconstruction is performed on the instantaneous feature pair sequence, and the instantaneous feature pair at each moment is combined with the feature pairs at several moments before and after the corresponding moment to form the high-dimensional embedding point at the corresponding moment. Arrange the high-dimensional embedding points of all moments in chronological order to obtain the high-dimensional phase space trajectory; The nonlinear duration warping module, based on the warping control parameters, inputs the acoustic feature sequence into the dynamic time warping algorithm. By calculating the globally optimal path between the acoustic feature sequence and the standard reference phoneme template, it performs nonlinear duration scaling mapping on the acoustic feature sequence and outputs the duration-warped feature sequence. The fuzzy phoneme segment recognition and reconstruction module identifies fuzzy phoneme segments in a regular feature sequence whose acoustic boundaries are blurred due to liaison. The fuzzy phoneme segments are input into a pre-trained conditional variational autoencoder network, and feature reconstruction is performed using the corresponding standard phoneme labels as conditions. The output is a clear feature sequence after replacing the fuzzy phoneme segments. The forced alignment and confidence scoring module forces the clear feature sequence to align with the standard reference phoneme template, determines the boundary of each phoneme, and calculates the pronunciation confidence of the acoustic features within each phoneme segment relative to the standard pronunciation features. The final oral assessment score is generated by combining the pronunciation confidence and fluency parameters.

Images