Artificial cochlea child vowel classification and objective evaluation method based on multi-view acoustic feature fusion

Abstract

The application provides a cochlear implant child vowel classification and objective evaluation method based on multi-view acoustic feature fusion, and belongs to the technical field of speech signal processing and intelligent rehabilitation evaluation. Child vowel speech signals are collected and stable intervals are located, F1, F2, MFCC and LPCC are extracted to form three complementary feature views; asymmetric independent Transformer branches are used for deep coding, wherein one layer of coding blocks are arranged in the formant branch, and two layers of coding blocks are arranged in the MFCC and LPCC branches respectively; the confidence is calculated by the maximum classification probability of each single view and is normalized, and the high-level features are dynamically weighted and fused according to samples; the fused features are input into a multi-layer perception to output the vowel category and probability. The application can reduce heterogeneous feature redundancy and unreliable view interference, improve the accuracy, robustness and objective quantification ability of fuzzy vowel classification, and is suitable for cochlear implant child speech rehabilitation training evaluation.

Description

Technical Field

[0001] This invention belongs to the fields of speech signal processing, machine learning, and intelligent speech rehabilitation assessment. Specifically, it relates to a method for the classification and objective assessment of vowels in children with cochlear implants, based on multi-perspective acoustic features of formants, Mel-frequency cepstral coefficients, and linearly predicted cepstral coefficients, and employing asymmetric multi-branch Transformer coding and dynamic fusion of classification confidence. Background Technology

[0002] Cochlear implants are a crucial technology for providing auditory input to children with severe to profound sensorineural hearing loss. However, limitations imposed by the number of implanted electrode channels, frequency-to-electrode mapping accuracy, and information compression in sound encoding strategies restrict the spectral resolution and temporal fine-structure information provided by cochlear implants. For children in the critical stage of language development, incomplete auditory feedback weakens the closed-loop regulation between articulation and auditory results, leading to phenomena such as acoustic spatial compression, concentrated formant distribution, increased intraclass fluctuations, and overlapping boundaries between adjacent vowel categories. Current assessments of vowel articulation quality in children with cochlear implants primarily rely on the subjective auditory perception of speech therapists or manual measurement of a limited number of acoustic parameters by professionals. These methods are time-consuming and easily affected by the assessor's experience, recording conditions, child cooperation, and parameter extraction errors, making it difficult to produce consistent, repeatable, and quantifiable assessment results. Therefore, developing a technology that can automatically process speech signals and output objective classification information is of great significance for monitoring and individualized training during the rehabilitation phase.

[0003] Vowel acoustic information can be represented from different perspectives. The first and second formants can reflect the degree of oral cavity opening, tongue position, and vocal tract resonance characteristics; Mel frequency cepstral coefficients characterize the overall spectral shape based on the nonlinear frequency perception of the human ear; linear prediction cepstral coefficients describe the vocal tract spectral envelope through an all-pole model. When using only one feature, it is difficult to simultaneously take into account the physiological information of articulation, auditory perception information, and spectral envelope information; while directly concatenating multiple features increases the input dimension, it assumes that each feature contributes equally to different samples and different vowel categories, which easily introduces redundant information and noise.

[0004] Furthermore, formant features and cepstral features differ significantly in dimensionality, statistical distribution, and information complexity. When using encoding networks with identical structures, low-dimensional formant features may suffer from parameter redundancy and overfitting due to excessive network depth, while high-dimensional cepstral features may struggle to form sufficient high-level representations due to insufficient network depth. When using fixed-weight fusion, if the reliability of a feature view decreases due to formant estimation bias, spectral perturbations, or individual abnormal pronunciation, its unreliable information will still be introduced in equal amounts. Therefore, existing technologies lack a method for cochlear implant-supported pediatric vowel classification and objective evaluation that can differentially model heterogeneous acoustic features and dynamically allocate fusion weights based on the reliability of each sample's view. Summary of the Invention

[0005] To overcome the problems of insufficient representation by a single acoustic feature, redundancy easily generated by static splicing of heterogeneous features, difficulty in adapting a unified coding structure to different feature complexities, and the inability of fixed fusion weights to reflect changes in sample-level reliability in existing technologies, this invention provides a method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion. This method learns high-level feature representations of formants, Mel-frequency cepstral coefficients, and linearly predicted cepstral coefficients, constructs sample-level classification confidence and dynamic fusion weights based on single-view category probabilities, and outputs vowel category probabilities, prediction results, and corresponding confidence levels, thereby improving the accuracy, stability, and quantifiability of fuzzy vowel recognition.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for vowel classification and objective assessment in children with cochlear implants based on multi-view acoustic feature fusion includes the following steps:

[0008] A. Acquire vowel speech signals of the child to be evaluated, preprocess the vowel speech signals, and extract formant features, Mel frequency cepstral coefficient features, and linear prediction cepstral coefficient features from the preprocessed vowel speech signals to form three complementary acoustic feature views.

[0009] B. Input the three acoustic feature views into independent multi-branch deep coding networks respectively, and use an asymmetric coding structure to learn feature representations according to the information complexity of different acoustic feature views to obtain the high-level feature representations corresponding to each acoustic feature view.

[0010] C. Perform single-view classification based on each of the high-level feature representations to obtain the category probability distribution corresponding to each acoustic feature view, and calculate the classification confidence corresponding to each acoustic feature view based on the category probability distribution.

[0011] D. Normalize the classification confidence scores to generate dynamic fusion weights, and use the dynamic fusion weights to perform weighted fusion of the high-level feature representations to obtain fused feature representations;

[0012] E. Input the fused feature representation into the classification network, output the vowel category probability and vowel category prediction result of the child to be evaluated, and use the vowel category probability, vowel category prediction result and corresponding classification confidence as objective evaluation information of vowel pronunciation.

[0013] Step A specifically includes: acquiring the target vowel speech signal naturally produced by the child to be evaluated; selecting effective speech segments from the recording; and excluding silence, obvious plosives, non-target pronunciations, and severe truncation distortion. Since the start and end positions of vowels are easily affected by consonant transitions and unstable pronunciation factors, a relatively stable vowel interval is selected from the middle of the vowel range. Preferably, the speech signal is acquired in an environment with background noise not exceeding 40 dB(A), with a sampling rate of 44.1 kHz and a quantization precision of 16 bits.

[0014] Extract the first formant frequency F1 and the second formant frequency F2 from the stable vowel interval to construct a two-dimensional formant feature vector:

[0015] X F =[F1, F2]

[0016] Mel frequency cepstral coefficients are extracted to construct a Mel frequency cepstral coefficient feature vector; in a preferred embodiment, the feature vector is twelve-dimensional.

[0017] X M =[MFCC1, MFCC2,..., MFCC 12 ]

[0018] The linear prediction cepstral coefficients are extracted to form a linear prediction cepstral coefficient feature vector; in a preferred embodiment, the feature vector is twelve-dimensional.

[0019] X L =[LPCC1, LPCC2,..., LPCC 12 ]

[0020] To eliminate differences in dimensions and numerical ranges between different acoustic feature views and between different feature dimensions within the same view, Z-score normalization was performed on each feature dimension of the three acoustic feature views:

[0021]

[0022] Where v∈{F, M, L} represents the formant, Mel frequency cepstral coefficient, and linear prediction cepstral coefficient view, respectively; d represents the feature dimension; n represents the sample number; x represents the original feature value; and μ and σ represent the mean and standard deviation of the corresponding feature dimension in the training samples, respectively. The mean and standard deviation are calculated from the training set and used in the validation set, test set, and subsequent samples to be evaluated, to avoid test information from being used in model training.

[0023] The standardized acoustic feature views are then input into the embedding layer, which consists of a fully connected layer and a ReLU activation function, to map them to a unified hidden space.

[0024]

[0025] Wherein, the standardized feature vector is denoted as W emb,v and b emb,v E represents the embedding weight matrix and bias vector of the corresponding view, respectively. v This represents the corresponding embedding feature. Preferably, the dimension of the unified hidden space is set to 64 dimensions, so that the three heterogeneous acoustic views can be input into subsequent independent coding branches.

[0026] Step B specifically includes: constructing three independent Transformer coding branches corresponding to formant features, Mel frequency cepstral coefficient features, and linearly predicted cepstral coefficient features, respectively. Each branch only receives embedded features from the corresponding acoustic feature view to reduce mutual interference between heterogeneous features during the coding stage. Different numbers of Transformer coding blocks are set according to the dimensionality and information complexity of different acoustic feature views, thereby forming an asymmetric multi-branch deep coding network.

[0027] Each Transformer encoding block includes a multi-head self-attention layer, a feedforward network, residual connections, and a dropout layer. For the embedded features Ev of the acoustic feature view v, the query matrix, key matrix, and value matrix corresponding to the h-th attention head are as follows:

[0028]

[0029]

[0030] Calculate the scaled dot product score based on the query matrix and the key matrix:

[0031]

[0032] The Sigmoid gate function is used to filter the value matrix element by element:

[0033] A v，h =σ(S v，h )⊙Vv，h

[0034] Among them, W Q W K and W V These represent the query mapping matrix, key mapping matrix, and value mapping matrix, respectively. Q b K and b V These represent the corresponding bias vectors, d and d. k σ represents the feature dimension of the key vector, σ represents the Sigmoid function, and ⊙ represents element-wise multiplication. The gating method is used to retain feature responses within each view that are beneficial for vowel classification and suppress invalid responses.

[0035] In a preferred embodiment, the formant features have low dimensionality and clear physical meaning, and their encoding branch uses a single Transformer encoding block. The Mel frequency cepstral coefficient features and the linearly predicted cepstral coefficient features have higher dimensionality and contain more complex spectral information; their encoding branches each use two Transformer encoding blocks. Each encoding block uses four attention heads. The feedforward network expands the 64-dimensional features to 256 dimensions before projecting them back to 64 dimensions, with a dropout rate of 0.3. After each encoding branch completes feature representation learning, a 128-dimensional high-level feature representation is output via a fully connected mapping.

[0036] Step C specifically includes: setting single-view classification heads after each of the three Transformer encoding branches, mapping the high-level feature representations corresponding to each acoustic feature view to a probability distribution of K preset vowel categories. For the nth sample, the maximum category probability in the single-view category probability distribution corresponding to the acoustic feature view v is taken as the classification confidence of that view for the current sample.

[0037]

[0038] Where K represents the number of preset vowel categories, This represents the high-level feature representation of the nth sample in the acoustic feature view v. This represents the classification confidence level of the corresponding acoustic feature view. The higher the classification confidence level, the more concentrated the classification judgment formed by the corresponding view for the current sample, and the higher its reliability in the subsequent fusion process.

[0039] Step D specifically includes: normalizing the classification confidence scores corresponding to the three acoustic feature views to obtain the dynamic fusion weight of the nth sample in view v.

[0040]

[0041] The dynamic fusion weights corresponding to each acoustic feature view satisfy the following:

[0042]

[0043] The three high-level feature representations are weighted and summed according to the dynamic fusion weights to obtain the fused feature representation:

[0044]

[0045] in, This represents the dynamic fusion weight of the nth sample under the acoustic feature view v. This represents the fused feature representation corresponding to the nth sample. Since the high-level feature dimensions output by the three encoding branches are consistent, the feature dimensions after weighted fusion remain unchanged. This late-stage fusion method can improve the contribution of high-confidence views and reduce the contribution of low-confidence views affected by noise, parameter estimation bias, or abnormal pronunciation.

[0046] Step E specifically includes inputting the fused feature representation into a multilayer perceptron classifier. The multilayer perceptron classifier sequentially comprises a fused feature input layer, a first fully connected layer, a first ReLU activation layer, a first Dropout layer, a second fully connected layer, a second ReLU activation layer, a second Dropout layer, and a Softmax output layer. Preferably, the fused feature input dimension is 128, the first fully connected layer outputs 256 dimensions with a first Dropout rate of 0.4, the second fully connected layer outputs 128 dimensions with a second Dropout rate of 0.3, and the output layer dimension is K.

[0047] Let the hidden representation of the multilayer perceptron be u(n), and the K-dimensional class score generated by the output layer be:

[0048] z (n) =W out u (n) +b out

[0049] The Softmax output layer outputs the predicted probability of the k-th vowel category:

[0050]

[0051] The vowel category prediction result for the nth sample is determined based on the maximum a posteriori probability:

[0052]

[0053] in, This indicates that the nth sample corresponds to the output value of the kth vowel category. This represents the corresponding predicted probability. In a preferred embodiment, K is 7, corresponding to the Mandarin vowels / a / , / i / , / u / , / γ / , / y / , and / ... and

[0054] The model training employs a two-stage training strategy. The first stage trains the embedding layers, Transformer encoding branches, and single-view classifiers corresponding to the three acoustic feature views separately, ensuring each branch forms a stable discriminative representation on its corresponding acoustic feature view. The second stage fixes the network parameters of the three Transformer encoding branches, constructs fused features based on the high-level feature representations and classification confidence scores corresponding to each acoustic feature view, and trains the fused classifier. The training process uses the cross-entropy loss function.

[0055]

[0056] Where N represents the number of training samples, y n，k This indicates that the nth sample corresponds to the true label of the kth vowel category. This represents the corresponding predicted probability. The network parameters are updated using a gradient optimization algorithm; in a preferred embodiment, the Adam optimizer is used with an initial learning rate of 0.001 and a maximum training epoch of 50.

[0057] For each vowel sample to be evaluated, the system outputs the probability distribution of the preset vowel category, the vowel category prediction result, the classification confidence of the three single-view samples, and the corresponding dynamic fusion weights. For multiple target vowel samples from the same child, the system can further analyze the classification accuracy, average prediction probability, proportion of low-confidence samples, and main confusion directions for each vowel category, forming repeatable and comparable objective assessment information for vowel pronunciation. This information can be used to assist in determining the focus of rehabilitation training and tracking phased changes.

[0058] The beneficial effects of the technical solution provided by this invention are:

[0059] This invention simultaneously utilizes the first and second formants, which have clear physiological significance in articulation, Mel-frequency cepstral coefficients that conform to the nonlinear frequency perception characteristics of the human ear, and linearly predicted cepstral coefficients that describe the vocal tract spectrum envelope, to characterize vowels from three complementary perspectives. This reduces information loss due to single acoustic features in children with ambiguous articulation under cochlear implant conditions. Asymmetric independent Transformer encoding branches are set according to the dimensionality and information complexity of different acoustic feature views, reducing parameter redundancy and overfitting risks caused by excessively deep networks in low-dimensional formant features, while improving the deep representation capability of high-dimensional cepstral features. Sample-level classification confidence is constructed based on the maximum class probability of a single view, and fusion weights are dynamically allocated. This approach, without relying on fixed weights or direct feature concatenation, automatically reduces the influence of unreliable acoustic feature views and maintains a compact dimensionality of the fused features, improving the robustness of vowel classification. Attached Figure Description

[0060] To more clearly illustrate the technical solution of the present invention, the accompanying drawings involved in the implementation of the present invention will be briefly described below.

[0061] Figure 1 This is a diagram of the overall architecture of the present invention.

[0062] Figure 2 This is a comparison chart of the classification performance of this invention on a dataset of hearing children.

[0063] Figure 3 This is a comparison chart of the classification performance of this invention on a dataset of children with cochlear implants.

[0064] Figure 4 This is the vowel classification confusion matrix of the present invention on test sets of hearing children and children with cochlear implants.

[0065] Figure 5 This is a training and validation curve of the present invention on datasets of hearing children and children with cochlear implants. Detailed Implementation

[0066] To more clearly illustrate the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are used to illustrate the present invention and not to limit the scope of protection of the present invention; equivalent substitutions for feature dimension, number of coding blocks, number of attention heads, classification categories or training parameters can be used to implement the present invention without departing from the technical concept of the present invention.

[0067] Example 1

[0068] This embodiment collects Mandarin vowel sounds from children with cochlear implants and age-matched hearing children to train and validate the vowel classification and objective evaluation method. The cochlear implant group includes 14 children (5 girls and 9 boys), aged 3.25 to 6.42 years; the hearing children group includes 56 children (33 girls and 23 boys), aged 3.17 to 6.42 years. A picture-based naming task guides children to naturally generate vowels containing / a / , / i / , / u / , / γ / , / y / , ... and Seven Mandarin target vowels were used to generate monosyllabic words, with four tone words for each vowel. 28 speech samples were generated from each child, resulting in a total of 1960 speech samples. Recordings were conducted in an environment with background noise no higher than 40 dB(A), at a sampling rate of 44.1 kHz, and with a quantization precision of 16 bits. After recording, valid speech segments were selected, and relatively stable analysis intervals were determined in the middle of the vowels.

[0069] Within the stable vowel range, the first formant frequency F1, the second formant frequency F2, 12-dimensional Mel frequency cepstral coefficients, and 12-dimensional linear predicted cepstral coefficients are extracted. The three acoustic feature views are Z-score normalized and mapped to a 64-dimensional unified hidden space through fully connected layers and ReLU activation functions. One Transformer encoding block is used for the formant branch, while two Transformer encoding blocks are used for the Mel frequency cepstral coefficient branch and the linear predicted cepstral coefficient branch. Each branch outputs a 128-dimensional high-level feature representation, and seven vowel probabilities and classification confidence scores are obtained through a single-view classification head. The three classification confidence scores are normalized to obtain dynamic fusion weights, and the three high-level feature representations are then weighted and fused. The fused features are input into a multilayer perceptron classifier with network dimensions of 128, 256, 128, and 7, respectively.

[0070] The data was divided into an 80% training set and a 20% test set, maintaining a relatively consistent ratio between the children's group and vowel category. The model employed a two-stage training strategy: first, three single-view encoding branches and a classifier head were trained separately; then, the encoding branches were fixed and a fusion classifier was trained. Training used the cross-entropy loss function and the Adam optimizer. Classification performance was evaluated using accuracy, precision, recall, and macro-average F1 score. Experimental results show that the multi-view fusion model of this invention outperforms single acoustic feature models and traditional feature concatenation methods on both hearing-sounding children and cochlear implant-impregnated children's datasets, validating the effectiveness of the dynamic fusion strategy based on classification confidence.

Claims

1. A method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion, characterized in that, Includes the following steps: A. Acquire vowel speech signals of the child to be evaluated, preprocess the vowel speech signals, and extract formant features, Mel frequency cepstral coefficient features, and linear prediction cepstral coefficient features from the preprocessed vowel speech signals to form three complementary acoustic feature views. B. Input the three acoustic feature views into independent multi-branch deep coding networks respectively, and use an asymmetric coding structure to learn feature representations according to the information complexity of different acoustic feature views to obtain the high-level feature representations corresponding to each acoustic feature view. C. Perform single-view classification based on each of the high-level feature representations to obtain the category probability distribution corresponding to each acoustic feature view, and calculate the classification confidence corresponding to each acoustic feature view based on the category probability distribution. D. Normalize the classification confidence scores to generate dynamic fusion weights, and use the dynamic fusion weights to perform weighted fusion of the high-level feature representations to obtain fused feature representations; E. Input the fused feature representation into the classification network, output the vowel category probability and vowel category prediction result of the child to be evaluated, and use the vowel category probability, vowel category prediction result and corresponding classification confidence as objective evaluation information of vowel pronunciation.

2. The method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion according to claim 1, characterized in that, In step A, a relatively stable vowel interval is selected from the middle of the vowel in the vowel speech signal, where the acoustic parameters change relatively smoothly; the formant feature is a two-dimensional feature vector composed of the first formant frequency F1 and the second formant frequency F2. X F =[F1,F2] The Mel frequency cepstral coefficients are characterized by a twelve-dimensional feature vector: X M =[MFCC1, MFCC2,..., MFCC 12 ] The linear prediction cepstral coefficient features are a twelve-dimensional feature vector: X L =[LPCC1,LPCC2,...,LPCC 12 ] Z-score normalization is performed on each feature dimension of the three acoustic feature views according to the following formula: Then, the normalized acoustic feature views are mapped to a unified hidden space through a fully connected layer and a ReLU activation function: Where v∈{F, M, L} represent the formant, Mel frequency cepstral coefficient, and linear prediction cepstral coefficient view, respectively; d represents the feature dimension; n represents the sample number; μ and σ represent the mean and standard deviation of the corresponding feature dimension, respectively; X represents the corresponding standardized feature vector; and W emb，v and b emb，v Let represent the embedding weight matrix and bias vector of the corresponding view, respectively, and E represent the corresponding embedding feature.

3. The method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion according to claim 1, characterized in that, In step B, the multi-branch deep coding network includes three Transformer coding branches corresponding to formant features, Mel frequency cepstral coefficient features, and linearly predicted cepstral coefficient features, respectively. Each Transformer coding block includes a multi-head self-attention layer, a feedforward network, a residual connection, and a Dropout layer. The Transformer coding branch corresponding to the formant features is configured with one Transformer coding block, and the Transformer coding branches corresponding to the Mel frequency cepstral coefficient features and the linearly predicted cepstral coefficient features are configured with two Transformer coding blocks, respectively. Each Transformer coding branch outputs the corresponding high-level feature representation after being mapped by a fully connected layer.

4. The method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion according to claim 1, characterized in that: (1) For the embedded feature E of the acoustic feature view v, the query matrix, key matrix and value matrix corresponding to the h-th attention head are calculated according to the following formula: Calculate the scaled dot product score based on the query matrix and the key matrix: The Sigmoid gating function is then used to filter the value matrix element by element. A v,h =σ(S v，h )⊙V v,h Among them, W Q W K and W V These represent the query mapping matrix, key mapping matrix, and value mapping matrix, respectively. Q b K and b V These represent the corresponding bias vectors, d and d. k σ represents the feature dimension of the key vector, σ represents the Sigmoid function, and ⊙ represents element-wise multiplication. (2) In step C, the maximum class probability of the nth sample in the single-view class probability distribution under the acoustic feature view v is taken as the classification confidence of the corresponding acoustic feature view: Where K represents the number of preset vowel categories, h represents the high-level feature representation of the nth sample in the acoustic feature view v, and c represents the classification confidence of the corresponding acoustic feature view.

5. The method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion according to claim 1, characterized in that, In step D, the classification confidence scores corresponding to each acoustic feature view are normalized according to the following formula to obtain dynamic fusion weights: The dynamic fusion weights corresponding to each acoustic feature view satisfy the following: The fused feature representation is obtained according to the following formula: in, This represents the dynamic fusion weight of the nth sample under the acoustic feature view v. This represents the fusion feature representation corresponding to the nth sample.

6. The method for vowel classification and objective evaluation in children with cochlear implants based on multi-view acoustic feature fusion according to claim 1, characterized in that, The classification network in step E is a multilayer perceptron classifier. (1) The multilayer perceptron classifier sequentially comprises a fused feature input layer, a first fully connected layer, a first ReLU activation layer, a first Dropout layer, a second fully connected layer, a second ReLU activation layer, a second Dropout layer, and a Softmax output layer; the Softmax output layer outputs the predicted probability of the k-th vowel category: The vowel category prediction result for the nth sample is determined according to the following formula: in, This indicates that the nth sample corresponds to the output value of the kth vowel category. This represents the corresponding predicted probability, and K represents the number of preset vowel categories. (2) The model training adopts a two-stage training strategy. In the first stage, the Transformer encoding branches and single-view classification heads corresponding to the three acoustic feature views are trained respectively. In the second stage, the network parameters of the three Transformer encoding branches are fixed, and the fusion classifier is trained according to the high-level feature representations and classification confidence of each acoustic feature view. The training process uses the cross-entropy loss function: Where N represents the number of training samples, y n，k This indicates that the nth sample corresponds to the true label of the kth vowel category. This represents the corresponding predicted probability, and the network parameters are updated using a gradient optimization algorithm.

Images