An intelligent adenoid and tonsil hypertrophy recognition system and method based on voice and facial image multi-modal fusion

Abstract

The application relates to the technical field of intelligent medical treatment, in particular to an adenoid and tonsil hypertrophy intelligent identification system, equipment and storage medium based on voice and facial image multi-modal fusion, which comprises the following steps: a voice signal acquisition module collects Chinese monophthong, consonant, word and short sentence; a facial image acquisition module collects multi-angle facial images; a pretreatment module obtains a voice frame sequence and pretreated facial images; a feature extraction and fusion screening module extracts voice features and facial features, constructs cross-modal coupling features, splices the cross-modal coupling features into a fusion feature vector, and screens an optimal feature dataset by using a correlation and redundancy constraint algorithm; an ADASYN is used in a data balancing module to balance samples; a disease diagnosis module is internally provided with an identification model, outputs disease type discrimination results and severity grading results, and performs hierarchical consistency constraint. The cross-modal coupling features and redundancy screening significantly improve the identification precision, and the hierarchical consistency constraint ensures that the diagnosis logic is self-consistent.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and in particular to an intelligent recognition system and method for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images. Background Technology

[0002] Adenoid and tonsil hypertrophy are common childhood conditions and a major cause of obstructive sleep apnea (OSA) in children. If left undiagnosed and untreated, this condition can cause short-term symptoms such as nasal congestion, snoring, and mouth breathing, and may lead to serious consequences such as maxillofacial deformities, growth retardation, learning disabilities, and long-term damage to cardiopulmonary function. Therefore, early, accurate, and convenient screening and assessment of this condition have significant clinical and social value.

[0003] Currently, the gold standard for diagnosing adenoid and tonsil hypertrophy in clinical practice mainly relies on imaging examinations and endoscopic examinations.

[0004] Lateral nasopharyngeal radiographs provide a quantitative assessment by measuring the ratio of adenoid thickness (A) to nasopharyngeal width (N), known as the A / N ratio. While this method provides objective indicators, it involves radiation exposure and relies on radiologists manually marking measurement points, making it time-consuming, subject to subjective variations, and unsuitable for large-scale screening or frequent follow-up. Nasal endoscopy allows direct observation of the morphology of the adenoids and tonsils, but it is an invasive procedure that requires inserting an endoscope into the nasal cavity. Children have poor tolerance and low cooperation, and the diagnostic results are highly dependent on the examiner's subjective experience and judgment, lacking a unified and objective quantitative standard.

[0005] Single-modal automatic identification technology is a non-invasive identification method based on a single signal. For example, it analyzes only speech signals and uses their acoustic features (such as formant shifts) for screening; or it analyzes only facial images and uses measurements of maxillofacial morphological features for auxiliary judgment. However, single-modal systems have significant limitations: speech recognition is easily affected by individual pronunciation habits, environmental noise, and age changes, resulting in insufficient robustness; while static facial morphology alone is insufficient to distinguish between physiological variations and pathological changes, and has low sensitivity for early or mild cases. Neither approach can comprehensively capture the coordinated abnormalities in vocal tract structure and maxillofacial development associated with disease.

[0006] In summary, existing technologies are either invasive, experience-dependent, and difficult to promote, or while non-invasive, their accuracy and robustness are limited, making it difficult to meet the needs for low-cost, high-precision, and widely applicable early screening. Therefore, there is an urgent need for an intelligent technology solution that can non-invasively, objectively, and efficiently integrate multi-dimensional physiological information to achieve accurate identification and grading of adenoid and tonsil hypertrophy. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, comprising: The speech signal acquisition module is used to acquire standardized speech signals including all Chinese monophthongs, consonants, words, and short sentences. The facial image acquisition module is used to acquire standardized facial images from multiple angles. The signal preprocessing module is connected to the speech signal acquisition module and the facial image acquisition module respectively. It is used to preprocess the standardized speech signal to obtain a speech frame sequence and to perform data cleaning and feature point completion on the standardized facial image to obtain a preprocessed facial image. A feature extraction-fusion-filtering module, connected to the signal preprocessing module, is used to extract several speech features from the speech frame sequence, extract several facial features from the preprocessed facial image, construct cross-modal coupling features based on the speech features and facial features, concatenate the speech features, facial features, and cross-modal coupling features into a fused feature vector, and use a filtering algorithm based on feature-target correlation and redundancy constraints to filter the optimal feature dataset from the fused feature vector; wherein, the cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal alar width; The data balancing module is used to perform sample balancing on the training set of the optimal feature dataset during the training phase using the Adaptive Synthetic Oversampling Algorithm ADASYN to obtain a balanced training set. The disease diagnosis module has a built-in recognition model trained on the balanced training set. It is used to receive the optimal feature dataset of the sample to be diagnosed during the application phase and output the disease type discrimination result and the disease severity classification result in sequence. The disease diagnosis module is also used to perform hierarchical consistency constraint on the disease severity classification result based on the disease type discrimination result, so that the output severity classification result and the disease type discrimination result are logically consistent.

[0008] Preferably, preprocessing the standardized speech signal to obtain a speech frame sequence further includes: The original speech signal is silenced by using a threshold method to obtain a filtered speech signal. A first-order high-pass filter is used to pre-emphasize the filtered speech signal to obtain a pre-emphasized speech signal; The pre-emphasized speech signal is processed by framing, and a Hanning window is applied to each frame of the pre-emphasized speech signal to obtain the speech frame sequence.

[0009] Preferably, extracting several speech features from the speech frame sequence further includes: Temporal features, including short-time energy, root mean square energy, short-time zero-crossing rate, short-time average amplitude, peak-to-peak value, and peak-to-peak interval, are extracted from the speech frame sequence. Frequency domain features, including spectral centroid, spectral flux, power spectral density, spectral bandwidth, and the difference mean, standard deviation, and difference mean of each order of the first to eighth order Mel cepstral coefficients, are extracted from the speech frame sequence. Extract prosodic features from the speech frame sequence, including fundamental frequency, mean, median, minimum, maximum, perturbation mean, and minimum and standard deviation of the first formant, second formant, and third formant. Furthermore, a nasalization index reflecting the degree of abnormal nasal resonance is extracted. This nasalization index is the ratio of low-frequency energy to mid-frequency energy, expressed as: ,in, This represents the cumulative spectral energy within the 250Hz to 1000Hz frequency band. It represents the cumulative spectral energy within the frequency band of 1000Hz to 3000Hz.

[0010] Preferably, several facial features are extracted from the preprocessed facial image, including: Features of the orbit and nose region, including the distance between the inner and outer canthi, the distance from the upper eyelid margin to the lower orbital margin, the inner and outer orbital diameters, the width of the nasal ala, and the distance from the nasal root to the zygomatic process, are extracted from the preprocessed facial image. Extract midface and jawbone region features, including zygomatic distance and midface height, from the preprocessed facial image; Mandibular and lip features, including mandibular angle distance, distance from chin apex to mandibular angle, upper lip thickness, lower lip thickness, mouth corner upturn angle, and upper and lower lip closure gap, are extracted from the preprocessed facial image. Extract overall contour and symmetry features from the preprocessed facial image, including face width, face length, nasal root protrusion, chin protrusion, left and right zygomatic symmetry, and left and right mandibular angle symmetry. Based on the aforementioned facial features, a morphological index reflecting adenoid facies is constructed, and the morphological index is expressed as follows: ,in, For the length of the face, For width, The gap between the upper and lower lips. G represents the width of the nasal alae, S represents the chin protrusion, and S represents the mean deviation of the symmetry of the left and right zygomatic processes and the symmetry of the left and right mandibular angles. , , , These are the weight coefficients determined through training.

[0011] Preferably, before concatenating the speech features and facial features into a fused feature vector, the feature extraction-fusion-filtering module is further configured to: Cross-modal difference features and cross-modal ratio features are constructed. The cross-modal difference features are the difference between the standard scores of speech features and facial features, and the cross-modal ratio features are the ratio between the standard scores of speech features and facial features, in order to enhance the ability to represent the relationship between vocal abnormalities and facial morphological abnormalities caused by oropharyngeal airway stenosis.

[0012] Preferably, the data balancing module includes: Analyze the disease category labels in the training set, label the category with the most samples as the majority class, and label the remaining categories as the minority class; For each minority class, calculate the difference in the number of samples between it and the majority class, and determine the total number of samples that need to be synthesized for that minority class based on a preset balance target coefficient; For each minority class sample, find its K nearest neighbor samples in the feature space, count the number of majority class samples among them, calculate the density distribution ratio of the minority class sample based on the number of majority class samples, and normalize the density distribution ratio of all minority class samples to obtain the composite weight of each sample. Based on the synthesis weight, the number of samples to be synthesized is allocated to each minority class sample. For each minority class sample to be synthesized, a nearest neighbor sample belonging to the same minority class is randomly selected from its K nearest neighbors, and a synthesized sample is generated by linear interpolation. The linear interpolation generation process is also subject to medical rationality boundary conditions, ensuring that the characteristic values ​​of the synthesized sample fall within the physiological measurement allowable range of subjects of the corresponding age group. The synthetic samples are merged with the training set samples to form a training dataset with a balanced class distribution, which is used for model training of the disease diagnosis module.

[0013] Preferably, the recognition model includes: The feature extraction network is used to perform deep feature mining on the training set. It includes 5 sequentially connected convolutional layers. Each convolutional layer kernel is a 3×3 convolutional kernel. It is also configured with ReLU non-linear activation function layer and 2×2 max pooling layer for progressive extraction and downsampling, and outputs high-level abstract features. A feature mapping network, connected to the feature extraction network, is used to reduce the dimensionality and fuse the high-level abstract features and output a deep feature vector, including three sequentially connected fully connected layers, each of which is configured with a ReLU non-linear activation function. A dual-path discriminant network, connected to the feature mapping network, is used to perform parallel disease type and severity classification based on the deep feature vector. It includes a first discriminant branch and a second discriminant branch. The first discriminant branch has a fully connected layer with 2 neurons and a Softmax classification layer, which is used to output the probability distribution of the deep feature vector belonging to the normal category or the hypertrophic category. The second discriminant branch has a fully connected layer with 4 neurons and a Softmax classification layer, which is used to output the probability distribution of the deep feature vector belonging to the normal category, the mild hypertrophic category, the moderate hypertrophic category, or the severe hypertrophic category. The consistency constraint unit, connected to the dual-path discrimination network, is used to perform logical consistency correction on the disease type probability output by the first discrimination branch and the severity probability output by the second discrimination branch. When the probability of the normal category output by the first discrimination branch is higher than a preset threshold, the probabilities of mild hypertrophy, moderate hypertrophy and severe hypertrophy in the second discrimination branch are suppressed, while the probability output of the normal category in the second discrimination branch is retained. The training objective function of the recognition model is: ,in, To determine the loss based on the type of disease, Losses are classified according to severity. For hierarchical consistency loss, , , The loss weight coefficient; the hierarchical consistency loss This is used to constrain the sum of probabilities that normal category samples are predicted as mild, moderate, or severe in the severity grading branch to not exceed a preset tolerance threshold.

[0014] Preferably, the filtering algorithm based on the correlation between features and the target selects the optimal feature dataset from the fused feature vector, further comprising: For each feature in the fused feature vector, extract the value sequence and obtain the corresponding disease diagnosis target variable; All features are sorted in descending order based on the absolute value of the Pearson correlation coefficient; Each candidate feature is examined sequentially according to the order. If the absolute value of the correlation coefficient between the current candidate feature and any feature in the selected feature set is greater than the redundancy threshold, the candidate feature is skipped; otherwise, it is added to the selected feature set until the number of selected features reaches K. The preset screening strategy also includes constraining the redundancy between features, with the redundancy threshold preset to a value between 0.7 and 0.9.

[0015] Preferably, the acquisition includes standardized speech signals containing all Chinese monophthongs, consonants, words, and short phrases, further including: Ten Chinese monophthongs, including a, o, e, i, u, ü, -i (front), -i (back), and er, were collected sequentially. Each vowel was pronounced once in each of the four tones, resulting in a total of 40 vowel samples. Twenty-one Chinese consonants, including b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, and s, as well as ng as a nasal final, were collected sequentially. Each consonant was combined with a fixed vowel and pronounced once in each of the four tones, for a total of 88 consonant samples. At least one pre-defined disyllabic or polysyllabic word; At least one pre-defined complete short sentence.

[0016] Based on the same concept, this invention also provides an intelligent recognition method for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, comprising the following steps: Collect standardized speech signals including all Chinese monophthongs, consonants, words, and short sentences; Acquire standardized facial images from multiple angles; The standardized speech signal is preprocessed to obtain a speech frame sequence, and the standardized facial image is cleaned and feature point completed to obtain a preprocessed facial image. Several speech features are extracted from the speech frame sequence, and several facial features are extracted from the preprocessed facial image. Cross-modal coupling features are constructed based on the speech features and the facial features. The speech features, the facial features, and the cross-modal coupling features are concatenated into a fusion feature vector. The optimal feature dataset is selected from the fusion feature vector using a filtering algorithm based on feature-target correlation and redundancy constraints. The cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal alar width. During the training phase, the Adaptive Synthetic Oversampling Algorithm ADASYN is used to perform sample balancing on the training set of the optimal feature dataset to obtain a balanced training set. During the application phase, the optimal feature dataset of the sample to be diagnosed is received, and the disease type discrimination result and the disease severity classification result are output sequentially. Based on the disease type discrimination result, the disease severity classification result is subject to hierarchical consistency constraint to ensure that the output severity classification result and the disease type discrimination result are logically consistent.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention acquires standardized speech signals, including all Chinese monophthongs, consonants, words, and short sentences, through a speech signal acquisition module, and standardized facial images from multiple angles through a facial image acquisition module. This enables non-invasive and simultaneous acquisition of the functional status of the vocal organs and facial morphological features of children, avoiding the radiation exposure and invasive discomfort of traditional nasopharyngeal lateral radiographs or nasal endoscopy. It has significant advantages such as being non-invasive, low-cost, highly repeatable, and well-accepted by children, providing a technical basis for large-scale community screening and home self-testing. This invention extracts several speech and facial features from speech frame sequences and preprocessed facial images through a feature extraction-fusion-screening module, and constructs cross-modal coupling features based on both. The speech features, facial features, and cross-modal coupling features are then concatenated into a fusion feature vector, achieving a leap from single-modal to multi-modal deep interactive fusion. The cross-modal coupling features directly capture the pathophysiological correlation between "upper airway obstruction → abnormal vocalization and abnormal facial development," overcoming the shortcomings of traditional simple feature concatenation that ignores the physical correlation between modalities, and significantly improving the disease discrimination ability of the fusion features. This invention employs a screening algorithm based on feature-target correlation and redundancy constraints to automatically select the optimal feature dataset from high-dimensional fused feature vectors. While ensuring that the features are strongly correlated with the gold standard of the disease, it effectively removes collinearity between features, reduces the risk of model overfitting, reduces computational complexity, and improves the generalization performance and training efficiency of the recognition model. This invention employs the Adaptive Synthetic Oversampling Algorithm ADASYN during the training phase to perform sample balancing on the training set of the optimal feature dataset. This allows for the adaptive generation of synthetic samples based on the distribution density of minority class samples, effectively solving the problem of class imbalance (more normal samples and fewer severe samples) in clinical data on adenoid / tonsil hypertrophy. It avoids model bias towards the majority class and significantly improves the model's sensitivity and specificity in identifying mild, moderate, and severe hypertrophy. This invention uses a built-in recognition model in the disease diagnosis module to sequentially output disease type discrimination results (normal / hypertrophic) and disease severity grading results (normal / mild / moderate / severe). Based on the disease type discrimination results, the severity grading results are subject to hierarchical consistency constraints, ensuring that the output severity grading results are logically consistent with the disease type discrimination results (for example, samples in the normal category will not be simultaneously graded as mild / moderate / severe). This eliminates logical contradictions in the diagnostic results, improves the clinical reliability of the output results, and avoids confusion for doctors regarding inconsistent results. This invention achieves simultaneous intelligent recognition of two diseases, adenoid and tonsillar hypertrophy, through a multimodal fusion architecture of voice and facial images and explicit construction of cross-modal coupling features. Compared with single-modal or simple splicing fusion schemes, this system can more comprehensively capture the phenotypic information of upper airway obstruction caused by the disease (reflecting both functional compensation and structural changes), significantly reducing the false positive rate while ensuring high sensitivity, and has higher diagnostic accuracy and clinical applicability. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0019] Figure 1 This is a structural diagram of the intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to the present invention. Figure 2 This is a flowchart of the intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to the present invention. Figure 3 This is a flowchart of the intelligent recognition method for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Obviously, the described embodiments are only some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0021] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a” and “an” used herein, and “the”, may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0022] First Embodiment Please see Figure 1 and Figure 2 As shown in the figure, this embodiment provides an intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, which includes the following steps: The speech signal acquisition module is used to collect standardized speech signals from the subjects, including all Chinese monophthongs, consonants, words, and short phrases. Specifically, in this embodiment, the microphone is 20cm away from the mouth, at a 15° angle to the horizontal, with a sampling rate of 44.1kHz and an ambient background noise of ≤35dB. The children are guided by professional assistants to familiarize themselves with the optimized vocabulary and short phrases, and the recording is made after ensuring standard pronunciation. This ensures that the data can reflect the acoustic abnormalities of adenoid and tonsil hypertrophy. A total of 134 speech acquisition samples are constructed (including 40 monophthong samples, 88 consonant samples, and several words and short phrases), from which 47 speech features are extracted.

[0023] Preferably, the acquisition includes standardized speech signals containing all Chinese monophthongs, consonants, words, and short phrases, further including: Ten Chinese monophthongs, including a, o, e, i, u, ü, -i (front), -i (back), and er, were collected in sequence. Each vowel was pronounced once in each of the four tones: high level, rising, falling-rising, and falling. A total of 40 vowel samples were collected. Specifically, in this embodiment, the acoustic characteristics of vocal tract obstruction in patients with adenoid and tonsil hypertrophy were designed. Twenty-one Chinese consonants, including b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, and s, as well as ng as a nasal final, were collected sequentially. Each consonant was combined with a fixed vowel and pronounced once in each of the four tones, for a total of 88 consonant samples. At least one preset disyllabic or polysyllabic word, specifically, in this embodiment, such as "mother", "father", "eat", "sleep", "go to school"; At least one pre-set complete short sentence, such as "I am very happy today".

[0024] The facial image acquisition module is used to acquire standardized facial images of the subject from multiple angles. Specifically, in this embodiment, the human head and face image acquisition system is used to acquire facial image data of the subject from multiple angles. The acquisition frame rate is no less than 30fps, and a multi-view structured light scanning mode is adopted, covering three-dimensional imaging: frontal face, left 45° angle, and right 45° angle. The acquisition resolution is no less than 500dpi. Standardized control is implemented during the acquisition process: the subject maintains an upright sitting posture, with both eyes horizontally aligned with the optical axis of the camera (deviation ≤ ±5°), and a natural and relaxed facial expression. A customized breathable head cover is worn to avoid hair obscuring key feature areas such as the nose, corners of the mouth, and corners of the eyes, ensuring that the acquired facial image data is complete and accurate, providing reliable morphological basis for the identification of adenoid and tonsil hypertrophy.

[0025] The signal preprocessing module is connected to both the speech signal acquisition module and the facial image acquisition module. It is used to preprocess the standardized speech signal to obtain a speech frame sequence and to perform data cleaning and feature point completion on the standardized facial images to obtain preprocessed facial images. Specifically, in this embodiment, invalid data with a key feature point missing rate > 5% is removed, deduplication is performed to eliminate redundant records from repeated acquisitions, and quantization encoding is used to convert character-type descriptive information into standardized digital features (e.g., "distance from the lower edge of the right nasal wing to the upper lip point is 3.2cm" is converted into 3.2). A small number of missing feature points are completed using interpolation to obtain standardized facial image data, ensuring the accuracy of subsequent adenoid and tonsil hypertrophy identification.

[0026] Preferably, preprocessing the standardized speech signal to obtain a speech frame sequence further includes: The original speech signal is silenced by using a threshold method to obtain a filtered speech signal. Specifically, in this embodiment, invalid frames with an amplitude of less than 30dB are removed by using a threshold method. A first-order high-pass filter is used to pre-emphasize the filtered speech signal to obtain a pre-emphasized speech signal. Specifically, in this embodiment, a first-order high-pass filter with a coefficient of 0.97 is used for processing. The pre-emphasized speech signal is processed by framing, and a Hanning window is applied to each frame of the pre-emphasized speech signal to obtain a speech frame sequence. Specifically, in this embodiment, the frame length is set to 20ms and the frame shift is set to 10ms.

[0027] The feature extraction-fusion-filtering module, connected to the signal preprocessing module, is used to extract several speech features from the speech frame sequence and several facial features from the preprocessed facial image. Based on the speech features and facial features, a cross-modal coupling feature is constructed. The speech features, facial features, and cross-modal coupling features are concatenated into a fused feature vector. The optimal feature dataset is selected from the fused feature vector using a filtering algorithm based on feature-target correlation and redundancy constraints. The cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal wing width.

[0028] Preferably, extracting several speech features from the speech frame sequence further includes: Temporal features, including short-time energy, root mean square energy, short-time zero-crossing rate, short-time average amplitude, peak-to-peak value, and peak-to-peak interval, are extracted from the speech frame sequence. Specifically, in this embodiment, six temporal features are extracted. Frequency domain features, including spectral centroid, spectral flux, power spectral density, spectral bandwidth, and the difference mean, standard deviation, and difference mean of each order of the first to eighth order Mel cepstral coefficients, are extracted from the speech frame sequence. Specifically, in this embodiment, 28 frequency domain features are extracted. The speech frame sequence is extracted with prosodic features including the fundamental frequency, mean, median, minimum, maximum, and mean of perturbation of the fundamental frequency, as well as the minimum and standard deviation of the first formant, second formant, and third formant. Specifically, in this embodiment, 12 prosodic features are extracted, and all features are designed around the disease characterization of adenoid and tonsil hypertrophy. Furthermore, a nasalization index reflecting the degree of abnormal nasal resonance is extracted. This nasalization index is the ratio of low-frequency energy to mid-frequency energy, expressed as: ,in, This represents the cumulative spectral energy within the 250Hz to 1000Hz frequency band. This represents the cumulative spectral energy within the 1000Hz to 3000Hz frequency band. Specifically, in this embodiment, there is one such energy, and the total number of the above speech features is 47.

[0029] Preferably, several facial features are extracted from the preprocessed facial image, including: Features of the orbit and nasal region, including medial canthal distance, lateral canthal distance, distance from upper eyelid margin to infraorbital margin, orbital diameter, nasal ala width, and distance from nasal root to zygomatic process, were extracted from the preprocessed facial images. Extract midface and jawbone region features, including zygomatic process distance and midface height, from preprocessed facial images; Mandibular and lip features, including mandibular angle distance, distance from chin apex to mandibular angle, upper lip thickness, lower lip thickness, mouth corner upturn angle, and upper and lower lip closure gap, are extracted from preprocessed facial images. The preprocessed facial image is used to extract overall contour and symmetry features, including face width, face length, nasal root protrusion, chin protrusion, left and right zygomatic symmetry, and left and right mandibular angle symmetry. Specifically, in this embodiment, 20 morphological features are extracted.

[0030] A morphological index reflecting adenoid facial features is constructed based on facial characteristics. The morphological index is expressed as: ,in, For the length of the face, For width, The gap between the upper and lower lips. G represents the width of the nasal alae, S represents the chin protrusion, and S represents the mean deviation of the symmetry of the left and right zygomatic processes and the symmetry of the left and right mandibular angles. , , , The weight coefficients are determined through training, and this index is added to the feature set as a derived feature.

[0031] Construct cross-modal coupling features: ① The ratio of formant offset (standard deviation offset between the second and third formants) to the lip closure gap; ② The coupling feature between fundamental frequency perturbation (mean of fundamental frequency perturbation) and mandibular angle distance (product normalization); ③ The normalized correlation feature between spectral bandwidth (mean of speech frame spectral bandwidth) and nasal alar width (difference divided by sum). A total of 3 new cross-modal coupling features were added.

[0032] Forty-seven speech features, 21 facial features, and three cross-modal coupling features were concatenated to obtain a 71-dimensional fused feature vector. Furthermore, constructed cross-modal difference features and cross-modal ratio features (with dimensions determined based on the number of selected speech-facial feature pairs) were added to the fused feature vector to enhance cross-modal association representation capabilities.

[0033] Preferably, before concatenating the speech features and facial features into a fused feature vector, the feature extraction-fusion-filtering module is further configured to: Cross-modal difference features and cross-modal ratio features are constructed. The cross-modal difference features are the difference between the standard scores of speech features and facial features, and the cross-modal ratio features are the ratio between the standard scores of speech features and facial features, in order to enhance the ability to represent the relationship between vocal abnormalities and facial morphological abnormalities caused by oropharyngeal airway stenosis.

[0034] Preferably, the filtering algorithm based on the correlation between features and the target selects the optimal feature dataset from the fused feature vector, further comprising: For each feature in the fused feature vector, the value sequence is extracted, and the corresponding disease diagnosis target variable is obtained. The disease diagnosis target variable is the ratio of adenoid thickness A to nasopharyngeal cavity width N determined according to the nasopharyngeal lateral radiograph, or the disease severity level label divided according to the ratio. All features are sorted in descending order based on the absolute value of the Pearson correlation coefficient; Each candidate feature is examined sequentially according to its ranking. If the absolute value of the correlation coefficient between the current candidate feature and any feature in the selected feature set is greater than the redundancy threshold, the candidate feature is skipped; otherwise, it is added to the selected feature set, until the number of selected features reaches K=35. The preset screening strategy also includes constraining the redundancy between features, with the redundancy threshold preset to a value between 0.7 and 0.9.

[0035] The data balancing module is used during the training phase to perform sample balancing on the training set of the optimal feature dataset using the Adaptive Synthetic Oversampling Algorithm ADASYN to obtain a balanced training set. Specifically, in this embodiment, by adaptively evaluating the degree of data imbalance, a balance coefficient β=1 is set, and synthetic samples are generated by interpolation in the K-nearest neighbor (K=5) space of minority class samples. The linear interpolation process is constrained by medical rationality boundary conditions: each feature value of the synthetic sample (such as mandibular angle distance, nasal wing width, etc.) must fall within the physiological measurement allowable range preset according to the age group (3-12 years old). If it exceeds the range, it is resampled to ensure that the number of samples in each class is balanced. The feature distribution similarity between the synthetic sample and the original sample is ≥95%, providing high-quality data support for the accurate diagnosis of adenoid and tonsil hypertrophy.

[0036] Preferably, the data balancing module includes: Analyze the disease category labels in the training set, label the category with the most samples as the majority class, and label the remaining categories as the minority class; For each minority class, calculate the difference in the number of samples between it and the majority class, and determine the total number of samples that need to be synthesized for that minority class based on the preset balance target coefficient; For each minority class sample, find its K nearest neighbor samples in the feature space, count the number of majority class samples among them, calculate the density distribution ratio of the minority class sample based on the number of majority class samples, and normalize the density distribution ratio of all minority class samples to obtain the composite weight of each sample. Based on the synthesis weight, the number of samples to be synthesized is allocated to each minority class sample. For each minority class sample to be synthesized, a neighboring sample belonging to the same minority class is randomly selected from its K nearest neighbors, and a synthesized sample is generated by linear interpolation. The linear interpolation generation process is also subject to medical rationality boundary conditions, ensuring that the characteristic values ​​of the synthesized sample fall within the physiological measurement allowable range of subjects of the corresponding age group. The synthetic samples are merged with the training set samples to form a training dataset with a balanced class distribution, which is used for model training in the disease diagnosis module.

[0037] The disease diagnosis module has a built-in recognition model trained on a balanced training set. During the application phase, it receives the optimal feature dataset of the sample to be diagnosed and sequentially outputs the disease type discrimination result and the disease severity classification result. The module also performs hierarchical consistency constraints on the disease severity classification result based on the disease type discrimination result, ensuring logical consistency between the output severity classification result and the disease type discrimination result. Specifically, in this embodiment, a cross-entropy loss function, the Adam optimizer, a learning rate of 0.001, and 50 training epochs are used. An early stopping strategy is also enabled (training stops if the validation set loss does not decrease after 3 epochs). This simultaneously optimizes the disease type discrimination and severity discrimination logic. The recognition model is equipped with a dedicated processing chip to ensure efficient operation of the feature mining and dual discrimination stages. It integrates the feature mining function and classification logic of the original deep learning network. During operation, it consists of two core stages: first, disease type discrimination, used to determine whether the subject has adenoid or tonsil hypertrophy; second, disease severity discrimination, used to accurately classify the severity of the patient's condition, achieving integrated processing from deep feature mining to complete diagnostic result output.

[0038] Preferably, the recognition model includes: The feature extraction network is used for deep feature mining of the training set. It includes 5 sequentially connected convolutional layers, each with a 3×3 convolutional kernel (with 256, 128, and 64 neurons respectively). It is also equipped with ReLU non-linear activation function layers and 2×2 max pooling layers for progressive extraction and downsampling, outputting high-level abstract features. The feature mapping network, connected to the feature extraction network, is used to reduce the dimensionality and fuse deep high-level abstract features, and output a deep feature vector. It includes three sequentially connected fully connected layers (with 256, 128 and 64 neurons respectively), and each fully connected layer is configured with the ReLU non-linear activation function. A dual-path discriminant network, connected to a feature mapping network, is used to perform parallel classification and determination of disease type and severity based on deep feature vectors. It includes a first discriminant branch and a second discriminant branch. The first discriminant branch has a fully connected layer with two neurons and a Softmax classification layer, outputting the probability distribution of the deep feature vector belonging to the normal or hypertrophic category. The second discriminant branch has a fully connected layer with four neurons and a Softmax classification layer, outputting the probability distribution of the deep feature vector belonging to the normal, mild, moderate, or severe hypertrophic category. Specifically, in this embodiment, the disease type and severity determination stages of the disease diagnosis module both use four metrics: accuracy, precision, recall, and F1 score. The performance was evaluated using a standard, combined with 10-fold cross-validation: the fused feature dataset was divided into 10 equal-sized subsets, with one subset used as the validation set and the remaining nine subsets used as the training set for model training and evaluation. The final performance index of the module was the average of the 10 evaluation metrics. Normal (A / N ratio ≤ 0.60), hypertrophic (A / N ratio > 0.60), and mildly hypertrophic (0.60 < A / N ratio ≤ 0.70), moderately hypertrophic (0.70 < A / N ratio ≤ 0.80), and severely hypertrophic (A / N ratio > 0.8) were used to quickly identify whether subjects had adenoid and tonsil hypertrophy.

[0039] The consistency constraint unit, connected to the dual-path discriminant network, is used to perform logical consistency correction on the disease type probability output by the first discriminant branch and the severity probability output by the second discriminant branch. When the probability of the normal category output by the first discriminant branch is higher than a preset threshold, the probabilities of mild hypertrophy, moderate hypertrophy and severe hypertrophy in the second discriminant branch are suppressed, while the probability output of the normal category in the second discriminant branch is retained. The training objective function of the recognition model is: ,in, To determine the loss based on the type of disease, Losses are classified according to severity. For hierarchical consistency loss, , , The loss weight coefficient; the hierarchical consistency loss This is used to constrain the sum of probabilities that normal category samples are predicted as mild, moderate, or severe in the severity grading branch to not exceed a preset tolerance threshold.

[0040] To verify the superiority of this embodiment, under the condition that other conditions (including data balancing algorithm and diagnostic module structure) remain unchanged, the recognition performance of voice features, facial image features and fusion features were tested respectively. The verification results of disease type discrimination (whether adenoids or tonsil hypertrophy are present) are shown in Table 1, and the verification results of disease severity discrimination (severity level classification) are shown in Table 2.

[0041] Table 1. Validation results of disease type identification Table 2 Validation Results of Disease Severity Determination As shown in Tables 1 and 2, the performance indicators of the speech and facial image fusion features are significantly better than those of single-modal features in both disease type discrimination and disease severity discrimination tasks. This fully demonstrates that the multimodal fusion scheme of the present invention can effectively integrate acoustic and morphological information and greatly improve the accuracy of adenoid and tonsil hypertrophy recognition and grading.

[0042] This invention is the first to deeply fuse full-volume voice dimension signals with facial image data for the identification of adenoid and tonsil hypertrophy. Through an integrated disease diagnosis module, it achieves dual discrimination function, realizes effective integration of multimodal features and simplifies and optimizes the diagnostic process, significantly improves the accuracy, robustness and clinical applicability of the identification, and the system is non-invasive, convenient and has broad application prospects.

[0043] Second Embodiment Please see Figure 3 As shown, based on the same concept, this invention also provides an intelligent recognition method for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, including the following steps: Collect standardized speech signals including all Chinese monophthongs, consonants, words, and short sentences; Acquire standardized facial images from multiple angles; The standardized speech signal is preprocessed to obtain a speech frame sequence, and the standardized facial image is cleaned and feature point completed to obtain a preprocessed facial image. Several speech features are extracted from the speech frame sequence, and several facial features are extracted from the preprocessed facial image. Cross-modal coupling features are constructed based on the speech features and the facial features. The speech features, the facial features, and the cross-modal coupling features are concatenated into a fusion feature vector. The optimal feature dataset is selected from the fusion feature vector using a filtering algorithm based on feature-target correlation and redundancy constraints. The cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal alar width. During the training phase, the Adaptive Synthetic Oversampling Algorithm ADASYN is used to perform sample balancing on the training set of the optimal feature dataset to obtain a balanced training set. During the application phase, the optimal feature dataset of the sample to be diagnosed is received, and the disease type discrimination result and the disease severity classification result are output sequentially. Based on the disease type discrimination result, the disease severity classification result is subject to hierarchical consistency constraint to ensure that the output severity classification result and the disease type discrimination result are logically consistent.

[0044] Third Embodiment Based on the same concept, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor causes the processor to execute a module of an intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images as described in the embodiments.

[0045] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions, characterized in that, when the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to execute a module of an intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images as described in any one of the embodiments.

[0046] It is understood that, regarding the aforementioned intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, if all components are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer server or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0047] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0048] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A smart recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, characterized in that, include: The speech signal acquisition module is used to acquire standardized speech signals including all Chinese monophthongs, consonants, words, and short sentences. The facial image acquisition module is used to acquire standardized facial images from multiple angles. The signal preprocessing module is connected to the speech signal acquisition module and the facial image acquisition module respectively. It is used to preprocess the standardized speech signal to obtain a speech frame sequence and to perform data cleaning and feature point completion on the standardized facial image to obtain a preprocessed facial image. A feature extraction-fusion-filtering module, connected to the signal preprocessing module, is used to extract several speech features from the speech frame sequence, extract several facial features from the preprocessed facial image, construct cross-modal coupling features based on the speech features and facial features, concatenate the speech features, facial features, and cross-modal coupling features into a fused feature vector, and use a filtering algorithm based on feature-target correlation and redundancy constraints to filter the optimal feature dataset from the fused feature vector; wherein, the cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal alar width; The data balancing module is used to perform sample balancing on the training set of the optimal feature dataset during the training phase using the Adaptive Synthetic Oversampling Algorithm ADASYN to obtain a balanced training set. The disease diagnosis module has a built-in recognition model trained on the balanced training set. It is used to receive the optimal feature dataset of the sample to be diagnosed during the application phase and output the disease type discrimination result and the disease severity classification result in sequence. The disease diagnosis module is also used to perform hierarchical consistency constraint on the disease severity classification result based on the disease type discrimination result, so that the output severity classification result and the disease type discrimination result are logically consistent.

2. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, Preprocessing the standardized speech signal to obtain a speech frame sequence further includes: The original speech signal is silenced by using a threshold method to obtain a filtered speech signal. A first-order high-pass filter is used to pre-emphasize the filtered speech signal to obtain a pre-emphasized speech signal; The pre-emphasized speech signal is processed by framing, and a Hanning window is applied to each frame of the pre-emphasized speech signal to obtain the speech frame sequence.

3. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, Extracting several speech features from the speech frame sequence further includes: Temporal features, including short-time energy, root mean square energy, short-time zero-crossing rate, short-time average amplitude, peak-to-peak value, and peak-to-peak interval, are extracted from the speech frame sequence. Frequency domain features, including spectral centroid, spectral flux, power spectral density, spectral bandwidth, and the difference mean, standard deviation, and difference mean of each order of the first to eighth order Mel cepstral coefficients, are extracted from the speech frame sequence. Extract prosodic features from the speech frame sequence, including fundamental frequency, mean, median, minimum, maximum, perturbation mean, and minimum and standard deviation of the first formant, second formant, and third formant. Furthermore, a nasalization index reflecting the degree of abnormal nasal resonance is extracted. This nasalization index is the ratio of low-frequency energy to mid-frequency energy, expressed as: ,in, This represents the cumulative spectral energy within the 250Hz to 1000Hz frequency band. It represents the cumulative spectral energy within the frequency band of 1000Hz to 3000Hz.

4. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, Several facial features are extracted from the preprocessed facial image, including: Features of the orbit and nose region, including the distance between the inner and outer canthi, the distance from the upper eyelid margin to the lower orbital margin, the inner and outer orbital diameters, the width of the nasal ala, and the distance from the nasal root to the zygomatic process, are extracted from the preprocessed facial image. Extract midface and jawbone region features, including zygomatic distance and midface height, from the preprocessed facial image; Mandibular and lip features, including mandibular angle distance, distance from chin apex to mandibular angle, upper lip thickness, lower lip thickness, mouth corner upturn angle, and upper and lower lip closure gap, are extracted from the preprocessed facial image. Extract overall contour and symmetry features from the preprocessed facial image, including face width, face length, nasal root protrusion, chin protrusion, left and right zygomatic symmetry, and left and right mandibular angle symmetry. Based on the aforementioned facial features, a morphological index reflecting adenoid facies is constructed, and the morphological index is expressed as follows: ,in, For the length of the face, For width, The gap between the upper and lower lips. G represents the width of the nasal alae, S represents the chin protrusion, and S represents the mean deviation of the symmetry of the left and right zygomatic processes and the symmetry of the left and right mandibular angles. , , , These are the weight coefficients determined through training.

5. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, Before concatenating the speech features and facial features into a fused feature vector, the feature extraction-fusion-filtering module is further configured to: Cross-modal difference features and cross-modal ratio features are constructed. The cross-modal difference features are the difference between the standard scores of speech features and facial features, and the cross-modal ratio features are the ratio between the standard scores of speech features and facial features, in order to enhance the ability to represent the relationship between vocal abnormalities and facial morphological abnormalities caused by oropharyngeal airway stenosis.

6. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, The data balancing module includes: Analyze the disease category labels in the training set, label the category with the most samples as the majority class, and label the remaining categories as the minority class; For each minority class, calculate the difference in the number of samples between it and the majority class, and determine the total number of samples that need to be synthesized for that minority class based on a preset balance target coefficient; For each minority class sample, find its K nearest neighbor samples in the feature space, count the number of majority class samples among them, calculate the density distribution ratio of the minority class sample based on the number of majority class samples, and normalize the density distribution ratio of all minority class samples to obtain the composite weight of each sample. Based on the synthesis weight, the number of samples to be synthesized is allocated to each minority class sample. For each minority class sample to be synthesized, a nearest neighbor sample belonging to the same minority class is randomly selected from its K nearest neighbors, and a synthesized sample is generated by linear interpolation. The linear interpolation generation process is also subject to medical rationality boundary conditions, ensuring that the characteristic values ​​of the synthesized sample fall within the physiological measurement allowable range of subjects of the corresponding age group. The synthetic samples are merged with the training set samples to form a training dataset with a balanced class distribution, which is used for model training of the disease diagnosis module.

7. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, The recognition model includes: The feature extraction network is used to perform deep feature mining on the training set. It includes 5 sequentially connected convolutional layers. Each convolutional layer kernel is a 3×3 convolutional kernel. It is also configured with ReLU non-linear activation function layer and 2×2 max pooling layer for progressive extraction and downsampling, and outputs high-level abstract features. A feature mapping network, connected to the feature extraction network, is used to reduce the dimensionality and fuse the high-level abstract features and output a deep feature vector, including three sequentially connected fully connected layers, each of which is configured with a ReLU non-linear activation function. A dual-path discriminant network, connected to the feature mapping network, is used to perform parallel disease type and severity classification based on the deep feature vector. It includes a first discriminant branch and a second discriminant branch. The first discriminant branch has a fully connected layer with 2 neurons and a Softmax classification layer, which is used to output the probability distribution of the deep feature vector belonging to the normal category or the hypertrophic category. The second discriminant branch has a fully connected layer with 4 neurons and a Softmax classification layer, which is used to output the probability distribution of the deep feature vector belonging to the normal category, the mild hypertrophic category, the moderate hypertrophic category, or the severe hypertrophic category. The consistency constraint unit, connected to the dual-path discrimination network, is used to perform logical consistency correction on the disease type probability output by the first discrimination branch and the severity probability output by the second discrimination branch. When the probability of the normal category output by the first discrimination branch is higher than a preset threshold, the probabilities of mild hypertrophy, moderate hypertrophy and severe hypertrophy in the second discrimination branch are suppressed, while the probability output of the normal category in the second discrimination branch is retained. The training objective function of the recognition model is: ,in, To determine the loss based on the type of disease, Losses are classified according to severity. For hierarchical consistency loss, , , The loss weight coefficient; the hierarchical consistency loss This is used to constrain the sum of probabilities that normal category samples are predicted as mild, moderate, or severe in the severity grading branch to not exceed a preset tolerance threshold.

8. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, The feature-based filtering algorithm, which selects the optimal feature dataset from the fused feature vector based on the correlation between features and the target, further includes: For each feature in the fused feature vector, extract the value sequence and obtain the corresponding disease diagnosis target variable; All features are sorted in descending order based on the absolute value of the Pearson correlation coefficient; Each candidate feature is examined sequentially according to the order. If the absolute value of the correlation coefficient between the current candidate feature and any feature in the selected feature set is greater than the redundancy threshold, the candidate feature is skipped; otherwise, it is added to the selected feature set until the number of selected features reaches K. The preset screening strategy also includes constraining the redundancy between features, with the redundancy threshold preset to a value between 0.7 and 0.

9.

9. The intelligent recognition system for adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images according to claim 1, characterized in that, Collect standardized speech signals including all Mandarin monophthongs, consonants, words, and short phrases, further including: Ten Chinese monophthongs, including a, o, e, i, u, ü, -i (front), -i (back), and er, were collected sequentially. Each vowel was pronounced once in each of the four tones, resulting in a total of 40 vowel samples. Twenty-one Chinese consonants, including b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, and s, as well as ng as a nasal final, were collected sequentially. Each consonant was combined with a fixed vowel and pronounced once in each of the four tones, for a total of 88 consonant samples. At least one pre-defined disyllabic or polysyllabic word; At least one pre-defined complete short sentence.

10. A method for intelligent recognition of adenoid and tonsil hypertrophy based on multimodal fusion of voice and facial images, characterized in that, Includes the following steps: Collect standardized speech signals including all Chinese monophthongs, consonants, words, and short sentences; Acquire standardized facial images from multiple angles; The standardized speech signal is preprocessed to obtain a speech frame sequence, and the standardized facial image is cleaned and feature point completed to obtain a preprocessed facial image. Several speech features are extracted from the speech frame sequence, and several facial features are extracted from the preprocessed facial image. Cross-modal coupling features are constructed based on the speech features and the facial features. The speech features, the facial features, and the cross-modal coupling features are concatenated into a fusion feature vector. The optimal feature dataset is selected from the fusion feature vector using a filtering algorithm based on feature-target correlation and redundancy constraints. The cross-modal coupling features include at least the ratio of formant offset to lip closure gap, the coupling feature of fundamental frequency perturbation and mandibular angle distance, and the normalized correlation feature of spectral bandwidth and nasal alar width. During the training phase, the Adaptive Synthetic Oversampling Algorithm ADASYN is used to perform sample balancing on the training set of the optimal feature dataset to obtain a balanced training set. During the application phase, the optimal feature dataset of the sample to be diagnosed is received, and the disease type discrimination result and the disease severity classification result are output sequentially. The severity classification results are subject to hierarchical consistency constraints based on the disease type identification results, ensuring that the output severity classification results are logically consistent with the disease type identification results.

Images