Speech disorder detection method, device, equipment and readable storage medium

Abstract

The present disclosure relates to a speech disorder detection method, device, equipment and readable storage medium. By acquiring standard audio-visual materials, in response to the pronunciation operation of the to-be-detected object to the standard audio-visual materials, multi-modal pronunciation data is collected, audio acoustic features are extracted based on the pronunciation audio, video visual features are extracted based on the video of the face and oral cavity activity, the audio acoustic features, the video visual features and the demographic information coding data are fused to obtain a fusion feature vector, and based on the fusion feature vector and a pre-trained prediction model, a speech disorder detection result of the to-be-detected object is obtained. Compared with the prior art, the embodiment of the present disclosure can improve the accuracy and comprehensiveness of speech disorder detection, improve the diagnosis efficiency, reduce the dependence on professionals, reduce the burden of medical resources, and clearly determine the specific type of pronunciation problem, thereby providing a scientific basis for subsequent individualized intervention treatment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a speech disorder detection method and device, equipment and readable storage medium. BACKGROUND

[0002] Functional speech disorder is a common language development problem in preschool children. Traditional speech disorder detection mostly uses auditory evaluation or visual evaluation, which has significant limitations. Auditory evaluation relies on subjective auditory analysis of speech by professionals, is easily disturbed by environmental noise, and cannot capture the coordinated movement information of the articulatory organs; visual evaluation relies on naked-eye observation of lip shape, tongue position and other movement characteristics, is limited by image resolution and observation angle, and is difficult to quantify subtle pronunciation errors.

[0003] Existing detection methods mostly use single-modal analysis, and auditory materials and visual materials are independent of each other, without joint analysis of multi-modal features, resulting in incomplete basis for judging speech disorders. Single-modal methods are greatly affected by environmental, individual difference and other factors, which may lead to high risk of misdiagnosis or missed diagnosis, especially for complex or mild functional speech disorders. In addition, traditional methods are inefficient, and a more comprehensive, accurate and efficient detection method is urgently needed to improve diagnostic accuracy and reduce medical resource consumption. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a speech disorder detection method, device, equipment and readable storage medium.

[0005] In a first aspect, the embodiments of the present disclosure provide a speech disorder detection method, which comprises:

[0006] obtaining standard audio-visual materials;

[0007] in response to pronunciation operation of a to-be-tested object on the standard audio-visual materials, collecting multi-modal pronunciation data, the multi-modal pronunciation data comprising pronunciation audio, video of face and oral cavity activity, and demographic information encoding data;

[0008] extracting audio acoustic features based on the pronunciation audio, and extracting video visual features based on the video of face and oral cavity activity;

[0009] performing multi-modal feature fusion on the audio acoustic features, the video visual features and the demographic information encoding data to obtain a fusion feature vector;

[0010] obtaining a speech disorder detection result of the to-be-tested object based on the fusion feature vector and a pre-trained prediction model.

[0011] In some embodiments, the collecting multi-modal pronunciation data comprises:

[0012] acquire a video of the face and oral cavity activity of the to-be-tested object through an image acquisition device, and acquire pronunciation audio of the to-be-tested object synchronously through a sound acquisition device;

[0013] acquire demographic information input by the to-be-tested object, encode the demographic information, and obtain demographic information encoding data.

[0014] In some embodiments, the audio acoustic features are extracted based on the pronunciation audio, including:

[0015] The pronunciation audio is subjected to data enhancement processing of time stretching and frequency moving, to obtain enhanced pronunciation audio;

[0016] The enhanced pronunciation audio is subjected to acoustic feature extraction, to obtain audio acoustic features of the to-be-tested object, including fundamental frequency, focal peak, and syllable duration.

[0017] In some embodiments, the video visual features include lip shape features, facial expression features, and tongue position features;

[0018] The video visual features are extracted based on the video of the face and oral cavity activity, including:

[0019] The video of the face and oral cavity activity is subjected to data enhancement processing of mirror flipping and angle rotating, to obtain enhanced video of the face and oral cavity activity;

[0020] Lip analysis is performed based on the enhanced video of the face and oral cavity activity by using a convolutional neural network, to obtain lip shape features of the to-be-tested object;

[0021] Dynamic expression changes are captured based on the enhanced video of the face and oral cavity activity by using a facial expression recognition algorithm, to obtain facial expression features of the to-be-tested object;

[0022] The motion and position of the tongue are analyzed based on the enhanced video of the face and oral cavity activity by using a deep learning model, to obtain tongue position features of the to-be-tested object.

[0023] In some embodiments, the multi-modal feature fusion of the audio acoustic features, the video visual features, and the demographic information encoding data is performed, to obtain a fusion feature vector, including:

[0024] The audio acoustic features, the video visual features, and the demographic information encoding data are subjected to multi-modal feature re-encoding by using a cross-modal model, to generate a re-encoding vector;

[0025] The importance of different modal features in the re-encoding vector is weighted by using a self-attention mechanism, to obtain a fusion feature vector.

[0026] In some embodiments, the pre-trained prediction model comprises a first-level classification prediction model and a second-level classification prediction model.

[0027] The voice disorder detection result of the to-be-detected object is obtained based on the fusion feature vector and the pre-trained prediction model, comprising:

[0028] The fusion feature vector is input into the first-level classification prediction model, and a first-level classification result is output by the first-level classification prediction model, the first-level classification result comprising one of normal or voice disorder;

[0029] When the first-level classification result is voice disorder, a second-level classification result is output by the second-level classification prediction model based on the fusion feature vector, the second-level classification result being used to represent the voice disorder type.

[0030] In some embodiments, the pre-trained prediction model is obtained by the following steps:

[0031] An initial prediction model is constructed, the initial prediction model comprising a first-level initial model and a second-level initial model;

[0032] A sample data set is obtained, and the sample data set is divided into a training set and a test set;

[0033] The initial prediction model is trained based on the training set, and when a preset prediction error threshold or a preset training number threshold is reached, a trained prediction model is obtained;

[0034] The trained prediction model is tested based on the test set, and a voice disorder prediction result of the test set is obtained;

[0035] When the voice disorder prediction result of the test set meets a preset test index, a pre-trained prediction model is obtained.

[0036] In a second aspect, the embodiments of the present disclosure provide a voice disorder detection device, comprising:

[0037] An acquisition module is configured to acquire standard audio-visual materials.

[0038] A collection module is configured to collect multi-modal pronunciation data in response to pronunciation operations of a to-be-detected object on the standard audio-visual materials, the multi-modal pronunciation data comprising pronunciation audio, video of facial and oral activities, and demographic information encoding data.

[0039] An extraction module is configured to extract audio acoustic features based on the pronunciation audio, and extract video visual features based on the video of facial and oral activities.

[0040] a fusion module configured to perform multi-modal feature fusion on the audio acoustic feature, the video visual feature, and the demographic information encoding data to obtain a fusion feature vector;

[0041] a detection module configured to obtain a speech disorder detection result of the to-be-detected object based on the fusion feature vector and a pre-trained prediction model.

[0042] In a third aspect, an electronic device is provided, including:

[0043] a memory;

[0044] a processor; and

[0045] a computer program;

[0046] The computer program is stored in the memory and configured to be executed by the processor to implement the method of the first aspect.

[0047] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the method of the first aspect.

[0048] In a fifth aspect, a computer program product is provided, which includes a computer program or instructions. The computer program or instructions are executed by a processor to implement the method of the first aspect.

[0049] The speech disorder detection method, device, equipment and readable storage medium provided by the embodiments of the present disclosure obtain standard audio-visual materials, collect multi-modal pronunciation data in response to pronunciation operations of a to-be-detected object on the standard audio-visual materials, the multi-modal pronunciation data including pronunciation audio, video of face and oral cavity activities, and demographic information encoding data. Further, audio acoustic features are extracted based on the pronunciation audio, video visual features are extracted based on the video of face and oral cavity activities, multi-modal feature fusion is performed on the audio acoustic features, the video visual features, and the demographic information encoding data to obtain a fusion feature vector, and a speech disorder detection result of the to-be-detected object is obtained based on the fusion feature vector and a pre-trained prediction model. Compared with the prior art, the embodiments of the present disclosure can improve the accuracy and comprehensiveness of speech disorder detection through multi-modal feature fusion, especially in the detection of complex environments or slight speech disorders, improve the diagnosis efficiency, reduce the dependence on professionals, reduce the burden of medical resources, provide technical support for large-scale, according to the multi-modal detection result, determine the specific type of pronunciation problem, and provide a scientific basis for subsequent individualized intervention treatment. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate preferred embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0052] Figure 1 The speech disorder detection method flowchart provided for the embodiments of the present disclosure;

[0053] Figure 2 The speech disorder detection method flowchart provided for another embodiment of the present disclosure;

[0054] Figure 3 The speech disorder detection method flowchart provided for another embodiment of the present disclosure;

[0055] Figure 4 The structural schematic diagram of the speech disorder detection device provided for the embodiments of the present disclosure;

[0056] Figure 5 The structural schematic diagram of the electronic device provided for the embodiments of the present disclosure. DETAILED DESCRIPTION

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings required by the embodiments or the prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0058] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some of the embodiments of the present disclosure, not all the embodiments.

[0059] Functional speech disorder is a common language development problem in preschool children. Traditional speech disorder detection mostly uses auditory evaluation or visual evaluation, which has significant limitations. Auditory evaluation relies on subjective auditory analysis of speech by professionals, is easily disturbed by environmental noise, and cannot capture the coordinated movement information of the articulatory organs; visual evaluation relies on naked eye observation of lip shape, tongue position and other movement characteristics, is limited by image resolution and observation angle, and is difficult to quantify subtle pronunciation errors.

[0060] The existing detection methods mostly use single-modal analysis, and the auditory materials and visual materials are independent of each other, and the joint analysis of multi-modal features cannot be achieved, which leads to that the judgment basis for speech disorders is not comprehensive, the single-modal method is greatly affected by environmental factors, individual differences and other factors, and may lead to a high risk of misdiagnosis or missed diagnosis, especially for complex or slight functional speech disorders. In addition, the traditional method is inefficient, and a more comprehensive, accurate and efficient detection method is urgently needed to improve the diagnostic accuracy and reduce the consumption of medical resources.

[0061] To solve the problem, the embodiment of the present disclosure provides a speech disorder detection method, which will be introduced below in combination with specific embodiments.

[0062] Figure 1 A flowchart of the speech disorder detection method provided by the embodiment of the present disclosure is shown. The execution subject of the method is an electronic device, which can be a portable mobile device such as a smartphone, a tablet computer, a notebook computer, etc.; or a fixed device such as a personal computer, a server, etc., wherein the server can be a single server, a server cluster, which can be a distributed cluster or a centralized cluster. The method can be applied to the scene of detecting speech disorders, and can improve the accuracy and comprehensiveness of speech disorder detection, reduce the dependence on professionals, and reduce the burden on medical resources. It can be understood that the speech disorder detection method provided by the embodiment of the present disclosure can also be applied in other scenes.

[0063] The speech disorder detection method shown in FIG. 1 will be introduced below, which includes the following specific steps: Figure 1 The speech disorder detection method shown in FIG. 1 will be introduced below, which includes the following specific steps:

[0064] S101, acquiring standard audio-visual materials.

[0065] In this step, the standard audio-visual materials are evaluated and selected by relevant professionals on the basis of Chinese phonetic system and articulatory phonetics, taking the initial consonants of Chinese pinyin as the unit, and then standard pictures (1024x768 pixels, 600 dpi, white background) are designed and made according to the objective things corresponding to the selected words, which are used for subsequent speech disorder detection. The relevant professionals include but are not limited to pediatricians, neurologists and professors of linguistics, etc. Further, the standard audio-visual materials are stored in an electronic device or a server, and the electronic device acquires the standard audio-visual materials from the electronic device or the server.

[0066] Optionally, in the standard audio-visual material, the initial consonant "p" corresponds to the vocabulary of "grapes", "pot", "milk bottle", "apple", "running"; the initial consonant "b" corresponds to the vocabulary of "pen", "bread", "white cloud", "biscuit"; the initial consonant "t" corresponds to the vocabulary of "rabbit", "stair", "sugar", "sun"; the initial consonant "d" corresponds to the vocabulary of "cake", "stool", "potato"; the initial consonant "k" corresponds to the vocabulary of "chopsticks", "trousers", "button", "cry"; the initial consonant "g" corresponds to the vocabulary of "dog", "watermelon", "toothpaste"; the initial consonant "m" corresponds to the vocabulary of "steamed buns", "towel", "building blocks", "noodles", "cat", "door"; the initial consonant "n" corresponds to the vocabulary of "ox", "bird", "milk"; the initial consonant "h" corresponds to the vocabulary of "cucumber", "window", "telephone", "fire"; the initial consonant "l" corresponds to the vocabulary of "radish", "dragon", "fawn", "moon"; the initial consonant "q" corresponds to the vocabulary of "piano", "eggplant", "skirt", "car", "balloon", "swimming ring"; the initial consonant "c" corresponds to the vocabulary of "green onion", "cabbage"; the initial consonant "z" corresponds to the vocabulary of "mouth", "writing", "soccer", "bath"; the initial consonant "s" corresponds to the vocabulary of "umbrella", "garlic"; the initial consonant "x" corresponds to the vocabulary of "panda", "shoes", "small fish", "circle"; the initial consonant "f" corresponds to the vocabulary of "hair", "rice", "airplane", "house"; the initial consonant "j" corresponds to the vocabulary of "scissors", "older sister", "pepper", "foot"; the initial consonant "r" corresponds to the vocabulary of "hot", "muscle", "throwing garbage"; the initial consonant "ch" corresponds to the vocabulary of "bed", "car", "ship", "police", "window"; the initial consonant "sh" corresponds to the vocabulary of "mobile phone", "book", "spoon", "television", "leaves"; the initial consonant "zh" corresponds to the vocabulary of "pig", "table".

[0067] S102, in response to the pronunciation operation of the to-be-tested object on the standard audio-visual material, collecting multi-modal pronunciation data, the multi-modal pronunciation data including pronunciation audio, video of face and oral cavity activity, and demographic information coding data.

[0068] In this step, the subject will perform pronunciation operation on the standard audio-visual material, and the electronic device will collect multi-modal pronunciation data in response to the pronunciation operation of the subject on the standard audio-visual material. In some embodiments, the subject includes 1000 patients aged 5-18 years old with functional speech disorders, and 1000 normal experimental participants matched in gender and age. All subjects need to be familiar with the pictures and the corresponding words and phrases of the pictures before the multi-modal pronunciation data collection. Optionally, a preset display is used to present the standardized pictures, and the pictures are located in the center of the screen. The preset display can be a 27-inch AOC Q27G2S / D, or other sizes and models of displays. The subject needs to speak out loud the corresponding words and phrases of the pictures according to the pictures, and press the space bar to switch the pictures. The electronic device collects multi-modal pronunciation data, including pronunciation audio, video of facial and oral activity, and demographic information coding data. The demographic information coding data is the data obtained by coding the demographic information such as age, gender, and grade.

[0069] S103, extracting audio acoustic features based on the pronunciation audio, and extracting video visual features based on the video of facial and oral activity.

[0070] In this step, after the multi-modal pronunciation data is collected, the electronic device extracts audio acoustic features based on the pronunciation audio, and extracts video visual features based on the video of facial and oral activity.

[0071] Optionally, the video visual features include lip shape features, facial expression features, and tongue position features.

[0072] S104, multi-modal feature fusion of the audio acoustic features, the video visual features, and the demographic information coding data to obtain a fusion feature vector.

[0073] In this step, after the audio acoustic features and the video visual features are extracted, the electronic device performs multi-modal feature fusion of the audio acoustic features, the video visual features, and the demographic information coding data to obtain a fusion feature vector. Through multi-modal feature fusion, the accuracy and comprehensiveness of detection can be improved, especially in complex environments or detection of slight speech disorders.

[0074] S105, obtaining a speech disorder detection result of the subject based on the fusion feature vector and a pre-trained prediction model.

[0075] In this step, after obtaining the fused feature vector, the electronic device uses the fused feature vector and a pre-trained prediction model to obtain the speech impairment detection result of the target object. Specifically, the fused feature vector is input into the pre-trained prediction model, and the pre-trained prediction model outputs the speech impairment detection result of the target object. In some optional implementations, diagnostic suggestions are also output along with the speech impairment detection result of the target object.

[0076] Optionally, the pre-trained prediction model includes a primary classification prediction model and a secondary classification prediction model. The primary classification prediction model is used to classify users as having "speech impairment" or "normal"; the secondary classification prediction model is used to further classify users with "speech impairment" to determine the type of speech impairment.

[0077] In some embodiments, the pre-trained prediction model is trained through the following steps:

[0078] S1. Construct an initial prediction model, which includes a first-level initial model and a second-level initial model;

[0079] In this embodiment, an initial prediction model is constructed, which includes a first-level initial model and a second-level initial model.

[0080] The initial level model is used to classify and predict whether a user has a "voice disorder" or is "normal." This initial level model uses the lightweight deep learning model EfficientNet and incorporates an adaptive optimization algorithm (AdamW) and a learning rate scheduler (Cosine Annealing) to optimize model parameters and the learning rate. The cross-entropy loss function is used as the prediction evaluation metric, and the initial level classification result for the user is output.

[0081] The secondary initial model is used to further classify users who were previously categorized as having "speech disorders" by the primary initial model, resulting in secondary classification results. These secondary classification results include errors such as tongue position errors, lip shape errors, vowel omission errors, elevation errors, vowel weakening errors, distortion errors, and other errors, which are not limited here. The secondary initial model uses an attention-based multi-class classification model and employs the Focal Loss algorithm to address class imbalance. To improve classification performance, the F1 score for a specific class is used as the prediction evaluation metric.

[0082] Furthermore, knowledge distillation is used to transfer knowledge from the lightweight model of the first-level initial model, and the hyperparameters of the first-level and second-level initial models are optimized using an automated tool (Hyperband). Optionally, parameter optimization can be achieved by defining a Bayesian optimization objective function and a hyperparameter search space.

[0083] S2. Obtain the sample dataset and divide the sample dataset into a training set and a test set;

[0084] In this step, the sample dataset is obtained and divided into training and test sets according to a preset ratio. The preset ratio can be 7:3, or any other ratio; there is no specific limitation. It is necessary to ensure that the division is random to prevent model overfitting and affecting detection performance.

[0085] S3. The initial prediction model is trained based on the training set. When the preset prediction error threshold or the preset training number threshold is reached, the trained prediction model is obtained.

[0086] In this step, the initial prediction model is trained based on the training set. Training stops when the model's prediction error reaches a preset prediction error threshold (minimum prediction error) or the number of training iterations reaches a preset training iteration threshold (maximum training iterations), resulting in the trained prediction model. Optionally, the preset prediction error threshold can be 1×10-1. -7 The preset training iteration threshold can be 1×10. 4 No specific limitations are imposed.

[0087] S4. Test the trained prediction model based on the test set to obtain the speech obstacle prediction results of the test set;

[0088] In this step, the trained prediction model is tested based on the test set, and the speech disorder prediction results are output.

[0089] In some embodiments, the electronic device analyzes the interpretability of the prediction results. Specifically, it calculates the SHAP value and uses Grad-CAM technology to interpret the model's decision-making process and prediction results.

[0090] S5. When the speech obstacle prediction results of the test set meet the preset test indicators, a pre-trained prediction model is obtained.

[0091] In this step, a pre-trained prediction model is obtained when the speech obstacle prediction results of the test set meet a preset test metric. For example, the preset test metric is the prediction accuracy; when the model's prediction accuracy is greater than an accuracy threshold, the pre-trained prediction model is obtained.

[0092] This disclosure embodiment acquires standard audiovisual materials and, in response to the pronunciation operations of the test subject in relation to these materials, collects multimodal pronunciation data. The multimodal pronunciation data includes pronunciation audio, video of facial and oral movements, and demographic information encoded data. Further, audio acoustic features are extracted based on the pronunciation audio, and video visual features are extracted based on the video of facial and oral movements. The audio acoustic features, video visual features, and demographic information encoded data are then fused using multimodal features to obtain a fused feature vector. Based on the fused feature vector and a pre-trained prediction model, the speech disorder detection result for the test subject is obtained. Compared to existing technologies, this disclosure embodiment, through multimodal feature fusion, can improve the accuracy and comprehensiveness of speech disorder detection, especially in complex environments or for mild speech disorders. It improves diagnostic efficiency, reduces reliance on specialists, lowers the burden on medical resources, provides technical support for large-scale deployment, and clarifies the specific type of pronunciation problem based on the multimodal detection results, providing a scientific basis for subsequent personalized intervention and treatment.

[0093] Figure 2 Here is a flowchart of a speech disorder detection method provided in another embodiment of this disclosure, as shown below. Figure 2 As shown, the method includes the following steps:

[0094] S201. Obtain standard audiovisual materials.

[0095] Specifically, the implementation process and principle of S201 and S101 are the same, and will not be repeated here.

[0096] S202. Acquire video of the facial and oral movements of the subject under test using an image acquisition device, and simultaneously acquire audio of the subject under test's pronunciation using a sound acquisition device.

[0097] In this step, video of the subject's facial and oral movements is captured using an image acquisition device such as a high-speed camera (frames per second ≥ 60fps), and audio is simultaneously recorded using a sound acquisition device such as a radio system (sampling rate ≥ 48kHz). Optionally, the image acquisition device can be an AZ 9502 high-speed camera, and the sound acquisition device can be an R∅DEWIRELESS PRO radio system; no specific limitations are imposed here.

[0098] S203. Obtain the demographic information input by the subject to be tested, encode the demographic information, and obtain the encoded demographic information data.

[0099] In this step, the test subject will input relevant demographic information, including age, gender, grade, etc., without specific limitations. Further, the electronic device will acquire the demographic information input by the test subject, encode the information, and obtain encoded demographic data. Specifically, age information retains the original continuous numerical format, two digits; gender information uses "01" binary encoding, with male represented by 1 and female by 0, without specific limitations; grade information uses one-hot encoding.

[0100] S204. Extract audio acoustic features based on the pronunciation audio, and extract video visual features based on the video of facial and oral movements.

[0101] Specifically, the implementation process and principle of S204 and S103 are the same, and will not be repeated here.

[0102] S205. The audio acoustic features, video visual features, and demographic information encoded data are recoded using a cross-modal model to generate a recoded vector.

[0103] In this step, the electronic device will re-encode the demographic information encoded data, audio acoustic features, and video visual features using the Transformer cross-modal model to generate a re-encoded vector.

[0104] S206. Use a self-attention mechanism to weight the importance of different modal features in the recoded vector to obtain a fused feature vector.

[0105] Furthermore, using a self-attention mechanism to weight the importance of features from different modalities to obtain a fused feature vector can improve the feature fusion effect.

[0106] S207. Input the fused feature vector into the primary classification prediction model, and output the primary classification result through the primary classification prediction model. The primary classification result includes either normal or speech disorder.

[0107] In this step, after obtaining the fused feature vector, the electronic device can input the fused feature vector into the primary classification prediction model. The primary classification prediction model outputs a primary classification result, which is used to characterize whether the subject has a speech disorder. The primary classification result includes either normal or speech disorder. When the subject is detected to have a speech disorder, the primary classification result is speech disorder; when the subject is not detected to have a speech disorder, the primary classification result is normal.

[0108] S208. When the primary classification result is a speech disorder, the secondary classification prediction model outputs a secondary classification result based on the fused feature vector. The secondary classification result is used to characterize the speech disorder type.

[0109] Furthermore, when the primary classification result indicates a speech disorder, the electronic device will output a secondary classification result based on the fused feature vector through the secondary classification prediction model. This secondary classification result is used to characterize the type of speech disorder. Optionally, the secondary classification result may include errors such as tongue position, lip shape, vowel omission, elevation, vowel weakening, distortion, and other errors, which are not limited here.

[0110] This embodiment of the disclosure acquires standard audiovisual materials, captures video of the subject's facial and oral movements using an image acquisition device, and simultaneously captures the subject's pronunciation audio using a sound acquisition device to obtain demographic information input from the subject. This demographic information is then encoded to obtain encoded demographic information data. Further, audio acoustic features are extracted based on the pronunciation audio, and video visual features are extracted based on the video of the facial and oral movements. The audio acoustic features, video visual features, and encoded demographic information data are then recoded using a cross-modal model to generate a recoded vector. A self-attention mechanism is used to weight the importance of different modal features in the recoded vector to obtain a fused feature vector. This fused feature vector is then input into a primary classification prediction model, which outputs a primary classification result. When the primary classification result indicates a speech disorder, a secondary classification prediction model outputs a secondary classification result based on the fused feature vector. This secondary classification result is used to characterize the speech disorder type. This method can improve the accuracy and comprehensiveness of speech disorder detection, especially in complex environments or for mild speech disorders. It can improve diagnostic efficiency, reduce reliance on professionals, reduce the burden on medical resources, provide technical support for large-scale testing, and clarify the specific type of pronunciation problem based on multimodal detection results, thus providing a scientific basis for subsequent personalized intervention and treatment.

[0111] Figure 3 Here is a flowchart of a speech disorder detection method provided in another embodiment of this disclosure, as shown below. Figure 3 As shown, the method includes the following steps:

[0112] S301. Obtain standard audiovisual materials.

[0113] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0114] S302. In response to the pronunciation operation of the test subject in response to standard audiovisual materials, multimodal pronunciation data is collected, including pronunciation audio, video of facial and oral movements, and demographic information encoded data.

[0115] Specifically, the implementation process and principle of S302 and S102 are the same, and will not be repeated here.

[0116] S303. Perform time-stretching and frequency-shifting data enhancement processing on the audio recording to obtain the enhanced audio recording.

[0117] In this step, the electronic device performs data enhancement processing on the audio output. The audio data enhancement processing methods include time stretching and frequency shifting to obtain enhanced audio output. Optionally, the audio is enhanced by time stretching (±5%) and frequency shifting (±3%), which is not limited.

[0118] S304. Perform acoustic feature extraction on the enhanced pronunciation audio to obtain the audio acoustic features of the test object. The audio acoustic features include fundamental frequency, focal peak, and syllable duration.

[0119] Furthermore, the electronic device can extract acoustic features from the enhanced audio output. For example, it can use Mel-frequency cepstral coefficients (MFCC) combined with a fundamental frequency tracking algorithm to extract acoustic features, obtaining the audio acoustic features of the test object. These audio acoustic features include the fundamental frequency, formants, and syllable duration. The fundamental frequency (F0) reflects pitch characteristics; the formants, including F1, F2, and F3, reflect formant information; and the syllable duration is used to analyze changes in speech rate and rhythm.

[0120] S305. Perform data enhancement processing on the video of the facial and oral movements by mirroring and rotating the angle to obtain the enhanced video of the facial and oral movements.

[0121] In this step, the electronic device performs data enhancement processing on the video of the facial and oral movements. The video data enhancement processing methods include mirror flipping and angle rotation to obtain enhanced video of the facial and oral movements. Optionally, the video is enhanced by mirror flipping and angle rotation (±10°), which is not limited.

[0122] S306. Using a convolutional neural network to perform lip analysis based on the enhanced facial and oral activity video, the lip shape features of the subject are obtained.

[0123] In this step, a convolutional neural network (CNN) is used to analyze information such as the closure, width, and roundness of the lips to obtain the lip shape features of the subject.

[0124] S307. Using a facial expression recognition algorithm, dynamic facial expression changes are captured based on the enhanced video of facial and oral movements to obtain the facial expression features of the test subject.

[0125] In this step, a Facial Action Coding System (FACS) algorithm is used to capture dynamic facial expression changes and obtain the facial expression features of the subject being tested.

[0126] S308. By using a deep learning model to analyze the movement and position of the tongue based on the enhanced facial and oral activity video, the tongue position characteristics of the test subject are obtained.

[0127] In this step, the movement and position of the tongue are analyzed using a deep learning model (HRNet) to obtain the tongue position characteristics of the subject.

[0128] S309. The audio acoustic features, the video visual features, and the demographic information encoded data are fused using multimodal features to obtain a fused feature vector.

[0129] Specifically, the implementation process and principle of S309 and S104 are the same, and will not be repeated here.

[0130] S310. Based on the fused feature vector and the pre-trained prediction model, the speech impairment detection result of the object to be tested is obtained.

[0131] Specifically, the implementation process and principle of S310 and S105 are the same, and will not be repeated here.

[0132] This embodiment of the disclosure acquires standard audiovisual materials, responds to the pronunciation operations of the test subject in response to the standard audiovisual materials, collects multimodal pronunciation data, and performs time-stretching and frequency-shifting data enhancement processing on the pronunciation audio to obtain enhanced pronunciation audio. Acoustic features are extracted from the enhanced pronunciation audio to obtain the audio acoustic features of the test subject, including fundamental frequency, focal peak, and syllable duration. Further, the video of facial and oral movements is mirrored and rotated to obtain enhanced video of facial and oral movements. A convolutional neural network is used to perform lip analysis based on the enhanced video of facial and oral movements to obtain the lip shape features of the test subject. A facial expression recognition algorithm is used to capture dynamic facial expression changes based on the enhanced video of facial and oral movements to obtain the facial expression features of the test subject. A deep learning model is used to analyze the movement and position of the tongue based on the enhanced video of facial and oral movements to obtain the tongue position features of the test subject. Finally, the audio acoustic features, the video visual features, and the demographic information encoded data are fused into a multimodal feature vector to obtain a fused feature vector. Based on the fused feature vector and a pre-trained prediction model, the speech disorder detection result of the test subject is obtained. This method can improve the accuracy and comprehensiveness of speech disorder detection, especially in complex environments or for mild speech disorders. It can improve diagnostic efficiency, reduce reliance on professionals, reduce the burden on medical resources, provide technical support for large-scale testing, and clarify the specific type of pronunciation problem based on multimodal detection results, thus providing a scientific basis for subsequent personalized intervention and treatment.

[0133] Figure 4 This is a schematic diagram of the structure of a speech disorder detection device provided in an embodiment of this disclosure. The speech disorder detection device can be an electronic device as described in the above embodiments, or it can be a component or assembly within that electronic device. The speech disorder detection device provided in this embodiment can execute the processing flow provided in the speech disorder detection method embodiments, such as... Figure 4As shown, the speech disorder detection device 40 includes: an acquisition module 41, a collection module 42, an extraction module 43, a fusion module 44, and a detection module 45. The acquisition module 41 acquires standard audiovisual materials. The collection module 42 collects multimodal speech data in response to the speech operation of the test subject on the standard audiovisual materials. The multimodal speech data includes speech audio, video of facial and oral movements, and demographic information encoded data. The extraction module 43 extracts audio acoustic features based on the speech audio and video visual features based on the video of facial and oral movements. The fusion module 44 fuses the audio acoustic features, video visual features, and demographic information encoded data to obtain a fused feature vector. The detection module 45 obtains the speech disorder detection result of the test subject based on the fused feature vector and a pre-trained prediction model.

[0134] Optionally, when the acquisition module 42 acquires multimodal vocal data, it is specifically used to: acquire video of the facial and oral movements of the subject under test through an image acquisition device, and simultaneously acquire the vocal audio of the subject under test through a sound acquisition device; obtain the demographic information input by the subject under test, encode the demographic information, and obtain demographic information encoded data.

[0135] Optionally, when the extraction module 43 extracts audio acoustic features based on the pronunciation audio, it is specifically used to: perform time-stretching and frequency-shifting data enhancement processing on the pronunciation audio to obtain enhanced pronunciation audio; and extract acoustic features from the enhanced pronunciation audio to obtain the audio acoustic features of the test object, wherein the audio acoustic features include fundamental frequency, focal peak, and syllable duration.

[0136] Optionally, the video visual features include lip shape features, facial expression features, and tongue position features;

[0137] When the extraction module 43 extracts video visual features based on the video of facial and oral activities, it is specifically used to: perform data enhancement processing such as mirror flipping and angle rotation on the video of facial and oral activities to obtain enhanced video of facial and oral activities; use a convolutional neural network to perform lip analysis based on the enhanced video of facial and oral activities to obtain the lip shape features of the subject; use a facial expression recognition algorithm to capture dynamic expression changes based on the enhanced video of facial and oral activities to obtain the facial expression features of the subject; and use a deep learning model to analyze the movement and position of the tongue based on the enhanced video of facial and oral activities to obtain the tongue position features of the subject.

[0138] Optionally, when the fusion module 44 fuses the audio acoustic features, the video visual features, and the demographic information encoded data into a multimodal feature vector, it specifically performs the following: re-encodes the audio acoustic features, the video visual features, and the demographic information encoded data into a multimodal feature vector using a cross-modal model to generate a re-encoded vector; and uses a self-attention mechanism to weight the importance of different modal features in the re-encoded vector to obtain the fusion feature vector.

[0139] Optionally, the pre-trained prediction model includes a primary classification prediction model and a secondary classification prediction model;

[0140] When the detection module 45 obtains the speech disorder detection result of the subject based on the fused feature vector and the pre-trained prediction model, it is specifically used to: input the fused feature vector into the first-level classification prediction model, and output the first-level classification result through the first-level classification prediction model, wherein the first-level classification result includes either normal or speech disorder; when the first-level classification result is a speech disorder, the second-level classification prediction model outputs the second-level classification result based on the fused feature vector, wherein the second-level classification result is used to characterize the speech disorder type.

[0141] Optionally, the pre-trained prediction model is obtained through the following steps: constructing an initial prediction model, which includes a first-level initial model and a second-level initial model; obtaining a sample dataset and dividing the sample dataset into a training set and a test set; training the initial prediction model based on the training set, and obtaining the trained prediction model when a preset prediction error threshold or a preset training iteration threshold is reached; testing the trained prediction model based on the test set to obtain the speech obstacle prediction results of the test set; and obtaining the pre-trained prediction model when the speech obstacle prediction results of the test set meet preset test indicators.

[0142] Figure 4 The speech impairment detection device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0143] This disclosure also provides a speech disorder detection system, which includes a standard audiovisual material unit, a data acquisition unit, a feature extraction unit, a classification prediction unit, and an output unit. The standard audiovisual material unit consists of standardized images of objective objects corresponding to characters and words based on the initials of Chinese Pinyin. The data acquisition unit acquires audio recordings of the test subject's pronunciation of the standard audiovisual material, videos of facial and oral movements, and coded demographic information data such as gender, age, and grade. The feature extraction unit extracts and integrates features from the audio, video, and demographic information. The classification prediction unit includes a primary classification prediction model and a secondary classification prediction model, outputting whether the test subject has a speech disorder and the type of disorder. The output unit provides classification results and diagnostic suggestions, and outputs a visualized explanation of the model.

[0144] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 5 It shows a schematic diagram of a structure suitable for implementing the electronic device 600 in the embodiments of this disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0145] like Figure 5 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the speech disorder detection method as described in the embodiments of this disclosure. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0146] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0147] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the speech impairment detection method as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0148] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0149] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0150] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0151] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0152] Obtain standard audiovisual materials;

[0153] In response to the pronunciation operation of the test subject in response to standard audiovisual materials, multimodal pronunciation data is collected, including pronunciation audio, video of facial and oral movements, and demographic information encoded data;

[0154] Audio acoustic features are extracted based on the pronunciation audio, and video visual features are extracted based on the video of facial and oral movements.

[0155] The audio acoustic features, the video visual features, and the demographic information encoded data are fused using multimodal features to obtain a fused feature vector;

[0156] Based on the fused feature vector and the pre-trained prediction model, the speech impairment detection result of the test object is obtained.

[0157] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0161] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0164] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0165] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for detecting speech disorders, characterized in that, The method includes: Obtain standard audiovisual materials; In response to the pronunciation operation of the test subject in response to standard audiovisual materials, multimodal pronunciation data is collected, including pronunciation audio, video of facial and oral movements, and demographic information encoded data; Audio acoustic features are extracted based on the pronunciation audio, and video visual features are extracted based on the video of facial and oral movements. The audio acoustic features, the video visual features, and the demographic information encoded data are fused using multimodal features to obtain a fused feature vector; Based on the fused feature vector and the pre-trained prediction model, the speech impairment detection result of the test object is obtained; The extraction of audio acoustic features based on the pronunciation audio includes: The audio recording is subjected to time-stretching and frequency-shifting data enhancement processing to obtain the enhanced audio recording. Acoustic features are extracted from the enhanced pronunciation audio to obtain the audio acoustic features of the test object, including fundamental frequency, focal peak, and syllable duration; The video visual features include lip shape features, facial expression features, and tongue position features; The extraction of video visual features based on the facial and oral movements includes: The video of the facial and oral movements is subjected to data enhancement processing, including mirror flipping and angle rotation, to obtain an enhanced video of the facial and oral movements. Lip analysis was performed on enhanced facial and oral video using a convolutional neural network to obtain the lip shape features of the subject. The facial expression recognition algorithm is used to capture dynamic facial expression changes based on enhanced video of facial and oral movements to obtain the facial expression features of the subject under test. By using a deep learning model to analyze the movement and position of the tongue based on enhanced facial and oral activity videos, the tongue position characteristics of the subject under test can be obtained. The time stretching range is -5% to +5%, the frequency shift range is -3% to +3%, and the angle rotation range is -10° to +10°. The step of fusing the audio acoustic features, the video visual features, and the coded demographic information data into a multimodal feature vector includes: The audio acoustic features, video visual features, and demographic information encoded data are recoded using a cross-modal model to generate a recoded vector. The importance of different modal features in the recoded vector is weighted using a self-attention mechanism to obtain a fused feature vector; The pre-trained prediction model includes a primary classification prediction model and a secondary classification prediction model; The process of obtaining the speech impairment detection result of the target object based on the fused feature vector and the pre-trained prediction model includes: The fused feature vector is input into a primary classification prediction model, and the primary classification prediction model outputs a primary classification result, which includes either normal or speech disorder. When the primary classification result is a speech disorder, the secondary classification prediction model outputs a secondary classification result based on the fused feature vector, and the secondary classification result is used to characterize the speech disorder type.

2. The method according to claim 1, characterized in that, The acquisition of multimodal speech data includes: The video of the subject's facial and oral movements is captured by an image acquisition device, and the audio of the subject's pronunciation is captured simultaneously by a sound acquisition device. The demographic information input by the subject to be tested is obtained, and the demographic information is encoded to obtain the encoded demographic information data.

3. The method according to claim 1, characterized in that, The pre-trained prediction model is obtained through the following steps: Construct an initial prediction model, which includes a first-level initial model and a second-level initial model; Obtain the sample dataset and divide the sample dataset into a training set and a test set; The initial prediction model is trained based on the training set. When the preset prediction error threshold or the preset number of training times threshold is reached, the trained prediction model is obtained. The trained prediction model is tested based on the test set to obtain the speech obstacle prediction results of the test set. When the speech impairment prediction results of the test set meet the preset test indicators, a pre-trained prediction model is obtained.

4. A speech disorder detection device, characterized in that, The device includes: The acquisition module is used to acquire standard audiovisual materials; The acquisition module is used to acquire multimodal vocalization data in response to the vocalization operation of the test subject in response to standard audiovisual materials. The multimodal vocalization data includes vocalization audio, video of facial and oral movements, and demographic information encoded data. The extraction module is used to extract audio acoustic features based on the pronunciation audio and to extract video visual features based on the video of facial and oral movements. The fusion module is used to fuse the audio acoustic features, the video visual features, and the demographic information encoded data into a multimodal feature vector. The detection module is used to obtain the speech impairment detection result of the object to be tested based on the fused feature vector and the pre-trained prediction model. When the extraction module extracts audio acoustic features based on the pronunciation audio, it is specifically used to: perform time-stretching and frequency-shifting data enhancement processing on the pronunciation audio to obtain enhanced pronunciation audio; and extract acoustic features from the enhanced pronunciation audio to obtain the audio acoustic features of the object under test, wherein the audio acoustic features include fundamental frequency, focal peak, and syllable duration. The video visual features include lip shape features, facial expression features, and tongue position features; When the extraction module extracts video visual features based on the video of facial and oral activities, it specifically performs the following: performs data enhancement processing on the video of facial and oral activities by mirroring and rotating it to obtain an enhanced video of facial and oral activities; uses a convolutional neural network to perform lip analysis based on the enhanced video of facial and oral activities to obtain the lip shape features of the subject; uses a facial expression recognition algorithm to capture dynamic expression changes based on the enhanced video of facial and oral activities to obtain the facial expression features of the subject; and uses a deep learning model to analyze the movement and position of the tongue based on the enhanced video of facial and oral activities to obtain the tongue position features of the subject. The time stretching range is -5% to +5%, the frequency shift range is -3% to +3%, and the angle rotation range is -10° to +10°. When the fusion module fuses the audio acoustic features, video visual features, and demographic information encoded data into a fused feature vector, it specifically performs the following: re-encodes the audio acoustic features, video visual features, and demographic information encoded data into a multimodal feature vector using a cross-modal model to generate a re-encoded vector; and uses a self-attention mechanism to weight the importance of different modal features in the re-encoded vector to obtain the fused feature vector. The pre-trained prediction model includes a primary classification prediction model and a secondary classification prediction model; When the detection module obtains the speech impairment detection result of the subject based on the fused feature vector and the pre-trained prediction model, it is specifically used for: The fused feature vector is input into a primary classification prediction model, and the primary classification prediction model outputs a primary classification result, which includes either normal or speech disorder. When the primary classification result is a speech disorder, the secondary classification prediction model outputs a secondary classification result based on the fused feature vector, and the secondary classification result is used to characterize the speech disorder type.

5. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images