A multilingual automatic recognition method and system

Abstract

The application relates to a multilingual automatic recognition method and system, and belongs to the technical field of language recognition. The recognition method comprises the following steps: receiving an original voice signal and performing pretreatment to obtain a pretreated voice frame sequence; simultaneously extracting an acoustic feature vector, a vocal organ movement feature matrix and a prosody feature vector from the voice frame sequence to form a feature triple; performing language family classification according to the acoustic feature vector and the vocal organ movement feature matrix to output a candidate language family set; inputting the candidate language family set and the prosody feature vector into a dialect clustering model to output a refined dialect cluster label; calculating acoustic feature weight values, vocal organ movement feature weight values and prosody feature weight values, performing weighted operation on the feature triple to generate a weighted feature vector; and inputting the weighted feature vector and the refined dialect cluster label into a language decision model to output a language recognition result containing a language label and a confidence value. The application improves the accuracy and robustness of multilingual recognition in a complex environment.

Description

Technical Field

[0001] This application relates to the field of language recognition technology, and in particular to a multilingual automatic recognition method and system. Background Technology

[0002] With the acceleration of globalization and the surge in demand for cross-language interaction, traditional monolingual or limited language recognition systems can no longer meet the requirements of rapid and accurate identification of multiple languages, dialects, and low-resource languages ​​in practical applications. There is an urgent need to build an automated language recognition architecture that can adapt to diverse voice inputs, has strong anti-interference capabilities, and is highly scalable.

[0003] Currently, most language recognition systems still rely primarily on acoustic features (such as MFCC and FBANK) combined with shallow classifiers (such as GMM and SVM) or deep neural networks for end-to-end modeling. The core idea is to distinguish the phoneme distribution patterns of different languages ​​by extracting the spectral characteristics of speech. However, in real-world applications, speech signals are often affected by environmental noise, channel distortion, individual speaker differences, and equipment acquisition quality, leading to a decrease in the stability of acoustic features. Recognition performance degrades significantly, especially under conditions of short speech, low signal-to-noise ratio, or accent variation.

[0004] Furthermore, existing systems lack hierarchical reasoning mechanisms and fail to fully utilize the hierarchical relationships between language families, dialects, and language types. This results in a lack of contextual guidance in the classification process, which easily leads to cross-language mismatches. Consequently, the practicality and reliability of multilingual recognition systems in complex real-world scenarios are limited, and they are unable to cope with the growing demand for cross-language interaction and technical challenges. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a multilingual automatic recognition method and system.

[0006] Firstly, this application provides a multilingual automatic recognition method, which adopts the following technical solution:

[0007] A multilingual automatic recognition method, the recognition method comprising:

[0008] The raw speech signal is received and preprocessed to obtain a preprocessed speech frame sequence.

[0009] Acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors are extracted synchronously from the speech frame sequence to form feature triples;

[0010] Based on the acoustic feature vectors and the speech organ motion feature matrix, language family classification is performed, and a candidate language family set is output.

[0011] Input the candidate language family set and the prosodic feature vector into the dialect clustering model, and output refined dialect cluster labels;

[0012] Based on the duration of the speech frame sequence and the prosodic feature vector, calculate the acoustic feature weight value, the speech organ movement feature weight value, and the prosodic feature weight value;

[0013] The feature triples are weighted based on the acoustic feature weights, the vocal organ movement feature weights, and the prosodic feature weights to generate a weighted feature vector.

[0014] The weighted feature vector and the refined dialect cluster label are input into the language decision model, and the output is the language identification result containing the language label and confidence value.

[0015] By adopting the above technical solution, efficient mapping from raw speech to precise language labels is achieved. By introducing the movement features of the vocal organs as a physiological dimension criterion, the limitations of traditional methods relying solely on acoustic features are overcome. Simultaneously, a hierarchical reasoning structure of language family → dialect → language is established, and generative data augmentation enhances low-resource language recognition capabilities. Furthermore, this application designs a dynamic weighting mechanism based on speech duration and signal-to-noise ratio, achieving adaptive optimization of feature fusion. Feature reuse and closed-loop transmission ensure the system's logical rigor and lack of information gaps. This application's solution significantly improves the accuracy and robustness of multilingual recognition in complex environments while maintaining real-time performance, making it particularly suitable for high-value application scenarios such as border security checks, cross-border customer service, and multilingual human-computer interaction.

[0016] Optionally, the step of simultaneously extracting acoustic feature vectors, articulatory organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples includes:

[0017] Perform linear prediction analysis on the speech frame sequence to generate a set of vocal tract parameter coefficients;

[0018] The set of vocal tract parameter coefficients is input into a pre-trained articulatory organ motion prediction model, and the output is an articulatory organ motion feature matrix representing changes in tongue position and lip shape.

[0019] Calculate the improved Mel-frequency cepstral coefficients of the speech frame sequence to generate an acoustic feature vector;

[0020] The pitch contours and syllable boundaries of the speech frame sequence are detected, and a prosodic feature vector is generated based on the pitch standard deviation and syllable interval entropy.

[0021] The motion feature matrix of the vocal organs, the acoustic feature vector, and the prosodic feature vector are combined into feature triples.

[0022] By employing the above technical solution, a deep mapping from speech frame sequences to a multi-dimensional feature space is achieved. This solution overcomes the limitations of traditional single acoustic features by introducing a speech organ motion prediction mechanism driven by LPC coefficients. It combines an improved MFCC with temporal differentials and prosodic features based on statistics to form a three-in-one feature system integrating physiology, perception, and rhythm. This not only enhances the discriminative ability of the features but also demonstrates stronger adaptability, especially in challenging scenarios such as low resources, short speech, and high noise. It provides high-quality, interpretable input representations for subsequent language recognition, dialect classification, and cross-language speech understanding tasks.

[0023] Optionally, the step of classifying language families based on the acoustic feature vectors and the speech organ motion feature matrix, and outputting a candidate language family set, includes:

[0024] The acoustic feature vectors are input into a pre-trained language family probability prediction model to generate an initial language family probability distribution;

[0025] Extract the positional parameters of key speech organs from the motion feature matrix of the speech organs;

[0026] The initial language family probability distribution is corrected based on the location parameters to generate a corrected language family probability distribution;

[0027] Language families whose probability values ​​exceed a preset probability threshold in the modified language family probability distribution are selected to form a candidate language family set.

[0028] By employing the aforementioned technical solutions, intelligent reasoning from multimodal features to language family candidates is achieved. Acoustic feature streams provide generalized linguistic statistical features, while articulatory organ feature streams introduce rigid constraints from articulatory physiology. These two approaches are fused at the probabilistic level to form a complementary and enhanced classification strategy. The multi-head attention TCN model ensures precise modeling of the temporal structure of speech, while conditional correction rules based on tongue position deviation and lip shape endow the system with linguistic cognitive capabilities. The introduction of a dynamic threshold function further enhances the system's adaptability to practical application scenarios, demonstrating superior performance, especially in short speech and low-resource language recognition. The overall solution not only improves the accuracy and robustness of language family classification but also enhances the model's transparency and credibility through interpretable correction logic, providing solid technical support for building next-generation multilingual speech recognition systems and possessing significant innovative value and industrialization prospects.

[0029] Optionally, after the step of outputting the candidate language family set, the following may also be included:

[0030] Calculate the topological distance between language families in the candidate language family set;

[0031] When there are conflicting language pairs with a topological distance less than a preset value, extract the tongue trajectory parameters from the speech organ motion feature matrix;

[0032] Based on the dynamic entropy value of the tongue trajectory parameters, language options with abnormal entropy values ​​are filtered from the conflicting language pairs and removed, and the filtered candidate language set is output to the dialect clustering model.

[0033] By adopting the above technical solutions, the problem of ambiguity in language identification between geographically proximate or similar languages ​​is effectively solved. The characteristics of articulatory organ movement are elevated from mere auxiliary features to a basis for decision-making arbitration. The temporal stability of tongue trajectory is used as a physiological fingerprint for language attribution, breaking through the bottleneck of traditional acoustic models in distinguishing highly similar languages. The introduction of topological distance provides a priori constraints at the linguistic level, while the calculation of dynamic entropy enables a quantitative assessment of the naturalness of articulation behavior. The combination of these two approaches forms a conflict resolution mechanism that combines cognitive rationality and engineering feasibility. This application not only improves the accuracy and reliability of language family identification, but also demonstrates superior performance, especially in multilingual regions, accent variations, or low-resource language identification scenarios, providing key technical support for building a high-precision, robust, and automatic multilingual recognition system.

[0034] Optionally, the step of inputting the candidate language family set and the prosodic feature vector into the dialect clustering model and outputting refined dialect cluster labels includes:

[0035] Determine whether a low-resource language family exists in the candidate language family set;

[0036] If so, then the pre-stored language topology map is used to generate enhanced training data;

[0037] The prosodic feature vectors are merged with the enhanced training data to form a clustering input dataset;

[0038] Based on a pre-trained dialect clustering model, spectral clustering analysis is performed on the input dataset to generate dialect clustering results.

[0039] Based on the dialect cluster segmentation results, output refined dialect cluster labels and confidence levels.

[0040] By employing the aforementioned technical solutions, high-precision identification of dialect variants is achieved under conditions of data scarcity. Combining linguistic knowledge (language topology maps) with deep generative artificial intelligence (GANs), a semantically aware dynamic data augmentation mechanism is proposed, significantly improving the clustering stability and generalization ability for low-resource languages. Simultaneously, spectral clustering and Mahalanobis distance confidence calculation enhance the modeling ability and decision interpretability of complex dialect continuums. This approach ensures efficient processing of high-resource languages ​​while achieving intelligent compensation for marginal languages, making it particularly suitable for real-world scenarios such as cross-border customer service, border identification, and the protection of minority languages, demonstrating strong technical adaptability and engineering practical value.

[0041] Optionally, the step of calculating the acoustic feature weight value, the articulatory organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector includes:

[0042] Receive the total duration parameter and prosodic feature vector of the speech frame sequence;

[0043] The syllable interval entropy value and pitch standard deviation parameter are extracted from the prosodic feature vector;

[0044] Calculate the duration decay factor based on the total duration parameter of the speech frame sequence;

[0045] A prosodic stability index is generated based on the syllable interval entropy value and pitch standard deviation parameter;

[0046] Input the duration decay factor and prosodic stability index into the dynamic weight function library;

[0047] Output acoustic feature weight values, vocal organ movement feature weight values, and prosodic feature weight values.

[0048] By adopting the above technical solution, the entire weight generation process fully considers the interactive effects of speech duration, rhythm regularity and intonation features, realizing a technical leap from static configuration to dynamic adaptation. This not only improves the scientificity and adaptability of feature fusion, but also shows stronger robustness, especially in challenging scenarios such as noisy environments, short speech, and accent variations, providing key support for the accurate decision-making of multilingual automatic recognition systems.

[0049] Optionally, the step of inputting the weighted feature vector and the refined dialect cluster label into the language decision model and outputting the language identification result containing the language label and confidence value includes:

[0050] The target sub-classifier is selected from the pre-stored set of sub-classifiers based on the refined dialect cluster labels;

[0051] The weighted feature vector is input into the target sub-classifier to generate an initial language probability distribution;

[0052] Based on the confidence level corresponding to the refined dialect cluster label, the initial language probability distribution is calibrated;

[0053] Output the language label with the highest probability value and the calibrated confidence value.

[0054] By adopting the above technical solution, intelligent mapping from weighted features to precise language labels is achieved. By deeply integrating dialect cluster labels and their confidence levels into the final decision-making process, the limitations of the traditional unidirectional pipeline of "feature → classification" are broken, forming a closed-loop feedback cognitive enhancement mechanism. The divide-and-conquer sub-classifier architecture improves the model's professionalism and scalability, while the probability decay rule based on geographical distance introduces spatial linguistic priors, significantly reducing the cross-regional misclassification rate. This technical solution maintains high computational efficiency while possessing high accuracy and strong robustness, making it particularly suitable for complex real-world scenarios such as border security checks, cross-border communications, and multilingual human-computer interaction.

[0055] Secondly, this application provides a multilingual automatic recognition system, which adopts the following technical solution:

[0056] A multilingual automatic recognition system, the recognition system comprising:

[0057] The preprocessing module is used to receive the raw speech signal and perform preprocessing to obtain a preprocessed speech frame sequence;

[0058] The feature extraction module is used to simultaneously extract acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples;

[0059] The language classification module is used to classify languages ​​based on the acoustic feature vector and the speech organ motion feature matrix, and output a candidate language set.

[0060] The dialect clustering module is used to input the candidate language family set and the prosodic feature vector into the dialect clustering model and output refined dialect cluster labels;

[0061] The weight calculation module is used to calculate the acoustic feature weight value, the vocal organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector.

[0062] The weighted operation module is used to perform weighted operations on the feature triples based on the acoustic feature weight values, the vocal organ movement feature weight values, and the prosodic feature weight values ​​to generate a weighted feature vector.

[0063] The language output module is used to input the weighted feature vector and the refined dialect cluster label into the language decision model, and output the language recognition result containing the language label and confidence value.

[0064] Thirdly, this application provides a computer device, which adopts the following technical solution:

[0065] A computer device includes a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to perform the steps of the method as described in the first aspect.

[0066] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0067] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as in any of the methods in the first aspect. Attached Figure Description

[0068] Figure 1 This is a first flowchart illustrating a multilingual automatic recognition method according to one embodiment of this application.

[0069] Figure 2 This is a second flowchart illustrating a multilingual automatic recognition method according to one embodiment of this application.

[0070] Figure 3 This is a schematic diagram of the third process of a multilingual automatic recognition method according to one embodiment of this application.

[0071] Figure 4 This is a schematic diagram of the fourth process of a multilingual automatic recognition method according to one embodiment of this application.

[0072] Figure 5 This is a fifth flowchart illustrating a multilingual automatic recognition method according to one embodiment of this application.

[0073] Figure 6 This is a schematic diagram of the sixth process of a multilingual automatic recognition method according to one embodiment of this application.

[0074] Figure 7 This is a schematic diagram of the seventh process of a multilingual automatic recognition method according to one embodiment of this application. Detailed Implementation

[0075] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1-7 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0076] This application discloses a method for automatic multilingual recognition.

[0077] Reference Figure 1 A multilingual automatic recognition method, the recognition method includes:

[0078] Step S101: Receive the original speech signal and perform preprocessing to obtain a preprocessed speech frame sequence;

[0079] In the actual acquisition process, the original speech signal is inevitably affected by environmental noise, channel distortion and equipment interference. Therefore, the signal must first be denoised to improve the accuracy and robustness of subsequent feature extraction.

[0080] Specifically, commonly used noise reduction methods include spectral subtraction, Wiener filtering, or deep learning-based time-frequency domain denoising models (such as DCCRN or SEGAN). These techniques can effectively suppress background noise while preserving the semantic and pronunciation structure information of speech.

[0081] Based on this, the denoised continuous speech signal is subjected to frame segmentation, which divides it into a series of short-time stationary speech frames, typically 20-30 milliseconds in length with a frame shift of about 10 milliseconds. This operation is based on the assumption of short-time stationarity of the speech signal, making the acoustic characteristics within each frame approximately constant, which facilitates subsequent feature analysis. After framing, windowing (such as Hamming window) is often applied to reduce spectral leakage, thereby obtaining a set of time-aligned, structurally clear speech frame sequences, providing a stable data foundation for the synchronous extraction of multi-dimensional features.

[0082] Step S102: Simultaneously extract acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples;

[0083] Among them, acoustic feature vectors usually refer to low-level acoustic representations such as Mel frequency cepstral coefficients (MFCC), filter bank energy (FBANK), or linear predictive cepstral coefficients (LPCC). They capture the energy distribution characteristics of speech signals in the frequency domain and are widely used in speech recognition and language classification tasks, with good discriminative ability and computational efficiency.

[0084] The speech organ motion feature matrix belongs to higher-order physiological features, reflecting the dynamic changes in vocal tract configurations such as tongue position, lip shape, and mandibular movement during articulation. This type of information cannot be directly observed from audio, but can be indirectly derived through linear predictive coding (LPC) coefficients. Specifically, the LPC model estimates the vocal tract transfer function through autoregression, and its coefficients implicitly contain formant positions and bandwidth information. This information is further input into a pre-trained neural network (such as a convolutional autoencoder or Transformer structure) to decode an approximate speech organ motion trajectory matrix. This matrix represents the coordinated pattern of articulation movements in the time-space dimension and plays a crucial role in distinguishing languages ​​with similar acoustic performance but different articulation mechanisms (such as French rounded consonants and Chinese unrounded consonants).

[0085] Prosodic feature vectors focus on suprasegmental information of speech, including fundamental frequency (F0) profile, energy variation, speech rate rhythm, pause patterns, etc. These features show significant differences between different language families—for example, tonal languages ​​(such as Chinese and Thai) have obvious fundamental frequency tone variations, while stressed languages ​​(such as English and Russian) depend on syllable intensity and duration distribution.

[0086] The three types of features mentioned above construct a three-dimensional speech representation system from three complementary dimensions: "auditory perception", "pronunciation physiology" and "language rhythm", forming "feature triples" that provide rich and structured input for subsequent hierarchical classification mechanisms.

[0087] Step S103: Classify language families based on acoustic feature vectors and speech organ movement feature matrices, and output a candidate language family set;

[0088] Language family classification can be achieved using a language family classifier, such as a multilayer perceptron (MLP), support vector machine (SVM), or lightweight neural network. The training data covers representative language samples from major global language families (such as Indo-European, Sino-Tibetan, Altaic, and Austronesian). The core logic of this classifier lies in its joint discriminative ability based on the phoneme distribution statistics provided by acoustic features and the vocal tract movement patterns revealed by the features of the articulatory organs.

[0089] For example, certain language families commonly use uvular or pharyngeal consonants (such as the Afro-Asian language family to which Arabic belongs), and their corresponding LPC high-frequency response and tongue root retraction movement will show specificity in the articulatory organ feature matrix; while the frequent use of rounded vowels (such as French / u / , German / ø / ) will lead to a significant increase in lip closure parameters. If such high rounded lip feature parameters are detected in the current speech segment and exceed a preset threshold, the possibility of East Asian language families (such as Chinese, Japanese, and Korean) can be excluded according to the logic of claim 3, because these languages ​​have fewer rounded vowels and smaller lip shape changes.

[0090] Understandably, this prior constraint mechanism based on physiological motivation not only improves classification accuracy but also enhances the model's robustness to low signal-to-noise ratios or short speech segments. The final output "candidate language family set" is not a single language family label but a candidate list ordered by probability (e.g., {Sino-Tibetan: 0.82, Indo-European: 0.15}), preserving uncertainty space for more refined dialect clustering downstream.

[0091] Step S104: Input the candidate language family set and prosodic feature vector into the dialect clustering model, and output refined dialect cluster labels;

[0092] The key to this step lies in introducing prior knowledge of language families to guide the clustering process, avoiding the misjudgments that traditional unsupervised clustering makes in cross-language confusion. For example, when the candidate language family set includes "Sino-Tibetan language family", the system will activate a spectral clustering model designed for Chinese dialects. Combining the tonal patterns in prosodic features (such as the four tones of Mandarin vs. the nine tones and six tones of Cantonese), speech rate distribution (Mandarin is faster vs. Wu is slower), and tone sandhi rules, the input speech will be classified into refined dialect clusters such as "Mandarin", "Cantonese", and "Min Nan".

[0093] Specifically, addressing the scarcity of samples from low-resource language families (such as the Tai-Kadai and Tibeto-Burman branches), the system can activate a Generative Adversarial Network (GAN) to synthesize enhanced training speech: the generator learns the joint acoustic-prosodic distribution of a small number of existing real speech samples, while the discriminator distinguishes between real and fake samples. Through adversarial training, realistic virtual speech data is generated to expand the training set and improve the generalization ability of the clustering model. This process not only alleviates the data bias problem but also gives the system a certain potential for zero-shot transfer learning. The final output of refined dialect cluster labels is not only a classification result but also a semantic anchor, providing contextual constraints for subsequent language decision-making.

[0094] Step S105: Calculate the acoustic feature weight value, the speech organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector.

[0095] Among them, speech duration is a key adjustment factor that directly affects the reliability of each feature: in very short speech (e.g., <1.5 seconds), acoustic features are difficult to form a stable statistical distribution due to insufficient frames, while prosodic features (e.g., overall intonation trend, initial energy) can still be extracted within a limited time. Therefore, the system sets the weight of prosodic features to increase linearly with the shortening of speech duration to ensure that rhythmic information dominates in short speech. Conversely, in long speech, acoustic features are fully accumulated, and their weights are increased accordingly.

[0096] Meanwhile, the acoustic feature weights are also proportional to the speech signal-to-noise ratio (SNR), meaning their contribution is reduced in high-noise environments to prevent erroneous features from misleading the classifier. The weights of articulatory organ movement features are affected by the clarity of lip and tongue movements and can be evaluated through LPC residual energy or the temporal consistency of the feature matrix.

[0097] Understandably, this weighting mechanism is not statically configured, but dynamically adjusted based on real-time voice quality and content attributes, realizing an intelligent fusion strategy that applies weights according to the specific needs of the user, significantly improving the system's adaptability in complex real-world scenarios.

[0098] Step S106: Perform a weighted operation on the feature triples based on the acoustic feature weight value, the speech organ movement feature weight value, and the prosodic feature weight value to generate a weighted feature vector;

[0099] Since acoustic feature vectors, articulatory organ motion feature matrices, and prosodic feature vectors differ in dimension, scale, and semantic space, direct concatenation may lead to optimization difficulties or information overload. Therefore, this scheme adopts a linear weighted fusion method, mapping the three to the same vector space and then superimposing them according to their weights to form a comprehensive high-dimensional feature vector. This operation is essentially a simplified implementation of an attention mechanism, with the weight values ​​acting as "soft gating" to determine the contribution ratio of each modality's information in the final representation.

[0100] For example, in a short, abrupt Chinese instruction in a noisy environment, the system might assign weights of 0.6 to prosodic features, 0.3 to articulatory features, and 0.1 to acoustic features, emphasizing intonation and lip movement information. In a clear English speech recording, however, the weights might be reversed to emphasize phoneme details. The weighted feature vector retains the multidimensional information of the original triples and, after being adjusted for signal-to-noise ratio, duration, and language family priors, possesses stronger discriminative power and robustness.

[0101] Step S107: Input the weighted feature vector and refined dialect cluster label into the language decision model, and output the language identification result containing language label and confidence value.

[0102] The language decision model is not a single global classifier, but a hierarchical architecture composed of multiple language sub-classifiers. The system first selects the corresponding sub-classifier based on refined dialect cluster labels (e.g., "Mandarin" corresponds to the Standard Mandarin recognition model, "Wu" corresponds to the Shanghai / Suzhou dialect model), avoiding parameter redundancy and confusion risks caused by a single model covering all languages. Each sub-classifier can be implemented using deep neural networks (e.g., TDNN, Conformer) or probabilistic graphical models (e.g., GMM-HMM), specifically for fine-grained differentiation of languages ​​within a particular dialect region (e.g., Standard Mandarin vs. Southwestern Mandarin vs. Beijing Mandarin).

[0103] After inputting the weighted feature vector, the model outputs the probability distribution of each candidate language and selects the maximum value as the language label. It also outputs confidence values ​​(such as softmax probability or Bayesian posterior) for subsequent decision-making. For example, in an immigration security check scenario, the system receives a 1.2-second voice message, "My passport is in my bag," which, after processing, is identified as "Chinese - Mandarin 96.3%." This high confidence stems from the synergistic effect of multiple mechanisms, including using articulatory features to exclude French interference, matching prosodic features to Mandarin tone patterns, and weighted fusion to highlight key rhythmic cues in short speech segments.

[0104] The above embodiments achieve efficient mapping from raw speech to precise language labels. By introducing the movement features of the vocal organs as a physiological dimension criterion, it overcomes the limitations of traditional methods that rely solely on acoustic features. Simultaneously, it establishes a hierarchical reasoning structure of language family → dialect → language, and enhances low-resource language recognition capabilities through generative data augmentation. Furthermore, this application designs a dynamic weighting mechanism based on speech duration and signal-to-noise ratio, achieving adaptive optimization of feature fusion. Feature reuse and closed-loop transmission ensure rigorous system logic and eliminate information gaps. This application's solution significantly improves the accuracy and robustness of multilingual recognition in complex environments while maintaining real-time performance, making it particularly suitable for high-value application scenarios such as border security checks, cross-border customer service, and multilingual human-computer interaction.

[0105] Reference Figure 2 As one implementation of step S102, the step of simultaneously extracting acoustic feature vectors, articulatory organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples includes:

[0106] Step S201: Perform linear prediction analysis on the speech frame sequence to generate a set of vocal tract parameter coefficients;

[0107] Among them, Linear Predictive Analysis (LPA) is used to model the resonance characteristics of the vocal tract. Its core principle is to assume that the current speech sampling point can be predicted by a linear combination of the previous few historical sampling points, thereby constructing an autoregressive (AR) model.

[0108] In this embodiment, the system performs 18th-order linear predictive coding (LPC) calculations on each speech frame, that is, using the previous 18 samples to predict the current sample and solving for the coefficient set that minimizes the prediction error. The reason for choosing 18th order is that it can capture the information of the first six formants (F1–F6) of the adult vocal tract relatively accurately. Each formant roughly corresponds to two LPC coefficients, and six formants are sufficient to cover the key frequency bands from low-frequency nasal resonances to high-frequency fricative energy distribution. These LPC coefficients calculated frame by frame are arranged in chronological order to form a time-varying parameter sequence, namely the "vocal tract parameter coefficient set". It not only contains the fine structure of the speech spectrum envelope, but also implies the dynamic evolution of the vocal tract shape during the articulation process, laying a theoretical foundation for subsequent physiological feature mapping.

[0109] Step S202: Input the set of vocal tract parameter coefficients into the pre-trained articulatory organ motion prediction model and output the articulatory organ motion feature matrix representing changes in tongue position and lip shape.

[0110] Among them, traditional speech recognition and language classification mostly rely on auditory perception features, while this application innovatively introduces the movement information of the vocal organs as the discrimination basis, enhancing the model's ability to analyze the essential differences in languages.

[0111] Specifically, this process relies on a pre-trained three-dimensional convolutional neural network (3D-CNN) as the vocal organ movement prediction model. The input of this model is not the original LPC coefficients themselves, but their organization into a time-frequency representation form. For example, the 18th-order LPC coefficients of several consecutive frames are stacked into a two-dimensional matrix (time × order), and then input as a temporal image through the short-time Fourier transform or directly. 3D-CNN can effectively capture the temporal continuity and spatial correlation of the coordinated movement patterns such as the up-and-down movement, forward-and-back movement of the tongue position, and the opening and closing of the lips in the LPC coefficient sequence due to its convolutional ability in both the temporal and spatial dimensions.

[0112] Furthermore, the output layer of the network is designed to generate a three-dimensional heatmap matrix of three key pronunciation parameters: the height of the tongue position, the front-back position of the tongue position, and the opening and closing degree of the lip shape. Among them, the two spatial dimensions correspond to the position coordinates of the vocal organs, and the temporal dimension corresponds to the time axis of the speech frame sequence. For example, when pronouncing the sound " / i / " (such as "衣"), the tongue position is high and forward, and the lips are slightly open; while when pronouncing the sound " / u / " (such as "乌"), the tongue position retracts and the lips are rounded. These physiological actions will present specific spatial activation patterns in the heatmap. This movement feature matrix not only reflects the surface acoustic performance of the speech, but more reveals the underlying pronunciation actions as the source, enabling the system to achieve more accurate discrimination between languages with similar acoustics but different pronunciation mechanisms (such as rich rounded vowels in French vs. mainly unrounded vowels in Chinese).

[0113] Step S203, calculate the improved Mel-frequency cepstral coefficients of the speech frame sequence to generate an acoustic feature vector;

[0114] Among them, different from the standard MFCC, the improved Mel-frequency cepstral coefficients proposed in this application introduce an explicit modeling of the temporal dynamic information.

[0115] First, the system extracts the energy values of a 64-channel Mel filter bank, covering the auditory sensitive frequency band from 20 Hz to 8 kHz. This high-resolution setting helps to capture the phoneme distribution characteristics unique to different languages (such as pharyngeal sounds in Arabic concentrated in the low frequency and fricative sounds in English prominent in the high frequency). Subsequently, not only the energy values of each frame are calculated, but also their first-order difference (delta) and second-order difference (delta-delta) are further calculated, which respectively represent the rate of change and acceleration of the energy over time, and are used to characterize the dynamic processes such as phoneme transitions and co-articulation. The three together form a three-dimensional tensor (number of frames × 64 frequency bands × 3 dynamic orders), and are subjected to dimensionality reduction processing by principal component analysis (PCA) to remove redundant information and retain the most discriminative orthogonal components, finally forming a fixed-length acoustic feature vector.

[0116] PCA, in particular, aims to compress data dimensionality while maximizing variance retention, ensuring that the feature vectors maintain information integrity while meeting the structural consistency requirements of subsequent model inputs. This acoustic feature representation method, which integrates static energy and dynamic changes, significantly improves sensitivity to key language units such as syllable boundaries and consonant clusters in fast-paced speech, and is especially suitable for distinguishing language variants with fast speech rates or frequent linking.

[0117] Step S204: Detect the pitch contour and syllable boundaries of the speech frame sequence, and generate a prosodic feature vector based on the pitch standard deviation and syllable interval entropy;

[0118] Pitch (fundamental frequency F0) is a core indicator for distinguishing between tonal and non-tonal languages, and its variation patterns are highly specific across different language families. The system estimates the fundamental frequency trajectory for each frame using the autocorrelation function method: it calculates the correlation between the signal and its delayed version in the time domain, finds the period corresponding to the maximum correlation delay, and then converts it into an F0 value. Subsequently, the standard deviation of the entire speech F0 sequence is calculated as a quantitative indicator of pitch variation. For example, the four tones of Mandarin result in larger F0 fluctuations and a higher standard deviation; while the F0 variation in English declarative sentences is relatively gradual, resulting in a lower standard deviation.

[0119] On the other hand, syllable boundary detection is achieved through short-time energy envelope analysis: the absolute value of energy in each frame is calculated and envelope extraction is performed (e.g., Hilbert transform or half-wave rectification + low-pass filtering). Energy spikes often correspond to the starting positions of new syllables. Based on the detected syllable start times, the temporal distribution of the intervals between adjacent syllables is calculated, and its information entropy is obtained to measure the regularity of speech rate rhythm. For example, Japanese has a highly isochronous syllable rhythm with low interval entropy; while Mandarin Chinese exhibits non-isochronism, uneven stress distribution, and high interval entropy. Finally, the F0 standard deviation and syllable interval entropy are combined into a two-dimensional prosodic feature vector, concisely and effectively encoding the rhythm and intonation style of the language, providing powerful high-level pragmatic cues for language identification.

[0120] Step S205: Combine the speech organ motion feature matrix, acoustic feature vector, and prosodic feature vector into feature triples.

[0121] This combination is not a simple vector concatenation, but rather a semantic alignment and formal normalization of three heterogeneous features under a unified time reference. The articulatory organ motion features exist in the form of a three-dimensional heatmap matrix, recording the dynamic trajectory of the articulatory organs in the three dimensions of space, time, and parameters; the acoustic features are transformed into fixed-dimensional vectors after dimensionality reduction by PCA, ensuring that they can be directly processed by the classifier; although the prosodic features are low-dimensional vectors, they carry global discourse-level rhythmic information.

[0122] Understandably, these three elements together constitute a "feature triplet," corresponding to the three essential levels of speech generation: vocal tract physical configuration (motor features), auditory perception representation (acoustic features), and language rhythmic patterns (prosodic features). This triplet structure design enables the subsequent language recognition system to collaboratively judge from three complementary perspectives: physiological motivation, acoustic performance, and language habits, greatly enhancing the robustness and interpretability of the model.

[0123] It should be noted that the entire feature extraction process adopts a parallel processing architecture, with the three branches sharing the same speech frame sequence as the input source. This ensures that all features are strictly aligned on the time axis, avoiding misalignment issues caused by processing delays. Furthermore, the vocal tract parameter coefficient set not only serves the articulator prediction model but can also be used as an auxiliary feature for syllable boundary detection (e.g., formant transitions are often accompanied by syllable switching), demonstrating the implicit correlation and resource reuse between features, further improving the overall efficiency and consistency of the system.

[0124] In the above implementation, a deep mapping from speech frame sequences to a multi-dimensional feature space is achieved. This scheme breaks through the limitations of traditional single acoustic features, introduces a speech organ motion prediction mechanism driven by LPC coefficients, and combines improved MFCC with temporal differential and prosodic features based on statistics to form a three-in-one feature system of physiology, perception, and rhythm. This not only improves the discriminative ability of features, but also shows stronger adaptability, especially in challenging scenarios such as low resources, short speech, and high noise. It provides high-quality and interpretable input representations for subsequent language recognition, dialect classification, and cross-language speech understanding tasks.

[0125] Reference Figure 3 As one implementation of step S103, the step of classifying language families based on acoustic feature vectors and speech organ movement feature matrices, and outputting a candidate language family set, includes:

[0126] Step S301: Input the acoustic feature vector into the pre-trained language family probability prediction model to generate the initial language family probability distribution;

[0127] The model employs a temporal convolutional neural network (TCN) architecture based on a multi-head attention mechanism, combining local temporal modeling capabilities with the advantage of global dependency capture. The TCN expands the receptive field through dilated convolutions, effectively handling long-term dependencies in speech frame sequences without introducing recurrent structures. Meanwhile, the multi-head attention mechanism allows the model to focus on language-related discriminative segments in the speech signal in parallel across different subspaces, such as initial consonant clusters, vowel transition trajectories, or stress patterns, thereby enhancing sensitivity to cross-language phonological rules.

[0128] In this embodiment, the model input is a time-slice sequence of acoustic feature vectors, i.e., a temporal tensor composed of several consecutive frames. The output is a 19-dimensional language family probability vector, with each dimension corresponding to the probability of belonging to a predefined language family (such as Indo-European, Sino-Tibetan, Altaic, Austronesian, etc.). The model is trained under supervision on a large-scale multilingual speech dataset, learning the mapping relationship from acoustic features to language family labels. Its output "initial language family probability distribution" is not a rigid classification result, but a soft, uncertain prior estimate. For example, a piece of Mandarin speech might be assigned a probability distribution of 0.78 for Sino-Tibetan, 0.12 for Indo-European, and 0.06 for Austroasiatic at this stage, reflecting the system's preliminary judgment of language family belonging based on acoustic cues.

[0129] However, predictions relying solely on acoustic features still carry the risk of misjudgment, especially in cases of low signal-to-noise ratio, short speech, or accent variations. Therefore, it is necessary to introduce more discriminative physiological features for correction.

[0130] Step S302: Extract the positional parameters of key speech organs from the motion feature matrix of speech organs;

[0131] Instead of directly using the original three-dimensional heatmap matrix, this process extracts the most language-discriminating low-dimensional indicators through statistical analysis to improve computational efficiency and enhance interpretability.

[0132] Specifically, the system first calculates the first three statistical moments of the tongue position height parameter: the mean reflects the average tongue position height (e.g., the high vowel / i / corresponds to a high tongue position, and the low vowel / a / corresponds to a low tongue position), the variance reflects the activity level of tongue position movement (e.g., large fluctuations in tongue position in fast speech), and the skewness describes the asymmetry of tongue position distribution (negative skewness indicates that the tongue position is in a high position more often, which is common in languages ​​dominated by high vowels).

[0133] Studies have shown that East Asian languages ​​(such as Chinese, Japanese, and Korean) generally have a high average tongue position and a significant negative skewness, due to their extensive use of high vowels such as / i / and / u / to form tonal syllables; while some languages ​​in the Indo-European language family (such as English and German) have a wider range of tongue position variations and a more symmetrical skewness.

[0134] Furthermore, the system detects the binary state index of lip opening and closing, that is, converting continuous lip shape parameters (such as lip crease width or lip curvature) into discrete states of "rounded lips" or "unrounded lips" by setting a threshold. This operation simulates the categorization process in human speech perception, highlighting the qualitative rather than quantitative changes in key articulation movements. For example, when the lip opening and closing is below a certain threshold (such as 0.6), it is judged as a rounded lip state, which is extremely common in Romance languages ​​(such as French, Spanish, and Italian), especially when the lips are clearly closed during the pronunciation of vowels such as / u / , / y / , and / o / ; while most dialects of the Sino-Tibetan language family are predominantly unrounded, with fewer and smaller lip rounding movements.

[0135] Step S303: Correct the initial language family probability distribution based on the position parameters to generate the corrected language family probability distribution;

[0136] This correction process is not a simple feature splicing or weighted fusion, but rather a conditional probability adjustment rule designed based on clear linguistic principles. When the system detects that the skewness value of the tongue height parameter is less than a predetermined lower limit (e.g., -0.8), it considers the pronunciation behavior to conform to the typical pattern of East Asian languages. However, to avoid misjudgment due to over-reliance on a single feature (such as certain non-East Asian languages ​​with dense high vowels), a strategy of reducing the probability value of East Asian languages ​​can be adopted to prevent the model from falling into the stereotypical association of "high tongue position = definitely Chinese". This counterintuitive design actually reflects a careful consideration of language diversity: although true native Chinese speakers often use a high tongue position, their tongue position dynamics have specific temporal distribution patterns. If only the skewness meets the standard while other features do not, it may be a distractor sample.

[0137] Conversely, when the lip opening / closing index indicates a sustained rounded lip state, the system significantly increases the probability value for Romance languages ​​(e.g., by multiplying by a gain factor of 1.4), because the high-frequency use of rounded vowels is one of the hallmark features of this language family. This rule-based probability modulation mechanism essentially constructs a lightweight symbol-connectivity hybrid reasoning system: the deep learning model provides initial probabilities, while the rule engine based on linguistic knowledge performs posterior corrections. The two work synergistically in the probability space, preserving data-driven generalization capabilities while introducing interpretable domain knowledge, thus improving the robustness and reliability of classification.

[0138] Step S304: Select language families whose probability values ​​exceed a preset probability threshold in the corrected language family probability distribution to form a candidate language family set.

[0139] Among them, the traditional fixed threshold (such as 0.3) is unstable when faced with different speech lengths: short speech (<1.5 seconds) often leads to a dispersed probability distribution due to insufficient information, making it difficult to reach a high threshold; while long speech may generate multiple high-probability language families, resulting in redundancy.

[0140] Therefore, in this embodiment, a dynamic probability threshold function can be introduced to calculate a preset probability threshold, the value of which is inversely proportional to the speech duration. Mathematically, it can be expressed as: Threshold(t) = 0.35 - 0.15 × sigmoid(5−2t), where t is the speech duration (seconds). When t is small (e.g., 1 second), the function outputs approximately 0.25, allowing more language families to enter the candidate set and avoiding missed detections due to insufficient information. As t increases (e.g., >3 seconds), the threshold gradually rises to 0.35, improving the strictness of the screening and preventing false alarms.

[0141] For example, in a 1.2-second French speech segment, the initial probability might show a Romance language family probability of 0.31, below the traditional threshold of 0.35, leading to exclusion. However, after correction using lip shape and rounding features, the probability increases to 0.42, and the dynamic threshold is automatically adjusted to 0.26, ultimately resulting in successful inclusion in the candidate set. The final "candidate language family set" typically contains 1-3 language families, serving as contextual priors for the next level of dialect clustering and language identification. This significantly reduces the search space and improves the overall system efficiency and accuracy.

[0142] In the above implementation, intelligent reasoning from multimodal features to language family candidates is achieved. The acoustic feature stream provides generalized linguistic statistical features, while the articulatory organ feature stream introduces hard constraints from articulatory physiology. The two are fused at the probabilistic level to form a complementary and enhanced classification strategy. The multi-head attention TCN model ensures precise modeling of the temporal structure of speech, while the conditional correction rules based on tongue position deviation and lip shape state endow the system with linguistic cognitive capabilities. The introduction of a dynamic threshold function further enhances the system's adaptability to practical application scenarios, especially demonstrating superior performance in short speech and low-resource language recognition. The overall solution not only improves the accuracy and robustness of language family classification but also enhances the transparency and credibility of the model through interpretable correction logic, providing solid technical support for building next-generation multilingual speech recognition systems and possessing significant innovative value and industrialization prospects.

[0143] Reference Figure 4 As a further implementation of the multilingual automatic identification method, after the step of outputting the candidate language family set, the method further includes:

[0144] Step S401: Calculate the topological distance between language families in the candidate language family set;

[0145] Topological distance, in this context, does not refer to physical distance in geographic space. Rather, it is a semantic similarity metric constructed based on linguistic genealogy and phonological features, reflecting the degree of kinship between different language families along their historical divergence paths. This distance is typically obtained by querying a pre-constructed language topological map, where each node represents a language family or major language. The edge weights are calculated by fusing multiple linguistic features, such as lexical cognition rate, phonological similarity, grammatical co-occurrence frequency, and cosine distance of vowel spatial distribution.

[0146] For example, Germanic languages ​​(such as German and Dutch) within the Indo-European language family have a relatively short topological distance to English (from the West Germanic branch), while the distance to Slavic languages ​​(such as Russian) is relatively long. When the system receives the candidate language family set (e.g., {German, Dutch}) output by the preceding module, it searches the graph for the shortest path length or comprehensive similarity score between the two languages. If the topological distance is less than a preset threshold (e.g., 0.3), it is determined to be a "conflicting language family pair".

[0147] Understandably, this judgment logic mimics the cautious approach of human linguists when faced with highly similar languages, avoiding misjudgments due to minor differences in acoustic features, which is particularly crucial in bilingual areas or language transition zones (such as the border area between northern Germany and southern Netherlands).

[0148] Step S402: When there are conflicting language pairs with a topological distance less than a preset value, extract the tongue trajectory parameters from the speech organ motion feature matrix;

[0149] Traditional language identification methods often rely on acoustic or prosodic features, which are often difficult to distinguish when faced with languages ​​with highly similar phonological structures. This application innovatively introduces the physiological motivation of pronunciation as a criterion, which enhances the ability to analyze the essential differences between languages.

[0150] Specifically, the tongue trajectory parameters are time-series data further extracted from the speech organ motion feature matrix generated in the preceding steps. Specifically, they include the continuous coordinate changes of the tongue body in the sagittal plane (anterior-posterior position), vertical plane (high-low position), and coronal plane (left-right offset). These parameters are decoded from the time-frequency representation of LPC coefficients by a three-dimensional convolutional neural network and have high spatiotemporal resolution.

[0151] For example, when pronouncing the German "ch" sound ( / x / ) and the Dutch "g" sound ( / ɣ / ), although both appear as velar fricatives on the frequency spectrum, there are subtle differences in the amplitude and duration of tongue back elevation: the German pronunciation emphasizes the retraction and tension of the tongue root, with a trajectory showing a steep rise followed by stability; while the Dutch pronunciation is relatively relaxed, with greater tongue position fluctuations. By tracing the dynamic evolution path of tongue position throughout the entire speech segment, the system can capture these coarticulation features that are difficult to detect by hearing alone, providing higher-order criteria for subsequent discrimination.

[0152] Step S403: Based on the dynamic entropy value of the tongue trajectory parameter, filter out language options with abnormal entropy values ​​from the conflict language pairs and remove them, and output the filtered candidate language set to the dialect clustering model.

[0153] The dynamic entropy value is not a static statistical representation of traditional information entropy, but rather a quantitative assessment of the complexity of tongue trajectory changes over time, reflecting the regularity and stability of the articulation action. Its calculation is typically based on the joint probability distribution of tongue velocity, acceleration, and rate of change of direction within a sliding window, yielding its Shannon entropy.

[0154] Specifically, low entropy values ​​indicate highly ordered and repetitive tongue movements, consistent with the natural and fluent pronunciation patterns of native speakers. High entropy values, on the other hand, signify disordered and disjointed movements, potentially corresponding to non-native speaker accents, excessively fast speech, or a mismatch with the target language. For example, in a speech segment authentically in German, if the entropy value of the tongue trajectory is significantly higher than the baseline model for that language (e.g., mean + 2 standard deviations), it indicates that the pronunciation movements do not conform to the typical stable fricative implementation of German, but are closer to the loose, diffused pronunciation habits of Dutch. In this case, although the two are highly similar in acoustic features and topological relationships, physiological motivation analysis can identify "Germany" as an abnormal option and eliminate it. This process essentially constructs a "reverse verification" mechanism: instead of directly confirming which language family is correct, it narrows down the candidate range by eliminating language families that do not conform to the rules of pronunciation behavior, thereby improving the robustness of the decision.

[0155] Furthermore, after the aforementioned conflict resolution process, redundant or contradictory language families that could potentially lead to misjudgments are effectively filtered out, retaining only the language family option that best conforms to the physiological laws of pronunciation. For example, in a multinational conference system instance, the initial candidate language families for the input speech are {Dutch, German}, with a topological distance of only 0.2, which is a typical conflict pair. After further extracting the tongue position trajectory and calculating its dynamic entropy value, it is found that the entropy value corresponding to Dutch is 1.8, far exceeding the normal range for this language (usually <1.2), while the entropy value of German is 1.0, which conforms to stable pronunciation characteristics. Therefore, Dutch is judged as an abnormal option and removed, and finally the single-element set {German} is output to enter the next stage.

[0156] Understandably, the streamlined candidate language family set not only reduces the search space of subsequent dialect clustering models but also avoids misguidance caused by language family confusion, ensuring that the clustering process focuses on the variations within the correct language families. More importantly, this mechanism enables cross-module knowledge transfer: the results of the preceding language family classification are no longer used as absolute premises but as hypotheses to be verified, and are cross-validated through the independent information source of articulatory organ movement features, forming a closed-loop feedback-based reliable reasoning architecture.

[0157] The above embodiments effectively solve the problem of identification ambiguity between geographically proximate or genealogically similar languages, elevating the characteristics of articulatory organ movement from mere auxiliary features to a basis for decision-making arbitration. Utilizing the temporal stability of tongue trajectory as a physiological fingerprint of language attribution breaks through the bottleneck of traditional acoustic models in distinguishing highly similar languages. The introduction of topological distance provides a priori constraints at the linguistic level, while the calculation of dynamic entropy enables a quantitative assessment of the naturalness of articulation behavior. The combination of these two approaches forms a conflict resolution mechanism that combines cognitive rationality and engineering feasibility. This application not only improves the accuracy and reliability of language family identification, but also demonstrates superior performance, especially in multilingual regions, accent variations, or low-resource language recognition scenarios, providing key technical support for building a high-precision, robust, and automatic multilingual recognition system.

[0158] Reference Figure 5 As one implementation of step S104, the step of inputting the candidate language family set and prosodic feature vector into the dialect clustering model and outputting refined dialect cluster labels includes:

[0159] Step S501: Determine whether there is a low-resource language family in the candidate language family set;

[0160] If yes, proceed to step S502; otherwise, proceed directly to the regular dialect clustering process, input the candidate language family set and prosodic feature vector into the dialect clustering model, and output refined dialect cluster labels.

[0161] Low-resource language families refer to language families with a small number of language samples, limited geographical coverage, and insufficient phonological diversity in the training dataset, such as the Tai-Kadai language family (including Thai, Lao, and Dai), the Austronesian language family (including Malay and Fijian), or some Tibeto-Burman languages. Due to the lack of large-scale labeled corpora, traditional clustering algorithms for these languages ​​are prone to unstable cluster structures, blurred boundaries, or even clustering failures due to data sparsity.

[0162] To address this, the system incorporates a predefined low-resource language family list. By querying whether each language family in the candidate language family set exists in this list, a binary judgment flag (existence / non-existence) is generated. This judgment is not statically configured but rather a dynamically updated metadata management mechanism, allowing the system to automatically adjust the resource level of newly added languages ​​based on the data accumulation.

[0163] Step S502: Use the pre-stored language topology map to generate enhanced training data;

[0164] When the judgment result is "existent", the system will activate the enhancement module that works in collaboration with the language topology map and the generative adversarial network (GAN) to make up for the lack of real training data; if the result is "no", it will directly enter the regular clustering process to avoid unnecessary computational overhead, which reflects the design concept of resource-sensitive processing.

[0165] Specifically, the language topology map is not a simple linguistic tree structure, but a language relationship network constructed based on phonological features. Its construction process is as follows: First, the spatial distribution of vowels in the speech of native speakers of each language is extracted, and the centroids of their vowel vectors on the F1-F2 formant plane (i.e., the average positions of the major vowels / i, a, u / , etc.) are calculated. Then, the inter-language distance function is defined as the complement of the cosine similarity. This measurement method can effectively reflect the similarity of different languages ​​in the structure of their vowel systems. For example, Thai, Cambodian, and Tai have a high geometric similarity in vowel distribution and are therefore classified as "adjacent languages".

[0166] Furthermore, when the system identifies the current speech as belonging to a low-resource language family (such as Thai), it retrieves its neighboring language set based on the atlas and activates a generative adversarial network (GAN) to synthesize mixed speech features. The GAN's generator uses the prosodic features of Thai and Cambodian as training samples, learning the joint distribution of these two languages ​​in terms of fundamental frequency variation rate, syllable duration distribution, and stress patterns. This generates a series of virtual prosodic vectors that lie between the two languages. These vectors retain the basic features of the target language family (Thai) while introducing the diversity and variations of neighboring languages, thus simulating the pronunciation habits of different regions, accents, and age groups. This language topology-guided data augmentation strategy is more linguistically sound than random noise injection or simple interpolation, effectively expanding the representation space of low-resource languages.

[0167] Step S503: Merge the prosodic feature vectors with the enhanced training data into a clustering input dataset;

[0168] The merging method typically employs matrix concatenation to form a hybrid dataset containing both real and synthetic samples. For non-low-resource language scenarios, the clustering input consists only of the original prosodic features; while for low-resource languages, the data scale is significantly expanded, improving the robustness of the clustering algorithm. This design implements an intelligent control mechanism of "on-demand enhancement," avoiding redundant computation for high-resource languages ​​while specifically addressing the data bottleneck problem for low-resource languages, reflecting a system-level resource optimization approach.

[0169] Step S504: Perform spectral clustering analysis on the clustering input dataset based on the pre-trained dialect clustering model to generate dialect clustering results;

[0170] Among them, spectral clustering, as a nonlinear clustering method based on graph theory, is particularly suitable for processing data distributions with complex manifold structures. Compared with traditional algorithms such as K-means, it is better able to capture the boundary characteristics of gradual transitions between dialects (such as the continuous variation between Mandarin and Jin dialect).

[0171] First, a similarity matrix is ​​constructed for all samples in the dataset (including original and synthetic prosodic features), where the similarity between any two samples is determined by their cosine similarity, reflecting their directional consistency in the prosodic space. Second, a Laplacian matrix (L=DW, where D is the degree matrix and W is the weight matrix) is calculated based on the similarity matrix, and eigenvalue decomposition is performed on it to extract the eigenvectors corresponding to the k smallest non-zero eigenvalues. These eigenvectors constitute a new representation of the original data in a low-dimensional embedding space, which can effectively separate the potential cluster structure. Finally, the K-means algorithm is used to cluster this low-dimensional feature space to complete the final dialect cluster division.

[0172] It should be noted that the k value (i.e. the number of clusters) can be automatically determined by the spectral gap method, that is, the index after the maximum interval in the feature value sequence is selected as the optimal number of clusters, thereby achieving adaptive clustering without manual preset, which is especially suitable for practical application environments with unknown dialect numbers.

[0173] Step S505: Based on the dialect cluster segmentation results, output refined dialect cluster labels and confidence levels.

[0174] This step not only provides classification labels but also introduces a quantitative confidence assessment mechanism, enhancing the system's decision-making transparency. Specifically, the system assigns a unique identifier to each identified dialect cluster (e.g., "Northern Thai Dialect," "Southwestern Mandarin-Chengdu-Chongqing Sub-dialect") and calculates the Mahalanobis distance between the prosodic feature vector of the current input speech and the centroids of each cluster. The advantage of Mahalanobis distance over Euclidean distance lies in its consideration of the covariance structure across dimensions, enabling a more accurate measurement of the relative position of samples within non-spherical distribution clusters.

[0175] For example, in a dialect cluster with drastic fundamental frequency variations but stable speech rate, Mahalanobis distance automatically weights and adjusts the importance of the two dimensions. Subsequently, the reciprocal of this distance is used as a preliminary confidence score, and after normalization, it is transformed into a probability value within the [0,1] interval, representing the relative certainty of the current speech belonging to this cluster. For instance, in an international call center scenario, a Thai speech segment, after processing, is classified as "Northern Thai dialect," which has the smallest Mahalanobis distance to the centroid of this cluster. The normalized reciprocal yields a confidence score of 0.92, indicating that the system has high confidence in this judgment. This mechanism effectively addresses the deficiency of traditional clustering methods in lacking confidence assessment, providing reliable contextual guidance for subsequent language decision-making models.

[0176] The above implementation achieves high-precision identification of dialect variants under data-scarce conditions. By combining linguistic knowledge (language topology map) with deep generative artificial intelligence (GAN), a semantically aware dynamic data augmentation mechanism is proposed, significantly improving the clustering stability and generalization ability for low-resource languages. Simultaneously, spectral clustering and Mahalanobis distance confidence calculation enhance the modeling ability and decision interpretability of complex dialect continuums. This approach ensures efficient processing of high-resource languages ​​while achieving intelligent compensation for marginal languages, making it particularly suitable for real-world scenarios such as cross-border customer service, border identification, and minority language protection, demonstrating strong technical adaptability and engineering practical value.

[0177] Reference Figure 6 As one implementation of step S105, the step of calculating the acoustic feature weight value, the vocal organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector includes:

[0178] Step S601: Receive the total duration parameter and prosodic feature vector of the speech frame sequence;

[0179] The total duration of the speech frame sequence is determined by the number of frames and frame shift in the framing stage (e.g., 120 frames × 10 milliseconds = 1.2 seconds), reflecting the richness of information in the original speech segment; while the prosodic feature vector comes from the feature extraction stage and usually includes suprasegmental features such as syllable interval entropy value and pitch standard deviation, used to characterize the rhythmic regularity and intonation variation of speech.

[0180] Step S602: Extract the syllable interval entropy value and pitch standard deviation parameter from the prosodic feature vector;

[0181] Among them, the syllable interval entropy value is a statistical indicator for measuring the regularity of speech rhythm. It is calculated based on the syllable boundary time points detected by short-time energy envelope, and then statistically analyzes the distribution of time intervals between adjacent syllables and calculates the information entropy of this distribution. The higher the entropy value, the more random the syllable duration variation and the more irregular the rhythm; the lower the entropy value, the more uniform the speech rate and the stronger the isochronism of the rhythm.

[0182] The standard deviation of pitch is the standard deviation of the fundamental frequency (F0) trajectory over the entire speech segment, reflecting the intensity of intonation fluctuations and serving as an important criterion for distinguishing between tonal and non-tonal languages. In Mandarin, a typical tonal language, the F0 of Mandarin exhibits significant jumps between the four tones, with a standard deviation reaching 25–35 Hz; while the F0 of English declarative sentences changes more gradually, with a standard deviation generally less than 20 Hz.

[0183] Step S603: Calculate the duration decay factor based on the total duration parameter of the speech frame sequence. ;

[0184] This process is implemented through a pre-defined piecewise function, fully considering the differences in information availability under different speech lengths. When the speech length is ≤1 second, the system determines it to be an extremely short speech. Due to the insufficient number of frames, the acoustic features are difficult to form stable spectral statistics, resulting in a low signal-to-noise ratio. Therefore, a relatively high attenuation factor of 0.8 is set to reduce the weight of the acoustic features. When the speech length is between 1 and 3 seconds, an exponential attenuation method is adopted. As the speech length increases, acoustic information gradually accumulates, feature reliability improves, and the attenuation factor decreases smoothly, reflecting the transition from "prosody-dependent" to "balanced use." When the speech length exceeds 3 seconds, the acoustic features are sufficiently stable, and the attenuation factor is fixed at 0.3 to ensure that they dominate in long speech.

[0185] Understandably, this nonlinear modeling approach avoids the problems of sudden changes or over-compression that may be caused by linear functions, is more in line with the perceptual patterns of the human auditory system under different speech lengths, and also provides an adjustable adjustment coefficient for the subsequent generation of acoustic feature weights.

[0186] Step S604: Generate a prosodic stability index based on syllable interval entropy value and pitch standard deviation parameter;

[0187] Specifically, the offset of the syllable interval entropy relative to a preset baseline entropy (e.g., 0.9) is first calculated. If the actual entropy value is higher than the baseline, it indicates that the rhythm is chaotic and the stability is reduced; otherwise, it indicates that the rhythm is regular. This offset is normalized and used as the first input.

[0188] Secondly, the pitch standard deviation is substituted into the Gaussian normalization function, which takes the form of: , where μ and σ are the empirical mean and standard deviation (e.g., μ=25, σ=10), used to measure whether the current F0 variation is within the typical tonal range. For example, when the standard deviation is close to 25Hz, the normalized value approaches 1, indicating a stable intonation pattern; if it is too high or too low, the value decreases, reflecting intonation abnormalities or non-tonal features.

[0189] Finally, the stability index is fused using the formula "Stability Index = 0.6 × Entropy Offset + 0.4 × Normalized Pitch Standard Deviation", giving higher weight to rhythmic regularity and highlighting its universal value in cross-language differentiation. The higher the value of this index, the clearer and more predictable the prosodic structure of the current speech, thus enhancing its correlation with the movement characteristics of the articulatory organs.

[0190] Step S605: Input the duration decay factor and prosodic stability index into the dynamic weight function library;

[0191] This dynamic weight function library is not a single function, but a set of predefined mapping rules used to transform the two regulatory factors into specific weight values ​​for the three types of features.

[0192] Specifically, the acoustic feature weight value is obtained by multiplying the base value of 0.5 by the duration decay factor, i.e. This ensures that it gradually increases with the duration of speech, maintaining its dominant position in long speech; the weight value of the articulatory organ motor features is set to a base value of 0.3, and superimposed with 0.2 times the prosodic stability index, i.e. This reflects the cognitive logic that "the more stable the rhythm, the more predictable the articulation." When the speech rhythm is regular and the intonation pattern is clear, the movement trajectories of the tongue position and lip shape are more consistent, resulting in higher feature reliability. The prosodic feature weight values ​​adopt a residual allocation strategy. This ensures that the sum of the three is always 1, avoiding inconsistencies caused by manually setting the total.

[0193] Understandably, this self-balancing mechanism not only simplifies design complexity but also enables dynamic competition among features: when acoustic or vocal organ features perform well, prosodic features automatically become secondary, and vice versa.

[0194] Step S606: Output acoustic feature weight values, vocal organ movement feature weight values, and prosodic feature weight values.

[0195] Before the final output, the system performs weight standardization, which involves calculating the sum of the three values. Each weight is then divided by S to ensure it strictly satisfies the probability distribution constraints. For example, in an in-vehicle voice system scenario, the input speech duration is 2.1 seconds, corresponding to an attenuation factor of approximately 0.52; the syllable entropy is 1.2, the pitch standard deviation is 28Hz, and the calculated prosodic stability index is 0.42. Therefore, the initial weights are: acoustic 0.26, vocal organ 0.38, and prosodic 0.36, which remain unchanged after standardization.

[0196] Furthermore, this triple will be passed to the subsequent weighted fusion module, which will use it to weight and combine the acoustic feature vector, the speech organ motion feature matrix, and the prosodic feature vector to form a more discriminative comprehensive feature representation.

[0197] In the above implementation, the entire weight generation process fully considers the interactive effects of speech duration, rhythmic regularity, and intonation features, realizing a technological leap from static configuration to dynamic adaptation. This not only improves the scientific nature and adaptability of feature fusion, but also demonstrates stronger robustness, especially in challenging scenarios such as noisy environments, short speech, and accent variations, providing key support for accurate decision-making in multilingual automatic recognition systems.

[0198] Reference Figure 7 As one implementation of step S107, the step of inputting the weighted feature vector and refined dialect cluster label into the language decision model and outputting the language identification result containing the language label and confidence value includes:

[0199] Step S701: Select the target sub-classifier from the pre-stored sub-classifier set based on the refined dialect cluster label;

[0200] Traditional language identification systems often use a single global classifier to process all languages, resulting in redundant model parameters, high training difficulty, and easy confusion when facing dialect-dense areas (such as between various southern Chinese dialects or between Taiwanese languages ​​in Southeast Asia).

[0201] To address this, this application constructs a set of dedicated sub-classifiers, each optimized for a specific dialect cluster. For example, the "Cantonese sub-classifier" focuses on distinguishing variants of Guangzhou, Taishan, and Wuzhou dialects, while the "Eastern Hindi sub-classifier" focuses on regional accents such as Bihari and Avad. This selection process relies on a predefined mapping table, uniquely or preferentially associating each refined dialect cluster label with a sub-classifier. When a label clearly corresponds to a single sub-classifier, it is directly used; however, when multiple candidates exist (e.g., "Wu" can correspond to three sub-classifiers: Shanghainese, Suzhou, and Ningbo), the sub-classifier with the largest training data volume is selected to ensure the model's strongest generalization ability.

[0202] Understandably, this data-scale-based prioritization mechanism effectively avoids the risk of misjudgment caused by model overfitting due to small samples, while also improving the maintainability of the system. Adding a new language only requires adding the corresponding sub-classifier and updating the mapping table, without having to reconstruct the entire system.

[0203] Step S702: Input the weighted feature vector into the target sub-classifier to generate the initial language probability distribution;

[0204] Among them, the target subclassifier usually adopts a deep neural network structure (such as Temporal Convolutional Network TCN, Transformer or X-vector architecture), and its training data is strictly limited to the relevant language within the corresponding dialect cluster, thus forming a highly specialized discrimination boundary.

[0205] For example, the "Northern Thai Dialect" subclassifier is trained only on Thai variants from Northern Thailand, Northern Laos, and Shan State in Myanmar, learning subtle differences in phoneme realization, tone contours, and consonant weakening. After the weighted feature vector is input, the model extracts deep features through multiple nonlinear transformations and ultimately outputs an initial language probability distribution vector, where each dimension represents the probability of belonging to a candidate language.

[0206] This distribution reflects the degree of acoustic matching between each language and the input speech under the constraints of the current dialect context. For example, in a speech input that is suspected to be Northern Thai, the model may output: [Northern Thai: 0.65, Lao: 0.18, Shan State Tai (Myanmar): 0.12, Others: 0.05], showing an initial preference for the target language.

[0207] Step S703: Based on the confidence level corresponding to the refined dialect cluster labels, calibrate the initial language probability distribution;

[0208] The dialect confidence scores from previous steps are used as calibration coefficients to adjust the probability weights of each language. Specifically, the system first identifies the languages ​​associated with the current dialect cluster label (e.g., under the label "Northern Thai Dialect," Northern Thai is an associated language, while Mandarin Chinese is an unrelated language), and then performs the following differentiation processing:

[0209] For relevant languages, their initial probability is multiplied by the dialect confidence value (e.g., 0.85) to achieve moderate enhancement—the higher the confidence, the stronger the enhancement, reflecting the system's confidence in the current language environment. For irrelevant languages, they are multiplied by a preset attenuation factor (e.g., 0.3) to significantly weaken their influence and prevent cross-language misjudgment.

[0210] Furthermore, this attenuation factor can be dynamically adjusted based on the principles of geolinguistics: when the geographical distance between the candidate language and the current dialect cluster center exceeds 1000 kilometers, the attenuation factor is set to 0.1×log 10 (Distance), for example, the distance between Beijing dialect and Northern Thai dialect is about 2500 kilometers, then the attenuation factor is 0.1×log 10 (2500)≈0.1×3.4=0.34, significantly lower than that of nearby languages. This design, which transforms physical spatial distance into a probability decay factor, simulates the prior knowledge of human linguists that "languages ​​at a distance have low similarity," effectively suppressing cross-regional mismatches and improving the semantic rationality of the system.

[0211] Step S704: Output the language label with the highest probability value and the calibrated confidence value.

[0212] Based on the calibrated probability distribution, the system sorts all candidate languages ​​in descending order of probability value and outputs the language with the highest probability as the main recognition result. At the same time, its calibrated normalized probability value is presented to the user or downstream application as the confidence level.

[0213] Furthermore, the system can output the top three language labels and their probabilities, forming a multi-level output mode of "main result + alternative suggestions," suitable for secondary confirmation mechanisms in high-security scenarios. For example, in the Southeast Asian language recognition example, the input is "Northern Thai" (confidence level 0.85) and a weighted feature vector. After processing by a sub-classifier to obtain the initial probability, calibration adjusts the probability of Northern Thai from 0.65 to 0.552 (×0.85), while distant languages ​​such as Burmese are attenuated to 0.036. Finally, after normalization, the main result is "Northern Thai (confidence level 0.83)," with Lao and Dai as alternatives. This output not only provides a clear language judgment but also offers an interpretable confidence level and alternative options, greatly enhancing the system's practicality and credibility.

[0214] The above implementation achieves intelligent mapping from weighted features to precise language labels. By deeply integrating dialect cluster labels and their confidence levels into the final decision-making process, it breaks the limitations of the traditional unidirectional pipeline of "feature → classification" and forms a closed-loop feedback cognitive enhancement mechanism. The divide-and-conquer sub-classifier architecture improves the model's professionalism and scalability, while the probability decay rule based on geographical distance introduces spatial linguistic priors, significantly reducing cross-regional misclassification rates. This technical solution maintains high computational efficiency while possessing high accuracy and strong robustness, making it particularly suitable for complex real-world scenarios such as border security checks, cross-border communications, and multilingual human-computer interaction.

[0215] This application also discloses a multilingual automatic recognition system.

[0216] A multilingual automatic recognition system, the recognition system includes:

[0217] The preprocessing module is used to receive the raw speech signal and perform preprocessing to obtain a preprocessed speech frame sequence;

[0218] The feature extraction module is used to simultaneously extract acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors from speech frame sequences to form feature triples;

[0219] The language classification module is used to classify languages ​​based on acoustic feature vectors and the motion feature matrix of articulatory organs, and output a set of candidate language families.

[0220] The dialect clustering module is used to input the candidate language family set and prosodic feature vector into the dialect clustering model and output refined dialect cluster labels;

[0221] The weight calculation module is used to calculate the acoustic feature weight value, the speech organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector.

[0222] The weighted operation module is used to perform weighted operations on feature triples based on acoustic feature weight values, vocal organ movement feature weight values, and prosodic feature weight values ​​to generate a weighted feature vector.

[0223] The language output module is used to input the weighted feature vector and refined dialect cluster labels into the language decision model, and output the language recognition result containing language labels and confidence values.

[0224] The multilingual automatic recognition system of this application embodiment can implement any of the above-described multilingual automatic recognition methods, and the specific working process of each module in the multilingual automatic recognition system can be referred to the corresponding process in the above-described method embodiments.

[0225] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a certain module is merely a logical functional division, and in actual implementation there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0226] This application also discloses a computer device.

[0227] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multilingual automatic recognition method as described above.

[0228] This application also discloses a computer-readable storage medium.

[0229] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in any of the multilingual automatic recognition methods.

[0230] The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0231] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0232] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0233] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A multilingual automatic recognition method, characterized in that, The identification method includes: The raw speech signal is received and preprocessed to obtain a preprocessed speech frame sequence. Acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors are extracted synchronously from the speech frame sequence to form feature triples; Based on the acoustic feature vectors and the speech organ motion feature matrix, language family classification is performed, and a candidate language family set is output. Input the candidate language family set and the prosodic feature vector into the dialect clustering model, and output refined dialect cluster labels; Based on the duration of the speech frame sequence and the prosodic feature vector, calculate the acoustic feature weight value, the speech organ movement feature weight value, and the prosodic feature weight value; The feature triples are weighted based on the acoustic feature weights, the vocal organ movement feature weights, and the prosodic feature weights to generate a weighted feature vector. The weighted feature vector and the refined dialect cluster label are input into the language decision model, and the language identification result containing the language label and confidence value is output. The steps of synchronously extracting acoustic feature vectors, articulatory organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples include: Perform linear prediction analysis on the speech frame sequence to generate a set of vocal tract parameter coefficients; The set of vocal tract parameter coefficients is input into a pre-trained articulatory organ motion prediction model, and the output is an articulatory organ motion feature matrix representing changes in tongue position and lip shape. Calculate the improved Mel-frequency cepstral coefficients of the speech frame sequence to generate an acoustic feature vector; The pitch contours and syllable boundaries of the speech frame sequence are detected, and a prosodic feature vector is generated based on the pitch standard deviation and syllable interval entropy. The motion feature matrix of the vocal organs, the acoustic feature vector, and the prosodic feature vector are combined into feature triples.

2. The multilingual automatic identification method according to claim 1, characterized in that, The steps for classifying language families based on the acoustic feature vectors and the speech organ movement feature matrix, and outputting a candidate language family set, include: The acoustic feature vectors are input into a pre-trained language family probability prediction model to generate an initial language family probability distribution; Extract the positional parameters of key speech organs from the motion feature matrix of the speech organs; The initial language family probability distribution is corrected based on the location parameters to generate a corrected language family probability distribution; Language families whose probability values ​​exceed a preset probability threshold in the modified language family probability distribution are selected to form a candidate language family set.

3. The multilingual automatic identification method according to claim 2, characterized in that, After the step of outputting the candidate language family set, the following is also included: Calculate the topological distance between language families in the candidate language family set; When there are conflicting language pairs with a topological distance less than a preset value, extract the tongue trajectory parameters from the speech organ motion feature matrix; Based on the dynamic entropy value of the tongue trajectory parameters, language options with abnormal entropy values ​​are filtered from the conflicting language pairs and removed, and the filtered candidate language set is output to the dialect clustering model.

4. The multilingual automatic identification method according to claim 1, characterized in that, The steps of inputting the candidate language family set and the prosodic feature vector into the dialect clustering model and outputting refined dialect cluster labels include: Determine whether a low-resource language family exists in the candidate language family set; If so, then the pre-stored language topology map is used to generate enhanced training data; The prosodic feature vectors are merged with the enhanced training data to form a clustering input dataset; Based on a pre-trained dialect clustering model, spectral clustering analysis is performed on the input dataset to generate dialect clustering results. Based on the dialect cluster segmentation results, output refined dialect cluster labels and confidence levels.

5. The multilingual automatic identification method according to claim 4, characterized in that, The steps for calculating the acoustic feature weight value, the articulatory organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector include: Receive the total duration parameter and prosodic feature vector of the speech frame sequence; The syllable interval entropy value and pitch standard deviation parameter are extracted from the prosodic feature vector; Calculate the duration decay factor based on the total duration parameter of the speech frame sequence; A prosodic stability index is generated based on the syllable interval entropy value and pitch standard deviation parameter; Input the duration decay factor and prosodic stability index into the dynamic weight function library; Output acoustic feature weight values, vocal organ movement feature weight values, and prosodic feature weight values.

6. The multilingual automatic identification method according to claim 5, characterized in that, The steps of inputting the weighted feature vector and the refined dialect cluster label into the language decision model and outputting the language identification result containing the language label and confidence value include: The target sub-classifier is selected from the pre-stored set of sub-classifiers based on the refined dialect cluster labels; The weighted feature vector is input into the target sub-classifier to generate an initial language probability distribution; Based on the confidence level corresponding to the refined dialect cluster label, the initial language probability distribution is calibrated; Output the language label with the highest probability value and the calibrated confidence value.

7. A multilingual automatic identification system, characterized in that, The identification system includes: The preprocessing module is used to receive the raw speech signal and perform preprocessing to obtain a preprocessed speech frame sequence; The feature extraction module is used to simultaneously extract acoustic feature vectors, speech organ motion feature matrices, and prosodic feature vectors from the speech frame sequence to form feature triples; The language classification module is used to classify languages ​​based on the acoustic feature vector and the speech organ motion feature matrix, and output a candidate language set. The dialect clustering module is used to input the candidate language family set and the prosodic feature vector into the dialect clustering model and output refined dialect cluster labels; The weight calculation module is used to calculate the acoustic feature weight value, the vocal organ motion feature weight value, and the prosodic feature weight value based on the duration of the speech frame sequence and the prosodic feature vector. The weighted operation module is used to perform weighted operations on the feature triples based on the acoustic feature weight values, the vocal organ movement feature weight values, and the prosodic feature weight values ​​to generate a weighted feature vector. The language output module is used to input the weighted feature vector and the refined dialect cluster label into the language decision model, and output the language recognition result containing the language label and confidence value. The feature extraction module is configured as follows: Perform linear prediction analysis on the speech frame sequence to generate a set of vocal tract parameter coefficients; The set of vocal tract parameter coefficients is input into a pre-trained articulatory organ motion prediction model, and the output is an articulatory organ motion feature matrix representing changes in tongue position and lip shape. Calculate the improved Mel-frequency cepstral coefficients of the speech frame sequence to generate an acoustic feature vector; The pitch contours and syllable boundaries of the speech frame sequence are detected, and a prosodic feature vector is generated based on the pitch standard deviation and syllable interval entropy. The motion feature matrix of the vocal organs, the acoustic feature vector, and the prosodic feature vector are combined into feature triples.

8. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 6.

Images