A language detection method based on multi-dimensional feature fusion

Abstract

The application discloses a kind of language detection method and system based on multi-dimension feature fusion, it is related to speech processing technical field.The method includes: audio pre-processing, multi-dimension acoustic feature extraction, attention mechanism feature fusion, light weight language classification, language determination and score verification.Through parallel extraction spectrum, prosody, phoneme three kinds of complementary features, adopt hierarchical attention weighted fusion, construct light weight model to realize cross-platform offline real-time language detection, and the consistency of evaluation score is verified, and the false high score caused by language misjudgment is corrected.The application has the advantages of light weight, high robustness, interpretable, easy to expand, etc., suitable for intelligent speech evaluation scene.

Description

Technical Field

[0001] This invention relates to the fields of speech processing, speech recognition and artificial intelligence, and specifically to a cross-platform real-time language detection method and system for intelligent speech evaluation. Background Technology

[0002] Pronunciation assessment systems are widely used in language education and language standardization verification. Existing pronunciation assessment systems rely on acoustic models or deep neural networks, mainly modeling fluency and clarity of speech, but not modeling the language itself. This leads to situations where fluent Chinese audio may receive inflated scores when assessing English sentences because it matches some English phonemes, seriously misleading learners, undermining the credibility of the teaching system, and rendering it unusable for serious exams or proficiency tests.

[0003] Existing language detection technologies have significant drawbacks: 1) Indirect verification based on general speech recognition models: Large-scale ASR models are computationally complex and resource-intensive, making them difficult to integrate into desktop applications with high real-time and resource efficiency requirements; recognition accuracy depends on training data, and they have poor support for minority languages ​​and accented speech, easily leading to language misjudgment when there is noise or non-standard pronunciation; they cannot handle meaningless syllables or mixed language scenarios. 2) End-to-end language detection models: These rely on massive amounts of balanced data, have poor generalization ability for minority languages ​​and accents; the models have a large number of parameters and complex structures, making them difficult to deploy efficiently offline on resource-constrained terminals; the decision-making process is a black box, lacking interpretability and failing to meet the feedback needs of teaching scenarios.

[0004] Therefore, there is an urgent need for a lightweight, cross-platform, offline, robust, and interpretable language detection method to solve the problem of scoring anomalies caused by language inconsistency in speech evaluation. Summary of the Invention

[0005] To address the technical problems in the background art, this invention proposes a lightweight language detection method based on multi-dimensional feature fusion.

[0006] The technical solution of this invention is a lightweight language detection method based on multi-dimensional feature fusion, comprising the following steps: Step 1: Audio preprocessing and input: Obtain the original audio signal to be detected, remove silence segments by VAD speech activity detection, and retain valid speech; perform pre-emphasis, framing and windowing processing on the valid speech to obtain a standardized audio frame sequence.

[0007] Step 2: Multi-dimensional acoustic feature extraction, parallel extraction of three types of complementary features: 1) Spectral characteristics: Extract Mel frequency cepstral coefficients or Mel filter bank energy to characterize the short-time spectral envelope of speech and reflect the differences in timbre between languages; 2) Prosodic features: Extract the fundamental frequency trajectory, energy envelope and its first-order difference to represent speech rhythm, intonation and stress patterns; 3) Phoneme-related features: Input the audio frames into a lightweight general phoneme classifier to obtain the posterior probability vector of the phoneme category for each frame, which represents the pronunciation content attribute.

[0008] Step 3: Feature fusion and joint representation generation: A weighted strategy based on attention mechanism is adopted: a lightweight attention sub-network is trained for each dimension to dynamically generate frame weights; the features of each dimension are weighted and converged to obtain a global representation; the dimension-level weights are calculated and normalized, and the weighted concatenation is used to obtain a joint feature representation.

[0009] Step 4: Language Classification and Output. Input the joint features into the lightweight language classification network and output the confidence score of the audio belonging to each preset language.

[0010] Step 5: Language Determination and Result Output. Select the language with the highest confidence level as the detection result; if the detected language is inconsistent with the target language, set the evaluation score to 0; if they are consistent, retain the original score.

[0011] The second technical solution of the present invention is a language detection system based on multi-dimensional feature fusion, comprising: a preprocessing module, a multi-dimensional feature extraction module, a feature fusion module, a language classification module, and a judgment and scoring processing module; the system executes the aforementioned language detection method.

[0012] The third technical solution of the present invention is a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned language detection method.

[0013] Beneficial effects This invention achieves lightweight and efficient deployment: by integrating highly discriminative multi-dimensional acoustic features, a dedicated language detection model with simplified parameters and optimized structure is constructed, supporting real-time offline operation on terminals such as Windows, Linux, and Android.

[0014] This invention ensures the interpretability of decisions and the effectiveness of feedback: it achieves white-box decision-making through multi-dimensional feature contribution analysis, outputs language detection results, and provides a basis for judgment.

[0015] This invention implements compliance verification of evaluation scores: it sets the scores to 0 for evaluation results with inconsistent languages, thus maintaining the fairness of the evaluation.

[0016] To verify the performance of this invention in lightweight deployment, the model was deployed on terminal devices (including Windows and Android mobile devices) for testing. The test audio duration was 10 seconds, and the sampling rate was 16kHz.

[0017] Experimental results show that, without relying on cloud computing power, the processing latency of the model of this invention is about 30ms in the Windows environment and about 35ms in the Android device, both of which can achieve real-time response.

[0018] Compared to larger model solutions, the model parameters in this invention are approximately 0.5M, corresponding to a memory footprint of approximately 2MB. This allows it to run on memory-constrained devices.

[0019] Based on the test results on the test set, the model of this invention can achieve an accuracy of about 95% to 98% in the language prediction task. While ensuring high recognition performance, it achieves significant savings in computing resources and low latency response.

[0020] The above results demonstrate that the present invention achieves a good balance between lightweight design and performance, and can meet the application requirements for real-time offline deployment on the terminal side.

[0021] To verify the robustness of this invention in complex speech environments, various interference conditions were introduced into the test set, including background noise (signal-to-noise ratio of 5dB~15dB), audio compression distortion, and speech text with significant accent differences. The test language length was 10s, and the sampling rate was 16kHz.

[0022] Comparative experiments were conducted using a single model based solely on the spectrum and the method of this invention for evaluation. Under the aforementioned complex conditions, the language recognition accuracy of the traditional method decreased to 80%–88%; while the method of this invention maintained an accuracy of approximately 94%–96% under the same conditions, showing a significantly smaller performance degradation. The fundamental reason is that it can adaptively select more stable feature information for discrimination based on a dynamic weight adjustment mechanism, thus significantly improving the robustness and reliability of the system.

[0023] To verify the performance of the system in terms of scalability and maintainability, Japanese was added as a language to be tested on the basis of the existing language recognition system (which supports Chinese and English).

[0024] During the expansion process, only the phoneme feature extraction module and its corresponding dictionary are expanded, and the classifier output layer is adapted. The remaining modules (including the spectral feature extraction module, prosodic feature extraction module, and attention fusion module) remain unchanged, and there is no need to reconstruct the overall model structure.

[0025] Experimental results show that after the addition of new languages, the system can perform language identification normally. The recognition accuracy of the newly introduced languages ​​can reach about 90% to 95%, and the recognition performance of the original languages ​​(Chinese and English) remains basically stable, with accuracy fluctuations of less than 0.1%.

[0026] Furthermore, when optimizing and replacing the prosodic feature extraction algorithm, only the corresponding module needs to be updated; there is no need to adjust the structure of other modules or redesign the overall model. The updated system performance is further improved without significantly impacting other feature branches.

[0027] Compared to traditional end-to-end models, which typically require overall retraining or structural adjustments when adding new languages ​​or optimizing algorithms, this invention achieves independent upgrades and replacements of local modules through modular decoupling design.

[0028] This invention has excellent cross-platform adaptability and can effectively solve the problem of inflated scores caused by language misjudgment in speech evaluation (such as speaking language B when evaluating language A), and achieve accurate language consistency verification and correction of unreasonable scores.

[0029] Compared with existing language detection technologies, this invention has the following outstanding advantages: First, lightweight deployment. Unlike large model solutions that rely on massive parameters and cloud computing power, this invention constructs a streamlined dedicated model through multi-dimensional feature fusion and a lightweight attention mechanism. It can run in real time offline on terminal devices such as Windows / Linux / Android, balancing accuracy and efficiency.

[0030] Secondly, it exhibits strong robustness. Under complex conditions such as noisy environments, audio distortion, or accent variations, the system can dynamically evaluate the reliability of each feature dimension, automatically reduce the weight of the disturbed dimensions, and instead rely on other relatively robust dimensions (such as prosodic features), thereby maintaining stable and reliable recognition performance.

[0031] Third, it is easy to maintain and scalable. The system adopts a modular and decoupled design, with feature extraction, attention fusion, and classifiers operating independently. When a new language needs to be added or an algorithm for a certain dimension needs to be optimized, there is no need to reconstruct the entire system; only the corresponding sub-modules need to be adjusted or replaced, which greatly reduces iteration costs and maintenance difficulty.

[0032] In summary, this invention can rationally correct scoring anomalies caused by language misjudgment for different terminal platform operating environments, significantly improving the accuracy and reliability of the speech evaluation system. Attached Figure Description

[0033] Figure 1 is a flowchart of the language detection process of this invention. Detailed Implementation

[0034] The present invention will be further described below.

[0035] To verify this invention, we conducted verification on a self-built database, which covers different languages ​​and has accumulated several thousand hours of data. The specific technical details are described below with reference to the accompanying drawings, mainly including the following steps: Step 1: Data preprocessing and input.

[0036] First, VAD (Voice Activity Detection) is used to segment the silent audio, removing the silent data and retaining only the audio intervals containing valid speech. The silence-removed data undergoes preprocessing operations such as pre-emphasis, framing, and windowing to obtain a standardized audio frame sequence.

[0037] (1) in This represents the feature matrix of the entire audio segment. This represents the output of the audio features at time t.

[0038] Step 2: Multi-dimensional acoustic feature extraction.

[0039] 1) Spectral Feature Dimension The goal of spectral feature extraction is to obtain the short-time spectral envelope of speech to reflect the acoustic differences in timbre among different languages. The specific implementation process is as follows: First, a Short-Time Fourier Transform (SFTF) is performed on each frame of the preprocessed speech signal to convert the time-domain signal to the frequency domain, and the power spectrum of each frame is calculated. Then, the power spectrum is filtered through a Mel filter bank (Mel) to simulate the characteristics of human auditory perception, obtaining the Mel spectrum. Finally, the logarithm of the Mel spectrum is taken, followed by a Discrete Cosine Transform (DCT) to obtain the Mel frequency cepstral coefficients.

[0040] The essence of this process is to perform orthogonal decomposition and dimensionality reduction representation of the logarithmic spectrum envelope. Because different languages ​​have different pronunciation mechanisms, their formant distributions (i.e., the location and bandwidth of vocal tract resonant frequencies) also vary. This difference in formant structure ultimately manifests in the distribution pattern of MFCC coefficients, making MFCC an effective acoustic feature for language identification.

[0041] The relationship between formants and language: Different languages ​​have different vowel systems, and their corresponding formant frequencies (F1, F2, F3) form different distribution patterns in acoustic space. For example, there may be subtle differences in the formant positions of / i / in Chinese and / i / in French. These differences, after MFCC transformation, manifest as a systematic shift in coefficient values, providing a basis for language identification.

[0042] 2) Prosodic Features Dimension The goal of prosodic feature extraction is to characterize the structural properties of speech over time, reflecting the differences in intonation patterns, rhythmic organization, and stress distribution among different languages. Unlike spectral features, which focus on the instantaneous acoustic structure of the vocal tract, prosodic features emphasize the dynamic patterns of speech changes over time. The specific implementation process is as follows: First, the fundamental frequency (F0) is extracted from the preprocessed and framed speech signal. The fundamental frequency reflects the vocal cord vibration frequency and is a direct physical representation of speech pitch. Periodic analysis is used to identify spoken frames, and the pitch value of each frame is estimated within a reasonable range of human voice frequencies, forming a continuous fundamental frequency trajectory. The formula is as follows: (2) in This represents the fundamental tone period. This represents the fundamental frequency of the t-th frame (the t-th time frame).

[0043] Secondly, the short-time energy or logarithmic energy of each frame is calculated to characterize the loudness variation of speech, forming an energy envelope curve. Energy information reflects the stress positions and rhythmic intensity distribution in speech. The formula is as follows: (3) in This represents the short-time energy of frame t. This represents the nth sample value in the t-th frame.

[0044] Subsequently, to characterize the dynamic changes in prosody, first-order difference features are calculated for both the fundamental frequency sequence and the energy sequence, yielding the pitch change rate and energy change rate. This step describes the rise and fall of intonation and the strong and weak fluctuations of rhythm, enabling the model to capture intonation contours and the dynamic structure of stress. The formula is as follows: (4) in This represents the first-order dynamic feature of the t-th frame. This represents the feature value of the (t+k)th frame. This represents the feature value of frame tk.

[0045] Finally, the fundamental frequency, energy, and differential features are combined to form a frame-level prosodic feature vector.

[0046] (5) in Let represent the prosodic feature vector of frame t. This frame contains four features, of which ... Indicates the fundamental frequency. The first-order difference representing the fundamental frequency, Indicates short-term energy. The first-order difference represents the short-time energy.

[0047] 3) Phoneme-related feature degree The goal of phoneme-related feature extraction is to characterize the categorical attributes of speech at the level of pronunciation content, reflecting the differences in phonological systems and phoneme combination patterns among different languages. Unlike spectral features, which focus on the acoustic structure of the vocal tract, and prosodic features, which focus on temporal organization, this dimension directly models the "distribution of pronunciation categories" of speech. The specific implementation process is as follows: First, the preprocessed and framed speech features (such as MFCC or Mel filter bank energy) are input into a lightweight, general-purpose phoneme classification model. This model is typically built on a deep neural network structure and trained with a large amount of multilingual or cross-lingual speech data to enable it to uniformly model basic phoneme categories.

[0048] Secondly, for each frame of speech signal, the classification model outputs a posterior probability vector indicating whether it belongs to each basic phoneme category. Each dimension of this vector represents the confidence level that the current frame belongs to a certain phoneme category. This representation does not make a hard decision but preserves the complete probability distribution information, thus more finely characterizing the features of the pronunciation content.

[0049] Subsequently, all frame-level phoneme posterior probability vectors are arranged in chronological order to form a phoneme probability trajectory. In higher-level modeling stages, these posterior vectors can be further statistically summarized or temporally modeled to extract stable language-level representations.

[0050] (6) in, This represents the overall phoneme posterior probability matrix. Let T represent the probability distribution of the T-th frame.

[0051] Step 3: Feature fusion and joint representation generation.

[0052] First, for the three acoustic feature dimensions generated in step two, a lightweight attention sub-network is trained for each dimension to calculate the importance score of each time frame within that dimension: (7) (8) (9) in, The attention score represents the spectral features in frame t. This represents the attention score for prosodic features in frame t. This represents the attention score of phoneme features in frame t.

[0053] Next, the attention scores for each dimension are softmax normalized to obtain the weight vector for the time dimension: (10) （11） (12) in, , , Let represent the unnormalized attention scores for frame t on the spectral, prosodic, and phonemic feature branches, respectively. The corresponding attention weights can be obtained by normalizing these scores over time using the Softmax function. , and , is used to characterize the relative importance of each frame in different feature branches, and satisfies that the sum of weights is 1.

[0054] By using the aforementioned time weight vector to weight and converge the frame-level features of each dimension, a global representation for each dimension is obtained: (13) (14) (15) in, , and Frame-level representation matrices representing spectral features, prosodic features, and phonemic features, respectively. , and These represent the attention weights of the corresponding branches in the time dimension. By weighted aggregation of frame-level features, the global representations of each branch can be obtained. , and .

[0055] The three global representation vectors mentioned above are concatenated into a holistic feature: (16) Where g represents the splicing of global representations of spectral features, prosodic features, and phonemic features at time T.

[0056] Based on this, calculate the weight of each dimension in the overall features: (17) (18) (19) in (.) is the Sigmoid activation function. These are learnable parameters. , and This is the bias term. The weight coefficients of each feature branch can be obtained through this mapping. , and It is used to characterize the relative importance of different features in the overall representation.

[0057] Softmax normalization is applied to the above weights: (20) in, , and It is the result of normalization based on the weights of the different features obtained above, and the sum is 1.

[0058] Finally, the three global representations are weighted and concatenated using normalized dimensional weights to obtain a joint feature representation for language discrimination: (twenty one) in, , and These represent the global representation vectors for spectral features, prosodic features, and phonemic features, respectively. , and represents the normalized weight coefficients for the corresponding feature branches. h represents the joint feature representation.

[0059] Step 4: Language classification and output.

[0060] The joint feature representation generated in step three is input into the trained lightweight language classification network, and the final output is the confidence score of the audio signal belonging to each preset language.

[0061] Specifically, this lightweight classification network employs a multilayer perceptron structure, typically consisting of two fully connected layers. The first layer maps the fused high-dimensional features to the hidden layer, introduces non-linearity through the ReLU activation function, and employs Dropout regularization to prevent overfitting. The second layer maps the hidden layer features to an output vector representing a preset number of language categories. After normalizing the output vector using the Softmax function, the probability distribution for each language is obtained and used as the judgment criterion for the next step.

[0062] Step 5: Language determination and result output.

[0063] Based on the output of the previous step, select the language with the highest confidence score as the detection result, and then determine whether to set the final evaluation score to 0 based on the result.

[0064] Example 1: Cross-platform lightweight offline deployment Step 1: Platform Adaptation and Preprocessing Three typical terminals—Windows, Linux, and Android—were selected, and a unified workflow of VAD silence removal + 16kHz sampling rate + 25ms frame splitting + 10ms frame shift + Hamming window preprocessing was adopted to ensure consistent input feature dimensions across platforms. On Android devices, floating-point quantization and memory optimization were enabled to compress the model parameters to within 10MB, meeting offline operation requirements.

[0065] Step 2: Lightweight Extraction of Multi-Dimensional Features 1. Spectral characteristics: 13-dimensional MFCC + first-order difference is used to reduce the amount of computation.

[0066] 2. Prosodic features: Fixed extraction of fundamental frequency, energy, and first-order difference, totaling 4 dimensions.

[0067] 3. Phoneme features: A lightweight CNN phoneme classifier is used, with inference time of less than 5ms / frame.

[0068] Step 3: Lightweight Attention Fusion A single hidden layer attention subnetwork is used to perform frame-level weighted convergence of spectrum, prosody, and phonemes, and then perform dimension-level weighted splicing to ensure that the number of parameters in the fusion module is less than 500k.

[0069] Step 4: Lightweight Language Classification Using a two-layer MLP (64-dimensional hidden layer + output layer), real-time inference is achieved on various platforms (single audio clip <30ms).

[0070] Step 5: Output Results The detection results are consistent across platforms, with an accuracy deviation of <0.5%, achieving lightweight offline language detection across platforms.

[0071] Example 2: Robust Complex Scene Detection Step 1: Constructing complex test audio Prepare four types of test audio: noisy, distorted, accented, and low signal-to-noise ratio, with a signal-to-noise ratio range of 0dB to 15dB. Samples include Chinese-English, Japanese-English, noisy Chinese, and noisy English.

[0072] Step 2: Preprocessing and Feature Extraction VAD (Voice-Audio Digest) removes silence segments, preserving a longer effective speech range for low signal-to-noise ratio audio and avoiding feature loss. Parallel extraction of three types of features—spectral, prosodic, and phonemic—is performed, with prosodic features showing greater stability under noise.

[0073] Step 3: Dynamic Adjustment of Attention Weights The model automatically evaluates the credibility of each dimension: When noise is high, reduce the spectrum and phoneme weights; Increase the weight of prosodic features and rely on intonation, rhythm, and stress to identify the language.

[0074] Step 4: Language Classification and Determination Under a signal-to-noise ratio of 5dB, the language recognition accuracy remains above 90%, which is significantly better than the single feature model.

[0075] Step 5: Scoring Verification For samples with inconsistent languages, the score is set to 0 to avoid misjudgments and inflated scores caused by noise / accent.

[0076] Example 3: Scalable addition of new languages ​​and module iteration Step 1: System modular decoupling Preprocessing, feature extraction, feature fusion, and classifier remain independent and decoupled, without altering the overall architecture.

[0077] Step Two: Preparing Data for New Languages Collect speech data for newly added languages ​​(such as Japanese and Korean), and annotate and preprocess them according to a unified format.

[0078] Step 3: Incremental Training and Module Replacement 1. Phoneme classifier: incremental fine-tuning to adapt to the distribution of phonemes in newly added languages.

[0079] 2. Attention Fusion Module: Retain the original weights and only use the new language data for small-batch updates.

[0080] 3. Classification Network: Replace the output layer and expand the number of categories to include the original language plus the newly added languages.

[0081] Step 4: Verification and Deployment By iterating only submodules without refactoring the system, the accuracy rate of new language detection meets the standard, and maintenance costs are reduced by more than 60%.

[0082] Step 5: Verify Expansion Capabilities It supports the addition of less commonly spoken languages ​​and dialects in the future, and has the ability to continuously expand.

[0083] Example 4: Explainability and Scoring Compliance Verification Step 1: Interpretable Output of Feature Contribution During the feature fusion stage, the contribution weights of the three dimensions—spectrum, prosody, and phonemes—are output to form an interpretable report. For example, in Chinese character recognition, phonemes and prosody contribute 70%; in English character recognition, spectrum and prosody contribute 65%.

[0084] Step 2: Language Consistency Verification Set the target language for the evaluation to English, and input audio in three categories: Chinese, English, and mixed languages.

[0085] Step 3: Scoring Processing 1. If the test is in English: the original pronunciation score will be retained; 2. If the detection is in Chinese / mixed languages: the evaluation score will be forcibly set to 0, and a language inconsistency warning will be given.

[0086] Step 4: Application Results To eliminate the anomaly of "speaking Chinese and getting high scores in English" and improve the credibility of the evaluation system.

Claims

1. A language detection method based on multi-dimensional feature fusion, characterized in that, Includes the following steps: S1 Audio Preprocessing and Input: Acquire the raw audio signal, remove silence via VAD, and perform pre-emphasis, framing, and windowing to obtain a standardized audio frame sequence; S2 Multidimensional Acoustic Feature Extraction: Parallel extraction of spectral features, prosodic features, and phoneme-related features; S3 Feature Fusion and Joint Representation Generation: Employing intra-dimensional attention-weighted convergence and inter-dimensional attention-weighted concatenation, including: 1) Train a lightweight attention subnetwork for each feature dimension, which dynamically generates a weight vector based on the current input feature sequence; 2) Use the generated weight vector to perform weighted concatenation of features in each dimension, focusing on the feature dimension that is most effective for the current discrimination; S4 Language Classification and Output: Input the joint feature representation generated in step S3 into the lightweight language classification network, and output the confidence score of the audio signal belonging to each preset language; S5 Language Determination and Scoring: The language with the highest confidence level is selected as the result; if there is a discrepancy, the evaluation score is set to 0.

2. The method according to claim 1, characterized in that, The spectral features are Mel frequency cepstral coefficients or Mel filter bank energy. The Mel frequency cepstral coefficients or Mel filter bank energy are extracted to characterize the short-time spectral envelope of speech.

3. The method according to claim 1, characterized in that, Prosodic features include fundamental frequency trajectory, energy envelope and its first-order difference. Extracting the fundamental frequency trajectory, energy envelope and its first-order difference can characterize speech rhythm, intonation and stress patterns.

4. The method according to claim 1, characterized in that, Phoneme-related features are frame-level phoneme posterior probability vectors output by a lightweight phoneme classifier. Audio frames are input into a lightweight general phoneme classifier to obtain the posterior probability vectors of the phoneme categories corresponding to each frame, representing the pronunciation content attributes.

5. The method according to claim 1, characterized in that, The language classification network in step S4 is a multilayer perceptron structure.

6. The method according to claim 5, characterized in that, The structure consists of two fully connected layers: the first layer maps the fused high-dimensional features to the hidden layer, introduces non-linearity through the ReLU activation function, and is supplemented by Dropout regularization to prevent overfitting; the second layer maps the hidden layer features to the output vector of the preset number of language categories, and the output vector is normalized by the Softmax function to obtain the probability distribution of each language, which is used as the judgment condition for the next step.

7. A language detection system based on multi-dimensional feature fusion, characterized in that, include: The module includes a preprocessing module, a multi-dimensional feature extraction module, a feature fusion module, a language classification module, and a judgment and scoring module. The system performs the language detection method according to any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the language detection method as described in any one of claims 1 to 6.

Images