English pronunciation semantic recognition compensation method using BERT model

Abstract

The invention relates to an English pronunciation semantic recognition and compensation method using a BERT model, which improves the recognition and compensation capability of diversified English pronunciation by combining acoustic features and semantic features. The scheme comprises the steps of data preparation, text preprocessing, BERT model fine tuning, feature extraction, speech feature fusion, model training, model optimization, model iteration and the like. A BERT model is finely adjusted on a specific English pronunciation data set, rich semantic features are extracted and combined with traditional acoustic features such as MFCC, and a comprehensive feature vector is formed. The model adopts a deep learning structure and comprises a feature fusion layer, a double-branch convolution layer, a bidirectional LSTM layer, an attention mechanism and the like, so that the recognition accuracy is improved. In addition, through transfer learning and regularization technologies, the robustness of the model is enhanced. The scheme can be widely applied to the fields of speech recognition, automatic subtitle generation and the like, and has remarkable technical advantages and market application potential.

Description

Technical Field

[0001] The present invention relates to the field of speech analysis, and in particular to an English pronunciation semantic recognition compensation method using a BERT model. Background Art

[0002] Different countries and regions have their own unique English accents and dialects, which not only affect the accuracy of speech recognition, but also create obstacles for communication between language learners and professionals. Traditional speech recognition technology is often based on standard pronunciation training, which is difficult to adapt to diverse pronunciation characteristics, resulting in a decrease in recognition accuracy.

[0003] In recent years, the development of deep learning technology has provided new possibilities for solving this problem. In particular, the emergence of the BERT (Bidirectional Encoder Representations from Transformers) model, which captures the contextual information of the text through a bidirectional encoder, greatly improves the performance of natural language processing tasks. The pre-training and fine-tuning mechanism of the BERT model enables it to adapt to specific data sets and extract rich semantic features.

[0004] However, most existing technologies focus on the acoustic features of speech, while ignoring the role of semantic information in pronunciation recognition. In addition, how to effectively integrate acoustic and semantic features to improve the adaptability and robustness of the model to different accents and dialects is still an unresolved problem. This scheme proposes a semantic recognition compensation method for English pronunciation that combines the BERT model and acoustic features, aiming to improve the recognition and compensation capabilities of diverse English pronunciations through deep learning technology. Summary of the invention

[0005] In order to solve the above problems, the present invention provides an English pronunciation semantic recognition compensation method using a BERT model, comprising the following steps:

[0006] Step A: Data preparation: collect English pronunciation data from different countries and regions, including different accents and dialects, and transcribe the data into text. The data set also contains metadata, including the nationality, age and gender of the speaker.

[0007] Step B: Text preprocessing: cleaning the transcribed text data, removing irrelevant noise words and punctuation, and performing standardization, including conversion to lowercase, elimination of repeated words, replacement of abbreviations, processing of spelling errors, and word segmentation and stemming;

[0008] Step C: BERT model fine-tuning: Use the pre-trained BERT model as a feature extractor, fine-tune the BERT model on a specific English pronunciation dataset, freeze most of the model's weights, and only train the top-level weights;

[0009] Step D: feature extraction, using the fine-tuned BERT model to extract semantic features of speech data and traditional acoustic features, including Mel-frequency cepstral coefficients (MFCC);

[0010] Step E: Speech feature fusion, combining semantic features and acoustic features through a feature fusion layer to form a comprehensive feature vector;

[0011] Step F, model training, training a deep learning model to identify and compensate for English pronunciation differences using a comprehensive feature vector, and using a cross entropy loss function to minimize the difference between predicted and actual pronunciations;

[0012] Step G: Model optimization, using a transfer learning strategy to fine-tune the top layer of the BERT model, using regularization techniques such as dropout and weight decay to prevent overfitting;

[0013] Step H: Model iteration: Establish a feedback loop, use user feedback to iterate and improve the model, and regularly update the model to reflect new data and user feedback.

[0014] The text preprocessing step also includes part-of-speech tagging and dependency syntax analysis of text data;

[0015] Part-of-speech tagging: assigning a corresponding part-of-speech category to each word in the preprocessed text, including nouns, verbs, adjectives, adverbs, prepositions, pronouns, conjunctions, and interjections;

[0016] Dependency syntactic analysis determines the dependency relationship of each word in the text to construct a dependency syntactic tree of the sentence, where each node represents a word and each edge represents a dependency relationship, including subject-predicate relationship, verb-object relationship, attributive relationship, and adverbial relationship.

[0017] Step C is specifically as follows:

[0018] Select the bert-base-uncased model as the feature extractor and load it using the transformers library from Hugging Face.

[0019] Freeze the weights of the first few layers of the model by setting the weight attribute requires_grad of these layers to False;

[0020] A new fully connected layer is added on top of the BERT model. This layer will be trained specifically on a country-specific English dataset. The purpose of this layer is to learn the characteristics of country-specific English pronunciation and be able to classify or regress these characteristics to facilitate future pronunciation compensation.

[0021] The preprocessed text data is input into the BERT model and trained on the newly added top layer. The model is trained using the cross entropy loss function. During the training process, only the weights of the top layer are updated, while the rest of the BERT model remains unchanged.

[0022] In step D:

[0023] Input text data into the BERT model, and the model outputs the embedding representation of each word and phoneme;

[0024] The speech signal is framed and pre-emphasized, and then the fast Fourier transform (FFT) is applied to convert the time domain signal into a frequency domain signal; the spectrum is then filtered through a series of Mel filters. After filtering, the energy of each filter group is logarithmically transformed, and the discrete cosine transform (DCT) is applied to obtain the MFCC coefficients;

[0025] In step E:

[0026] The semantic features extracted by BERT and the extracted MFCC coefficient features are fused through a feature fusion layer to form a comprehensive feature vector combining semantic and acoustic information.

[0027] The specific feature fusion is:

[0028] Use the BERT model to extract the semantic features of text data and obtain the semantic feature vector S;

[0029] Extract the MFCC acoustic features of the speech data to obtain the acoustic feature vector A;

[0030] Normalize the semantic feature vector S and the acoustic feature vector A to ensure that they are at the same scale;

[0031] Assign weights w to semantic features and acoustic features S and w A ; Form a comprehensive feature vector F:

[0032] F=w S ·S norm +w A ·A norm ;

[0033]

[0034]

[0035] The structure of the deep learning model is:

[0036] Input layer: receives two inputs, one is the semantic feature vector processed by the BERT model, and the other is the extracted MFCC acoustic feature vector;

[0037] Feature fusion layer: concatenates the semantic feature vector and the acoustic feature vector to form a comprehensive feature vector;

[0038] Dual-branch convolutional layer: The semantic branch uses a series of convolutional layers, each followed by batch normalization and ReLU activation function to extract local features of semantic features; the acoustic branch uses convolution kernels of different sizes, including 1×3, 3×1, and 3×3, to capture different scale information of acoustic features;

[0039] Bidirectional LSTM layer: A bidirectional LSTM layer is applied on the output of the two-branch convolutional layer to capture dependencies on the time series;

[0040] Attention mechanism: Attention mechanism is applied to the output of the bidirectional LSTM layer, allowing the model to focus on key pronunciation features;

[0041] Fusion layer: The output of the attention mechanism layer is again feature-fused using a fully connected layer and ReLU activation function;

[0042] Depthwise separable convolutional layer: Apply depthwise separable convolutional layer on the fused features to further extract features and reduce model parameters;

[0043] Densely connected layer: Each layer of the densely connected layer is connected to all previous layers, transmitting more information and enhancing feature transmission;

[0044] Bottleneck layer: A bottleneck layer is added in the middle of the model. The number of neurons in this layer is much smaller than that in the previous and next layers to encourage the model to learn more abstract feature representations.

[0045] Global average pooling layer: A global average pooling layer is used before the fully connected layer to reduce the spatial size of the feature map while retaining important feature information;

[0046] Output layer: The number of neurons in the last fully connected layer is set according to the task requirements. If the task is to recognize multiple pronunciation features, the output layer has a corresponding number of neurons. The softmax activation function is used to output the probability distribution of each category.

[0047] A weighted cross entropy loss function is used to handle the class imbalance problem.

[0048] Step G is specifically as follows:

[0049] Fine-tune the model on a larger and more diverse dataset of English pronunciation that includes more accents and dialects, as well as different speaking contexts;

[0050] Choose to fine-tune the top layer of the BERT model, which is closer to the model's output and contains more task-related information. Unfreeze the weights of this layer and train it on a new dataset to adapt to a wider range of pronunciation features.

[0051] A dropout layer is introduced into the model to randomly shut down some neurons during training, forcing the model to learn more robust features.

[0052] The beneficial effects of the present invention are:

[0053] The beneficial effect of this solution lies in its innovative combination of the advanced technology in the field of deep learning and natural language processing - the BERT model - with acoustic feature extraction, forming a new method for compensating English pronunciation semantic recognition. This method can not only capture the deep semantic information of the speech signal by fine-tuning the pre-trained BERT model on a specific English pronunciation dataset, but also adapt to the characteristics of different accents and dialects, greatly improving the accuracy and robustness of the speech recognition system. During the model training process, by introducing regularization techniques such as dropout and weight decay, and adopting a transfer learning strategy, the model can focus more on learning those features that are more critical and common for pronunciation recognition, effectively avoiding overfitting and enhancing the stability and reliability of the model under diverse pronunciation conditions.

[0054] In addition, this solution uses advanced deep learning technologies in the model structure design, such as feature fusion layer, dual-branch convolution layer, bidirectional LSTM layer and attention mechanism. These structures can not only fully extract and utilize the information in semantic features and acoustic features, but also capture the time series dependencies and key pronunciation features in speech signals, further improving the performance of the model. The introduction of deep separable convolution layer and bottleneck layer not only improves the efficiency of feature extraction, but also reduces the number of model parameters, making the model more lightweight and easy to deploy in practical applications.

[0055] More importantly, this solution has established a mechanism for continuous iteration and optimization. By collecting user feedback and regularly updating the model, the system can continuously adapt to new data and environmental changes, maintaining its long-term adaptability and advancement. This ability to continuously evolve makes this solution not only highly practical at present, but also has continuous development potential in the future.

[0056] In general, the beneficial effects of this solution are reflected in many aspects: improving the accuracy of speech recognition, enhancing the robustness and generalization ability of the model, optimizing the model structure, improving the efficiency of feature extraction, and establishing a continuous iteration mechanism. These effects not only make this solution innovative in technology, but also have wide applicability and huge market potential in practical applications. It can provide users in different countries and regions with more accurate and efficient speech recognition services, promote cross-cultural exchanges and cooperation, and promote the development of the globalization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Attached Figure 1 This is a flow chart of the English pronunciation semantic recognition compensation method using the BERT model in the present invention. DETAILED DESCRIPTION

[0059] Embodiment 1:

[0060] See also Figure 1 The present invention provides an English pronunciation semantic recognition compensation method using a BERT model, comprising the following steps:

[0061] Step A: Data preparation: collect English pronunciation data from different countries and regions, including different accents and dialects, and transcribe the data into text. The data set also contains metadata, including the nationality, age and gender of the speaker.

[0062] Step B: Text preprocessing: cleaning the transcribed text data, removing irrelevant noise words and punctuation, and performing standardization, including conversion to lowercase, elimination of repeated words, replacement of abbreviations, processing of spelling errors, and word segmentation and stemming;

[0063] Step C: BERT model fine-tuning: Use the pre-trained BERT model as a feature extractor, fine-tune the BERT model on a specific English pronunciation dataset, freeze most of the model's weights, and only train the top-level weights;

[0064] Step D: feature extraction, using the fine-tuned BERT model to extract semantic features of speech data and traditional acoustic features, including Mel-frequency cepstral coefficients (MFCC);

[0065] Step E: Speech feature fusion, combining semantic features and acoustic features through a feature fusion layer to form a comprehensive feature vector;

[0066] Step F, model training, training a deep learning model to identify and compensate for English pronunciation differences using a comprehensive feature vector, and using a cross entropy loss function to minimize the difference between predicted and actual pronunciations;

[0067] Step G: Model optimization, using a transfer learning strategy to fine-tune the top layer of the BERT model, using regularization techniques such as dropout and weight decay to prevent overfitting;

[0068] Step H: Model iteration: Establish a feedback loop, use user feedback to iterate and improve the model, and regularly update the model to reflect new data and user feedback.

[0069] The text preprocessing step also includes part-of-speech tagging and dependency syntax analysis of text data;

[0070] Part-of-speech tagging: assigning a corresponding part-of-speech category to each word in the preprocessed text, including nouns, verbs, adjectives, adverbs, prepositions, pronouns, conjunctions, and interjections;

[0071] Dependency syntactic analysis determines the dependency relationship of each word in the text to construct a dependency syntactic tree of the sentence, where each node represents a word and each edge represents a dependency relationship, including subject-predicate relationship, verb-object relationship, attributive relationship, and adverbial relationship.

[0072] Step C is specifically as follows:

[0073] Select the bert-base-uncased model as the feature extractor and load it using the transformers library from Hugging Face.

[0074] Freeze the weights of the first few layers of the model by setting the weight attribute requires_grad of these layers to False;

[0075] A new fully connected layer is added on top of the BERT model. This layer will be trained specifically on a country-specific English dataset. The purpose of this layer is to learn the characteristics of country-specific English pronunciation and be able to classify or regress these characteristics to facilitate future pronunciation compensation.

[0076] The preprocessed text data is input into the BERT model and trained on the newly added top layer. The model is trained using the cross entropy loss function. During the training process, only the weights of the top layer are updated, while the rest of the BERT model remains unchanged.

[0077] In step D:

[0078] Input text data into the BERT model, and the model outputs the embedding representation of each word and phoneme;

[0079] The speech signal is framed and pre-emphasized, and then the fast Fourier transform (FFT) is applied to convert the time domain signal into a frequency domain signal; the spectrum is then filtered through a series of Mel filters. After filtering, the energy of each filter group is logarithmically transformed, and the discrete cosine transform (DCT) is applied to obtain the MFCC coefficients;

[0080] In step E:

[0081] The semantic features extracted by BERT and the extracted MFCC coefficient features are fused through a feature fusion layer to form a comprehensive feature vector combining semantic and acoustic information.

[0082] The specific feature fusion is:

[0083] Use the BERT model to extract the semantic features of text data and obtain the semantic feature vector S;

[0084] Extract the MFCC acoustic features of the speech data to obtain the acoustic feature vector A;

[0085] Normalize the semantic feature vector S and the acoustic feature vector A to ensure that they are at the same scale;

[0086] Assign weights w to semantic features and acoustic features S and w A ; Form a comprehensive feature vector F:

[0087] F=w S ·S norm +w A ·A norm ;

[0088]

[0089] The structure of the deep learning model is:

[0090] Input layer: receives two inputs, one is the semantic feature vector processed by the BERT model, and the other is the extracted MFCC acoustic feature vector;

[0091] Feature fusion layer: concatenates the semantic feature vector and the acoustic feature vector to form a comprehensive feature vector;

[0092] Dual-branch convolutional layer: The semantic branch uses a series of convolutional layers, each followed by batch normalization and ReLU activation function to extract local features of semantic features; the acoustic branch uses convolution kernels of different sizes, including 1×3, 3×1, and 3×3, to capture different scale information of acoustic features;

[0093] Bidirectional LSTM layer: A bidirectional LSTM layer is applied on the output of the two-branch convolutional layer to capture dependencies on the time series;

[0094] Attention mechanism: Attention mechanism is applied to the output of the bidirectional LSTM layer, allowing the model to focus on key pronunciation features;

[0095] Fusion layer: The output of the attention mechanism layer is again feature-fused using a fully connected layer and ReLU activation function;

[0096] Depthwise separable convolutional layer: Apply depthwise separable convolutional layer on the fused features to further extract features and reduce model parameters;

[0097] Densely connected layer: Each layer of the densely connected layer is connected to all previous layers, transmitting more information and enhancing feature transmission;

[0098] Bottleneck layer: A bottleneck layer is added in the middle of the model. The number of neurons in this layer is much smaller than that in the previous and next layers to encourage the model to learn more abstract feature representations.

[0099] Global average pooling layer: A global average pooling layer is used before the fully connected layer to reduce the spatial size of the feature map while retaining important feature information;

[0100] Output layer: The number of neurons in the last fully connected layer is set according to the task requirements. If the task is to recognize multiple pronunciation features, the output layer has a corresponding number of neurons. The softmax activation function is used to output the probability distribution of each category.

[0101] A weighted cross entropy loss function is used to handle the class imbalance problem.

[0102] Step G is specifically as follows:

[0103] Fine-tune the model on a larger and more diverse dataset of English pronunciation that includes more accents and dialects, as well as different speaking contexts;

[0104] Choose to fine-tune the top layer of the BERT model, which is closer to the model's output and contains more task-related information. Unfreeze the weights of this layer and train it on a new dataset to adapt to a wider range of pronunciation features.

[0105] Introduce a dropout layer in the model, randomly turn off some neurons during training, and force the model to learn more robust features.

[0106] Example 2:

[0107] Step A: Data Preparation

[0108] Collect English pronunciation data from different countries and regions such as the United States, the United Kingdom, India, Australia, and South Africa.

[0109] The data includes English with different accents and dialects, such as Southern American accent, London accent in the UK, Indian English, Australian English, and South African English.

[0110] The dataset contains metadata recording the nationality, age, and gender of the speakers. For example: Alice, 34 years old, female, from the United States; Bob, 42 years old, male, from India.

[0111] The dataset contains 1000 hours of audio recordings and corresponding text transcripts.

[0112] Step B: Text Preprocessing

[0113] Clean the text data, remove irrelevant noise words and punctuation marks, such as filler words like "um", "ah", etc.

[0114] Perform normalization, convert all text to lowercase, eliminate duplicate words, replace abbreviations such as "can't" with "cannot", and handle spelling mistakes.

[0115] Tokenization and stemming, for example, extract "running" to "run".

[0116] Perform part-of-speech tagging and dependency syntactic analysis on the text data. For example, tag "run" as a verb and analyze the subject-predicate relationship in "Alice is running".

[0117] Step C: BERT Model Fine-tuning

[0118] Select the bert-base-uncased model as the feature extractor and use the transformers library of Hugging Face to load it.

[0119] Freeze the weights of the first few layers of the model by setting the requires_grad attribute of these layer weights to False.

[0120] Add a new fully connected layer on top of the BERT model, and this layer will be specifically trained for the English dataset of a specific country.

[0121] Step D: Feature extraction

[0122] Text data is input into the BERT model, and the model outputs embedding representations for each word and phoneme.

[0123] The speech signal is framed and pre-emphasized, and then the fast Fourier transform (FFT) is applied to convert the time domain signal into a frequency domain signal.

[0124] The spectrum is filtered through a series of Mel filters. After filtering, the energy of each filter bank is logarithmically transformed and discrete cosine transform (DCT) is applied to obtain MFCC coefficients.

[0125] Step E: Speech feature fusion

[0126] Use the BERT model to extract the semantic features of text data and obtain the semantic feature vector S.

[0127] Extract the MFCC acoustic features of the speech data to obtain the acoustic feature vector A.

[0128] The semantic feature vector S and the acoustic feature vector A are normalized to ensure that they are at the same scale.

[0129] Assign weights w to semantic features and acoustic features S and w A , forming a comprehensive feature vector F: F = w S ·S norm +w A ·A norm .

[0130] Step F: Model training

[0131] Train a deep learning model to recognize and compensate for English pronunciation differences using comprehensive feature vectors.

[0132] The cross entropy loss function is used to minimize the difference between the predicted and actual pronunciation.

[0133] Step G: Model optimization

[0134] Fine-tune the model on a larger and more diverse dataset of English pronunciation that includes more accents and dialects, as well as different speaking contexts.

[0135] We choose to fine-tune the top layer of the BERT model, which is closer to the output of the model and contains more task-related information. We unfreeze the weights of this layer and train it on a new dataset to adapt to a wider range of pronunciation features.

[0136] A dropout layer is introduced into the model to randomly shut down some neurons during training, forcing the model to learn more robust features.

[0137] Step H: Model iteration

[0138] Establish a feedback loop to iterate and improve the model using user feedback.

[0139] Update the model regularly to reflect new data and user feedback, for example, every three months based on user feedback and newly collected data.

[0140] Thus far, the description of the above-described embodiments is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present disclosure. The individual elements or features of a specific embodiment are generally not subject to the limitations of the specific embodiment, but when applicable, they can be interchanged and used for selected embodiments even if not specifically shown or described. In many respects, the same element or feature also can be changed. This variation is not considered to be a departure from the present disclosure, and all such modifications are intended to be included in the scope of the present disclosure.

[0141] Example embodiments are provided so that the present disclosure will be thorough and the scope will be fully conveyed to those skilled in the art. In order to thoroughly understand the embodiments of the present disclosure, numerous details are set forth, such as examples of specific parts, devices, and methods. Obviously, for those skilled in the art, it is not necessary to use specific details, the example embodiments can be implemented in many different forms, and neither should be construed as limiting the scope of the present disclosure. In certain example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

[0142] Here, specialized vocabulary is used only for the purpose of describing specific example embodiments, and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a" and "the" used herein may be intended to include plural forms as well. The terms "including" and "having" are inclusive, and therefore specify the presence of the claimed features, wholes, steps, operations, elements and / or components, but do not exclude the presence or additional presence of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof. Unless the order of execution is clearly indicated, the method steps, processing and operations described herein are not interpreted as necessarily needing to be performed in the specific order discussed and shown. It should also be understood that additional or optional steps may be adopted.

Claims

1. A method for compensating English pronunciation semantic recognition using a BERT model, characterized in that The steps include: Step A: Data preparation: collect English pronunciation data from different countries and regions, including different accents and dialects, and transcribe the data into text. The data set also contains metadata, including the nationality, age and gender of the speaker. Step B: Text preprocessing: cleaning the transcribed text data, removing irrelevant noise words and punctuation, and performing standardization, including conversion to lowercase, elimination of repeated words, replacement of abbreviations, processing of spelling errors, and word segmentation and stemming; Step C: BERT model fine-tuning: Use the pre-trained BERT model as a feature extractor, fine-tune the BERT model on a specific English pronunciation dataset, freeze most of the model's weights, and only train the top-level weights; Step D: feature extraction, using the fine-tuned BERT model to extract semantic features of speech data and traditional acoustic features, including Mel-frequency cepstral coefficients (MFCC); Step E: Speech feature fusion, combining semantic features and acoustic features through a feature fusion layer to form a comprehensive feature vector; Step F, model training, training a deep learning model to identify and compensate for English pronunciation differences using a comprehensive feature vector, and using a cross entropy loss function to minimize the difference between predicted and actual pronunciations; Step G: Model optimization, using a transfer learning strategy to fine-tune the top layer of the BERT model, using regularization techniques such as dropout and weight decay to prevent overfitting; Step H: Model iteration: Establish a feedback loop, use user feedback to iterate and improve the model, and regularly update the model to reflect new data and user feedback.

2. The English pronunciation semantic recognition compensation method using the BERT model according to claim 1, characterized in that: The text preprocessing step also includes part-of-speech tagging and dependency syntax analysis of text data; Part-of-speech tagging: assigning a corresponding part-of-speech category to each word in the preprocessed text, including nouns, verbs, adjectives, adverbs, prepositions, pronouns, conjunctions, and interjections; Dependency syntactic analysis determines the dependency relationship of each word in the text to construct a dependency syntactic tree of the sentence, where each node represents a word and each edge represents a dependency relationship, including subject-predicate relationship, verb-object relationship, attributive relationship, and adverbial relationship.

3. The English pronunciation semantic recognition compensation method using the BERT model according to claim 1, characterized in that: Step C is specifically as follows: Select the bert-base-uncased model as the feature extractor and load it using the transformers library from Hugging Face. Freeze the weights of the first few layers of the model by setting the weight attribute requires_grad of these layers to False; A new fully connected layer is added on top of the BERT model. This layer will be trained specifically on a country-specific English dataset. The purpose of this layer is to learn the characteristics of country-specific English pronunciation and be able to classify or regress these characteristics to facilitate future pronunciation compensation. Feed the preprocessed text data into the BERT model and train it on the newly added top layer; The model is trained using the cross entropy loss function. During the training process, only the weights of the top layer are updated, while the rest of the BERT model remains unchanged.

4. The English pronunciation semantic recognition compensation method using the BERT model according to claim 3, characterized in that: In step D: The text data is input into the BERT model, and the model outputs the embedding representation of each word and phoneme; the speech signal is framed and pre-emphasized, and then the fast Fourier transform (FFT) is applied to convert the time domain signal into a frequency domain signal; the spectrum is then filtered through a series of Mel filters. After filtering, the energy of each filter group is logarithmically transformed, and the discrete cosine transform (DCT) is applied to obtain the MFCC coefficients; In step E: The semantic features extracted by BERT and the extracted MFCC coefficient features are fused through a feature fusion layer to form a comprehensive feature vector combining semantic and acoustic information.

5. The English pronunciation semantic recognition compensation method using the BERT model according to claim 4, characterized in that: The specific feature fusion is: Use the BERT model to extract the semantic features of text data and obtain the semantic feature vector S; Extract the MFCC acoustic features of the speech data to obtain the acoustic feature vector A; Normalize the semantic feature vector S and the acoustic feature vector A to ensure that they are at the same scale; Assign weights w to semantic features and acoustic features S and w A ; Form a comprehensive feature vector F: F=w S ·S norm +w A ·A norm ; 6. The English pronunciation semantic recognition compensation method using the BERT model according to claim 5, characterized in that: The structure of the deep learning model is: Input layer: receives two inputs, one is the semantic feature vector processed by the BERT model, and the other is the extracted MFCC acoustic feature vector; Feature fusion layer: concatenates the semantic feature vector and the acoustic feature vector to form a comprehensive feature vector; Two-branch convolutional layer: The semantic branch uses a series of convolutional layers, each followed by batch normalization and ReLU activation function to extract local features of semantic features; The acoustic branch uses convolution kernels of different sizes, including 1×3, 3×1, and 3×3, to capture different scale information of acoustic features. Bidirectional LSTM layer: A bidirectional LSTM layer is applied on the output of the two-branch convolution layer to capture dependencies on the time series. Attention mechanism: Attention mechanism is applied to the output of the bidirectional LSTM layer, allowing the model to focus on key pronunciation features; Fusion layer: The output of the attention mechanism layer is again feature-fused using a fully connected layer and ReLU activation function; Depthwise separable convolutional layer: Apply depthwise separable convolutional layer on the fused features to further extract features and reduce model parameters; Densely connected layer: Each layer of the densely connected layer is connected to all previous layers, transmitting more information and enhancing feature transmission; Bottleneck layer: A bottleneck layer is added in the middle of the model. The number of neurons in this layer is much smaller than that in the previous and next layers to encourage the model to learn more abstract feature representations. Global average pooling layer: A global average pooling layer is used before the fully connected layer to reduce the spatial size of the feature map while retaining important feature information; Output layer: The number of neurons in the last fully connected layer is set according to the task requirements. If the task is to recognize multiple pronunciation features, the output layer has a corresponding number of neurons. The softmax activation function is used to output the probability distribution of each category. A weighted cross entropy loss function is used to handle the class imbalance problem.

7. The English pronunciation semantic recognition compensation method using the BERT model according to claim 6, characterized in that: Step G is specifically as follows: Fine-tune the model on a larger and more diverse dataset of English pronunciation that includes more accents and dialects, as well as different speaking contexts; Choose to fine-tune the top layer of the BERT model, which is closer to the model's output and contains more task-related information. Unfreeze the weights of this layer and train it on a new dataset to adapt to a wider range of pronunciation features. A dropout layer is introduced into the model to randomly shut down some neurons during training, forcing the model to learn more robust features.

Images