A Vietnamese grammar correction method based on large language model detection and phoneme enhancement

Abstract

The present invention discloses a Vietnamese grammar correction method based on large language model detection and phoneme enhancement, which belongs to the technical field of natural language processing. The present invention first uses the multilingual pre-training model mBART to build an end-to-end grammar correction basic model, and uses the powerful language representation ability of mBART to improve the correction effect; a phoneme encoding module is also designed to assist the correction model in learning the unique language characteristics of Vietnamese; further, the phoneme information containing Vietnamese features is extracted through the average pooling layer, and its dimensional consistency with the text information representation is ensured; then, the phoneme information is fused with the text information through the filtering gating mechanism, and the interference components in the phoneme information are eliminated to ensure the accuracy and reliability of the text information representation; finally, an error detection layer is added after the filtering gating mechanism to accurately identify errors, reduce the over-correction phenomenon of the model, and improve the accuracy and stability of the overall grammar correction.

Description

Technical Field

[0001] The invention relates to a Vietnamese grammar error correction method based on large language model detection and phoneme enhancement, and belongs to the technical field of natural language processing. Background Art

[0002] With the deepening of economic globalization, Vietnam, as an important economy in Southeast Asia, and its language, Vietnamese, are increasingly entering the field of international communication. The development of Vietnamese grammar correction technology is particularly critical. The core goal of Vietnamese grammar correction is to accurately identify and correct grammatical irregularities in texts, which may include: errors in word order, improper part of speech collocation, and sentence structure problems. The existence of these grammatical errors will seriously affect the clarity and readability of the text, and thus affect the effectiveness of communication. Therefore, in order to improve the accuracy and professionalism of Vietnamese texts, the demand for Vietnamese grammar correction technology is growing. The advancement of this technology not only helps to improve the Vietnamese writing ability of non-native speakers, but also has important significance for promoting the popularization and development of Vietnamese as an international communication language. With the continuous advancement and application of technology, Vietnamese grammar correction technology will become an important tool for improving the quality of language education and optimizing cross-cultural communication.

[0003] Vietnamese grammar correction technology is uniquely challenging in the field of natural language processing. Since the grammatical structure of Vietnamese is different from that of Chinese, its correction process needs to take into account the unique language characteristics of Vietnamese, such as tones, complex word ending changes, and unique syntactic rules.

[0004] The significance of Vietnamese grammar correction technology is not only reflected in improving text quality, but also in its irreplaceable role in optimizing user experience and meeting the needs of different application scenarios. As the core of ensuring text accuracy and professionalism, this technology effectively reduces misunderstandings and obstacles in communication and improves communication efficiency.

[0005] In summary, for non-native speakers, Vietnamese grammar correction technology is a powerful auxiliary tool that can help them improve their writing skills, thereby promoting the development of language learning and education. In other areas of natural language processing, such as machine translation and speech recognition, the improvement of grammar detection technology has also directly promoted the accuracy and practicality of these technologies. Summary of the invention

[0006] The present invention provides a Vietnamese grammar correction method based on large language model detection and phoneme enhancement, which is used to build a Vietnamese grammar correction model based on the mBART basic model architecture, and to add a phoneme encoding module, an average pooling layer, a filtering gating mechanism and a detection layer to solve the problem of low accuracy in Vietnamese grammar correction.

[0007] The technical solution of the present invention is:

[0008] A Vietnamese grammar error correction method based on large language model detection and phoneme enhancement, the specific steps of the method are as follows:

[0009] Step 1. Select the pre-trained model mBART as the basic model architecture;

[0010] Step 2, collect Vietnamese phonemes and build a Vietnamese phoneme dictionary;

[0011] Step 3: Based on the basic model architecture of Step 1, a phoneme encoding module is added to obtain the context phoneme information representation;

[0012] Step 4: After the phoneme encoding module, an average pooling layer and a filtering gating mechanism are introduced;

[0013] Step 5: By integrating a detection layer after the encoding end of the basic model and jointly optimizing and adjusting the parameters of this layer and the model, the model integrating Step 3 and Step 4 is fully trained and optimized to achieve the best performance indicators.

[0014] Furthermore, the specific steps of Step 1 are as follows:

[0015] Step 1.1. First, obtain the open-source and pre-trained multi-language mBART basic model. The mBART basic model adopts the encoding end to decoding end architecture design;

[0016] Step 1.2, download the relevant speech recognition data set and image recognition data set, and obtain the error correction parallel data set containing errors after speech recognition and image recognition; the error correction parallel data set will be used as a training set to train the mBART basic model;

[0017] Step 1.3: Conduct preliminary tests on the mBART basic model on a public Vietnamese error correction dataset to obtain the model’s baseline performance results.

[0018] Furthermore, the specific steps of Step 2 are as follows:

[0019] Step 2.1. Collect all possible phoneme representations P from online resources and then construct an initial basic phoneme dictionary P v1 ;

[0020] Step 2.2, construct the maximum matching algorithm, and split the Vietnamese syllables by implementing the maximum matching algorithm, based on the initial basic phoneme dictionary P v1In the process, all Vietnamese syllables in the training set are traversed and the maximum matching algorithm is used to construct them into phoneme representations P. Subsequently, the phoneme representations P are counted and any character not in the initial basic phoneme dictionary P is v1 The representation Pi in is added to the initial basic phoneme dictionary P v1 In order to expand and form the final basic phoneme dictionary P v As a Vietnamese phoneme dictionary;

[0021] Step 2.3: Final basic phoneme dictionary P v All the phoneme representations in the are sorted, ensuring that similar phonemes are arranged in close proximity according to their similarity.

[0022] Furthermore, the specific steps of Step 3 are as follows:

[0023] Step 3.1. Convert the input sentence S into the corresponding phoneme representation P through the maximum matching algorithm, and convert each phoneme in the input sentence into the corresponding phoneme id representation P through the Vietnamese phoneme dictionary. id ;

[0024] Step 3.2: Add a phoneme embedding layer to the mBART basic model and use the phoneme embedding layer to represent the sparse phoneme ids P id Mapping to dense phonetic vector representation ;

[0025] Step 3.3: Represent the dense phonetic vector obtained in Step 3.2 As the input of the bidirectional long short-term memory network, it is encoded by the bidirectional long short-term memory network to obtain the context phoneme information representation .

[0026] Furthermore, the specific steps of Step 4 are as follows:

[0027] Step 4.1, when obtaining context phoneme information representation After that, we first obtain the context phoneme information representation The average value in the second dimension is repeated until it matches the contextual text representation. Dimensionally consistent temporary hidden layer representation of phonemes ; Then the context phoneme temporary hidden layer representation is extracted through the average pooling layer A single dimension feature ; Then, by combining the text representation in the third dimension Temporal hidden layer representation of phonemes , by fusing phoneme information with text information and obtaining text representation with phoneme information ;

[0028] Step 4.2: Represent the text with phoneme information obtained in Step 4.1 Through filtering gating, useless information is filtered out to obtain the final text representation with phoneme information. ; By using the final text representation with phoneme information As the input of the mBART basic model decoding end.

[0029] Furthermore, the specific steps of Step 5 are as follows:

[0030] Step 5.1: Represent the context text output by the mBART basic model encoding end As the input of the bidirectional gated recurrent unit, the bidirectional gated recurrent unit captures the possible erroneous information between the text information and obtains the vector representation with detection information ;

[0031] Step 5.2. Vector representation with detection information After the classification layer, the dense vector representation is mapped into sparse detection information ;

[0032] Step 5.3. Try to use cross entropy to detect information Modeling, optimizing text encoder parameters by calculating the negative log-likelihood function.

[0033] The beneficial effects of the present invention are as follows: the present invention first uses the multilingual pre-training model mBART to build an end-to-end grammatical error correction basic model, and uses the powerful language representation ability of mBART to improve the error correction effect; on the above basis, the present invention further designs a dedicated phoneme encoding module to assist the error correction model in learning the unique language characteristics of Vietnamese; further, the phoneme information containing Vietnamese characteristics is extracted through the average pooling layer, and its dimensional consistency with the text information representation is ensured; then, the phoneme information is fused with the text information through a filtering gating mechanism, and the interference components in the phoneme information are eliminated to ensure the accuracy and reliability of the text information representation; finally, the present invention adds an error detection layer after the filtering gating mechanism, and reduces the over-correction phenomenon of the model by accurately identifying errors, thereby improving the accuracy and stability of the overall grammatical error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily without conflict.

[0036] Example 1: Figure 1 As shown, a Vietnamese grammar error correction method based on large language model detection and phoneme enhancement, the specific steps of the method are as follows:

[0037] Step 1. Select the pre-trained model mBART as the basic model architecture. It should be noted that after selecting the basic model, the public Vietnamese grammar correction dataset was used to test the basic model, so as to comprehensively evaluate the accuracy and robustness of the model.

[0038] Step 2, collect Vietnamese phonemes and build a Vietnamese phoneme dictionary. Specifically, comprehensively collect and organize all Vietnamese phonemes, and then build a Vietnamese phoneme dictionary that contains all phoneme information. This dictionary aims to provide accurate phoneme data support for subsequent models.

[0039] Step 3: Based on the basic model architecture of Step 1, a phoneme encoding module is added to obtain the representation of contextual phoneme information. This module is used to enable the model to effectively learn and understand Vietnamese phoneme information, thereby improving the overall performance of the model.

[0040] Step 4: After the phoneme encoding module, an average pooling layer and a filtering gating mechanism are introduced; the average pooling layer is designed to extract key features from the phoneme information and ensure its consistency with the text information. In order to optimize the accuracy of information processing, a filtering gating mechanism is added to effectively filter out harmful information that may exist after fusion and improve the overall performance of the model;

[0041] Step 5. The present invention integrates a detection layer after the text encoding stage, and jointly optimizes and adjusts the parameters of this layer and the model, thereby comprehensively training and optimizing the model integrating Step 3 and Step 4 to achieve the optimal performance indicators.

[0042] Furthermore, the specific steps of Step 1 are as follows:

[0043] Step 1.1. First, obtain the open-source and pre-trained multi-language mBART basic model. The mBART basic model adopts an encoding-end to decoding-end architecture design, which has excellent performance for grammatical error correction tasks.

[0044] Step 1.2, download relevant speech recognition data sets and image recognition data sets from the Internet, and obtain error correction parallel data sets after speech recognition and image recognition. The error correction parallel data sets will be used as training sets to train the mBART basic model;

[0045] Step 1.3, download the public Vietnamese error correction dataset VSEC, preprocess the original file, and obtain parallel sentence pairs of incorrect sentences and correct sentences; conduct preliminary tests on the basic model on the public Vietnamese error correction dataset VSEC to obtain the benchmark performance results of the model.

[0046] Furthermore, the specific steps of Step 2 are as follows:

[0047] Step 2.1. Collect all possible phoneme representations P from online resources and then construct an initial basic phoneme dictionary P v1 ;

[0048] Step 2.2, write Python code, construct the maximum matching algorithm, and split the Vietnamese syllables by implementing the maximum matching algorithm based on the initial basic phoneme dictionary P v1 In the process, all Vietnamese syllables in the training set are traversed and the maximum matching algorithm is used to construct them into phoneme representations P. Subsequently, the phoneme representations P are counted and any character not in the initial basic phoneme dictionary P is v1 The representation Pi in is added to the initial basic phoneme dictionary P v1 In order to expand and form the final basic phoneme dictionary P v As a Vietnamese phoneme dictionary;

[0049] Step 2.3: Final basic phoneme dictionary P v All the phoneme representations in the model are sorted to ensure that similar phonemes are arranged in adjacent positions according to their similarity. This step is intended to improve the model's ability to learn similar phonemes. By organizing the phoneme representations in this way, the model's recognition accuracy is enhanced when processing similar phonemes.

[0050] Furthermore, the specific steps of Step 3 are as follows:

[0051] Step 3.1. Convert the input sentence S into the corresponding phoneme representation P through the maximum matching algorithm, and convert each phoneme in the input sentence into the corresponding phoneme id representation P through the Vietnamese phoneme dictionary. id ;

[0052] Step 3.2: Add a phoneme embedding layer to the mBART basic model and use the phoneme embedding layer to represent the sparse phoneme ids P id Mapping to dense phonetic vector representation ;

[0053] Step 3.3: Represent the dense phonetic vector obtained in Step 3.2 As the input of the bidirectional long short-term memory network, it is encoded by the bidirectional long short-term memory network to obtain the context phoneme information representation .

[0054] Furthermore, the specific steps of Step 4 are as follows:

[0055] Step 4.1, when the context phoneme information of Step 3.3 is obtained After that, in order to ensure that the contextual phoneme information is represented Contextual text representation output by the mBART base model encoder The dimensions are consistent. First, we obtain the context phoneme information representation. The average value in the second dimension is repeated until it matches the contextual text representation. Dimensionally consistent temporary hidden layer representation of phonemes ; Then the context phoneme temporary hidden layer representation is extracted through the average pooling layer A single dimension feature ; Then, by combining the text representation in the third dimension Temporal hidden layer representation of phonemes , by fusing phoneme information with text information and obtaining text representation with a large amount of phoneme information ;

[0056] Step 4.2: Represent the text with phoneme information obtained in Step 4.1 Through filtering gating, useless information is filtered out to obtain the final text representation with phoneme information. ; Among them, the filtering gate is composed of a single dimension feature of the phoneme , as well as sigmoid function and linear layer; by using the final text representation with phoneme information As the decoding input of the mBART basic model, it improves the grammatical error correction model's ability to utilize Vietnamese phoneme information.

[0057] Furthermore, the specific steps of Step 5 are as follows:

[0058] Step 5.1: Represent the context text output by the mBART basic model encoding end As the input of the bidirectional gated recurrent unit, the bidirectional gated recurrent unit captures the possible erroneous information between the text information and obtains the vector representation with detection information ;

[0059] Step 5.2. Vector representation with detection information After the classification layer, the dense vector representation is mapped into sparse detection information ;

[0060] Step 5.3. Try to use cross entropy to detect information Modeling, optimizing text encoder parameters by calculating the negative log-likelihood function.

[0061] Example 2: Figure 1 As shown, a Vietnamese grammar error correction method based on large language model detection and phoneme enhancement, the specific steps of the method are as follows:

[0062] a1. Download the pre-trained multi-language mBART model, construct a dataset for the Vietnamese grammar correction task, and perform preprocessing, as follows:

[0063] 1) Download the multi-language mBART model.

[0064] 2) Download the speech recognition dataset and the image recognition dataset and generate a grammatical error correction parallel dataset with incorrect sentences through the speech recognition model and the image recognition model. The processing of the dataset mainly includes deleting some datasets that cannot be annotated. Finally, 563,134 grammatical error correction training data are formed as the training set, that is, the total training corpus.

[0065] Table 1 Number of grammatical error correction datasets

[0066] Dataset name Number of datasets / sentences Speech recognition model generation 256486 Image recognition model generation 306648 VSEC 9341 Total training corpus 563134

[0067] a2. First, collect all the pinyin representations and construct the final phoneme dictionary P V Then align the wrong sentences with the correct sentences, and construct the grammar detection task label by constructing the edit sequence corresponding to each word segmentation. Then use the BART model in fairseq as the basic model architecture, and selectively load the parameters of the pre-trained mBART model. Finally, add the phoneme encoding module, average pooling layer, filter gating mechanism and detection layer to the mBART model to build a Vietnamese grammar error correction model.

[0068] 1) Collect all phoneme representations and construct the final phoneme dictionary P V .

[0069] 2) In order to match the input and output of the detection task, the present invention attempts to use the edit sequence to construct the grammar detection task label.

[0070] Specifically, we first align the source sentence segmentation with the target sentence segmentation, then generate the edit tag by constructing the edit distance from the wrong sentence to the target sentence. Finally, we determine whether the current segmentation is wrong based on the edit operation in the edit tag.

[0071] 3) The adopted BART model has an encoder and decoder structure. The encoder is used to encode text representation, and the decoder converts the text representation into text. The encoder and decoder complement each other to ensure the grammatical and semantic accuracy of the output sentence.

[0072] 4) Add an additional phoneme encoding module to convert discrete phoneme IDs into dense phonetic vector representations Then, in order to better encode the phoneme information, the present invention uses a phoneme-private bidirectional long short-term memory network to encode the contextual phoneme information representation , the specific formula is as follows:

[0073] (1)

[0074] 5) After passing through the bidirectional long-short time memory network, the context phoneme information is represented After that, in order to ensure that the contextual phoneme information is represented Contextual text representation output by the mBART base model encoder The dimensions are consistent, and we get The average value in the second dimension is repeated until it matches the contextual text representation. The dimensions are consistent. And by using the average pooling layer to extract the single-dimensional features of the temporary hidden layer representation of the context phoneme , the specific formula is as follows:

[0075] (2)

[0076] in, yes The temporary hidden layer representation of the phoneme is obtained by averaging and repeating in the second dimension. Then, the temporary hidden layer representation of the phoneme is combined in the last dimension by concat With text representation .

[0077] (3)

[0078] in, is the weight matrix. Finally, through the single feature of the phoneme , construct filtering gating to filter harmful information:

[0079] (4)

[0080] in, , is the weight matrix; It represents element-by-element multiplication, and sigmoid is the activation function. The final prediction result is then obtained through the decoding end and softmax.

[0081] (5)

[0082] in, , is the weight matrix; is the previous true value of the current predicted value. During the training process, the objective function is to minimize the cross entropy loss function, the formula is as follows:

[0083] )(6)

[0084]

[0085] in, are trainable model parameters, Is with Etymological sentences, Is with The correct sentence of the word It is in words visible at time steps;

[0086] In the prediction phase, beam search decoding is used to maximize the conditional probability Find an optimal sequence ;

[0087] 6) This invention attempts to add a detection layer after the encoding end to assist in syntax error correction. And sets an independent loss function for it , the loss function is mainly calculated by cross entropy. Add together to get the total loss function , optimize the entire model parameters. The following is the formula describing the total loss function:

[0088] (7)

[0089] in, is the detection layer loss function, is the error correction loss function.

[0090] 7) In order to better evaluate the Vietnamese grammar correction model, the present invention uses Vietnamese grammar evaluation indicators based on syllables, and evaluates the tasks at the correction level and the tasks at the detection level. The evaluation indicators used for Vietnamese grammar correction are precision P, recall R and F1, and the calculation results are as follows:

[0091] (8)

[0092] (9)

[0093] (10)

[0094] in:

[0095] TP (True Positive): The syllables where the model corrects the wrong word.

[0096] FP (False Positive): The model changes the correct word into an incorrect syllable.

[0097] FN (False Negative): The model did not correct the incorrect syllable.

[0098] 8) Use the total training corpus as the training set to train the Vietnamese grammar correction model, adjust the hyperparameters, and obtain the detection model with the best performance.

[0099] a3. Construct 1545 parallel sentence pairs as a validation set, and obtain the best results by adjusting the performance of different hyperparameters on the validation set. Furthermore, the model of the present invention and other models are tested on the VSEC test set, and the test results are shown in Table 2. VSEC improves the Vietnamese grammar error correction performance by artificially constructing a large number of data sets. Although the training set constructed by the present invention has only 563K text pairs, it can be seen that the effect achieved by the present invention on the test set far exceeds the F1 score obtained by VSEC using 500K text pairs. Moreover, compared with the effect achieved by VSEC using 5M text pairs, the score of the model of the present invention is also highly improved, which proves the effectiveness of the method of the present invention.

[0100] Table 2 Test results of the model of the present invention and other models on the VSEC test set

[0101]

[0102] In order to verify the performance improvement of the phoneme coding module and the detection layer on Vietnamese, the present invention tested the performance of the detection layer and the phoneme coding module on the basis of training the basic model. The results are shown in Table 3. After adding the detection layer, the accuracy of the correction result is significantly improved, proving that the detection layer can alleviate the problem of over-correction. When the phoneme coding module is integrated at the same time, the F1 score is improved from 82.3 to 85.5 points. When the two parts are integrated at the same time, the F1 value is improved from 82.3 to 88.0. This proves the effectiveness of the method proposed by the present invention.

[0103] Table 3 Detection-level test results of the base model and the detection and phoneme-enhanced model on the VSEC test set

[0104]

[0105] In order to explore more effective filtering methods, the present invention has conducted the following filtering method tests: ordinary gating means that the obtained fusion information is not filtered by the average pooling layer and the sigmoid function; original gating means selecting the forget gate and the output gate as the fusion method; filtering gating is the method selected by the present invention. The results are shown in Table 4, and the filtering gating mechanism obtains the highest score. At the same time, it is also confirmed that the method proposed by the present invention can be used as a powerful method to improve the grammatical error correction performance.

[0106] Table 4 Test results of different filtering gating methods on VSEC dataset

[0107]

[0108] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A Vietnamese grammar error correction method based on large language model detection and phoneme enhancement, characterized in that: The specific steps of the method are as follows: Step 1. Select the pre-trained model mBART as the basic model architecture; Step 2, collect Vietnamese phonemes and build a Vietnamese phoneme dictionary; Step 3: Based on the basic model architecture of Step 1, a phoneme encoding module is added to obtain the context phoneme information representation; Step 4: After the phoneme encoding module, an average pooling layer and a filtering gating mechanism are introduced; Step 5: By integrating a detection layer after the encoding end of the basic model and jointly optimizing and adjusting the parameters of this layer and the model, the model integrating Step 3 and Step 4 is fully trained and optimized; The specific steps of Step 3 are as follows: Step 3.

1. Convert the input sentence S into the corresponding phoneme representation P through the maximum matching algorithm, and convert each phoneme in the input sentence into the corresponding phoneme id representation P through the Vietnamese phoneme dictionary. id ; Step 3.2: Add a phoneme embedding layer to the mBART basic model and use the phoneme embedding layer to represent the sparse phoneme ids P id Mapping to dense phonetic vector representation ; Step 3.3: Represent the dense phonetic vector obtained in Step 3.2 As the input of the bidirectional long short-term memory network, it is encoded by the bidirectional long short-term memory network to obtain the context phoneme information representation ; The specific steps of Step 4 are as follows: Step 4.1, when obtaining context phoneme information representation After that, we first obtain the context phoneme information representation The average value in the second dimension is repeated until it matches the contextual text representation. Dimensionally consistent temporary hidden layer representation of phonemes ; Then the context phoneme temporary hidden layer representation is extracted through the average pooling layer A single dimension feature ; Then, by combining the text representation in the third dimension Temporal hidden layer representation of phonemes , by fusing phoneme information with text information and obtaining text representation with phoneme information ; Step 4.2: Represent the text with phoneme information obtained in Step 4.1 Through filtering gating, useless information is filtered out to obtain the final text representation with phoneme information. ; By using the final text representation with phoneme information As the input of the mBART basic model decoding end; The specific steps of Step 5 are as follows: Step 5.1: Represent the context text output by the mBART basic model encoding end As the input of the bidirectional gated recurrent unit, the bidirectional gated recurrent unit captures the possible erroneous information between the text information and obtains the vector representation with detection information ; Step 5.

2. Vector representation with detection information After the classification layer, the dense vector representation is mapped into sparse detection information ; Step 5.

3. Try to use cross entropy to detect information Modeling, optimizing text encoder parameters by calculating the negative log-likelihood function.

2. The Vietnamese grammar error correction method based on large language model detection and phoneme enhancement according to claim 1 is characterized in that: The specific steps of Step 1 are as follows: Step 1.

1. First, obtain the open-source and pre-trained multi-language mBART basic model. The mBART basic model adopts the encoding end to decoding end architecture design; Step 1.2, download the relevant speech recognition data set and image recognition data set, and obtain the error correction parallel data set containing errors after speech recognition and image recognition; the error correction parallel data set will be used as a training set to train the mBART basic model; Step 1.3: Conduct preliminary tests on the mBART basic model on a public Vietnamese error correction dataset to obtain the model’s baseline performance results.

3. The Vietnamese grammar error correction method based on large language model detection and phoneme enhancement according to claim 1 is characterized in that: The specific steps of Step 2 are as follows: Step 2.

1. Collect all possible phoneme representations P from online resources and then construct an initial basic phoneme dictionary P v1 ; Step 2.2, construct the maximum matching algorithm, and split the Vietnamese syllables by implementing the maximum matching algorithm, based on the initial basic phoneme dictionary P v1 In the process of processing, all Vietnamese syllables in the training set are traversed, and the maximum matching algorithm is used to try to construct them into phoneme representation P; Then, statistics are taken on the phoneme representation P. For any phoneme representation not in the initial basic phoneme dictionary P v1 The representation Pi in is added to the initial basic phoneme dictionary P v1 In order to expand and form the final basic phoneme dictionary P v As a Vietnamese phoneme dictionary; Step 2.3: Final basic phoneme dictionary P v All the phoneme representations in the are sorted, ensuring that similar phonemes are arranged in close proximity according to their similarity.

Images