Regularization method for non-autoregressive decoding of Vietnamese text based on edit alignment algorithm

Abstract

This paper provides a non-autoregressive decoding Vietnamese text regularization method based on an edit alignment algorithm. Vietnamese text regularization faces challenges due to the diversity of linguistic expressions and the semantic ambiguity of non-standard words. This method constructs a regularized annotated corpus of Vietnamese text and expands it with character-level alignment. In the first stage, the annotated corpus is used to pre-train an indicator. The edit alignment algorithm is used to train the edit alignment vectors in the indicator from the original sequence to the target sequence, simultaneously improving the model's ability to model contextual semantic information on the encoder side. In the second stage, a non-autoregressive architecture is used to fine-tune the pre-trained model. By modifying the position vectors, the adjusted sequences of non-standard words are extracted, narrowing the search space on the decoder side and avoiding the non-regularized text prediction errors caused by the limitations of the sequence generation model. Experiments have demonstrated that this method achieves 97% accuracy, effectively addressing the semantic disambiguation problem in Vietnamese text regularization.

Description

Technical Field

[0001] The present invention relates to a non-autoregressive decoding Vietnamese text regularization method based on an edit alignment algorithm, and belongs to the technical field of natural language processing. Background Art

[0002] Currently, in previous text-to-speech (TTS) text regularization work, the main method used is a rule-based hard-coding method. The encoding method in MITalk is one of the earliest TTS text regularization modules. Bell Labs' multilingual TTS system introduced a weighted finite state transducer for text regularization. In addition, machine learning models combined with handwritten grammars have been proposed for specific NSW categories. Roark et al. reported the use of maximum entropy to classify letter sequences, and Sproat et al. proposed a method for text regularization based on identified abbreviation expansions.

[0003] Inspired by the recent success of neural networks in various natural language processing tasks, many neural network models have been proposed to solve the problem of TTS text regularization. Sproat et al. proposed an RNN-based architecture for text regularization, as well as an open source corpus of corresponding written and spoken sentence pairs. Subsequent research focused on convolutional neural network models. Another approach is to regard TTS text regularization as a machine translation problem. Tyagi et al. proposed a hybrid system consisting of a tagging model and tokenization mechanism that enables the system to automatically learn most categories and combine them with pre-encoded language knowledge categories for NSW expansion. Lai et al. proposed a unified framework for building a single bidirectional system that can handle TTS text regularization and reverse TTS text regularization.

[0004] In the Vietnamese TTS text regularization task, Trang et al. first developed a classification system for 21 types of Vietnamese non-standard words and proposed a decision tree classifier and an alphabetic language model for non-standard word classification, as well as a hybrid normalizer for non-standard word expansion. Trang et al. proposed a method that uses a random forest classifier to identify non-standard words and combines it with a sequence-to-sequence model and rule-based abbreviation and loanword expansion. In 2012, a rule-based method used regular expressions and decision lists to classify and expand non-standard words.

[0005] Hochreiter et al. proposed the Long Short-Term Memory (LSTM) network, which optimizes the problems of RNN networks in sequence labeling tasks and offers significant advantages. In 2019, Park et al. used the BERT pre-trained language model to complete the sequence labeling task of predicting Chinese polyphones, achieving good results.

[0006] Recently, Dang et al. combined BERT, BiLSTM, and CNN to propose a neural network model with 19 non-standard word categories. While the model demonstrated promising results on sequence classification tasks, it still required the construction of a rule-based matching algorithm for Vietnamese corpus conversion. The construction of rules is often complex and extensive, requiring significant time for subsequent updating and maintenance, and is difficult to port and adapt to other languages. Recently, Ro et al. proposed a Transformer-based text regularization framework, comprising preprocessing, encoder, decoder, and postprocessing steps. This method offers superior performance compared to baseline and other existing methods and can be trained end-to-end without the need for additional feature engineering. Therefore, this paper aims to develop a more versatile and scalable text regularization method. Drawing on current related work, this paper summarizes the main difficulties in Vietnamese text regularization, including the following: First, non-standard words and symbols in Vietnamese are ambiguous. As shown in Table 1, different symbols require contextual relationships to accurately translate into readable words; second, end-to-end neural network models can easily convert words that do not require regularization into incorrect words. To address these two problems, the present invention proposes a non-autoregressive decoding text regularization method based on edit alignment algorithm.

[0007] Table 1 shows some difficulties in regularizing Vietnamese text

[0008] Summary of the Invention

[0009] The present invention provides a non-autoregressive decoding Vietnamese text regularization method based on an edit alignment algorithm, which is used to solve the problem of ambiguity between non-standard words and symbols in Vietnamese.

[0010] The technical solution of the present invention is: the specific steps of the method are:

[0011] Step 1: Preprocessing: We crawled text corpora from various Vietnamese news and media websites, modified the text corpora, and used annotation to expand the Vietnamese dataset.

[0012] The specific steps of Step 1 are as follows:

[0013] Step 1.1. The present invention crawls a large amount of text corpus from various major Vietnamese news and media websites. In the preprocessing module, the crawled Vietnamese original text is deleted from redundant space characters, ASCII codes, emoticons, HTML entities, and unreadable words;

[0014] Step 1.2: Segment the large paragraph corpus. Specifically, a regular expression-based tokenizer is used to segment long sentences into sentences with a length of 20-150 characters based on punctuation marks. All the resulting text corpus is then corrected to make it more standardized and reasonable.

[0015] Step 1.3. Finally, the processed text is embedded into the vector space as an unlabeled dataset, and the obtained dataset is labeled in combination with the text regularization characteristics of Vietnamese.

[0016] Tyagi et al. (2020) proposed a text regularization task under scarce resources. The main problem they solved was how to deal with the text regularization task under limited (small-scale) data. The proposed method Proteno treats the text regularization task as a sequence labeling task, separately labels and classifies the character sequences, and obtains a high accuracy rate on less than 30,000 training data.

[0017] The present invention combines the idea of completing English text regularization under low-resource conditions by Tyagi et al., and adopts the method of annotating and expanding data under low-resource conditions to complete the Vietnamese text regularization task.

[0018] The Vietnamese language structure is based on words, each of which is composed of multiple characters. Vietnamese also has a unique linguistic phenomenon of postposition of adjectives. This paper proposes a new annotation method based on the text regularization characteristics of Vietnamese.

[0019] The non-standard word vectors of the original text (i.e., the text sequence that needs regularization) and the target text (i.e., the text sequence after regularization) are annotated. The annotation specifications for the regularized Vietnamese text are as follows: characters that need regularization in the text sequence are marked with '_', and a single word vector is marked each time, that is, the data is marked using the word-granular Vietnamese text regularization label segmentation method; if a sentence contains multiple words that need regularization, it is expanded into multiple sequences according to the number of non-standard words and annotated separately.

[0020] Step 2, Pre-training: Improve the matching algorithm based on edit alignment to detect and obtain the edit distance of non-standard words in the original sequence. At the same time, use the expanded annotated data to pre-train the indicator in the text regularization model and calculate the edit alignment vector between the original sequence and the target sequence.

[0021] As a further solution of the present invention, in Step 2, the edit alignment algorithm in the original FastCorret model is improved. Sequence alignment is achieved by overall matching of the same word vectors in the original sequence and the target sequence. The position coordinates of the replacement words and inserted words in the original sequence are extracted based on the rules. '-1' is used to represent replacement and '-2' is used to represent insertion. A new edit alignment vector is calculated, so that the model can more accurately identify the precise changes in the feature vector.

[0022] Step 3, Decoding and Fine-tuning: During the decoding and fine-tuning phase, the present invention divides the word vectors in the original sequence into two categories: immutable words and non-standard words (words that require regularization). Based on Step 2, the pre-trained model is fine-tuned using a non-autoregressive architecture. The position vector of the edit alignment vector in the indicator is modified to adjust the length of the original sequence to the target sequence, thereby ensuring that the prediction of the text regularization model only includes the regularized words corresponding to the non-standard words.

[0023] As a further solution of the present invention, in Step 3, based on the formed pre-trained model, all unchanged marker vectors '1' are replaced with deleted marker vectors '0' by modifying the position vector of the edit alignment vector in the indicator; at the same time, the target set dictionary of the decoding stage is reconstructed, and the prediction difficulty of the model is further reduced by reducing the size of the predicted word candidate set.

[0024] As a further solution of the present invention, in Step 3, the length adjustment from the original sequence to the target sequence includes the following:

[0025] The edit distance between the source and target sentences is calculated by analyzing the insertion, deletion, and substitution operations in the edit distance to obtain the number of target tokens corresponding to each source token after editing. A non-autoregressive encoder-to-decoder structure and indicator are used to solve the length mismatch problem between the encoder and decoder. The obtained number of target tokens is used to train the indicator to predict the length of each source token after correction, and each source token is adjusted and sent to the decoder for parallel generation.

[0026] As a further solution of the present invention, in Step 3, obtaining the edit distance specifically includes the following steps:

[0027] The edit distance measures the dissimilarity between two sentences by calculating the minimum number of edit operations required to transform a source sentence into a target sentence. Valid edit operations include insertion, deletion, and substitution. Given a source sentence S = (s1, s2, ..., sM) and a target sentence T = (t1, t2, ..., tN), where M and N are the lengths of the source and target sentences, the edit distance between S and T is obtained by recursively calculating the edit distance of the prefix sentences. The specific steps are as follows:

[0028] D(i,j)=min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+T(si≠tj))

[0029] In the above formula, D(i,j) is the edit distance between the source prefix sentence (s1,s2,...,si) and the target prefix sentence (t1,t2,...,tj), 1(·) is an indicator function that outputs 1 when the condition is true and 0 otherwise; the boundary conditions are D(i,0)=i,D(0,j)=j.

[0030] As a further solution of the present invention, in Step 3, combined with the regularization characteristics of non-standard words in Vietnamese, when a replacement or matching operation occurs, the source token is aligned with the replaced or unchanged token in the target sentence; the tag vectors of all training data are calculated separately using the original dataset and the target dataset, and are subsequently passed as parameters into the model training process to provide key information for indicator training.

[0031] Step 4, post-processing: Based on step 3, the regularized words corresponding to the non-standard words predicted by the text regularization model are inserted into the invariant words of the original sequence to obtain a complete target prediction sequence.

[0032] As a further solution of the present invention, in Step 4, the position of the regularized word corresponding to the non-standard word in the target sequence is found, and the inference result is inserted into the target sequence. The final complete sequence effectively avoids the situation where the text regularization model incorrectly predicts the immutable word.

[0033] The beneficial effects of the present invention are:

[0034] 1. This invention solves the current problems faced by Vietnamese text regularization, such as the diversity of language expressions and the semantic ambiguity of non-standard words;

[0035] 2. This paper establishes a more adaptable end-to-end text regularization method. Taking into account the characteristics of the Vietnamese language, it constructs several categories of Vietnamese text regularization annotated corpora and expands the annotated corpora with character-level alignment. In the first stage, the present invention uses the annotated corpus to pre-train the indicator and uses the edit alignment algorithm to train the edit alignment vector from the original sequence to the target sequence in the indicator, while also improving the contextual semantic information modeling capability of the model encoding end. In the second stage, the pre-trained model is fine-tuned using a non-autoregressive architecture. By modifying the position vector, the adjusted sequence of non-standard words is extracted, which reduces the search space on the decoding end and avoids the problem of non-regularized text prediction errors caused by the limitations of the sequence generation model itself.

[0036] 3. The present invention has proved through experiments that the method of the present invention has achieved an accuracy rate of 97%, effectively solving the semantic disambiguation problem in Vietnamese text regularization. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flow chart of the reasoning stage of the method proposed in the present invention;

[0038] Figure 2 This is a structural diagram of the model in the decoding stage of the method proposed in the present invention. DETAILED DESCRIPTION

[0039] Example 1: Figure 1-Figure 2 As shown, a non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm, the specific steps of the method are:

[0040] Step 1. Preprocessing: We crawled text data (including news, business, finance, sports, etc.) from the vnexpress.net website and modified the text data. We expanded the annotations of 30,000 text data items to 100,000 annotated data pairs. The data annotation work can be completed by native Vietnamese linguists.

[0041] The specific steps of Step 1 are as follows:

[0042] Step 1.1. The present invention crawls a large amount of text corpus from various major Vietnamese news and media websites. In the preprocessing module, the crawled Vietnamese original text is deleted from redundant space characters, ASCII codes, emoticons, HTML entities (for example, "www.xxx...", "Email@...", " ", "<", ">", "&", """), and unreadable words (for example, ":))");

[0043] Step 1.2, and the corpus of large paragraphs was segmented. Specifically, a regular expression-based tokenizer was used to segment long sentences into sentences with a character length of 20-150 according to punctuation marks (for example, dot '.', comma ',', ellipsis '...', exclamation mark '!', etc.). Then all the obtained text corpus was corrected to make the text corpus more standardized and reasonable; because there are numbers and characters in the text, and irregular splicing between characters, such as: "SP3", "100GB", "7X", “USD / ngu'ò'i”, “euro / "wait;

[0044] Step 1.3. Finally, the processed text is embedded into the vector space as an unlabeled dataset. In order to improve the training effect of the indicator in the pre-training stage, the obtained dataset is labeled in combination with the text regularization characteristics of Vietnamese.

[0045] First, the non-standard word vectors of the original text (i.e., the text sequence to be regularized) and the target text (i.e., the text sequence after regularization) are annotated. The annotation specifications for the regularized Vietnamese text are as follows: characters in the text sequence that need to be regularized are marked with '_', and a single word vector is annotated each time. This is done by using the word-granular Vietnamese text regularization label segmentation method to label the data.

[0046] Assume that the text sequence to be regularized is: "trong làm 18,19%”, the corresponding target text is: “trong làm mu'ò'i tám chín If a sentence contains multiple words that need to be regularized, such as: "có2 tour 4ngày 3 và2ngày 1 ", the present invention expands it into multiple sequences according to the number of non-standard words and marks them separately. The comparison of the sequences before and after marking is shown in Table 2:

[0047] Table 2 shows the comparison of Vietnamese annotation data before and after

[0048]

[0049]

[0050] In order to solve the problem of scarcity of Vietnamese corpus, the present invention uses the expanded text sequence as training corpus for pre-training and decoding. In addition, in most cases, the present invention needs to add '0' to the non-annotated word numbers in the original sequence. Since there are a lot of non-standard words in the original text, such as "12345" is pronounced as "mu'ò'i hai ngànba mu'ò'i ", the present invention combines the language characteristics of Vietnamese and the algorithm characteristics of the model, and tries to keep the number of characters of non-standard words in the original text the same as the number of their corresponding target text word vectors. The '0' padding of the present invention is as follows: Since 12345 has 5 digits, and "mu'ò'i hai ngàn ba mi " has 8 words. In this context, according to relevant Vietnamese knowledge, 1 is pronounced as "mu'ò'i", 2 is pronounced as "hai ngàn", 3 is pronounced as "ba ”,4 is read as “ mu'o'i", 5 pronounced as The present invention uses "000" to represent "ngàn" and "00" to represent We use '0' to represent "mu'o'i". Therefore, the annotation of "12345" in this paper is "_1 2 000 3 00 4 0 5_", so that the new annotation sequence and the target sequence form the same number of word vectors. The non-standard words before and after some annotations are shown in Table 3:

[0051] Table 3 shows the Vietnamese part of the labeled data

[0052]

[0053] For some specific cases, for example, '%' is read as "1 / 4" is read as There is no need to fill in the words to force the source sequence to align with the target sequence.

[0054] For the text sequence in the above example, the training data in this paper is divided into unlabeled and labeled data. This labeling method not only expands the dataset size, improving the language understanding capabilities of the text regularization model in low-resource scenarios, but also fully resolves the length mismatch between the original and target sequences, further improving the model's accuracy.

[0055] Step 2, Pre-training: Improve the matching algorithm based on edit alignment to detect and obtain the edit distance of non-standard words in the original sequence. At the same time, use the expanded annotated data to pre-train the indicator in the text regularization model and calculate the edit alignment vector between the original sequence and the target sequence.

[0056] Leng et al. achieved high accuracy in Chinese error correction tasks by pre-training a length predictor using the edit alignment algorithm. The edit alignment algorithm has a significant effect on compensating for the prediction error caused by the mismatch between the length of the original sequence and the target sequence in the sequence generation task.

[0057] The present invention uses the improved edit alignment algorithm to test the accuracy of Vietnamese text regularization tasks under low resources.

[0058] The input length of the neural network model is l sequence defined as X={x1,...,x m ,...,x l}, where x tFor unreadable words that need regularization, the output of the neural model is a label Y = y m , where y m is x m The prediction result of z is defined as m is x m The true label of such a set of labeled data is represented as {x m ,z m The learned conditional probability distribution is defined as:

[0059]

[0060] Where Score(x,y) is the score obtained by the predicted value.

[0061] The limited training data under scarce resource conditions can lead to insufficient model training and poor model reasoning performance. To address this issue, the present invention constructs 100,000 annotated data from 30,000 original data to pre-train the model. To verify the prediction performance of the model under different annotated data, the present invention refers to the classification standard proposed by Dang et al. and divides the original data into 11 test sets according to the non-standard word categories (as shown in Table 4):

[0062] Table 4 shows the Vietnamese training data categories

[0063]

[0064]

[0065] The original model selects the alignment method between the original sequence and the target sequence by calculating the matching score of each path. This paper uses the original calculation method to test a large amount of labeled data. Since there are more replacement operations and fewer insertion operations in the Vietnamese regularization task, such as '%' is converted to It is often located at the end of the non-standard sequence. The vector representation calculated by the original edit alignment algorithm is to perform an insertion operation at the beginning of the non-standard word sequence, and then all replacement operations are performed. This does not conform to the law of change from the original sequence of non-standard words to the target sequence, resulting in a decrease in the accuracy of the model in predicting long sequences. The original calculation method cannot accurately calculate the sequence position of the insertion operation, thereby affecting the prediction ability of the model. Therefore, the present invention improves the edit alignment algorithm in the original FastCorret model, realizes sequence alignment by overall matching the same word vectors in the original sequence and the target sequence, extracts the position coordinates of the replacement word and the inserted word in the original sequence based on the rules, uses '-1' to represent replacement, '-2' to represent insertion, and calculates a new edit alignment vector, so that the model can more accurately identify the precise changes of the feature vector.

[0066] Step 3, Decoding and Fine-tuning: During the decoding and fine-tuning phase, the present invention divides the word vectors in the original sequence into two categories: immutable words and non-standard words (words that require regularization). Based on Step 2, the pre-trained model is fine-tuned using a non-autoregressive architecture. The position vector of the edit alignment vector in the indicator is modified to adjust the length of the original sequence to the target sequence, thereby ensuring that the prediction of the text regularization model only includes the regularized words corresponding to the non-standard words.

[0067] As a further solution of the present invention, due to the training of the model with a large amount of labeled data, the pre-trained model has deepened the embedded memory of non-standard words. The neural network model often has the problem of unpredictability in sequence generation tasks, that is, it is easy to predict words that do not need to be changed as incorrect words. In order to avoid this problem and simultaneously achieve end-to-end regularization of non-standard words, in the Step 3, based on the formed pre-trained model, by modifying the position vector of the edit alignment vector in the indicator, all unchanged tag vectors '1' are replaced with deleted tag vectors '0'; at the same time, the target set dictionary of the decoding stage is reconstructed, and the prediction difficulty of the model is further reduced by reducing the size of the predicted word candidate set.

[0068] As a further solution of the present invention, in Step 3, the length adjustment from the original sequence to the target sequence includes the following:

[0069] The model used in this paper utilizes non-autoregressive decoding generation with edit alignment to accelerate the inference process of an autoregressive model. The edit distance between the source and target sentences is calculated by analyzing the insertion, deletion, and substitution operations in the edit distance to obtain the number of target tokens corresponding to each source token after editing (i.e., 0 for deletion, 1 for unchanged, -1 for replacement, and -2 for insertion). A non-autoregressive encoder-to-decoder structure and indicator are used to address the length mismatch between the encoder (source sentence) and decoder (target sentence). The obtained target token number is used to train the indicator to predict the length of each source token after correction, and each source token is adjusted. The adjusted source tokens are then sent to the decoder for parallel generation.

[0070] As a further solution of the present invention, in Step 3, obtaining the edit distance specifically includes the following steps:

[0071] The edit distance measures the dissimilarity between two sentences by calculating the minimum number of edit operations required to transform a source sentence into a target sentence. Valid edit operations include insertion, deletion, and substitution. Given a source sentence S = (s1, s2, ..., sM) and a target sentence T = (t1, t2, ..., tN), where M and N are the lengths of the source and target sentences, the edit distance between S and T is obtained by recursively calculating the edit distance of the prefix sentences. The specific steps are as follows:

[0072] D(i,j)=min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+T(si≠tj))

[0073] In the above formula, D(i,j) is the edit distance between the source prefix sentence (s1,s2,...,si) and the target prefix sentence (t1,t2,...,tj), 1(·) is an indicator function that outputs 1 when the condition is true and 0 otherwise; the boundary conditions are D(i,0)=i,D(0,j)=j.

[0074] As a further solution of the present invention, in the Step 3, when performing edit alignment, since the transformation of non-annotated words in the regularization task only contains insertion and replacement operations, the path of the edit distance is often constant. Therefore, the impact on the model performance depends on the selection of a suitable edit alignment method. The alignment method proposed in the present invention is different from that in the original paper. Since there are more replacement operations and fewer insertion operations in the Vietnamese regularization task, the method of selecting the alignment method by calculating the matching score of the path is not applicable to the Vietnamese regularization task. Therefore, the present invention combines the regularization characteristics of non-standard words in Vietnamese. When a replacement or matching operation occurs, the present invention aligns the source tag with the replaced or unchanged tag in the target sentence. When an insertion operation occurs in the source sequence (the data annotation work of the present invention avoids the situation where a deletion operation occurs in the source sequence), such as: '%' is read as When the algorithm matches the '%' in the source sequence, the present invention needs to perform an insertion operation at this position. The present invention designs an algorithm that, when the '%' character vector in the source sentence matches the '%' character vector in the target sentence, When the word vector is matched, a '-2' mark is inserted into the mark vector of the edit alignment operation, so that the model deepens the memory of the insertion operation at the '%' position. For example: the source sequence of the present invention is: "Nhànu'ó'ctháng 11 1 4,7% so tháng 10 / 2021", the target sequence is: "Nhànu'ó'c tháng 11 mu'ó'i so tháng 10 / 2021", the label vector calculated by this method is "1 1 1 1 1-1-1-1-1-21 1 1". The present invention uses the original data set and the target data set to separately calculate the label vectors of all participating training data, which are subsequently passed into the model training process as parameters to provide key information for the training of the indicator.

[0075] The present invention uses the default model architecture in FastCorrect, which uses Transformer as the basic model architecture, such as Figure 2 As shown, it consists of a 6-layer encoder and a 6-layer decoder, with a hidden layer dimension size of 512 and a feedforward layer dimension size of 1024. The indicator of the present invention consists of 5 layers of one-dimensional convolutional networks with ReLU activation function and 2 linear layers, all of which have a hidden size of 512 dimensions in terms of output scalars. Each convolutional layer is followed by layer normalization and dropout. The kernel size of the convolutional network is 3. FastCorrect is implemented on Fairseq. The encoder takes the source sentence as input and outputs a hidden sequence that: 1) is fed to the indicator to predict the number of target tags corresponding to each source tag (i.e., the tag vector obtained in the previous subsegment), 2) is used for the decoder through encoder-decoder attention. The model structure is as follows Figure 1 As shown, the predictor is optimized using the mean squared error loss.

[0076] The encoder's output vector is fed into the indicator, which trains the token vector for edits between the input sequence and the target sequence, and into the decoder for training. The model adjusts the input sequence based on the token vector and feeds it into the decoder. Due to the design of the edit alignment and indicator in the model, deletion and insertion errors are detected by predicting a length of 0 or -2 on the corresponding source token. For substitution errors, the indicator predicts a length of 1, the same as the length of the unchanged / correct source token. In this case, all 1-tokens in the token vector are set to 0 and then adjusted with the source input sequence, indicating that all unchanged words in the input sequence are deleted and only the non-standard words are retained. Therefore, when the new adjusted sequence is fed into the decoder for training, the decoder output only retains the regularized words corresponding to the non-standard words in the target set.

[0077] Step 4, post-processing: Based on step 3, the regularized words corresponding to the non-standard words predicted by the text regularization model are inserted into the invariant words of the original sequence to obtain a complete target prediction sequence.

[0078] As a further solution of the present invention, in Step 4, the position of the regularized word corresponding to the non-standard word in the target sequence is found, and the inference result is inserted into the target sequence. The final complete sequence effectively avoids the situation where the text regularization model incorrectly predicts the immutable word.

[0079] In order to illustrate the effect of the present invention, the present invention divides the training data into 11 categories according to the categories of non-standard words from 100,000 labeled data pairs, of which 80,000 are used as training data sets, 20,000 are used as verification data sets, and 20,000 are used as test data sets.

[0080] The following experimental verification was conducted. The experimental setup consisted of three groups of experiments. In the first group, the test conditions included training data, improved edit alignment, dictionary reduction, and modification indicator position vectors. The main experiment in the first group was to integrate the four improvements into the FastCorrect model. Five additional ablation experiments (using different training methods) were also conducted to investigate their effects on text regularization accuracy.

[0081] The first training method uses an unlabeled dataset to train and fine-tune the FastCorrect model. To verify the impact of the proposed labeling method on experimental results, the second training method uses a labeled dataset to train and fine-tune the model. Furthermore, to examine the optimization effects of the three different improvements on the model, three comparative experiments were conducted using improved editing algorithms, reduced dictionary size, and modified position vectors. The precision rate reflects the model's prediction accuracy for non-standard words only. The experimental results are shown in Table 5.

[0082] Table 5 Text regularization accuracy of six different training methods

[0083]

[0084] The model trained on annotated text data achieved a 2.6-fold reduction in word error rate compared to unannotated data, significantly improving both accuracy and precision. Furthermore, the three types of comparative experiments improved the model's performance to varying degrees. Furthermore, the main experiment achieved an accuracy rate of 97%. Furthermore, the indicator is a key component influencing the model's performance. Therefore, in a second set of experiments, the authors calculated the impact of different training methods on the indicator's prediction accuracy. The experimental results are shown in Table 6.

[0085] Table 6 Prediction accuracy of indicator for target sequence length changes

[0086]

[0087] Experimental data shows that modifying the position vector has the best impact on the indicator's prediction performance. Setting the position vector to 0 shortens the input sequence, and the target training set also becomes shorter than other training methods. This makes it easier for the indicator to identify the length change from the input sequence to the target sequence.

[0088] In order to analyze the prediction of the final experimental results for different categories of data sets, the third group of experiments set up by the present invention chose to perform classification experiments on the main experiment with the best experimental results. 11 categories of non-standard words were extracted from the test set to calculate their accuracy. The experimental results are shown in Table 7.

[0089] Table 7 Accuracy of the test data of FastCorrect model + annotated corpus + reduced dictionary + modified editing algorithm + modified position vector

[0090]

[0091] Visualize the text regularization results of different training methods. In the inference stage, the test input sequence is converted into a labeled sequence after a pre-processing (see Table 3), and then enters the model for training. For example, the test input sample is:

[0092] trong 2 xin là69%

[0093] sang ngày 1–1 0–2 0 0 1mó'i

[0094] ngày tù'3 0-3 5k g

[0095] Table 8 shows the output sequences of different training methods during the decoding process.

[0096] Table 8 Regularization results of three different decoding methods

[0097]

[0098] It can be seen that the regularization methods implemented in this invention are all implemented through sequence generation. The input of all training methods is a complete Vietnamese text. The data samples after pre-processing are closer in length to the word correspondence of the target sequence, which can greatly reduce the difficulty of model training. <unk>This marker indicates word vectors that the model cannot recognize. In the target set encoding, we did not define the IDs of these words that do not require regularization, so that only regularized words are retained in the output. As you can see, the model regularizes all non-standard words at different positions in the original input sequence. This approach can solve the problem of a sentence containing multiple words that require text regularization. Subsequent post-processing can be used to insert unchanged words to complete the target sequence. The final output only includes the regularized output for the non-standard word "6 9%." This approach, by modifying the indicator's position vector, enables the model to accurately identify the coordinates of the words that require regularization while filtering out information interference from other positions, thereby significantly improving the model's prediction accuracy.

[0099] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.< / unk>

Claims

1. A non-autoregressive decoding Vietnamese text regularization method based on edit alignment algorithm, characterized by: The specific steps of the method are: Step 1: Preprocessing: We crawled text corpora from various Vietnamese news and media websites, modified the text corpora, and used annotation to expand the Vietnamese dataset. Step 2, Pre-training: Improve the matching algorithm based on edit alignment to detect and obtain the edit distance of non-standard words in the original sequence. At the same time, use the expanded annotated data to pre-train the indicator in the text regularization model and calculate the edit alignment vector between the original sequence and the target sequence. Step 3: Decoding and fine-tuning: Based on Step 2, the pre-trained model is fine-tuned using a non-autoregressive architecture. The position vector of the edit alignment vector in the indicator is modified to adjust the length of the original sequence to the target sequence. This ensures that the prediction of the text regularization model only includes the regularized words corresponding to the non-standard words. Step 4, post-processing: Based on Step 3, the regularized words corresponding to the non-standard words predicted by the text regularization model are inserted into the invariant words of the original sequence to obtain a complete target prediction sequence; The Step 1 includes: The non-standard word vectors of the original text (i.e., the text sequence to be regularized) and the target text (i.e., the text sequence after regularization) are annotated. The annotation specifications for the regularized Vietnamese text are as follows: characters in the text sequence that need to be regularized are marked with '_', and a single word vector is annotated at a time. This is done by using the word-granular Vietnamese text regularization label segmentation method to label the data. If a sentence contains multiple words that need to be regularized, it is expanded into multiple sequences according to the number of non-standard words and annotated separately. In Step 2, the edit alignment algorithm in the original FastCorret model is improved. Sequence alignment is achieved by matching the same word vectors in the original sequence and the target sequence. The position coordinates of the replacement and insertion words in the original sequence are extracted based on the rules. '-1' is used to represent replacement and '-2' is used to represent insertion. A new edit alignment vector is calculated, which enables the model to more accurately identify the precise changes in the feature vector. In Step 3, obtaining the edit distance specifically includes the following steps: The edit distance is a measure of the dissimilarity between two sentences by calculating the minimum number of editing operations required to transform a source sentence into a target sentence. Valid editing operations include insertion, deletion, and replacement. Given a source sentence S = (s1, s2, ..., sM) and a target sentence T = (t1, t2, ..., tN), where M and N are the lengths of the source and target sentences, the edit distance between S and T is obtained by recursively calculating the edit distance of the prefix sentences.

2. The non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.1: For the original Vietnamese text obtained by crawling, remove extra spaces, ASCII codes, emoticons, HTML entities, and unreadable words; Step 1.2: Use a regular expression-based tokenizer to segment long sentences into sentences with a character length of 20-150 based on punctuation marks, and then perform correction work on all the resulting text corpus; Step 1.

3. Finally, the processed text is embedded into the vector space as an unlabeled dataset, and the obtained dataset is labeled in combination with the text regularization characteristics of Vietnamese.

3. The non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm according to claim 1, characterized in that: In Step 3, based on the pre-trained model, all unchanged marker vectors '1' are replaced with deleted marker vectors '0' by modifying the position vector of the edit alignment vector in the indicator. At the same time, the target set dictionary of the decoding stage is reconstructed, and the prediction difficulty of the model is further reduced by reducing the size of the predicted word candidate set.

4. The non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm according to claim 1, characterized in that: In Step 3, the length adjustment from the original sequence to the target sequence includes the following: By analyzing the insertion, deletion, and substitution operations in the edit distance between the source and target sentences, the number of target tokens corresponding to each source token after editing is obtained; a non-autoregressive encoder-to-decoder structure and indicator are adopted to solve the problem of length mismatch between the encoder and decoder. The obtained number of target tokens is used to train the indicator to predict the length of each source token after correction, and each source token is adjusted and sent to the decoder for parallel generation.

5. The non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm according to claim 1, characterized in that: In Step 3, taking into account the regularization characteristics of non-standard words in Vietnamese, when a replacement or matching operation occurs, the source token is aligned with the replaced or unchanged token in the target sentence; The label vectors of all training data are calculated separately using the original dataset and the target dataset, and then passed into the model training process as parameters to provide key information for the indicator training.

6. The non-autoregressive decoding Vietnamese text regularization method based on the edit alignment algorithm according to claim 1, characterized in that: In Step 4, the position of the regularized word corresponding to the non-standard word in the target sequence is found, and the inference result is inserted into the target sequence. The final complete sequence effectively avoids the situation where the text regularization model mispredicts the immutable word.

Images