Self-supervised pre-training method, system and medium for Chinese pinyin spelling correction

Abstract

The present invention discloses a self-supervised pre-training method, system and medium for Chinese pinyin spelling error correction, wherein the method comprises: obtaining a Chinese text sequence, converting the Chinese text sequence into an input sentence X; obtaining a list of characters that need to be replaced using a pinyin confusion set from the input sentence X, and recording it as PYList(X); for each character x in PYList(X), obtaining the pinyin of the character x, obtaining a list of homophones based on the pinyin, and replacing the character x with a new character based on the homophone list; after processing all the characters in PYList(X), obtaining a new input sentence PYInput(X), obtaining the input Input(X) of a BERT model based on the input sentence PYInput(X); using Input(X) as the input of a BERT model and training it, and then predicting the correct value of each character in Input(X) through a masked language model in the BERT model. The present invention replaces the pinyin confusion set and converts the MLM task in BERT into predicting the correct value of the masked characters, thereby enhancing the error correction capability of BERT and being widely applicable in the field of natural language processing.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a self-supervised pre-training method, system and medium for Chinese phonetic spelling error correction. Background Art

[0002] Text error correction is an important subtask and research direction in natural language processing (NLP). Its main goal is to automatically verify, identify, and correct text errors through machine learning, thereby improving the accuracy of linguistic expression while reducing the cost of manual verification. Compared to English text error correction research, Chinese text error correction faces more difficulties and challenges. In particular, Chinese characters lack delimiters and morphological variations, making the syntactic and semantic interpretation of Chinese highly dependent on context. As a key component of Chinese text error correction research, Chinese spelling error correction (CSC) aims to detect and correct spelling errors in Chinese characters (or words). Early research on Chinese spelling error correction primarily employed rule-based or statistical methods based on accumulated correction rules and correction dictionaries. Error correction was divided into three steps: error detection, candidate recall, and error correction. Major drawbacks of CSC include the need for manually formulated complex rules, a lack of generalizability, and a heavy reliance on the quality and quantity of training data.

[0003] In the past two years, as two-stage approaches using pre-trained language models and task-specific fine-tuning have achieved new state-of-the-art results on multiple natural language processing tasks, some researchers have begun applying pre-trained language models like BERT and RoBERTa to Chinese spelling correction. These models have significantly outperformed traditional rule-based or statistical machine translation methods on various corpora, such as Sighan 2013, Sighan 2014, and Sighan 2015. Representative works include FASPell, Soft-Masked BERT, SpellGCN, and DCN.

[0004] Compared to traditional rule-based or statistical machine translation methods, CSC methods based on BERT / RoBERTa pre-trained language models can significantly improve the model's performance in Chinese spelling error correction tasks through pre-training on massive amounts of text and fine-tuning on downstream tasks. However, BERT and other models primarily use the Masking Language Model (MLM) and Next Sentence Prediction (NSP) tasks during pre-training, masking words with [MASK] before predicting their correct values. This results in a lack of error recognition and correction capabilities, and leads to inconsistencies in the model's pre-training and fine-tuning objectives. Summary of the Invention

[0005] In order to at least to some extent solve one of the technical problems existing in the prior art, the object of the present invention is to provide a self-supervised pre-training method, system and medium for Chinese pinyin spelling error correction.

[0006] The technical solution adopted in the present invention is:

[0007] A self-supervised pre-training method for Chinese pinyin spelling correction includes the following steps:

[0008] Obtain a Chinese text sequence and convert it into an input sentence X that meets the requirements of the BERT model based on preset tags;

[0009] Get the list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and record it as PYList(X);

[0010] For each character x in PYList(X), get the pinyin of the character x, get a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones;

[0011] After processing all the words in PYList(X), a new input sentence PYInput(X) is obtained, and the input Input(X) of the BERT model is obtained based on the input sentence PYInput(X);

[0012] After taking Input(X) as the input of the BERT model and training it, the masked language model BertForMaskedLM in the BERT model is used to predict the correct values ​​of each word in Input(X) except [CLS] and [SEP].

[0013] Furthermore, the method of obtaining a list of characters that need to be replaced by the pinyin confusion set from the input sentence X and recording it as PYList(X) includes:

[0014] For an input sentence X, BERT's masking strategy is used to select the corresponding masked characters. That is, 15% of the characters in the input sentence X are selected for replacement. 80% of the selected characters are replaced with characters in the pinyin confusion set, 10% are randomly replaced with a character in the vocabulary, and 10% remain unchanged.

[0015] For ease of representation, the list of all characters in the input sentence X that are selected to be replaced using the pinyin confusion set is denoted as PYList(X).

[0016] Furthermore, for each character x in PYList(X), obtaining the pinyin of the character x, obtaining a list of homophones based on the pinyin, and replacing the character x with a new character based on the list of homophones includes:

[0017] For each word x in PYList(X) perform the following steps:

[0018] Use the Chinese character to pinyin tool pypinyin in Python to obtain the pinyin and monotone of the word;

[0019] According to the pinyin of the word, use the Pinyin to Chinese character tool Pinyin2Hanzi in Python to obtain a list of homophones under the pinyin; if the list is not empty, obtain the list of homophones and homophones with different tones of the word SamePYList(x); if the list is empty, set the list SamePYList(x) to empty;

[0020] If the pinyin of the word ends with g, remove the g and use the pinyin to Chinese character tool in Python to obtain the word's similar pronunciation list DiffPYList(x);

[0021] Select the replacement word according to the list SamePYList(x) and the list DiffPYList(x); if the list SamePYList(x) or the list DiffPYList(x) is empty, keep the original word unchanged.

[0022] Furthermore, after processing all the words in PYList(X), a new input sentence PYInput(X) is obtained, and the input Input(X) of the BERT model is obtained according to the input sentence PYInput(X), including:

[0023] Convert each word in PYInput(X) into a sequence number in the vocabulary, combine the position word vector, word vector information and segmentation information as the input sentence of the BERT layer; at the same time, set the sequence numbers of the words that do not need to be replaced and the padding flag [PAD] in the input sentence X to -100 to obtain the input Input(X) of the BERT model.

[0024] Furthermore, during the training process, the loss function uses the standard cross entropy, and the loss value is recorded as LossMAPYSC;

[0025] If the training data is a single sentence, the loss value only includes LossMAPYSC; if the training data is multiple sentences, the adjacent sentence relationship prediction NSP in BERT or the sentence order relationship SOP in ALBERT is combined to further predict the relationship between sentences, and the corresponding loss value is calculated and summed with LossMAPYSC as the loss value of the model.

[0026] Furthermore, in order to better utilize the grammatical and semantic knowledge learned by the pre-trained language model on massive texts, the pre-training process uses a preset pre-trained language model as the basic model for initialization.

[0027] Furthermore, the pre-training process adopts batch training with a learning rate of 5e -5 , trained for 10 rounds in total, and optimized using the Adam optimizer.

[0028] Another technical solution adopted in the present invention is:

[0029] A self-supervised pre-training system for Chinese pinyin spelling correction, including:

[0030] The Chinese text input module is used to obtain Chinese text sequences and convert them into input sentences X that meet the requirements of the BERT model according to preset tags;

[0031] The replacement character selection module is used to obtain a list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and is recorded as PYList(X);

[0032] The character replacement module is used to obtain the pinyin of each character x in PYList(X), obtain a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones;

[0033] The training set acquisition module is used to obtain a new input sentence PYInput(X) after processing all the words in PYList(X), and obtain the input Input(X) of the BERT model based on the input sentence PYInput(X);

[0034] The model training module is used to take Input(X) as the input of the BERT model and perform training. Then, the masked language model BertForMaskedLM in the BERT model is used to predict the correct values ​​of all words in Input(X) except [CLS] and [SEP].

[0035] Another technical solution adopted in the present invention is:

[0036] A self-supervised pre-training system for Chinese pinyin spelling correction, including:

[0037] at least one processor;

[0038] at least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0040] Another technical solution adopted in the present invention is:

[0041] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0042] The beneficial effects of the present invention are as follows: the present invention combines the Chinese character to pinyin and pinyin to Chinese character tools to construct a pinyin confusion set of masked characters and then replaces them, and converts the MLM task in BERT into the prediction of the correct value of the masked characters. Finally, by re-pre-training BERT, without changing the structure of the model, BERT's ability to detect and correct errors in Chinese pinyin spellings is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 2 is a schematic diagram of the BERT pre-training model structure based on MAPYSC in an embodiment of the present invention;

[0045] Figure 2 4 is a flow chart of a self-supervised pre-training method for Chinese pinyin spelling error correction in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0047] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0048] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0049] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0050] In response to the deficiencies in the prior art, the present invention provides a self-supervised pre-training method MAPYSC (Masked Language Model as PinYin Spelling Correction) for Chinese pinyin spelling correction. The masked language model (MLM) used by the pre-trained language model BERT in the pre-training stage lacks the ability to recognize and correct Chinese pinyin spelling errors, and has inconsistent goals with the fine-tuning stage. The method converts the masking and correct value prediction tasks of characters in the MLM into the recognition and correction of pinyin-spelled characters. The method also combines Chinese character to pinyin and pinyin to Chinese character tools to construct a pinyin-confused character set of masked characters from homophones and near-phonetics, and then replaces them. The method also changes the training goal of the model to predicting correct characters. Finally, MAPYSC is used for re-training based on the public pre-trained language model and dataset, thereby further enhancing BERT's ability to detect and correct Chinese pinyin spelling errors without changing the model structure. The method can also be directly applied to various BERT-based Chinese spelling error models and improve the model's effectiveness.

[0051] MAPYSC is used to convert the original BERT MLM's masking and correct value prediction tasks of characters into the recognition and correction of pinyin spelling errors. The Chinese character to pinyin and pinyin to Chinese character tools are combined to construct a pinyin confusion character set of the masked characters from the aspects of homophones and near-phonetics and replace them, thus forming a pre-training dataset containing Chinese pinyin spelling errors. On this basis, the public pre-trained language model based on massive text and MLM is retrained in combination with the model's pre-training and loss function, thereby further enhancing BERT's ability to detect and correct Chinese pinyin spelling errors. Finally, the pre-trained BERT model obtained after re-training is applied to fine-tuning and testing of downstream tasks. Experiments on the public corpora such as Sighan2013, Sighan2014, and Sighan2015 have shown that the proposed MAPYSC method can not only improve the error detection and correction capabilities of the pre-trained language model BERT for Chinese pinyin spelling correction, but can also be directly applied to other BERT-based Chinese spelling correction models such as Soft-Masked BERT, SpellGCN, and DCN without changing the BERT model structure, and further improve the model's performance.

[0052] like Figure 1 and Figure 2As shown, this embodiment provides a self-supervised pre-training method for Chinese pinyin spelling correction. It takes advantage of the masked language model MLM and pre-training in BERT in learning the contextual information of text sequences and their relationships, and proposes a self-supervised pre-training method MAPYSC for Chinese pinyin spelling correction. It combines the Chinese character to pinyin and pinyin to Chinese character tools to construct a pinyin confusion set of masked characters and then replaces them. It also converts the MLM task in BERT into a prediction of the correct value of the masked characters. Finally, through re-pre-training, BERT further enhances its ability to detect and correct Chinese pinyin spelling errors without changing the structure of the model. The method specifically includes the following steps:

[0053] S1. Obtain a Chinese text sequence and convert it into an input sentence X that meets the requirements of the BERT model according to the preset flags.

[0054] For the input Chinese text sequence, special flags such as [CLS] and [SEP] are combined to merge the text sequence into an input sequence X that meets the BERT requirements.

[0055] S2. Obtain a list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and record it as PYList(X).

[0056] For an input sentence X, BERT's original masking strategy is used to select the corresponding masked characters. Specifically, 15% of the characters in the input sentence (excluding special markers such as [CLS] and [SEP]) are selected for replacement. These characters have an 80% probability of being replaced with a character from the phonetic confusion set, a 10% probability of being replaced with a random character from the vocabulary, and a 10% probability of remaining unchanged. For ease of representation, the list of all characters in the input sentence X selected for replacement using the phonetic confusion set is denoted as PYList(X).

[0057] S3. For each character x in PYList(X), obtain the pinyin of the character x, obtain a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones.

[0058] For each character x in PYList(X), MAPYSC uses the following strategy and process: 1. Use the Python Chinese character to pinyin tool pypinyin to obtain the pinyin and homonym of the character. By default, pypinyin does not use the polyphonic mode, that is, heteronym is set to False; 2. Based on the pinyin of the character, use the Python pinyin to Chinese character tool Pinyin2Hanzi to obtain a list of homophones under the pinyin. If the list is not empty, obtain the list of homophones and homophones of the character SamePYList(x). If the list is empty, set SamePYList(x) to empty. 3. If the pinyin of the character ends with g, remove the g and use the above process 2 to obtain the list of similar sounds of the character DiffPYList(x); 4. Based on SamePYList(x) and DiffPYList(x), randomly select a character from SamePYList(x) and DiffPYList(x) with a probability of 70% and 30%, respectively. If SamePYList(x) or DiffPYList(x) is empty, the original word is kept unchanged.

[0059] S4. After processing all the words in PYList(X), a new input sentence PYInput(X) is obtained, and the input Input(X) of the BERT model is obtained according to the input sentence PYInput(X).

[0060] Continue executing step S3 until all characters in PYList(X) have been processed, ultimately obtaining a new input sentence PYInput(X). Each character in PYInput(X) is converted into a sequence number in the vocabulary, and the positional word vector, word vector information, and segmentation information are combined to form the input sentence for the BERT layer. The sequence numbers of the unselected characters and the padding flag [PAD] in the original X are set to -100, ultimately obtaining the BERT model's input, Input(X).

[0061] S5. After taking Input(X) as the input of the BERT model and training it, the masked language model BertForMaskedLM in the BERT model is used to predict the correct values ​​of each word in Input(X) except [CLS] and [SEP].

[0062] Input(X) is used as the input of BertForMaskedLM and trained to predict the correct value of each word. Its loss function uses standard cross entropy and the loss value is recorded as Loss MAPYSCIn order to better utilize the grammatical and semantic knowledge learned by the pre-trained language model on massive text, the pre-training process uses a public pre-trained language model (such as BERT-base or BERT-large) as the base model for initialization. The pre-training process uses batch training with a learning rate of 5e-5, a total of 10 rounds of training, and the Adam optimizer. If the training data is a single sentence, the loss value only includes Loss MAPYSC If the training data consists of multiple sentences, the relationship between sentences can be further predicted by combining the adjacent sentence relationship prediction NSP in BERT or the sentence order relationship SOP in ALBERT, and the corresponding loss value can be calculated and combined with Loss MAPYSC The sum is taken as the loss value of the model.

[0063] Steps S4 and S5 construct the model input and train the correct value prediction. By replacing the original sentence with the constructed confusion set, a new input sentence is generated. Based on this, BertForMaskedLM is used to predict the correct values ​​of the masked words. Standard cross-entropy is used to calculate the corresponding loss and optimize the model parameters. The output of step S5 is the cross-entropy value between the vocabulary index of each masked word and the correct value index. Smaller values ​​indicate more accurate model predictions, while larger values ​​indicate less accurate predictions.

[0064] As an optional implementation method, a self-supervised pre-training method for Chinese pinyin spelling correction in this embodiment is completed by automatically constructing confusion sets and training data based on the pre-trained language model BERT. Step S1 is the data preprocessing link. Step S2 is the masking strategy and masking process, that is, selecting 15% of the characters in the input sentence (excluding special tags such as [CLS] and [SEP]) for replacement, wherein the selected characters have an 80% probability of being replaced with characters in the pinyin confusion set of the character, a 10% probability of being randomly replaced with a character in the vocabulary, and a 10% probability of remaining unchanged. Step S3 is the pinyin confusion set construction and the replacement of the masked characters, that is, using the Chinese character to pinyin tool and the pinyin to Chinese character tool to obtain the homophone and near-phonetic confusion sets of the masked characters respectively, and selecting the homophone and near-phonetic characters in the pinyin confusion set for replacement with a probability of 70% and 30% respectively. Step S4 is the input construction of BERT, and the sequence numbers of the non-selected characters and the padding flag [PAD] are set to -100. Step S5 is to determine the correct value of the masked word based on the BertForMaskedLM model, and optimize the model parameters after calculating the loss value using the standard cross entropy. During the model pre-training process, the value of the loss function is optimized and combined with the Adam optimizer, with a unified learning rate of 5e -5 .

[0065] In summary, the method of this embodiment has the following advantages and beneficial effects compared to the prior art: The present invention addresses the problem that the masked language model (MLM) used in the pre-training phase of the pre-trained language model BERT (BERT) lacks the ability to identify and correct Chinese pinyin spelling errors, and suffers from inconsistent goals with the fine-tuning phase. Therefore, a self-supervised pre-training method for Chinese pinyin spelling error correction, MAPYSC, is proposed. First, the input sentence is pre-processed to form an input that meets BERT's requirements. Then, the MLM masking strategy in BERT is used to select the corresponding masked characters. Next, a set of pinyin-confused characters is constructed based on homophones and near-phonetics, and then replaced with the masked characters. Meanwhile, the model's training objective is changed to predicting the correct character. Finally, MAPYSC is used for re-training based on the publicly available pre-trained language model and dataset. This further enhances BERT's ability to detect and correct Chinese pinyin spelling errors without changing the model's structure. Furthermore, the resulting pre-trained BERT model can be directly applied to various other BERT-based Chinese spelling error correction methods, effectively improving the model's performance.

[0066] This embodiment also provides a self-supervised pre-training system for Chinese pinyin spelling correction, including:

[0067] The Chinese text input module is used to obtain Chinese text sequences and convert them into input sentences X that meet the requirements of the BERT model according to preset tags;

[0068] The replacement character selection module is used to obtain a list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and is recorded as PYList(X);

[0069] The character replacement module is used to obtain the pinyin of each character x in PYList(X), obtain a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones;

[0070] The training set acquisition module is used to obtain a new input sentence PYInput(X) after processing all the words in PYList(X), and obtain the input Input(X) of the BERT model based on the input sentence PYInput(X);

[0071] The model training module is used to take Input(X) as the input of BertForMaskedLM and predict the correct value of each word after training.

[0072] The self-supervised pre-training system for Chinese pinyin spelling correction of this embodiment can execute the self-supervised pre-training method for Chinese pinyin spelling correction provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0073] This embodiment also provides a self-supervised pre-training system for Chinese pinyin spelling correction, including:

[0074] at least one processor;

[0075] at least one memory for storing at least one program;

[0076] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 2 The method shown.

[0077] The self-supervised pre-training system for Chinese pinyin spelling correction of this embodiment can execute the self-supervised pre-training method for Chinese pinyin spelling correction provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0078] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 2 The method shown.

[0079] This embodiment also provides a storage medium storing instructions or programs that can execute a self-supervised pre-training method for Chinese pinyin spelling correction provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0080] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0081] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0082] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0083] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0084] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0085] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0086] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0087] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0088] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A self-supervised pre-training method for Chinese pinyin spelling correction, characterized in that: The following steps are involved: Obtain a Chinese text sequence and convert it into an input sentence X that meets the requirements of the BERT model based on preset tags; Get the list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and record it as PYList(X); For each character x in PYList(X), get the pinyin of the character x, get a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones; After processing all the words in PYList(X), a new input sentence PYInput(X) is obtained, and the input Input(X) of the BERT model is obtained based on the input sentence PYInput(X); After taking Input(X) as the input of the BERT model and training it, the masked language model in the BERT model is used to predict the correct value of each word in Input(X); The list of characters that need to be replaced by the pinyin confusion set is obtained from the input sentence X and is recorded as PYList(X), including: For an input sentence X, BERT's masking strategy is used to select the corresponding masked characters. That is, 15% of the characters in the input sentence X are selected for replacement. 80% of the selected characters are replaced with characters in the pinyin confusion set, 10% are randomly replaced with a character in the vocabulary, and 10% remain unchanged. For ease of representation, the list of all characters in the input sentence X that are selected to be replaced using the pinyin confusion set is denoted as PYList(X); For each character x in PYList(X), obtaining the pinyin of the character x, obtaining a list of homophones based on the pinyin, and replacing the character x with a new character based on the list of homophones, including: For each word x in PYList(X) perform the following steps: Use the Chinese character to pinyin tool pypinyin in Python to obtain the pinyin and tone of the word; According to the pinyin of the word, use the Pinyin to Chinese character tool Pinyin2Hanzi in Python to obtain a list of homophones under the pinyin; if the list is not empty, obtain the list of homophones and homophones with different tones of the word SamePYList(x); if the list is empty, set the list SamePYList(x) to empty; If the pinyin of the word ends with g, remove the g and use the pinyin to Chinese character tool in Python to obtain the word's similar pronunciation list DiffPYList(x); Select the replacement word according to the list SamePYList(x) and the list DiffPYList(x); if the list SamePYList(x) or the list DiffPYList(x) is empty, keep the original word unchanged; After processing all the words in PYList(X), a new input sentence PYInput(X) is obtained, and the input Input(X) of the BERT model is obtained according to the input sentence PYInput(X), including: Convert each word in PYInput(X) into a sequence number in the vocabulary, combine the position word vector, word vector information and segmentation information as the input sentence of the BERT layer; at the same time, set the sequence numbers of the words that do not need to be replaced and the padding flag [PAD] in the input sentence X to -100 to obtain the input Input(X) of the BERT model.

2. A self-supervised pre-training method for Chinese phonetic spelling correction according to claim 1, characterized in that: During the training process, the loss function uses the standard cross entropy and the loss value is Loss MAPYSC ; If the training data are all single sentences, the loss value only includes Loss MAPYSC If the training data consists of multiple sentences, the adjacent sentence relationship in BERT is combined to predict NSP, or the sentence order relationship in ALBERT is combined to further predict the relationship between sentences, and the corresponding loss value is calculated and then combined with Loss MAPYSC The sum is taken as the loss value of the model.

3. A self-supervised pre-training method for Chinese phonetic spelling error correction according to claim 2, characterized in that: In order to better utilize the grammatical and semantic knowledge learned by the pre-trained language model on massive texts, the pre-training process uses a preset pre-trained language model as the basic model for initialization.

4. A self-supervised pre-training method for Chinese phonetic spelling error correction according to claim 3, characterized in that: The pre-training process adopts batch training method, and the learning rate is , trained for 10 rounds in total, and optimized using the Adam optimizer.

5. A self-supervised pre-training system for Chinese phonetic spelling correction, applied to the method according to any one of claims 1 to 4, characterized in that: include: The Chinese text input module is used to obtain Chinese text sequences and convert them into input sentences X that meet the requirements of the BERT model according to preset tags; The replacement character selection module is used to obtain a list of characters that need to be replaced using the pinyin confusion set from the input sentence X, and is recorded as PYList(X); The character replacement module is used to obtain the pinyin of each character x in PYList(X), obtain a list of homophones based on the pinyin, and replace the character x with a new character based on the list of homophones; The training set acquisition module is used to obtain a new input sentence PYInput(X) after processing all the words in PYList(X), and obtain the input Input(X) of the BERT model based on the input sentence PYInput(X); The model training module is used to take Input(X) as the input of the BERT model and perform training. Then, the masked language model in the BERT model is used to predict the correct value of each word in Input(X).

6. A self-supervised pre-training system for Chinese pinyin spelling correction, characterized by: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 4 when executed by the processor.

Images