Chinese wrongly written character correction data enhancement method and device, equipment and medium

Abstract

The invention provides a Chinese wrongly written character correction data enhancement method and device, equipment and a medium. The method comprises the steps of obtaining a wrongly written character correction training set; performing dependency syntactic analysis on samples in the training set to obtain an analysis result; combining the samples in the training set in pairs, and calculating the probability that the two samples belong to the context to obtain a calculation result; and expanding the training set according to the analysis result or the calculation result. According to the scheme, on the basis of an existing training set, original samples are segmented or expanded through dependency syntax analysis and context judgment, so that training data in a unique field is increased, and the wrongly written character correction effect in the corresponding field is improved.

Description

Technical Field

[0001] The present application relates to the field of data enhancement technology, and in particular to a method, device, electronic device and storage medium for enhancing data for correcting Chinese typos. Background Art

[0002] Chinese character correction is widely used, especially in education and publishing. This task primarily corrects similar-sounding and similar-looking errors in input text without changing the length of the input text. Currently, the effectiveness of Chinese character correction is limited by the limited availability of annotated data.

[0003] Existing data augmentation methods propose several approaches: first, they crawl large amounts of Chinese text from various web pages, then break this text into sentences. The processed sentences are assumed to be completely correct, and finally, using confusion sets, certain Chinese characters or words in the sentences are replaced with easily confused ones, thereby constructing incorrect sentences. However, the quality of text collected from web pages is not high, and it may contain errors. Furthermore, most of the text is general-purpose, so correcting typos in specific fields is limited. Summary of the Invention

[0004] The purpose of this application is to provide a method and device for enhancing Chinese typo correction data, an electronic device, and a computer-readable storage medium.

[0005] The first aspect of the present application provides a method for enhancing data for correcting Chinese typos, comprising:

[0006] Get the typo correction training set;

[0007] Performing dependency syntax analysis on the samples in the training set to obtain analysis results;

[0008] Combining two samples in the training set, calculating the probability that the two samples belong to the context, and obtaining a calculation result;

[0009] The training set is expanded according to the analysis result or the calculation result.

[0010] In one possible implementation, performing dependency syntax analysis on the samples in the training set to obtain analysis results includes:

[0011] Segmenting the samples in the training set and tagging each word with a part-of-speech tag;

[0012] Perform dependency syntactic analysis on each word in the sample to obtain the analysis results.

[0013] In one possible implementation, expanding the training set according to the analysis result or the calculation result includes:

[0014] Determining independent sentence components in the sample based on the analysis results and a preset independent sentence recognition rule;

[0015] The independent sentence components are removed from the sample or only the independent sentence components are retained to construct a new sample.

[0016] In one possible implementation, expanding the training set according to the analysis result or the calculation result includes:

[0017] Determining non-essential components in the sample based on the analysis results and preset non-essential identification rules;

[0018] The non-essential components are removed from the sample to construct a new sample.

[0019] In one possible implementation, expanding the training set according to the analysis result or the calculation result includes:

[0020] Determining at least one high-frequency interjection based on the analysis result;

[0021] At least one of the high-frequency interjections is inserted into the sample to construct a new sample.

[0022] In one possible implementation, expanding the training set according to the analysis result or the calculation result includes:

[0023] According to the calculation result, the probability of the two samples being the context is determined. If the probability exceeds a preset probability threshold and the sample length after the two samples are combined does not exceed a preset length threshold, the sample after the two samples are combined is used as a new training sample.

[0024] A second aspect of the present application provides a Chinese typo correction data enhancement device, comprising:

[0025] An acquisition module is used to obtain a training set for correcting typos;

[0026] An analysis module, configured to perform dependency syntax analysis on the samples in the training set to obtain analysis results;

[0027] A calculation module, configured to combine two samples in the training set, calculate the probability that the two samples belong to the context, and obtain a calculation result;

[0028] An expansion module is used to expand the training set according to the analysis result or the calculation result.

[0029] In one possible implementation, the analysis module is specifically configured to:

[0030] Segmenting the samples in the training set and tagging each word with a part-of-speech tag;

[0031] Perform dependency syntactic analysis on each word in the sample to obtain the analysis results.

[0032] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect of the present application.

[0033] A fourth aspect of the present application provides a computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method described in the first aspect of the present application.

[0034] Compared to existing technologies, the Chinese typo correction data enhancement method provided in this application obtains a typo correction training set; performs dependency syntactic analysis on samples in the training set to obtain analysis results; combines samples in the training set with each other to calculate the probability that the two samples belong to the context to obtain calculation results; and expands the training set based on the analysis results or the calculation results. In this solution, based on the existing training set, the original samples are segmented or expanded through dependency syntactic analysis and context judgment, thereby increasing training data within a unique domain and improving the typo correction effect in the corresponding domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0036] Figure 1 A flowchart of a Chinese typo correction data enhancement method provided by some embodiments of the present application is shown;

[0037] Figure 2 One of the schematic diagrams showing the dependency parsing results provided by this application;

[0038] Figure 3 The second schematic diagram shows the dependency parsing result provided by this application;

[0039] Figure 4 A schematic diagram of a Chinese typo correction data enhancement device provided by some embodiments of the present application is shown;

[0040] Figure 5A schematic diagram of an electronic device provided in some embodiments of the present application is shown;

[0041] Figure 6 A schematic diagram of a computer-readable storage medium provided in some embodiments of the present application is shown. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which this application belongs.

[0044] In addition, the terms "first" and "second" are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0045] The existing data augmentation method is to crawl a large amount of Chinese text from various web pages, then divide the text into sentences. The processed sentences are considered to be completely correct. Finally, the confusion set shown in Table 1 below is used to replace a Chinese character or word in the sentence with an easily confused Chinese character or word, thereby constructing an incorrect sentence.

[0046] Table 1: Confusion set

[0047]

[0048] However, the quality of texts collected on web pages is not high, and they may contain errors. Moreover, most of them are texts in general fields, and the effect of correcting typos in a specific field is limited.

[0049] In view of this, the embodiments of the present application provide a method and device for enhancing Chinese typo correction data, as well as an electronic device and a storage medium based on the method. To further illustrate the solution of the embodiments of the present application, the following description will be combined with the accompanying drawings. It is understood that in the following embodiments, the same or corresponding content can be referenced to each other, and for the sake of simplicity, it will not be repeated in detail.

[0050] Please refer to Figure 1 , which shows a flowchart of a Chinese typo correction data augmentation method provided by some embodiments of the present application, as Figure 1 shown, the method includes the following steps S101 - S104:

[0051] S101. Obtain a typo correction training set.

[0052] The training set includes a lot of typo correction corpora. The purpose of the present application is to construct a more accurate Chinese typo correction corpus that is closer to the distribution of the test set through the original typo correction training set and the dependency syntax tree. The present application mainly focuses on data augmentation based on the original training set, which can be used as a supplement to the previous data augmentation using a large amount of web corpora.

[0053] The typo correction corpus contains incorrect sentences and correct sentences. For example: "I think they have the heart that is not afraid of solid difficulties and is brave in facing them, which we who are not in the disaster area do not have." and "I think they have the heart that is not afraid of difficulties and is brave in facing them, which we who are not in the disaster area do not have.", the lengths of the two sentences are the same, and the original sentence contains some errors, such as "solid difficulties".

[0054] S102. Perform dependency syntax analysis on the samples in the training set to obtain an analysis result.

[0055] Specifically, step S102 can be implemented as: segment the samples in the training set and perform词性标注 (pos tagging) on each word; perform dependency syntax analysis on each word in the sample to obtain an analysis result.

[0056] For example, for the input correct sentence "I think they have the heart that is not afraid of difficulties and is brave in facing them, which we who are not in the disaster area do not have.", first perform word segmentation, then perform词性标注 (pos tagging), and finally perform dependency syntax analysis. The above analysis result includes the词性标注 (pos tagging) result and the dependency syntax analysis result.

[0057] Pos tagging gives the词性 (pos) of each word. For example, PRON is a pronoun, VERB is a verb, ADV is an adverb, PUNCT is a punctuation mark, etc.

[0058] Dependency syntax analysis mainly gives the dependency relationship between words. As Figure 2 shown, the arrow points from the governing word to the subordinate word, where ROOT represents the central word, nsubj represents the subject - predicate relationship, donj represents the verb - object relationship, ccomp represents the complement of the sentence, conj represents the connection relationship, and so on.

[0059] There are many existing dependency syntax analysis tools, such as corenlp, spacy, etc. Figure 2 It should be noted that the term "词性标注" in the original text is directly translated as "词性标注" in the English translation for the purpose of maintaining consistency with the original text. In a more formal English expression, it can be translated as "pos tagging".The dependency syntactic analysis results are obtained using the spacy tool. In addition to using existing tools, other methods can also be used to implement this function, and this application does not limit this. Table 2 below lists the part of speech of each word in the sentence and the results of the dependency syntactic analysis.

[0060] Table 2: Sentence part-of-speech tagging and dependency syntax analysis results

[0061]

[0062] S103 , combining samples in the training set in pairs, calculating the probability that the two samples belong to the context, and obtaining a calculation result.

[0063] The pre-trained BERT language model has a pre-training task: determining whether two sentences are in context and outputting the probability that the two sentences are in context. Therefore, you can combine two samples from the training set and input them into the pre-trained BERT model to obtain the probability that the two samples are in context. Of course, other methods can also be used to determine whether two samples are in context, and this application does not limit this.

[0064] S104: Expand the training set according to the analysis result or the calculation result.

[0065] In practical applications, the samples in the training set may be segmented into sentences or expanded according to the analysis result of step S102 or the calculation result of step S103.

[0066] In some embodiments, step S104 can be implemented as follows: determining the independent sentence components in the sample based on the analysis results and preset independent sentence recognition rules; removing the independent sentence components from the sample or retaining only the independent sentence components to construct a new sample.

[0067] In this embodiment, the analysis results of step S102 are used to set rules to identify independent sentence components in the sentence, and new samples are constructed by removing independent sentence components or retaining only independent sentence components. It should be noted that the part where the incorrect Chinese characters are located is retained. For example, the incorrect and correct sentence pairs, "I think they have the courage to face difficulties, which is something we don't have when we are not in the disaster area." and "I think they have the courage to face difficulties, which is something we don't have when we are not in the disaster area." According to the above analysis results, the independent sentence components labeled with ccomp are separated or retained, and two sentence pairs containing incorrect Chinese characters and one sentence pair that does not contain incorrect Chinese characters can be obtained, as shown in Table 3 below.

[0068] Rules for identifying independent components need to be set in advance. For example: ① A sample is segmented into independent components according to full stops, question marks, and exclamation marks; ② Sentences with dependent tags such as ccomp (complementary clause), conj (coordinate clause), and advcl (adverbial clause) as the root node are regarded as independent components.

[0069] Table 3

[0070]

[0071] In some embodiments, step S104 can also be implemented as: determining non-essential components in the sample according to the analysis result and a preset non-essential identification rule; removing the non-essential components in the sample to construct a new sample.

[0072] The preset non-essential identification rule includes non-essential components to be identified. Non-essential components can be time, place, modifiers, etc.

[0073] In this embodiment, the analysis result of step S102 and the preset non-essential identification rule are used to identify non-essential components in the sentence, and a new sample is constructed by removing non-essential components. For example, in the sentence in Table 4 below, the error is "拙壮" (assuming this is a wrong word). Figure 3 The following shows the dependency syntactic analysis result of the sample.

[0074] Table 4

[0075]

[0076] In some embodiments, step S104 can also be implemented as: determining at least one high-frequency parenthetical expression according to the analysis result; inserting at least one of the high-frequency parenthetical expressions into the sample to construct a new sample.

[0077] In this embodiment, some high-frequency parenthetical expressions can be counted in advance through the result of dependency syntactic analysis, such as phrases like "in fact", "I think", "here", "think about it", "sometimes", "generally speaking", "more precisely", etc., for constructing new samples. Then, according to the analysis result of the current sample, at least one high-frequency parenthetical expression that matches the context of the analysis result is determined, and then at least one of the high-frequency parenthetical expressions is inserted into the sample to construct a new sample. For example, in the sentence in Table 5 below, the error is "早以" (assuming this is a wrong phrase).

[0078] [[ID=3A]]

[0079] In some embodiments, step S104 can also be implemented as: based on the calculation result, determine the probability of the two samples being the context; if the probability exceeds a preset probability threshold and the length of the sample after the two samples are combined does not exceed a preset length threshold, then the sample after the two samples are combined is used as a new training sample.

[0080] In this embodiment, the calculation result of step S103 is used to obtain the probability of the two samples serving as context. If the probability exceeds a preset probability threshold and the length of the combined samples does not exceed a preset length threshold, the two samples are used as new training samples. For example, the preset probability threshold can be set to 90%, and the preset length threshold can be set to 50 words. The specific settings are not limited.

[0081] In this application, the samples constructed by the above two methods are used together with the original training set to train the typo correction model, which can correct more errors and thus improve the error correction effect.

[0082] Compared to the prior art, the aforementioned Chinese typo correction data enhancement method provided in the embodiments of the present application obtains a typo correction training set; performs dependency syntactic analysis on the samples in the training set to obtain an analysis result; combines the samples in the training set with each other to calculate the probability that the two samples belong to the context to obtain a calculation result; and expands the training set based on the analysis result or the calculation result. In this solution, based on the existing training set, the original samples are segmented or expanded through dependency syntactic analysis and context judgment, thereby increasing training data within a unique domain and improving the typo correction effect in the corresponding domain.

[0083] In the above embodiment, a method for enhancing Chinese typo correction data is provided. Correspondingly, the present application also provides a device for enhancing Chinese typo correction data. Figure 4 , which shows a schematic diagram of a Chinese typo correction data enhancement device provided by some embodiments of the present application. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment. The device embodiment described below is merely illustrative.

[0084] like Figure 4 As shown, the Chinese typo correction data enhancement device 10 may include:

[0085] An acquisition module 101 is used to acquire a typo correction training set;

[0086] An analysis module 102 is configured to perform dependency syntax analysis on the samples in the training set to obtain analysis results;

[0087] A calculation module 103 is used to combine two samples in the training set, calculate the probability that the two samples belong to the context, and obtain a calculation result;

[0088] The expansion module 104 is configured to expand the training set according to the analysis result or the calculation result.

[0089] In a possible implementation, the analysis module 102 is specifically configured to:

[0090] Segmenting the samples in the training set and tagging each word with a part-of-speech tag;

[0091] Perform dependency syntactic analysis on each word in the sample to obtain the analysis results.

[0092] In a possible implementation, the expansion module 104 is specifically configured to:

[0093] Determining independent sentence components in the sample based on the analysis results and a preset independent sentence recognition rule;

[0094] The independent sentence components are removed from the sample or only the independent sentence components are retained to construct a new sample.

[0095] In a possible implementation, the expansion module 104 is specifically configured to:

[0096] Determining non-essential components in the sample based on the analysis results and preset non-essential identification rules;

[0097] The non-essential components are removed from the sample to construct a new sample.

[0098] In a possible implementation, the expansion module 104 is specifically configured to:

[0099] Determining at least one high-frequency interjection based on the analysis result;

[0100] At least one of the high-frequency interjections is inserted into the sample to construct a new sample.

[0101] In a possible implementation, the expansion module 104 is specifically configured to:

[0102] According to the calculation result, the probability of the two samples being the context is determined. If the probability exceeds a preset probability threshold and the sample length after the two samples are combined does not exceed a preset length threshold, the sample after the two samples are combined is used as a new training sample.

[0103] The Chinese typo correction data enhancement device 10 provided in the embodiment of the present application is based on the same inventive concept as the Chinese typo correction data enhancement method provided in the aforementioned embodiment of the present application and has the same beneficial effects.

[0104] The embodiments of the present application also provide an electronic device corresponding to the Chinese typo correction data enhancement method provided in the aforementioned embodiments, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the aforementioned Chinese typo correction data enhancement method.

[0105] Please refer to Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and when the processor 200 runs the computer program, it executes the Chinese typo correction data enhancement method provided in any of the aforementioned embodiments of the present application.

[0106] The electronic device provided in the embodiment of the present application and the Chinese typo correction data enhancement method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0107] The present application also provides a computer-readable storage medium corresponding to the Chinese typo correction data enhancement method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the Chinese typo correction data enhancement method provided by any of the aforementioned embodiments.

[0108] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0109] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the Chinese typo correction data enhancement method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and description of the present application.

Claims

1. A data augmentation method for Chinese typos correction, It is characterized in that include: Get the typo correction training set; Performing dependency syntactic analysis on the samples in the training set to obtain analysis results; Combining samples in the training set in pairs, calculating the probability that the two samples belong to the context, and obtaining a calculation result; The training set is expanded according to the analysis result or the calculation result.

2. The method according to claim 1, It is characterized in that The performing dependency syntactic analysis on the samples in the training set to obtain analysis results includes: Segmenting the samples in the training set and tagging each word by part of speech; Perform dependency syntactic analysis on each word in the sample to obtain the analysis results.

3. The method according to claim 1, It is characterized in that The expanding the training set according to the analysis result or the calculation result includes: Determine the independent sentence components in the sample according to the analysis results and the preset independent sentence recognition rules; The independent sentence components are removed from the sample or only the independent sentence components are retained to construct a new sample.

4. The method according to claim 1, It is characterized in that The expanding the training set according to the analysis result or the calculation result includes: Determine the non-essential components in the sample according to the analysis results and the preset non-essential identification rules; The non-essential components are removed from the sample to construct a new sample.

5. The method according to claim 1, It is characterized in that The expanding the training set according to the analysis result or the calculation result includes: According to the analysis result, determining at least one high-frequency interjection; At least one of the high-frequency interjections is inserted into the sample to construct a new sample.

6. The method according to claim 1, It is characterized in that The expanding the training set according to the analysis result or the calculation result includes: According to the calculation result, the probability of the two samples being the context is determined. If the probability exceeds a preset probability threshold and the length of the sample after the two samples are combined does not exceed a preset length threshold, the sample after the two samples are combined is used as a new training sample.

7. A data enhancement device for correcting Chinese typos, It is characterized in that include: An acquisition module is used to obtain a training set for correcting typos; An analysis module, used to perform dependency syntax analysis on the samples in the training set to obtain analysis results; A calculation module, used for combining samples in the training set in pairs, calculating the probability that the two samples belong to the context, and obtaining a calculation result; An expansion module is used to expand the training set according to the analysis result or the calculation result.

8. The device according to claim 7, It is characterized in that The analysis module is specifically used for: Segmenting the samples in the training set and tagging each word by part of speech; Perform dependency syntactic analysis on each word in the sample to obtain the analysis results.

9. An electronic device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, It is characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 6.

Images