Pinyin error correction method

Abstract

This invention provides a Pinyin error correction method that simplifies the noise channel model used in most current Pinyin error correction algorithms by employing a real-time frequency counting method, effectively improving the efficiency and lightweighting of the algorithm. Furthermore, this invention uses a direct character substitution method instead of the traditional edit distance calculation method, avoiding frequent edit distance calculations in the Pinyin error correction algorithm. The dictionary database itself is localized to the individual user, resulting in a highly personalized, targeted, and small-scale dictionary that provides accurate candidate words. Compared to existing algorithms, this invention offers significant improvements in error checking rate, candidate word accuracy, and execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of machine learning model technology, and in particular to a method for correcting pinyin errors. Background Technology

[0002] Language input is indispensable in various human-computer interaction processes. When people input text, the machine can automatically check for errors and provide accurate candidate words, which brings great convenience. Due to the characteristics of the Chinese language, there are a variety of error correction schemes to choose from when inputting Chinese Pinyin, such as those based on Burkhard-Keller trees, keyboard layout, binary search trees, and noisy channel models.

[0003] Clearly, these pinyin correction methods either start from the structure of pinyin or evolve from English spelling correction. The noisy channel model (Shannon 1948) has been successfully applied in a wide range of fields, especially in communications, including pinyin correction.

[0004] The noisy channel model for pinyin error correction treats errors caused by noise interference in the transmission channel from the keyboard to the system as errors that occur when people mistype pinyin or press the wrong keys. In terms of error detection and correction, the noisy channel model is based on the likelihood function from Bayes' theorem.

[0005] This shows that among the many unique pinyin correction methods available, they all stem from a problem-solving model. People have evolved from this initial model to improve pinyin correction capabilities and the accuracy of alternative words. These improvements are highly significant and valuable, with multiple methods adapting to the spelling and input habits of different groups. Summary of the Invention

[0006] In view of the above-mentioned practical needs, the purpose of this invention is to provide a Pinyin error correction method that improves the Pinyin error correction model for noisy channels. By using a real-time frequency counting method, it simplifies the noisy channel model used in most current Pinyin error correction algorithms, effectively improving the efficiency of the error correction algorithm; at the same time, it makes the algorithm itself lightweight. Furthermore, this invention uses a direct character substitution method to replace the edit distance calculation method in traditional Pinyin error correction algorithms, avoiding frequent calculations of edit distance. The dictionary database used by this Pinyin error correction method is localized to the individual user, resulting in high personalization, strong targeting, small scale, and the ability to provide accurate candidate words. Compared with existing algorithms, this invention's method significantly improves error checking rate, candidate word accuracy, and execution efficiency.

[0007] To achieve the above and other related objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for correcting pinyin errors, including the following steps:

[0009] S1) Input a dictionary of commonly used Chinese characters with pinyin, where dictionary D contains about 10,000 commonly used pinyin characters with homophones. Each pinyin character is a one-letter string, which is not case-sensitive and will be referred to as a pinyin string below.

[0010] S2) Establish a dictionary of popular pinyin characters G. The method for establishing dictionary G is as follows: record each pinyin character entered by the user and its frequency of occurrence, and sort them in descending order of frequency of occurrence.

[0011] S3) Construct a Bayesian dictionary S. Each record in Bayesian dictionary S contains the pinyin strings of two Chinese characters and their frequencies. Let the current pinyin string be A, and the string preceding A be B. The method for constructing Bayesian dictionary S is as follows: if B is an empty string, ignore A; if B is not an empty string, treat BA as a pinyin string. If BA is in Bayesian dictionary S, increment the frequency of BA by 1; if BA is not in Bayesian dictionary S, add BA to Bayesian dictionary S and mark its frequency as 1.

[0012] S4) Input the pinyin string x;

[0013] S5) Match x in the pinyin dictionary D. If x is contained in the pinyin dictionary D, no error correction is needed, and return directly to step S4.

[0014] S6) Let l be the length of x, and let x be... i Let x be the i-th letter of the pinyin string, where 1 ≤ i ≤ l;

[0015] S7) Establish a candidate pinyin string set Y, which is initially an empty set;

[0016] S8) Starting from i=1 to i=1, repeat steps S81 to S85 below;

[0017] S81) Let Q be the letter x i Let m be the set of letters on adjacent keys on a keyboard, and let m be the length of set Q.

[0018] S82) Let T be the set of new pinyin strings after the i-th element x of the pinyin string x is replaced one by one with the elements in set Q. So the set T has m elements.

[0019] S83) Delete elements in set T that are not in the pinyin dictionary D;

[0020] S84) Merge set T into set Y;

[0021] S85)i increases by 1;

[0022] S9) Establish an extended candidate pinyin string set Z, initialized as an empty set;

[0023] S10) Let n be the length of set Y, and let y be... j (y j (∈Y) is the j-th element of Y. Obviously, 0≤j≤n. Repeat steps S101 to S117 from j=1 to j=n.

[0024] S101) If y j Let the first letter be 'c', and let t be y. j The string that replaces the first letter 'c' with 'ch', and if t is a string in the pinyin dictionary D, add t to Z;

[0025] S102) If y j The first two letters are 'ch', and t is y. j The string that replaces the first two letters 'ch' with 'c', and if t is a string in the pinyin dictionary D, add t to z;

[0026] S103) If y j The first letter is 's', and t is denoted as y. j The string that replaces the first letter 's' with 'sh', and if t is a string in the pinyin dictionary D, add t to z;

[0027] S104) If y j The first two letters are 'sh', and t is y. i The string that replaces the first two letters 'sh' with 's', and if t is a string in the pinyin dictionary D, add t to Z;

[0028] S105) If y j The first letter is 'z', and t is denoted as y. j The string that replaces the first letter 'z' with 'zh', and if t is a string in the pinyin dictionary D, add t to Z;

[0029] S106) If y j The first two letters are 'zh', and t is y. j The string that replaces the first two letters 'zh' with 'z', and if t is a string in the pinyin dictionary D, add t to z;

[0030] S107) If y j The first letter is 'f', and t is denoted as y. j The string that replaces the first letter 'f' with 'h', and if t is a string in the pinyin dictionary D, add t to Z;

[0031] S108) If y jThe first letter is 'h', and t is denoted as y. j The string that replaces the first letter 'h' with 'f', and if t is a string in the pinyin dictionary D, add t to Z;

[0032] S109) If y j The last two letters are 'an', and t is y. j The string that replaces the last two letters 'an' with 'ang', and if t is a string in the pinyin dictionary D, add t to z;

[0033] S110) If y i The last two letters are 'en', and t is y. j The string that replaces the last two letters 'en' with 'eng', and if t is a string in the pinyin dictionary D, add t to Z;

[0034] S111) If y j The last two letters are 'in', and t is y. j The string that replaces the last two letters 'in' with 'ing', and if t is a string in the pinyin dictionary D, add t to z;

[0035] S112) If y j The last two letters are 'on', and t is y. j The string that replaces the last two letters 'on' with 'ong', and if t is a string in the pinyin dictionary D, add t to z;

[0036] S113) If y j The last three letters are 'ang', and t is y. j The string that replaces the last three letters 'ang' with 'an', and if t is a string in the pinyin dictionary D, add t to Z;

[0037] S114) If y j The last three letters are 'eng', and t is y. j The string that replaces the last three letters 'eng' with 'en', and if t is a string in the pinyin dictionary D, add t to z;

[0038] S115) If y j The last three letters are 'ing', and t is y. j The string that replaces the last three letters 'ing' with 'in', and if t is a string in the pinyin dictionary D, add t to Z;

[0039] S116) If y i The last three letters are 'ong', and t is y. jThe string that replaces the last three letters 'ong' with 'in', and if t is a string in the pinyin dictionary D, add t to Z;

[0040] S117)j increases by 1;

[0041] S11) Merge set z into set Y. If x is the first pinyin string of the input sentence, sort set Y in descending order according to the hot character pinyin dictionary G. If x is not the first pinyin string of the input sentence, denote w as the previous pinyin string of x, denote u as the string connecting pinyin string x to the end of pinyin string w, denote f as the frequency of u in the Bayesian dictionary S, and denote r1, r2, ..., r h Let m be the frequency of the pinyin string in the Bayesian dictionary S that ends with x, and let m be the cumulative sum of the frequencies of all words in the Bayesian dictionary s. Calculate the conditional probability. and Where α takes the value (0.5, 0.9), due to conditional probability Since the denominators are the same, division by m is not required in actual calculations. Set Y is sorted in descending order of size;

[0042] S12) Output the top three candidate pinyin characters in descending order of set Y.

[0043] S13) Record the candidate pinyin characters selected by the user and update them in dictionary G and dictionary S.

[0044] Optionally, the steps of updating the dictionary G in step S11) are as follows: if the selected word is not in G, add the selected word to G and record its frequency as 1; if the selected word is in G, increase the frequency of the selected word in G by 1.

[0045] Optionally, the steps of updating the dictionary S in step S11) are as follows: if the selected word is not in S, add the selected word to S and record its frequency as 1; if the selected word is in S, increase the frequency of the selected word in S by 1; and increase the frequency of words in S that contain the selected word by 1.

[0046] The beneficial effects of this invention are as follows: This invention simplifies the noise channel model used in most current pinyin error correction algorithms by employing a real-time frequency counting method, effectively improving the efficiency of the error correction algorithm. In contrast, error correction algorithms such as search optimization (Martins and Silva, 2004; Gao et al., 2010), machine translation (Belinkov and Bisk, 2017), and part-of-speech tagging (Van Rooy and Schafer, 2002; Sak-aguchi et al., 2012) require a comprehensive understanding of word similarity, language modeling, and reasoning, resulting in high complexity and large scale. This invention is a very lightweight method compared to the scale and complexity of these algorithms. Furthermore, this invention uses a direct character substitution method instead of the traditional edit distance calculation method, avoiding the frequent calculation of edit distances required by other pinyin error correction algorithms. For the establishment and management of dictionary databases, which are localized for individual users, this invention offers high personalization, strong targeting, and small scale. Attached Figure Description

[0047] Figure 1 This is a system block diagram of a pinyin error correction method.

[0048] Figure 2 This is a schematic diagram illustrating the implementation of a pinyin error correction method. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.

[0050] In one embodiment, a pinyin error correction method has the following overall architecture: Figure 1 As shown; the implementation process is as follows: Figure 2 As shown, the steps include:

[0051] S1) Input a dictionary of commonly used Chinese characters with pinyin, where dictionary D contains about 10,000 commonly used pinyin characters with homophones. Each pinyin character is a one-letter string, which is not case-sensitive and will be referred to as a pinyin string below.

[0052] S2) Establish a dictionary of popular pinyin characters G. The method for establishing dictionary G is as follows: record each pinyin character entered by the user and its frequency of occurrence, and sort them in descending order of frequency of occurrence.

[0053] S3) Construct a Bayesian dictionary S. Each record in Bayesian dictionary S contains the pinyin strings of two Chinese characters and their frequencies. Let the current pinyin string be A, and the string preceding A be B. The method for constructing Bayesian dictionary S is as follows: if B is an empty string, ignore A; if B is not an empty string, treat BA as a pinyin string. If BA is in Bayesian dictionary S, increment the frequency of BA by 1; if BA is not in Bayesian dictionary S, add BA to Bayesian dictionary S and mark its frequency as 1.

[0054] S4) Input the pinyin string x;

[0055] S5) Match x in the pinyin dictionary D. If x is contained in the pinyin dictionary D, no error correction is needed, and return directly to step S4.

[0056] S6) Let l be the length of x, and let x be... i Let x be the i-th letter of the pinyin string, where 1 ≤ i ≤ l;

[0057] S7) Establish a candidate pinyin string set Y, which is initially an empty set;

[0058] S8) Starting from i=1 to i=1, repeat steps S81 to S85 below;

[0059] S81) Let Q be the letter x i Let m be the set of letters on adjacent keys on a keyboard, and let m be the length of set Q.

[0060] S82) Let T be the i-th element of the pinyin string x. i The set of new pinyin strings is formed by replacing each element in set Q one by one, so set T has m elements.

[0061] S83) Delete elements in set T that are not in the pinyin dictionary D;

[0062] S84) Merge set T into set Y;

[0063] S85)i increases by 1;

[0064] S9) Establish an extended candidate pinyin string set Z, initialized as an empty set;

[0065] S10) Let n be the length of set Y, and let y be... j (y j (∈Y) is the j-th element of Y. Obviously, 0≤j≤n. Repeat steps S101 to S117 from j=1 to j=n.

[0066] S101) If y j Let the first letter be 'c', and let t be y. jThe string that replaces the first letter 'c' with 'ch', and if t is a string in the pinyin dictionary D, add t to z;

[0067] S102) If y j The first two letters are 'ch', and t is y. j The string that replaces the first two letters 'ch' with 'c', and if t is a string in the pinyin dictionary D, add t to Z;

[0068] S103) If y j The first letter is 's', and t is denoted as y. j Replace the first letter 's' with 'sh' in the string. If t is a string in the pinyin dictionary D, add t to Z.

[0069] S104) If y j The first two letters are 's', and t is y. j Replace the first two letters 's' with the string containing 's'. If t is a string in the pinyin dictionary D, add t to Z.

[0070] S105) If y j The first letter is 'z', and t is denoted as y. i Replace the string whose first letter 'z' is 'zh'. If t is a string in the pinyin dictionary D, add t to z.

[0071] S106) If y j The first two letters are 'zh', and t is y. j The string that replaces the first two letters 'zh' with 'z', and if t is a string in the pinyin dictionary D, add t to Z;

[0072] S107) If y j The first letter is 'f', and t is denoted as y. j The string that replaces the first letter 'f' with 'h', and if t is a string in the pinyin dictionary D, add t to Z;

[0073] S108) If y j The first letter is 'h', and t is denoted as y. j The string that replaces the first letter 'h' with 'f', and if t is a string in the pinyin dictionary D, add t to Z;

[0074] S109) If y j The last two letters are 'an', and t is y. j The string that replaces the last two letters 'an' with 'ang', and if t is a string in the pinyin dictionary D, add t to Z;

[0075] S110) If yj The last two letters are 'en', and t is y. j The string that replaces the last two letters 'en' with 'eng', and if t is a string in the pinyin dictionary D, add t to Z;

[0076] S111) If y j The last two letters are 'in', and t is y. j The string that replaces the last two letters 'in' with 'ing', and if t is a string in the pinyin dictionary D, add t to Z;

[0077] S112) If y j The last two letters are 'on', and t is y. j The string that replaces the last two letters 'on' with 'ong', and if t is a string in the pinyin dictionary D, add t to Z;

[0078] S113) If y j The last three letters are 'ang', and t is y. j The string that replaces the last three letters 'ang' with 'an', and if t is a string in the pinyin dictionary D, add t to Z;

[0079] S114) If y j The last three letters are 'eng', and t is y. j The string that replaces the last three letters 'eng' with 'en', and if t is a string in the pinyin dictionary D, add t to Z;

[0080] S115) If y j The last three letters are 'ing', and t is y. j The string that replaces the last three letters 'ing' with 'in', and if t is a string in the pinyin dictionary D, add t to Z;

[0081] S116) If y j The last three letters are 'ong', and t is y. i The string that replaces the last three letters 'ong' with 'in', and if t is a string in the pinyin dictionary D, add t to Z;

[0082] S117)j increases by 1;

[0083] S11) Merge set z into set Y. If x is the first pinyin string of the input sentence, sort set Y in descending order according to the hot character pinyin dictionary G. If x is not the first pinyin string of the input sentence, denote w as the previous pinyin string of x, denote u as the string connecting pinyin string x to the end of pinyin string w, denote f as the frequency of u in the Bayesian dictionary S, and denote r1, r2, ..., r h Let m be the frequency of the pinyin string in the Bayesian dictionary S that ends with x, and let m be the cumulative sum of the frequencies of all words in the Bayesian dictionary s. Calculate the conditional probability. and Where α takes the value (0.5, 0.9), due to conditional probability Since the denominators are the same, division by m is not required in actual calculations. Set Y is sorted in descending order of size;

[0084] S12) Output the top three candidate pinyin characters in descending order of set Y.

[0085] S13) If the selected word is not in G, add the selected word to G and record its frequency as 1; if the selected word is in G, increase the frequency of the selected word in G by 1; if the selected word is not in S, add the selected word to S and record its frequency as 1; if the selected word is in S, increase the frequency of the selected word in S by 1; words in S that contain the selected word increase their frequency by 1.

[0086] The beneficial effects of this invention are as follows: This invention simplifies the noise channel model used in most current pinyin error correction algorithms by employing a real-time frequency counting method, effectively improving the efficiency of the error correction algorithm. In contrast, error correction algorithms such as search optimization (Martins and Silva, 2004; Gao et al., 2010), machine translation (Belinkov and Bisk, 2017), and part-of-speech tagging (Van Rooy and Schafer, 2002; Sak-aguchi et al., 2012) require a comprehensive understanding of word similarity, language modeling, and reasoning, resulting in high complexity and large scale. This invention is a very lightweight method compared to the scale and complexity of these algorithms. Furthermore, this invention uses a direct character substitution method instead of the traditional edit distance calculation method, avoiding the frequent calculation of edit distances required by other pinyin error correction algorithms. For the establishment and management of dictionary databases, which are localized for individual users, this invention offers high personalization, strong targeting, and small scale.

[0087] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for correcting pinyin errors, characterized in that, Including the following steps: S1) Input the commonly used Chinese character pinyin dictionary D; S2) Establish a dictionary of popular pinyin characters (G); S3) Construct a Bayesian dictionary S; S4) Input the pinyin string x; S5) Match x in the pinyin dictionary D. If x is contained in the pinyin dictionary D, no error correction is needed, and return directly to step S4. S6) Let l be the length of x, and let xi be the i-th letter of the pinyin string x, 1≤i≤l; S7) Establish a candidate pinyin string set Y, which is initially an empty set; S8) Starting from i=1 to i=1, repeat steps S81 to S85 below; S81) Let Q be the set of letters xi on adjacent keys on the keyboard, and let m be the length of set Q; S82) Let T be the set of new pinyin strings after replacing each i-th element xi of the pinyin string x with the elements in set Q. Therefore, set T has m elements. S83) Delete elements in set T that are not in the pinyin dictionary D; S84) Merge set T into set Y; S85)i increases by 1; S9) Establish an extended candidate pinyin string set Z, initialized as an empty set; S10) Let n be the length of set Y, and let yj be the j-th element of Y, yj ∈ Y, 0≤j≤n. Repeat steps S101 to S117 from j=1 to j=n. S101) If the first letter of yj is 'c', denote t as the string in which the first letter of yj is replaced with 'ch'. If t is a string in the pinyin dictionary D, add t to z. S102) If the first two letters of yj are 'ch', denote t as the string in which the first two letters of yj are replaced with 'c'. If t is a string in the pinyin dictionary D, add t to Z. S103) If the first letter of yj is 's', denote t as the string in which the first letter of yj is replaced with 'sh'. If t is a string in the pinyin dictionary D, add t to Z. S104) If the first two letters of yj are 'sh', denote t as the string in which the first two letters of yj are replaced with 's'. If t is a string in the pinyin dictionary D, add t to Z. S105) If the first letter of yj is 'z', denote t as the string in which yj replaces the first letter 'z' with 'zh'. If t is a string in the pinyin dictionary D, add t to z. S106) If the first two letters of yj are 'zh', denote t as the string in which the first two letters of yj are replaced with 'z'. If t is a string in the pinyin dictionary D, add t to Z. S107) If the first letter of yj is 'f', denote t as the string in which the first letter of yj is replaced with 'h'. If t is a string in the pinyin dictionary D, add t to Z. S108) If the first letter of yj is 'h', denote t as the string in which the first letter of yj is replaced with 'f'. If t is a string in the pinyin dictionary D, add t to Z. S109) If the last two letters of yj are 'an', denote t as the string in which yj replaces the last two letters 'an' with 'ang'. If t is a string in the pinyin dictionary D, add t to Z. S110) If the last two letters of yj are 'en', denote t as the string in which the last two letters of yj are replaced with 'eng'. If t is a string in the pinyin dictionary D, add t to Z. S111) If the last two letters of yj are 'in', denote t as the string in which the last two letters of yj are replaced with 'ing'. If t is a string in the pinyin dictionary D, add t to Z. S112) If the last two letters of yj are 'on', denote t as the string in which the last two letters of yj are replaced with 'ong'. If t is a string in the pinyin dictionary D, add t to Z. S113) If the last three letters of yj are 'ang', denote t as the string in which the last three letters of yj are replaced with 'an'. If t is a string in the pinyin dictionary D, add t to Z. S114) If the last three letters of yj are 'eng', denote t as the string in which the last three letters of yj are replaced with 'en'. If t is a string in the pinyin dictionary D, add t to Z. S115) If the last three letters of yj are 'ing', denote t as the string in which the last three letters of yj are replaced with 'in'. If t is a string in the pinyin dictionary D, add t to Z. S116) If the last three letters of yj are 'ong', denote t as the string in which the last three letters of yj are replaced with 'in'. If t is a string in the pinyin dictionary D, add t to z. S117)j increases by 1; S11) Merge set Z into set Y; S12) Output the three candidate pinyin characters that are first in the descending sorted set Y; S13) Record the candidate pinyin characters selected by the user and update them in dictionary G and dictionary S.

2. The pinyin error correction method according to claim 1, characterized in that, Step S1) The dictionary D contains 10,000 commonly used homophones in pinyin, each pinyin character being a one-letter string, and is not case-sensitive.

3. The pinyin error correction method according to claim 1, characterized in that, The method for establishing dictionary G in step S2) is as follows: record each pinyin character input by the user and its frequency of occurrence, and sort them in descending order of frequency of occurrence.

4. The pinyin error correction method according to claim 1, characterized in that, Each record in the Bayesian dictionary S described in step S3) contains the pinyin strings of two Chinese characters and their frequencies. Let the current pinyin string be A, and the string preceding A be B. The Bayesian dictionary S is constructed as follows: if B is an empty string, ignore A; if B is not an empty string, treat BA as a pinyin string. If BA is in the Bayesian dictionary S, increment the frequency of BA by 1; if BA is not in the Bayesian dictionary S, add BA to the Bayesian dictionary S and mark its frequency as 1.

5. The pinyin error correction method according to claim 1, characterized in that, In step S11), the set Z is merged into the set Y. If x is the first pinyin string of the input sentence, the set Y is sorted in descending order according to the hot word pinyin dictionary G. If x is not the first pinyin string of the input sentence, w is the previous pinyin string of x, u is the string that pinyin string x is connected to the end of pinyin string w, f is the frequency of u in the Bayesian dictionary S, r1, r2, ..., rh are the frequencies of pinyin strings that contain x at the end in the Bayesian dictionary S, and m is the cumulative sum of the frequencies of all words in the Bayesian dictionary s. The conditional probability is calculated, where α takes the value (0.5, 0.9). Since the denominator of the conditional probability is the same, it is not necessary to divide by m in the actual calculation. The set Y is sorted in descending order according to its size.

6. The pinyin error correction method according to claim 1, characterized in that, The dictionary G mentioned in step S13) is as follows: if the selected word is not in G, add the selected word to G and record its frequency as 1; if the selected word is in G, increase the frequency of the selected word in G by 1.

7. The pinyin error correction method according to claim 1, characterized in that, The dictionary S mentioned in step S13) consists of the following steps: if the selected word is not in S, add the selected word to S and record its frequency as 1; if the selected word is in S, increase the frequency of the selected word in S by 1; and increase the frequency of words in S that contain the selected word by 1.

Images