Target domain sample library establishment method and device, computer device, and storage medium

Abstract

This application relates to a method, apparatus, computer device, storage medium, and computer program product for establishing a target domain sample library. The method includes: obtaining the confusion correlation degree between each target letter and a confused letter; obtaining the character to be replaced in training text of the target domain, and its corresponding initial pinyin; performing confusion processing on the initial pinyin to obtain multiple reference pinyin; calculating the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree; determining the character to be replaced from each of the reference pinyin based on the edit distance; generating a confused character based on the character to be replaced; replacing the character to be replaced in the training text with the confused character to obtain multiple confused texts; establishing the association relationship between the training text and each of the confused texts to obtain an error correction sample library. The text error correction model trained using the error correction sample library is targeted and practical, improving the accuracy of error correction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, computer equipment, and storage medium for establishing a sample library in a target domain. Background Technology

[0002] With the development of artificial intelligence, automatic speech recognition technology has been widely used in various industries. Automatic speech recognition technology refers to the technology of converting acquired speech into text content. In order to improve the accuracy of the text, it is necessary to use text correction models to correct the text content.

[0003] In existing technologies, a public text correction model is obtained by training a neural network model using a public sample library. However, the accuracy of using this public text correction model to correct speech recognition text in a specialized field is low. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, system, computer equipment, and computer-readable storage medium for establishing a target domain sample library to address the aforementioned technical problems and improve the accuracy of text error correction in the target domain.

[0005] This application provides a method for establishing a sample library in a target domain, the method comprising:

[0006] Obtain the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0007] Obtain the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced;

[0008] The initial pinyin is obfuscated to obtain multiple reference pinyin corresponding to the character to be replaced;

[0009] Based on the confusion correlation degree, the edit distance between the initial pinyin and each of the reference pinyin is calculated;

[0010] Based on the edit distance, the pinyin to be replaced is determined from each of the reference pinyin, and a confused character corresponding to the pinyin to be replaced is generated based on the pinyin to be replaced. The character to be replaced in the training text is replaced with the confused character to obtain multiple confused texts corresponding to the training text.

[0011] Establish the association between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

[0012] In one embodiment, obtaining the confusion correlation degree between each target letter and the confused letter includes:

[0013] Obtain the target pinyin and the corresponding confused pinyin from a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters.

[0014] Based on the positional relationship between each target letter and each confused letter, a reference association pair between the target letter and the confused letter is established;

[0015] The first statistical value is obtained by counting the number of reference association pairs that have the same target letter and the same confusion letter.

[0016] Based on the first statistical value, the number of reference association pairs corresponding to the same target letter in the reference association pairs is counted to obtain the second statistical value;

[0017] Calculate the ratio between the first statistical value and the second statistical value to obtain the confusion correlation between the target letter and the confused letter.

[0018] In one embodiment, establishing a reference association pair between the target letters and the confused letters based on the positional relationship between each target letter and each confused letter includes:

[0019] The first target letter of the target pinyin is obtained as the current target letter, and the first confused letter of the confused pinyin is obtained as the current confused letter.

[0020] Create a reference association pair by combining the current target letter and the current confused letter;

[0021] Obtain the next target letter of the target pinyin as the current target letter, and obtain the next confusing letter of the confused pinyin as the current confusing letter. If the next target letter of the target pinyin is empty and the next confusing letter of the confused pinyin is not empty, then use the target character as the current target letter. If the next confusing letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then use the target character as the current confusing letter. Repeat the step of forming a reference association pair between the current target letter and the current confusing letter until both the next target letter of the target pinyin and the next confusing letter of the confused pinyin are empty.

[0022] In one embodiment, calculating the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree includes:

[0023] Establish multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin;

[0024] Based on the obfuscation correlation, query the target obfuscation correlation corresponding to each target association pair;

[0025] The confusion correlation degree of each target is statistically analyzed to obtain the edit distance between the initial pinyin and the reference pinyin.

[0026] In one embodiment, the step of obfuscating the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced includes:

[0027] Obtain the initial initial and initial final corresponding to the initial pinyin;

[0028] Obtain a confusion pinyin rule base, and based on the confusion pinyin rule base, match the corresponding reference initials for the initial initials and the corresponding reference finals for the initial finals;

[0029] Based on the reference initials and reference finals, a reference pinyin corresponding to the character to be replaced is formed.

[0030] In one embodiment, the step of generating a confused character corresponding to the character to be replaced based on the pinyin to be replaced, and replacing the character to be replaced in the training text with the confused character, results in multiple confused texts corresponding to the training text, including:

[0031] The pinyin to be replaced is combined with each tone to obtain the target confused pinyin;

[0032] Based on the target confused pinyin, generate multiple confused characters;

[0033] The characters to be replaced in the training text are replaced with the obfuscated characters to obtain multiple reference texts;

[0034] Each of the reference texts is input into the text detection model, and the obfuscated texts corresponding to the training texts are obtained by filtering.

[0035] In one embodiment, obtaining the word to be replaced in the training text of the target domain includes:

[0036] Obtain training text in the target domain;

[0037] The training text is segmented into words to obtain multiple constituent words;

[0038] Keywords are obtained by filtering the constituent words;

[0039] Randomly select characters from the keywords and use them as the replacement characters in the training text.

[0040] This application also provides an apparatus for establishing a target domain sample library, the apparatus comprising:

[0041] The confusion correlation degree acquisition module is used to acquire the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0042] The module for obtaining characters to be replaced is used to obtain the characters to be replaced in the training text of the target domain, as well as the initial pinyin corresponding to the characters to be replaced;

[0043] The obfuscation module is used to obfuscate the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced;

[0044] The calculation module is used to calculate the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree;

[0045] The replacement module is used to determine the pinyin to be replaced from each of the reference pinyin based on the edit distance, generate a confused character corresponding to the pinyin to be replaced based on the pinyin to be replaced, replace the pinyin to be replaced in the training text with the confused character, and obtain multiple confused texts corresponding to the training text.

[0046] The association module is used to establish the association relationship between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

[0047] This application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0048] Obtain the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0049] Obtain the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced;

[0050] The initial pinyin is obfuscated to obtain multiple reference pinyin corresponding to the character to be replaced;

[0051] Based on the confusion correlation degree, the edit distance between the initial pinyin and each of the reference pinyin is calculated;

[0052] Based on the edit distance, the pinyin to be replaced is determined from each of the reference pinyin, and a confused character corresponding to the pinyin to be replaced is generated based on the pinyin to be replaced. The character to be replaced in the training text is replaced with the confused character to obtain multiple confused texts corresponding to the training text.

[0053] Establish the association between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

[0054] This application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0055] Obtain the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0056] Obtain the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced;

[0057] The initial pinyin is obfuscated to obtain multiple reference pinyin corresponding to the character to be replaced;

[0058] Based on the confusion correlation degree, the edit distance between the initial pinyin and each of the reference pinyin is calculated;

[0059] Based on the edit distance, the pinyin to be replaced is determined from each of the reference pinyin, and a confused character corresponding to the pinyin to be replaced is generated based on the pinyin to be replaced. The character to be replaced in the training text is replaced with the confused character to obtain multiple confused texts corresponding to the training text.

[0060] Establish the association between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

[0061] The method, apparatus, computer equipment, and storage medium for establishing the target domain sample library involve: obtaining the confusion correlation between target letters and confused letters in a public sample library; obtaining training text in the target domain; selecting characters to be replaced from the training samples; obtaining the initial pinyin of the characters to be replaced; confusing the initial pinyin to obtain multiple reference pinyin corresponding to the characters to be replaced; calculating the edit distance between the initial pinyin and each reference pinyin based on the confusion correlation; selecting the characters to be replaced from the multiple initial pinyin based on the edit distance; generating confused characters based on the characters to be replaced; replacing the characters to be replaced in the training text with the confused characters to obtain multiple confused texts; and establishing the association between the training text and the confused texts to obtain the error correction sample library. The initial pinyin is obfuscated to obtain reference pinyin, ensuring the quantity of the error correction sample library. The edit distance between the initial pinyin and the reference pinyin is calculated based on the obfuscation correlation of each letter in the public sample library. The reference pinyin with the shortest edit distance is selected as the pinyin to be replaced. The pinyin with the highest obfuscation degree is selected to increase the utilization rate of obfuscated text in the error correction sample library. The text error correction model is obtained by training a neural network model using the error correction sample library. The text error correction model is targeted and practical, improving the accuracy of error correction. Attached Figure Description

[0062] Figure 1 This is a diagram illustrating the application environment of a method for establishing a target domain sample library in one embodiment.

[0063] Figure 2 This is a flowchart illustrating a method for establishing a target domain sample library in one embodiment;

[0064] Figure 3 This is a flowchart illustrating the method for calculating obfuscation correlation in one embodiment;

[0065] Figure 4 This is a schematic diagram of the process of a pinyin confusion method in one embodiment;

[0066] Figure 5 This is a flowchart illustrating the process of generating obfuscated text in one embodiment;

[0067] Figure 6 This is a flowchart illustrating the process of selecting the character to be replaced in one embodiment;

[0068] Figure 7 This is a schematic diagram of the process for establishing a sample library in one embodiment;

[0069] Figure 8 A structural block diagram of a target domain sample library establishment device in one embodiment;

[0070] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The method for establishing a target domain sample library provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 sends a request to server 104 to establish an error correction sample library. Server 104 receives training text from the target domain, obtains the confusion correlation between target letters and confused letters in the public sample library, filters out characters to be replaced from the training samples, obtains the initial pinyin of the characters to be replaced, confuses the initial pinyin to obtain multiple reference pinyin corresponding to the characters to be replaced, calculates the edit distance between the initial pinyin and each reference pinyin based on the confusion correlation, filters out the pinyin to be replaced from the multiple initial pinyin based on the edit distance, generates confused characters based on the pinyin to be replaced, replaces the characters to be replaced in the training text with the confused characters, obtains multiple confused texts, establishes the association between the training text and the confused texts, and obtains the error correction sample library. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0073] In one embodiment, such as Figure 2 As shown, a method for establishing a target domain sample library is provided. This embodiment illustrates the method by applying it to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step 202: Obtain the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0075] Among them, the target letters refer to each pinyin letter in the pinyin of the character to be replaced. For example, the pinyin of "方" is fang, and the target letters are "f", "a", "n", and "g". The confusing letters refer to each pinyin letter in the pinyin of the confusing character used to replace the character to be replaced. For example, using "放" to replace "方", the pinyin of "放" is fang, and the confusing letters are "f", "a", "n", and "g". It can be understood that the target letters and the confusing letters can be the same or different. The confusion correlation degree can be understood as the probability that the target letter becomes the confusing letter. The public sample library refers to the publicly available and existing sample library. For example, the SIGHAN international Chinese word segmentation dataset, and there are multiple sample pairs in the public sample library. A sample pair can be understood as the correlation pair of the correct text and the incorrect text. For example, the correct text "我想要咨询存款期限的问题" and the corresponding incorrect text "我享要咨询存钱时间的方案", etc.

[0076] Specifically, calculate the confusion correlation degree corresponding to the target letter and the confusing letter according to the sample pairs included in the public sample library, and form the confusion correlation degree set corresponding to the public sample library with all the confusion correlation degrees corresponding to the target letter and the confusing letter.

[0077] Step 204, obtain the character to be replaced in the training text of the target field, and the initial pinyin corresponding to the character to be replaced.

[0078] Among them, the target field refers to the industry used. For example, the financial field, the communication field, the Internet field, etc. The training text refers to the text to be trained in the target field. It can be understood as the common sentences in the target field. For example, in the financial field, "我想了解现在的理财产品", "现在定期存款产品的利息", etc. The character to be replaced refers to the character selected from the training text for replacement. The character to be replaced can be screened according to the set rules or generated randomly. The character to be replaced can be one character or multiple characters. The initial pinyin refers to the correct pinyin of the character to be replaced.

[0079] Specifically, obtain the training text from the training sample library of the target field, select one or more characters to be replaced from the training text, and respectively obtain the initial pinyin corresponding to each character to be replaced.

[0080] Step 206, perform confusion processing on the initial pinyin to obtain multiple reference pinyins corresponding to the character to be replaced.

[0081] Among them, the confusion processing refers to performing operations such as replacement, deletion, and addition on one or more letters in the initial pinyin. The process of confusion processing can be set according to requirements. For example, the process of confusion processing can be set to replace the initial consonant or vowel in the pinyin. The reference pinyin refers to the pinyin obtained after performing confusion processing on the initial pinyin.

[0082] Specifically, perform obfuscation processing on the initial pinyin according to the set obfuscation processing rules. After the obfuscation processing, multiple reference pinyins corresponding to the characters to be replaced are obtained.

[0083] Step 208: Calculate the edit distance between the initial pinyin and each reference pinyin based on the obfuscation correlation degree.

[0084] The edit distance refers to the steps for the initial pinyin to become the reference pinyin. It can be understood as the steps required for the initial pinyin to become the reference pinyin. For example, in the process of "fang" becoming "guo", the initial pinyin of "方" is "fang", the obfuscated pinyin of "方" is "guo", changing "f" to "g" is one step, changing "a" to "u" is one step, changing "n" to "o" is one step, and changing "g" to a space is one step. There are a total of four steps, which can be understood as the edit distance being four.

[0085] In one embodiment, a weight value can also be set according to the initial letters and obfuscated letters involved in each step. This weight value is used to calculate the edit distance between the initial pinyin and the reference pinyin.

[0086] Specifically, according to the set of obfuscation correlation degrees corresponding to the public sample library, respectively obtain the target obfuscation correlation degrees corresponding to the initial letters in the initial pinyin and the reference letters in the reference pinyin, and calculate the edit distance between the initial pinyin and the reference pinyin based on the target obfuscation correlation degrees.

[0087] Step 210: Determine the pinyin to be replaced from each reference pinyin based on the edit distance, generate the obfuscated characters corresponding to the pinyin to be replaced, and replace the characters to be replaced in the training text with the obfuscated characters to obtain multiple obfuscated texts corresponding to the training text.

[0088] The pinyin to be replaced refers to the pinyin corresponding to the initial pinyin of the finally determined character to be replaced. The obfuscated character refers to the character generated by the pinyin to be replaced. The obfuscated character can be one or more.

[0089] Specifically, compare the edit distances corresponding to each reference pinyin, select one or more reference pinyins with shorter distances as the pinyin to be replaced, and replace the characters to be replaced in the training text with the obfuscated characters generated by the pinyin to be replaced to obtain multiple obfuscated texts.

[0090] Step 212: Establish the association relationship between the training text and each obfuscated text to obtain an error correction sample library; the error correction sample library is used to train the text error correction model for the target domain.

[0091] In this context, the association relationship refers to the correspondence between training texts and obfuscated texts; this relationship can be one-to-one or one-to-many. The error correction sample library is a text database containing multiple pairs of correct and incorrect texts; an upper or lower limit can be set for the number of such pairs. A text correction model is a model that takes a potentially grammatically incorrect Chinese sentence as input and outputs a correct Chinese sentence. Types of grammatical errors include extra characters, missing characters, and typos; currently, typos are the most common type.

[0092] Specifically, the association between training text and error-correcting text is established, and the training text, error-correcting text, and the association between them are packaged into an error-correcting sample library. The neural network model is trained using this sample library to obtain a domain-specific text correction model, which is used to correct text obtained from speech recognition or other methods within that domain.

[0093] In the above method for establishing a target domain sample library, the following steps are taken: obtaining the confusion correlation between target letters and confused letters in a public sample library; obtaining training texts in the target domain; selecting characters to be replaced from the training samples; obtaining the initial pinyin of the characters to be replaced; confusing the initial pinyin to obtain multiple reference pinyin corresponding to the characters to be replaced; calculating the edit distance between the initial pinyin and each reference pinyin based on the confusion correlation; selecting the characters to be replaced from the multiple initial pinyin based on the edit distance; generating confused characters based on the characters to be replaced; replacing the characters to be replaced in the training text with the confused characters to obtain multiple confused texts; and establishing the association between the training texts and the confused texts to obtain the error correction sample library. The initial pinyin is obfuscated to obtain reference pinyin, ensuring the quantity of the error correction sample library. The edit distance between the initial pinyin and the reference pinyin is calculated based on the obfuscation correlation of each letter in the public sample library. The reference pinyin with the shortest edit distance is selected as the pinyin to be replaced. The pinyin with the highest obfuscation degree is selected to increase the utilization rate of obfuscated text in the error correction sample library. The text error correction model is obtained by training a neural network model using the error correction sample library. The text error correction model is targeted and practical, improving the accuracy of error correction.

[0094] In one embodiment, such as Figure 3 As shown, step 202 includes:

[0095] Step 302: Obtain the target pinyin and the corresponding confused pinyin from the public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters.

[0096] In this context, target pinyin refers to the pinyin of the correct characters in the correct text. Confusing pinyin refers to the pinyin of the incorrect characters in the incorrect text that correspond to the correct characters. Target pinyin and confusing pinyin appear in pairs, with one target pinyin corresponding to one confusing pinyin. The public sample library contains pairs of target pinyin and confusing pinyin.

[0097] Specifically, all target pinyin and their corresponding confused pinyin are obtained from sample pairs in a public sample library. Each target pinyin contains multiple target letters, and each confused pinyin contains multiple confused letters.

[0098] Step 304: Based on the positional relationship between each target letter and each confused letter, establish a reference association pair between the target letters and the confused letters.

[0099] Positional relationships refer to the relationship between the position of the target letter in the target pinyin and the position of the confusing letter in the confusing pinyin. For example, the relationship between the first letter of the target pinyin and the first letter of the confusing pinyin, or the relationship between the second letter of the target pinyin and the second letter of the confusing pinyin. Reference pairs refer to the correspondence established between letters. Reference pairs can be represented by two letters, or by two letters plus numbers or symbols. For example, the first letter "f" in the target pinyin and the first letter "h" in the confusing pinyin form a reference pair, which can be represented as "fh".

[0100] Specifically, reference association pairs are formed by combining the target letter in the target pinyin with the confused letter in the confused pinyin at the same position. Multiple reference association pairs are formed by combining one target pinyin and one confused pinyin.

[0101] Step 306: Count the number of reference association pairs that have the same target letter and the same confusion letter to obtain the first statistical value.

[0102] The first statistic refers to the total number of reference association pairs that share the same target letter and the confusing letter. For example, if there are 6 reference association pairs, namely "fh", "ao", "fh", "fh", "ao", "ou", "ff", "au", and "fp", then the number of "fh" is 3, the number of "ao" is 2, the number of "ou" is 1, the number of "ff" is 1, the number of "au" is 1, and the number of "fp" is 1.

[0103] Specifically, among all reference association pairs composed of target letters and confusion letters in the statistical public sample library, reference association pairs with the same target letter and confusion letter are regarded as the same type of reference association pair. The number of reference association pairs in each type is calculated and used as the first statistical value of each type of reference association pair.

[0104] Step 308: Based on the first statistical value, count the number of reference association pairs corresponding to the same target letter in the reference association pairs to obtain the second statistical value.

[0105] The second statistic refers to the total number of reference association pairs that share the same target letter. For example, if the number of "fh" is 3, the number of "ao" is 2, the number of "ou" is 1, the number of "ff" is 1, the number of "au" is 1, and the number of "fp" is 1, then the number of reference association pairs corresponding to the target letter h is the sum of the numbers of "fh", "ff", and "fp", which is 5.

[0106] Specifically, among all reference association pairs composed of target letters and confusion letters in the statistical public sample library, reference association pairs with the same target letter are regarded as reference association pairs with the same target letter. The first statistical values ​​with the same target letter are added together to obtain the second statistical value corresponding to each target letter.

[0107] Step 310: Calculate the ratio between the first statistical value and the second statistical value to obtain the confusion correlation between the target letter and the confused letter.

[0108] In this context, proportion refers to the percentage of each part in a whole relative to the total number of parts, reflecting the composition or structure of the whole. For example, if the number of "fh" is 3, the number of "ff" is 1, the number of "fp" is 1, the number of reference association pairs corresponding to the target letter h is 5, and the confusion association degree between the target letter f and the confusing letter h is 3 / 5.

[0109] Specifically, the confusion correlation degree between the target letter and the confusing letter in the reference association pair corresponding to the first statistical value is obtained by dividing the first statistical value by the second statistical value corresponding to the target letter in the reference association pair corresponding to the first statistical value.

[0110] In this embodiment, the confusion correlation degree between the target letter and the confused letter in the public sample library is statistically analyzed. This confusion correlation degree represents the probability that the target letter is confused into a confused letter, providing a basis for subsequent training samples and improving the accuracy of calculating the edit distance.

[0111] In one embodiment, establishing a reference association pair between the target letter in the target pinyin and the confused letter in the confused pinyin includes:

[0112] Get the first target letter of the target pinyin as the current target letter, and get the first confused letter of the confused pinyin as the current confused letter; form a reference association pair between the current target letter and the current confused letter; get the next target letter of the target pinyin as the current target letter, and get the next confused letter of the confused pinyin as the current confused letter. If the next target letter of the target pinyin is empty and the next confused letter of the confused pinyin is not empty, then use the target character as the current target letter. If the next confused letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then use the target character as the current confused letter. Repeat the step of forming a reference association pair between the current target letter and the current confused letter until both the next target letter of the target pinyin and the next confused letter of the confused pinyin are empty.

[0113] The target character is the character used to replace a space. The target character can be a number, a symbol, etc. For example, "*" can be used as the target character to replace a space.

[0114] Specifically, the target letter in the target pinyin is paired with the confused letter in the confused pinyin that is in the same position and order to form a reference association pair; if the target letter in the target pinyin is empty, the target character is used as the target letter to form a reference association pair with the confused letter in the confused pinyin; if the confused letter in the confused pinyin is empty, the target character is used as the confused letter to form a reference association pair with the target letter.

[0115] In this embodiment, a reference association pair between the target letter and the corresponding confused letter is established based on the positional order relationship between the target letter in the target pinyin and the confused letter in the confused pinyin. This provides a large number of samples for calculating the confusion association degree and improves the accuracy of the confusion association degree calculation.

[0116] In one embodiment, calculating the edit distance between the initial pinyin and the reference pinyin includes:

[0117] Establish multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; based on the confusion association degree, query the target confusion association degree corresponding to each target association pair; calculate the edit distance between the initial pinyin and the reference pinyin by calculating the confusion association degree of each target pair.

[0118] Among them, the target association pair refers to the association pair formed by the initial letter in the initial pinyin and the reference letter in the reference pinyin.

[0119] Specifically, target association pairs are established between the initial letters in the initial pinyin and the reference letters in the reference pinyin. The confusion degree corresponding to the reference association pair is queried as the target confusion degree of the reference association pair. The edit distance between the reference pinyin and the confused pinyin is obtained by adding up the target confusion degrees.

[0120] In one embodiment, the initial consonant of the initial pinyin and the reference initial consonant of the reference pinyin are combined to form a target association pair, and the initial vowel of the initial pinyin and the reference vowel of the reference pinyin are combined to form a target association degree. The target confusion association degree corresponding to the above two target association pairs is obtained by querying, and the two target confusion association degrees are added together to obtain the edit distance.

[0121] In one embodiment, the tone of the initial pinyin and the tone of the reference pinyin are used as a target correlation. If the tone of the initial pinyin is the same as the tone of the reference pinyin, the confusion correlation of the target correlation pair is 0. If the tone of the initial pinyin is different from the tone of the reference pinyin, the confusion correlation of the target correlation pair is 1.

[0122] In this embodiment, the edit distance between the reference pinyin and the confused pinyin is calculated based on the confusion correlation in the public sample library, which provides a basis for selecting the pinyin to be replaced from the reference pinyin and improves the accuracy of the edit distance calculation.

[0123] In one embodiment, such as Figure 4 As shown, the reference pinyin for the character to be replaced includes:

[0124] Step 402: Obtain the initial initials and initial finals corresponding to the initial pinyin.

[0125] In phonics, initials refer to the consonants preceding finals, which together with the finals form a complete syllable. Consonants are sounds produced when airflow is obstructed in the oral cavity during phonation. The pinyin alphabet includes 23 initials: b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, s, y, w; and 24 finals: a, o, e, i, u, ü, ai, ei, ui, ao, ou, iu, ie, üe, er, an, en, in, un, ün, ang, eng, ing, ong.

[0126] Specifically, the initial consonant in the initial pinyin is taken as the initial initial consonant, and the final vowel in the initial pinyin is taken as the initial final vowel.

[0127] Step 404: Obtain the confusion pinyin rule base, and match the corresponding reference initials for the initial initials and the corresponding reference finals for the initial finals based on the confusion pinyin rule base.

[0128] The confusion rule base refers to a collection containing confused initials and finals corresponding to each initial and final. It can contain confused initials for each of the 23 initials and confused finals for each of the 24 finals. The confusion rule base can also be categorized into confusions between retroflex and alveolar consonants, front and back nasal sounds, tones, and regional accents. For example, in the confusion rule base, the initial "c" corresponds to the confused initials "ch" and "zh".

[0129] Specifically, the initial consonant in the confusion pinyin rule library is matched with the corresponding reference initial consonant, and the final consonant in the confusion pinyin rule library is matched with the corresponding reference final consonant.

[0130] Step 406: Based on the reference initials and reference finals, generate the reference pinyin corresponding to the character to be replaced.

[0131] Specifically, the reference initials and reference finals are combined to form the reference pinyin for the character to be replaced. This can be understood as randomly combining the reference initials and reference finals to form one or more reference pinyin.

[0132] In this embodiment, a reference initial is matched to the initial initial by a confusion pinyin rule base, and a reference final is matched to the initial final. The reference initials and reference finals are then combined to form the reference pinyin corresponding to the sub-subject to be replaced. Multiple reference pinyin are formed by combining reference initials and reference pinyin, thus expanding the number of confusion samples in the error correction sample base.

[0133] In one embodiment, such as Figure 5 As shown, creating obfuscated text includes:

[0134] Step 502: Combine the pinyin to be replaced with each tone to obtain the target confused pinyin.

[0135] Among them, tone refers to the pitch changes of Chinese characters. There are four tones in Hanyu Pinyin: high level tone, rising tone, falling-rising tone, and falling tone.

[0136] Specifically, the pinyin to be replaced is combined with the first tone, second tone, third tone, and fourth tone respectively. After combining one pinyin to be replaced with a tone, four target confused pinyin are obtained.

[0137] Step 504: Generate multiple confused characters based on the target confused pinyin.

[0138] Specifically, match the obfuscated characters corresponding to each target obfuscated pinyin, and use all the obfuscated characters corresponding to the target pinyin as the obfuscated characters corresponding to the character to be replaced.

[0139] Step 506: Replace the characters to be replaced in the training text with obfuscated characters to obtain multiple reference texts.

[0140] Specifically, each character to be replaced is replaced with a corresponding obfuscated character in the training text, resulting in multiple reference texts corresponding to the training text.

[0141] In one embodiment, the number of characters to be replaced each time is set. For example, in the first round of replacement, only one character to be replaced is replaced, and each character to be replaced is replaced in turn. In the second round, two characters to be replaced are replaced, and the two characters to be replaced are randomly selected from all the characters to be replaced. In the third round, three characters to be replaced are replaced, and the three characters to be replaced are randomly selected from all the characters to be replaced, and so on, until all the characters to be replaced are replaced at once.

[0142] In one implementation, the obfuscated characters corresponding to each character to be replaced are grouped into a set, and in each round, an obfuscated character is randomly selected from each set to replace all the characters to be replaced in the training text.

[0143] Step 508: Input each reference text into the text detection model and filter to obtain the obfuscated text corresponding to the training text.

[0144] Among them, text detection models refer to models used to detect the semantics or compositional logic of text. For example, models that determine whether text is fluent and reasonable by examining its subject-verb-object structure, modifiers, and complements.

[0145] Specifically, multiple reference texts are input into a pre-defined text detection model, and the reference texts are filtered according to the rules set by the text detection model to select one or more obfuscated texts that meet the rules.

[0146] In this embodiment, the target confused pinyin is obtained by combining the pinyin to be replaced with its tone. Multiple confused characters are then generated based on the target confused pinyin. These confused characters replace the characters to be replaced in the training text, resulting in multiple reference texts, thus increasing the number of reference texts. The reference texts are then input into a text detection model for filtering, improving the quality of the confused texts.

[0147] In one embodiment, such as Figure 6 As shown, the steps to determine the character to be replaced include:

[0148] Step 602: Obtain training text for the target domain.

[0149] Specifically, training texts are obtained from a training sample library of the target domain.

[0150] In one embodiment, the training sample library for the target domain contains multiple training texts, which are common terms in the target domain. The training samples for the target domain can be updated by operators or obtained from the target system being interfaced with.

[0151] Step 604: Segment the training text to obtain multiple constituent words.

[0152] Word segmentation refers to the process of dividing a sequence of Chinese characters into individual words. It can be understood as dividing a sentence into multiple words. Word segmentation methods include, but are not limited to, rule-based segmentation, statistical segmentation, semantic segmentation, and understanding-based segmentation.

[0153] Specifically, the selected word segmentation method is used to divide the Chinese character sequence in the training text into multiple individual words.

[0154] Step 606: Select keywords from the constituent words.

[0155] Specifically, keywords in the training text are selected from each constituent word according to the set rules.

[0156] In one embodiment, multiple constituent words are filtered based on a keyword list set for the target domain. The related word list contains high-frequency words in the target domain, such as high-frequency words in the financial field like deposits, terms, wealth management, and funds.

[0157] Step 608: Randomly select one character from each keyword as the character to be replaced in the training text.

[0158] In one embodiment, one character is randomly selected from each keyword, and all selected characters are used as the characters to be replaced.

[0159] In one embodiment, based on the frequency of each character in the keyword, the characters with the highest frequency in the keyword are selected as the characters to be replaced.

[0160] In this embodiment, the training text is segmented into words, keywords are selected from the multiple constituent words obtained from the segmentation, and then the characters to be replaced are obtained from the keywords. This accurately obtains the characters to be replaced and improves the targeting of the characters to be replaced.

[0161] In a specific embodiment, the process of generating a target domain sample library is as follows: Figure 7 As shown:

[0162] Obtain training text from the training sample library corresponding to the target domain, extract keywords from the training text, randomly select characters to be replaced from the keywords, and obtain the initial pinyin corresponding to the characters to be replaced.

[0163] The initial pinyin is used to obtain the reference pinyin through the confusion pinyin rule library. One or more of the following rules can be selected from the confusion pinyin rule library: confusion between flat and retroflex consonants, confusion between front and back nasal consonants, tone mixing, and regional special confusion. The reference pinyin is then confused based on the confusion correlation between each pinyin letter in the public sample and the confused letter. The edit distance between the initial pinyin and the reference pinyin is then calculated, and the reference pinyin with the shorter edit distance is selected as the confused pinyin.

[0164] The confused characters are generated based on the confused pinyin. The characters to be replaced in the training text are replaced with the confused characters to obtain the confused text. The training text and the confused text are combined into sample pairs for input. All input sample pairs form the error correction sample library of the target domain.

[0165] In this embodiment, the initial pinyin is obfuscated to obtain reference pinyin, ensuring the quantity of the error correction sample library. The edit distance between the initial pinyin and the reference pinyin is calculated based on the obfuscation correlation of each letter in the public sample library. The reference pinyin with the shortest edit distance is selected as the pinyin to be replaced. The pinyin to be replaced with a high degree of obfuscation is selected to increase the utilization rate of obfuscated text in the error correction sample library. The text error correction model trained using the error correction sample library is targeted and practical, improving the efficiency and effectiveness of error correction.

[0166] It should be understood that, although Figure 1-7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1-7 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0167] Based on the same inventive concept, this application also provides an apparatus for establishing a target domain sample library as described above. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations of the one or more target domain sample library establishment apparatuses provided below can be found in the limitations of the target domain sample library establishment method described above, and will not be repeated here.

[0168] In one embodiment, such as Figure 8As shown, a target domain sample library establishment device is provided, including a confusion correlation degree acquisition module, a character to be replaced acquisition module, a confusion module, a calculation module, a replacement module, and a correlation module, wherein:

[0169] The confusion correlation degree acquisition module 802 is used to acquire the confusion correlation degree between each target letter and the confused letter. The confusion correlation degree is determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in the public sample library.

[0170] The replacement character acquisition module 804 is used to acquire the replacement character in the training text of the target domain, as well as the initial pinyin corresponding to the replacement character;

[0171] The obfuscation module 806 is used to obfuscate the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced;

[0172] The calculation module 808 is used to calculate the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree;

[0173] The replacement module 810 is used to determine the pinyin to be replaced from each of the reference pinyin based on the edit distance, generate a confused character corresponding to the pinyin to be replaced based on the pinyin to be replaced, replace the pinyin to be replaced in the training text with the confused character, and obtain multiple confused texts corresponding to the training text.

[0174] The association module 812 is used to establish the association relationship between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

[0175] In one embodiment, the confusion association degree acquisition module 802 is further configured to: acquire a target pinyin and a confused pinyin corresponding to the target pinyin in a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters; establish a reference association pair between the target letters and the confused letters based on the positional relationship between each target letter and each confused letter; count the number of reference association pairs with the same target letter and confused letter in the reference association pair to obtain a first statistical value; based on the first statistical value, count the number of reference association pairs corresponding to the same target letter in the reference association pair to obtain a second statistical value; calculate the ratio between the first statistical value and the second statistical value to obtain the confusion association degree between the target letter and the confused letter.

[0176] In one embodiment, the confusion association degree acquisition module 802 is further configured to: acquire the first target letter of the target pinyin as the current target letter, acquire the first confused letter of the confused pinyin as the current confused letter; form a reference association pair between the current target letter and the current confused letter; acquire the next target letter of the target pinyin as the current target letter, acquire the next confused letter of the confused pinyin as the current confused letter; if the next target letter of the target pinyin is empty and the next confused letter of the confused pinyin is not empty, then use the target character as the current target letter; if the next confused letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then use the target character as the current confused letter; repeat the step of forming a reference association pair between the current target letter and the current confused letter until both the next target letter of the target pinyin and the next confused letter of the confused pinyin are empty.

[0177] In one embodiment, the calculation module 808 is further configured to: establish multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; query the target confusion association degree corresponding to each target association pair based on the confusion association degree; and calculate the edit distance between the initial pinyin and the reference pinyin by calculating the confusion association degree of each target association pair.

[0178] In one embodiment, the obfuscation module 806 is further configured to: obtain the initial initial and initial final corresponding to the initial pinyin; obtain an obfuscation pinyin rule library, and match the initial initial with a corresponding reference initial and the initial final with a corresponding reference final based on the obfuscation pinyin rule library; and form a reference pinyin corresponding to the character to be replaced based on the reference initial and the reference final.

[0179] In one embodiment, the replacement module 810 is further configured to: combine the pinyin to be replaced with each tone to obtain a target confused pinyin; generate multiple confused characters based on the target confused pinyin; replace the character to be replaced in the training text with the confused character to obtain multiple reference texts; input each of the reference texts into a text detection model, and filter to obtain the confused text corresponding to the training text.

[0180] In one embodiment, the word to be replaced acquisition module 804 is further configured to: acquire training text in the target domain; segment the training text to obtain multiple constituent words; select keywords from the constituent words; and randomly acquire characters from the keywords as words to be replaced in the training text.

[0181] Each module in the aforementioned target domain sample library establishment device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0182] Figure 9 An internal structural diagram of a computer device in one embodiment is shown. This computer device can be either a terminal or a server. Figure 9 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store computer programs. When executed by the processor, these computer programs enable the processor to implement a video compositing method. The internal memory may also store computer programs, which, when executed by the processor, enable the processor to implement a business document conversion method.

[0183] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: obtaining the confusion correlation degree between each target letter and the confused letter, the confusion correlation degree being determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in a public sample library; obtaining the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced; performing confusion processing on the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced; calculating the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree; determining the character to be replaced from each of the reference pinyin based on the edit distance; generating the confused character corresponding to the character to be replaced based on the character to be replaced; replacing the character to be replaced in the training text with the confused character to obtain multiple confused texts corresponding to the training text; establishing the correlation relationship between the training text and each of the confused texts to obtain an error correction sample library; the error correction sample library being used to train a text error correction model in the target domain.

[0185] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a target pinyin and a confused pinyin corresponding to the target pinyin from a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters; establishing reference association pairs between the target letters and the confused letters based on the positional relationship between each target letter and each confused letter; counting the number of reference association pairs with the same target letter and confused letter in the reference association pairs to obtain a first statistical value; counting the number of reference association pairs corresponding to the same target letter in the reference association pairs based on the first statistical value to obtain a second statistical value; and calculating the ratio between the first statistical value and the second statistical value to obtain the confusion association degree between the target letter and the confused letter.

[0186] In one embodiment, when the processor executes the computer program, it further implements the following steps: obtaining the first target letter of the target pinyin as the current target letter, and obtaining the first confused letter of the confused pinyin as the current confused letter; forming a reference association pair between the current target letter and the current confused letter; obtaining the next target letter of the target pinyin as the current target letter, and obtaining the next confused letter of the confused pinyin as the current confused letter; if the next target letter of the target pinyin is empty and the next confused letter of the confused pinyin is not empty, then using the target character as the current target letter; if the next confused letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then using the target character as the current confused letter; repeating the step of forming a reference association pair between the current target letter and the current confused letter until both the next target letter of the target pinyin and the next confused letter of the confused pinyin are empty.

[0187] In one embodiment, when the processor executes the computer program, it further performs the following steps: establishing multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; querying the target confusion association degree corresponding to each target association pair based on the confusion association degree; and calculating the edit distance between the initial pinyin and the reference pinyin by calculating the confusion association degree of each target pinyin.

[0188] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the initial initial and initial final corresponding to the initial pinyin; obtaining a confusion pinyin rule library, matching a corresponding reference initial for the initial initial and a corresponding reference final for the initial final based on the confusion pinyin rule library; and forming a reference pinyin corresponding to the character to be replaced based on the reference initial and the reference final.

[0189] In one embodiment, when the processor executes the computer program, it further performs the following steps: combining the pinyin to be replaced with each tone to obtain a target confused pinyin; generating multiple confused characters based on the target confused pinyin; replacing the character to be replaced in the training text with the confused character to obtain multiple reference texts; inputting each of the reference texts into a text detection model, and filtering to obtain the confused text corresponding to the training text.

[0190] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring training text in the target domain; segmenting the training text to obtain multiple constituent words; selecting keywords from the constituent words; and randomly selecting characters from the keywords as characters to be replaced in the training text.

[0191] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: obtaining the confusion correlation degree between each target letter and the confused letter, the confusion correlation degree being determined based on the correlation relationship between the target pinyin and the confused pinyin corresponding to each public sample in a public sample library; obtaining the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced; performing confusion processing on the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced; calculating the edit distance between the initial pinyin and each of the reference pinyin based on the confusion correlation degree; determining the character to be replaced from each of the reference pinyin based on the edit distance; generating the confused character corresponding to the character to be replaced based on the character to be replaced; replacing the character to be replaced in the training text with the confused character to obtain multiple confused texts corresponding to the training text; establishing the correlation relationship between the training text and each of the confused texts to obtain an error correction sample library; the error correction sample library being used to train a text error correction model in the target domain.

[0192] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: obtaining a target pinyin and a confused pinyin corresponding to the target pinyin from a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters; establishing reference association pairs between the target letters and the confused letters based on the positional relationship between each target letter and each confused letter; counting the number of reference association pairs with the same target letter and confused letter in the reference association pairs to obtain a first statistical value; counting the number of reference association pairs corresponding to the same target letter in the reference association pairs based on the first statistical value to obtain a second statistical value; calculating the ratio between the first statistical value and the second statistical value to obtain the confusion association degree between the target letter and the confused letter.

[0193] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: obtaining the first target letter of the target pinyin as the current target letter, obtaining the first confused letter of the confused pinyin as the current confused letter; forming a reference association pair between the current target letter and the current confused letter; obtaining the next target letter of the target pinyin as the current target letter, obtaining the next confused letter of the confused pinyin as the current confused letter; if the next target letter of the target pinyin is empty and the next confused letter of the confused pinyin is not empty, then using the target character as the current target letter; if the next confused letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then using the target character as the current confused letter; repeating the step of forming a reference association pair between the current target letter and the current confused letter until both the next target letter of the target pinyin and the next confused letter of the confused pinyin are empty.

[0194] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: establishing multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; querying the target confusion association degree corresponding to each target association pair based on the confusion association degree; and calculating the edit distance between the initial pinyin and the reference pinyin by calculating the confusion association degree of each target pinyin.

[0195] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the initial initial and initial final corresponding to the initial pinyin; obtaining a confusion pinyin rule library, matching a corresponding reference initial for the initial initial and a corresponding reference final for the initial final based on the confusion pinyin rule library; and forming a reference pinyin corresponding to the character to be replaced based on the reference initial and the reference final.

[0196] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: combining the pinyin to be replaced with each tone to obtain a target confused pinyin; generating multiple confused characters based on the target confused pinyin; replacing the character to be replaced in the training text with the confused character to obtain multiple reference texts; inputting each of the reference texts into a text detection model, and filtering to obtain the confused text corresponding to the training text.

[0197] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: acquiring training text in the target domain; segmenting the training text to obtain multiple constituent words; selecting keywords from the constituent words; and randomly selecting characters from the keywords as characters to be replaced in the training text.

[0198] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for establishing a sample library in a target domain, characterized in that, The method includes: Obtain the target pinyin and the corresponding confused pinyin from a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters. Based on the positional relationship between each target letter and each confused letter, a reference association pair between the target letter and the confused letter is established; The first statistical value is obtained by counting the number of reference association pairs that have the same target letter and the same confusion letter. Based on the first statistical value, the number of reference association pairs corresponding to the same target letter in the reference association pairs is counted to obtain the second statistical value; Calculate the ratio between the first statistical value and the second statistical value to obtain the confusion correlation between the target letter and the confused letter; Obtain the character to be replaced in the training text of the target domain, and the initial pinyin corresponding to the character to be replaced; The initial pinyin is obfuscated to obtain multiple reference pinyin corresponding to the character to be replaced; Establish multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; Based on the obfuscation correlation, query the target obfuscation correlation corresponding to each target association pair; The confusion correlation degree of each target is calculated to obtain the edit distance between the initial pinyin and the reference pinyin; Based on the edit distance, the pinyin to be replaced is determined from each of the reference pinyin, and a confused character corresponding to the pinyin to be replaced is generated based on the pinyin to be replaced. The character to be replaced in the training text is replaced with the confused character to obtain multiple confused texts corresponding to the training text. Establish the association between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

2. The method according to claim 1, characterized in that, The target pinyin refers to the pinyin of the correct character in the correct text, and the confusing pinyin refers to the pinyin of the incorrect character in the incorrect text corresponding to the correct character.

3. The method according to claim 1, characterized in that, The step of establishing reference association pairs between the target letters and the confused letters based on the positional relationship between each target letter and each confused letter includes: The first target letter of the target pinyin is obtained as the current target letter, and the first confused letter of the confused pinyin is obtained as the current confused letter. Create a reference association pair by combining the current target letter and the current confused letter; Obtain the next target letter of the target pinyin as the current target letter, and obtain the next confusing letter of the confused pinyin as the current confusing letter. If the next target letter of the target pinyin is empty and the next confusing letter of the confused pinyin is not empty, then use the target character as the current target letter. If the next confusing letter of the confused pinyin is empty and the next target letter of the target pinyin is not empty, then use the target character as the current confusing letter. Repeat the step of forming a reference association pair between the current target letter and the current confusing letter until both the next target letter of the target pinyin and the next confusing letter of the confused pinyin are empty.

4. The method according to claim 1, characterized in that, The establishment of multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin includes: The initial initial consonant of the initial pinyin and the reference initial consonant of the reference pinyin are combined to form a target association pair; The initial vowel of the initial pinyin and the reference vowel of the reference pinyin are combined to form a target association pair; The statistical analysis of the confusion correlation of each target, to obtain the edit distance between the initial pinyin and the reference pinyin, includes: The edit distance between the initial pinyin and the reference pinyin is obtained by summing the target confusion correlation degrees corresponding to the two target association pairs respectively.

5. The method according to claim 1, characterized in that, The process of obfuscating the initial pinyin to obtain multiple reference pinyin for the character to be replaced includes: Obtain the initial initial and initial final corresponding to the initial pinyin; Obtain a confusion pinyin rule base, and based on the confusion pinyin rule base, match the corresponding reference initials for the initial initials and the corresponding reference finals for the initial finals; Based on the reference initials and reference finals, a reference pinyin corresponding to the character to be replaced is formed.

6. The method according to claim 1, characterized in that, The process involves generating a confused character corresponding to the character to be replaced based on the pinyin to be replaced, replacing the character to be replaced in the training text with the confused character, and obtaining multiple confused texts corresponding to the training text, including: The pinyin to be replaced is combined with each tone to obtain the target confused pinyin; Based on the target confused pinyin, generate multiple confused characters; The characters to be replaced in the training text are replaced with the obfuscated characters to obtain multiple reference texts; Each of the reference texts is input into the text detection model, and the obfuscated texts corresponding to the training texts are obtained by filtering.

7. The method according to claim 1, characterized in that, The process of obtaining the word to be replaced in the training text of the target domain includes: Obtain training text in the target domain; The training text is segmented into words to obtain multiple constituent words; Keywords are obtained by filtering the constituent words; Randomly select characters from the keywords and use them as the replacement characters in the training text.

8. An apparatus for compiling a sample library, characterized in that, The device includes: The confusion association degree acquisition module is used to acquire target pinyin and corresponding confused pinyin from a public sample library; the target pinyin contains multiple target letters, and the confused pinyin contains multiple confused letters; based on the positional relationship between each target letter and each confused letter, a reference association pair between the target letter and the confused letter is established; the number of reference association pairs with the same target letter and confused letter is counted to obtain a first statistical value; based on the first statistical value, the number of reference association pairs corresponding to the same target letter is counted to obtain a second statistical value; the ratio between the first statistical value and the second statistical value is calculated to obtain the confusion association degree between the target letter and the confused letter. The module for obtaining characters to be replaced is used to obtain the characters to be replaced in the training text of the target domain, as well as the initial pinyin corresponding to the characters to be replaced; The obfuscation module is used to obfuscate the initial pinyin to obtain multiple reference pinyin corresponding to the character to be replaced; The calculation module is used to establish multiple target association pairs between the initial letters in the initial pinyin and the reference letters in the reference pinyin; based on the confusion association degree, query the target confusion association degree corresponding to each target association pair; and calculate the edit distance between the initial pinyin and the reference pinyin by calculating the confusion association degree of each target pinyin. The replacement module is used to determine the pinyin to be replaced from each of the reference pinyin based on the edit distance, generate a confused character corresponding to the pinyin to be replaced based on the pinyin to be replaced, replace the pinyin to be replaced in the training text with the confused character, and obtain multiple confused texts corresponding to the training text. The association module is used to establish the association relationship between the training text and each of the obfuscated texts to obtain an error correction sample library; the error correction sample library is used to train the text error correction model in the target domain.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images