Generation method and device for text error correction task model, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a text error correction task model generation method and device, equipment and a medium. The method comprises the following steps of: acquiring an original Chinese text and a corresponding error correction label from a preset data set, training a preset text error correction task model by using the original Chinese text and the corresponding error correction label to obtain an initially trained text error correction task model; according to the method, a target Chinese text with an accurate label and a corresponding error correction label are screened out from a data set, the target Chinese text and the corresponding error correction label are used for training an initially trained text error correction task model, and a secondarily trained text error correction task model is obtained and used for conducting error correction on a to-be-recognized Chinese text. A text error correction model is trained twice by using a Chinese text, the first training is to learn a large amount of data and improve the application range of the model, and the second training is to improve the precision of the model, so that the trained model is used for text error correction.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for generating a text error correction task model. Background Art

[0002] At present, Chinese text error correction plays a very important role. If the input is wrong, it will affect the understanding of the context and the performance of any downstream tasks. Due to the development of deep learning technology, the development of natural language processing has been rapid, and good progress has been made in the Chinese text error correction task. However, there are still the following problems in the implementation: the Chinese text error correction task is mainly divided into two subtasks, one is Chinese spelling error correction, and the other is grammar error correction. The two tasks are mostly developed independently and are not unified; when the user enters a sentence, the outside world cannot know in advance what type of error is in the sentence, so it can be connected in series, first go through spelling correction, then go through grammar correction, but the connection form is not only time-consuming, but may also affect each other; in the Chinese text error correction task, in addition to spelling errors and grammar errors, there is also a common punctuation error. At present, there is no relevant work to solve the problem of punctuation errors.

[0003] Therefore, how to improve the training accuracy of the model in the Chinese text correction scenario to improve the accuracy of model use has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, device, equipment and medium for generating a text error correction task model to solve the problem of how to improve the training accuracy of the model in the Chinese text error correction scenario to improve the accuracy of the model use.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a text error correction task model, comprising:

[0006] Obtaining original Chinese texts and corresponding error correction annotations from a preset data set, wherein the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0;

[0007] Using the original Chinese text and the corresponding error correction annotations, training a preset text error correction task model to obtain an initially trained text error correction task model;

[0008] Filtering out target Chinese text and corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations;

[0009] The target Chinese text and the corresponding error correction annotations are used to train the initially trained text error correction task model to obtain a secondary trained text error correction task model, and the secondary trained text error correction task model is used to correct the Chinese text to be recognized.

[0010] In a second aspect, an embodiment of the present application provides a device for generating a text error correction task model, comprising:

[0011] A text acquisition module, used to acquire original Chinese text and corresponding error correction annotations from a preset data set, wherein the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0;

[0012] A first training module is used to train a preset text error correction task model using the original Chinese text and the corresponding error correction annotations to obtain an initially trained text error correction task model;

[0013] A target screening module, used to screen out target Chinese text and corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations;

[0014] The second training module is used to use the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model to obtain a secondary trained text error correction task model, and the secondary trained text error correction task model is used to correct the Chinese text to be recognized.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating a text error correction task model as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a text error correction task model as described in the first aspect is implemented.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0018] The original Chinese text and the corresponding error correction annotations are obtained from a preset data set, wherein the data set includes N Chinese texts and the error correction annotations corresponding to each Chinese text. The preset text error correction task model is trained using the original Chinese text and the corresponding error correction annotations to obtain an initially trained text error correction task model. The target Chinese text and the corresponding error correction annotations are screened out from the data set, wherein the target Chinese text is a Chinese text with accurate annotations. The initially trained text error correction task model is trained using the target Chinese text and the corresponding error correction annotations to obtain a second-trained text error correction task model, which is used to correct the Chinese text to be recognized. The text error correction model is trained twice using Chinese texts. The first training is to learn a large amount of data and improve the scope of application of the model without screening the data, so that both high-quality and low-quality data can be used for model training. The second training is to improve the accuracy of the model and screen out high-quality data for model training, so that the trained model can be used for text error correction, and the model has a high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 It is a schematic diagram of an application environment of a method for generating a text error correction task model provided in Example 1 of the present application;

[0021] Figure 2 It is a flowchart of a method for generating a text error correction task model provided in Example 2 of the present application;

[0022] Figure 3 It is a flowchart of a method for generating a text error correction task model provided in Example 3 of the present application;

[0023] Figure 4 It is a flowchart of a method for generating a text error correction task model provided in Example 4 of the present application;

[0024] Figure 5 It is a flowchart of a method for generating a text error correction task model provided in Example 5 of the present application;

[0025] Figure 6 It is a flowchart of a method for generating a text error correction task model provided in Example 6 of the present application;

[0026] Figure 7 It is a structural schematic diagram of a device for generating a text error correction task model provided in Example 7 of the present application;

[0027] Figure 8 It is a structural diagram of a computer device provided in Example 8 of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0035] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0036] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0037] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0038] A method for generating a text error correction task model provided in the first embodiment of the present application can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0039] See also Figure 2 , is a flow chart of a method for generating a text error correction task model provided in the second embodiment of the present application. The method for generating a text error correction task model can be applied to Figure 1 The server side, the computer device corresponding to the server side connects to the client side, the database, etc., to obtain the corresponding data, and operates and processes the data accordingly. Of course, the model of the present application is a model for error correction tasks for Chinese text, so the model can also be deployed on the client side, that is, the generation method of the present application can also be applied to the client side. In this embodiment, the method is applied to the server side, and any device connected to the server side can send corresponding data to the server side, so that the server side relies on the data to complete the corresponding training model and generation model tasks, and the specific content will be described in detail below.

[0040] like Figure 2 As shown, the method for generating a text error correction task model may include the following steps:

[0041] Step S201, obtaining original Chinese text and corresponding error correction annotations from a preset data set.

[0042] In this embodiment, the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0.

[0043] In this embodiment, Chinese text may refer to a text consisting of characters in the Chinese language, and the text may be provided to the database by the client. Of course, the data in the database may also be collected through other collection methods to obtain the original Chinese text and the corresponding error annotations.

[0044] Error marking refers to the marking result of the error in the original Chinese text, which may include error type and / or correct result, etc. Error type may include word error, punctuation error, grammar error, etc. In addition, errors may also be divided into similar sound error, character shape error, missing character error, etc.

[0045] The data set includes at least one original Chinese text, and the data set is stored in the above-mentioned database. Specifically, public data sets can be collected from the Internet, and divided into high quality and low quality according to the source of the data set. For example, high quality: manually annotated, and the source is the mistakes made by native speakers; low quality: automatically generated, or the source is the mistakes made by non-native speakers. Of course, some specific rules can also be used to automatically generate some data on Chinese text errors. This part can make the scale of the data exceed 100 million as much as possible, and cover as many fields as possible, such as law, general, daily life, medicine, finance, literature and other industries and fields.

[0046] Step S202, using the original Chinese text and the corresponding error correction annotations, train the preset text error correction task model to obtain an initially trained text error correction task model.

[0047] In this embodiment, the original Chinese text and the corresponding error correction annotations can form supervised training samples to train a model, which is specifically a text error correction task model, that is, the purpose of the model is to identify and correct errors in the input text and output the correct text.

[0048] In the past, relevant error correction tasks were divided into two tasks: word errors and grammatical errors. Word errors are often equal in input and output, while grammatical errors are variable in input and output. In the present application, if the above-mentioned error correction annotations can cover all error types, the trained file error correction task model can solve all error types at the same time. Therefore, the embodiments of the present application can give priority to using a generative model to train a unified model, for example, the Transformer (Bidirectional and Auto-Regressive Transformers, BART) and T5 of the encoder-decoder structure with both bidirectional language modeling and autoregressive mechanism, and the natural language processing model (Generative Pretrained Transformer, GPT) and LLAMA based on deep learning corresponding to the decoder model. Preferably, the present application uses the T5 model for relevant experiments.

[0049] Due to the above data conditions and the different specific challenges for different error types, this solution proposes a two-stage training solution. In the first stage, massive data, including high-quality and low-quality data, as well as more generated data, are used to train the T5 model. It is expected that the model will have general performance for general spelling corrections, as well as some obvious grammatical errors and punctuation errors, and can cover a wider range of fields.

[0050] Step S203, filtering out target Chinese text and corresponding error correction annotations from the data set.

[0051] The target Chinese text is a Chinese text with accurate annotations. Accurate annotations may refer to annotations that are generally recognized as correct, for example, manual annotations are considered accurate annotations, and other annotations are considered inaccurate annotations.

[0052] When collecting data, the annotation types corresponding to the error correction annotations (ie, manual annotations and non-manual annotations) are collected, so as to be used in step S203 to implement the screening of the target Chinese text.

[0053] The target Chinese text is essentially high-quality data, where high quality is a relative concept. For example, the criteria for judging high-quality data are: manually annotated, and the corpus source is native speakers, and its errors are consistent with the errors of native speakers. Of course, low-quality data refers to data other than high-quality data.

[0054] Step S204, using the target Chinese text and the corresponding error correction annotations, the initially trained text error correction task model is trained to obtain a secondary trained text error correction task model, and the secondary trained text error correction task model is used to correct the Chinese text to be recognized.

[0055] In this embodiment, the target Chinese text and the corresponding error correction annotations can form supervised training samples to train a model. Specifically, the model is the text error correction task model that has been initially trained as mentioned above. Similarly, the purpose of the model is to identify and correct errors in the input text and output the correct text.

[0056] This training is the second stage compared to the first stage. In this second stage, only all high-quality data are used. Based on the model parameters of the first stage, the model is further fine-tuned. It is expected that the model can further improve its performance in more detailed grammatical correction and other aspects. The final secondary trained text correction task model is used to correct Chinese text to be recognized.

[0057] The embodiment of the present application obtains the original Chinese text and the corresponding error correction annotations from a preset data set, the data set includes N Chinese texts and the error correction annotations corresponding to each Chinese text, uses the original Chinese text and the corresponding error correction annotations to train the preset text error correction task model, obtains the initially trained text error correction task model, and screens out the target Chinese text and the corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations, uses the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model, obtains the second trained text error correction task model, and the second trained text error correction task model is used to correct the Chinese text to be recognized. Among them, the text error correction model is trained twice using Chinese text, the first training is to learn a large amount of data and improve the scope of application of the model, without the need to screen the data, so that both high and low quality data can be used for model training, the second training is to improve the model accuracy, screen out high-quality data for model training, so that the trained model is used for text error correction, and the model has a high accuracy.

[0058] See also Figure 3 , is a flow chart of a method for generating a text error correction task model provided in Example 3 of the present application. Figure 3 As shown, the error correction annotation includes the error correction target result and the error type. In the above step S202, the original Chinese text and the corresponding error correction annotation are used to train the preset text error correction task model to obtain the initially trained text error correction task model, including the following steps:

[0059] Step S301, splicing the original Chinese text with the corresponding error correction target result and error type to obtain an original splicing result.

[0060] Step S302: input the original splicing result into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

[0061] In this embodiment, the original Chinese text, the correct Chinese text (i.e., the error correction target result) and the error type are concatenated to form the following format:

[0062] {"input":"I am in a good mood today.","target":"I am in a good mood today.||Error type is word error"}

[0063] {"input":"I am in a good mood today. I want to go out and play.!","target":"I am in a good mood today. I want to go out and play!||Error type is punctuation error"}

[0064] {"input":"I am in a good mood today.","target":"I am in a good mood today.||Error type is syntax error"}

[0065] {"input":"I am in a good mood today.","target":"I am in a good mood today."}

[0066] Input refers to the input of the sentence, which may contain word errors, grammatical errors or punctuation errors. Of course, under some verification conditions, there may be no errors. Target is the target sentence (i.e. the error correction target result), which is the part of the expected model output. Here, the error type is integrated, and the model is expected to determine the error type of the sentence while correcting the error sentence.

[0067] In the embodiment of the present application, the model is trained by connecting the error type and the error correction target result, so that the obtained model has the error classification function as well as the error correction function, realizing the functional integration of the model, improving the accuracy of the model, and broadening the application scenarios of the model.

[0068] See also Figure 4 , is a flow chart of a method for generating a text error correction task model provided in Example 4 of the present application. Figure 4 As shown, in the above step S204, using the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model to obtain a secondarily trained text error correction task model may include the following steps:

[0069] Step S401, splicing the target Chinese text with the corresponding error correction target result and error type to obtain a target splicing result.

[0070] Step S402: input the target splicing result into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

[0071] In this embodiment, the original Chinese text, the correct Chinese text (i.e., the error correction target result) and the error type are concatenated to form the following format:

[0072] {"input":"I am in a good mood today.","target":"I am in a good mood today.||Error type is word error"}

[0073] {"input":"I am in a good mood today. I want to go out and play.!","target":"I am in a good mood today. I want to go out and play!||Error type is punctuation error"}

[0074] {"input":"I am in a good mood today.","target":"I am in a good mood today.||Error type is syntax error"}

[0075] {"input":"I am in a good mood today.","target":"I am in a good mood today."}

[0076] Input refers to the input of the sentence, which may contain word errors, grammatical errors or punctuation errors. Of course, under some verification conditions, there may be no errors. Target is the target sentence (i.e. the error correction target result), which is the part of the expected model output. Here, the error type is integrated, and the model is expected to determine the error type of the sentence while correcting the error sentence.

[0077] In the embodiment of the present application, the model is trained by connecting the error type and the error correction target result, so that the obtained model has the error classification function as well as the error correction function, realizing the functional integration of the model, improving the accuracy of the model, and broadening the application scenarios of the model.

[0078] See also Figure 5 , is a flow chart of a method for generating a text error correction task model provided in Example 5 of the present application. Figure 5 As shown, the step S203 of selecting the target Chinese text and the corresponding error correction annotation from the data set may include the following steps:

[0079] Step S501, filtering out a Chinese text subset with manual error correction annotations from the data set.

[0080] Step S502 , selecting texts from users whose native language is Chinese as target Chinese texts from the Chinese text subset, and determining error correction annotations corresponding to the target Chinese texts.

[0081] Among them, due to the uneven quality of public data sets, some public data sets are sampled from foreign language learners' examination websites, and there are significant differences between their expressions and those of native speakers, and some public data sets are generated. In addition, considering that it is difficult to obtain clean corpus with no grammatical errors in the case of generated data sets, the data in the database will be divided into high-quality data and low-quality data. Among them, the judgment criteria for high-quality data are: manual annotation, and the source of the corpus is native speakers, and its errors are consistent with the errors of native speakers.

[0082] Spelling errors do not actually focus on whether punctuation and grammar are correct. They are mainly typos, including glyph errors or pronunciation errors. For example, "Today is very high-tech", you only need to combine the context to determine whether there are wrong words in the sentence. Compared with grammatical errors and punctuation errors, the task is simpler, but the data required is richer, because there are many kinds of typos. The requirement for the model is: it is expected that the corpus data covers different fields and different majors, and involves glyph errors and pronunciation errors.

[0083] There is not much public data on grammatical errors, and most of them are data from second language learners, which is not very consistent with the situation of native speakers. Or, based on the correct corpus, generate corpus with grammatical errors according to the rules. Grammatical errors have high requirements on the correctness of the corpus, and there are very few perfect corpora without grammatical errors. Therefore, the biggest challenge of grammatical errors is the lack of high-quality training data. The total amount of high-quality grammatical error data in all public data sets does not exceed 50,000. For punctuation error data sets, due to the lack of relevant public data sets, it is the same as grammatical errors. This application can collect some punctuation corrections involved in elementary school, junior high school, and high school textbooks, and use them as high-quality data after manual annotation. In addition, some data can be generated based on the correct corpus using rules.

[0084] In the embodiment of the present application, conditions such as manual annotation and native language are used to realize the screening of target Chinese text, which can simply and effectively obtain the corresponding text data, thereby helping to improve the efficiency of building the training data set, thereby improving the efficiency and accuracy of model training.

[0085] See also Figure 6 , is a flow chart of a method for generating a text error correction task model provided in Example 6 of the present application. Figure 6 As shown, before obtaining the original Chinese text and the corresponding error correction annotation from the preset data set in the above step S201, the following steps may also be included:

[0086] Step S601, collecting correct alternative Chinese corpora.

[0087] In this embodiment, a database can be constructed to store relevant corpora. The corpora are from public data on the Internet as candidate Chinese corpora. The requirements for this part of Chinese corpora are:

[0088] 1. The data should be as clean as possible, without obvious word errors or other incoherent expressions;

[0089] 2. The sources should be as broad as possible, covering various fields, such as finance, medicine, science, physics, chemistry, literature, etc.;

[0090] 3. Try to be in the form of a sentence. If it is in the form of a passage, it needs to be processed into the form of a single sentence before being put into the database.

[0091] Step S602: For any alternative Chinese corpus, determine at least one Chinese character from the alternative Chinese corpus and determine the pinyin data of each Chinese character.

[0092] Step S603: Use the input method interface to generate characters based on the pinyin data of each Chinese character to obtain homophone words corresponding to each Chinese character.

[0093] Step S604: For any Chinese character, replace the Chinese character in the alternative Chinese corpus with the corresponding homophone word respectively to obtain the first Chinese text, and determine the error correction annotation corresponding to the alternative Chinese corpus and the homophone error type for the first Chinese text.

[0094] Among them, the homophone error generation step can utilize existing application programming interfaces (APIs), such as Google, Baidu, Sogou, etc. This API can generate a corresponding list of characters according to the input pinyin. For example, when inputting "nihao", the return is: 你好, 你号, 拟好, 倪浩, 你哈哦, 泥嚎, 尼, 拟, 妮.

[0095] Among them, after obtaining the list of characters, the rules for error generation are as follows:

[0096] 1. The list of characters is classified according to length. Those with a length greater than or equal to the original Chinese character form a complete candidate set, and those with a length of 1 form a first-character candidate set;

[0097] 2. If there is a complete candidate set, randomly select one from the complete candidate set to replace the original Chinese character. If not, randomly select one from the first-character candidate set to replace the first character of the original Chinese character.

[0098] The embodiments of the present application can utilize existing interfaces, so that they can be used for generating homophone words and can efficiently generate Chinese texts of the homophone error type.

[0099] Optionally, after collecting the correct alternative Chinese corpus, it further includes:

[0100] For any alternative Chinese corpus, determine at least one target Chinese character from the alternative Chinese corpus and determine the Chinese characters with similar glyphs to each target Chinese character;

[0101] For any target Chinese character, replace the target Chinese character in the alternative Chinese corpus with the corresponding Chinese character with a similar glyph to obtain the second Chinese text, and determine the error correction annotation corresponding to the alternative Chinese corpus and the glyph error type for the second Chinese text.

[0102] Among them, the generation process of the glyph confusion set is to download 3500 commonly used Chinese characters, convert the 3500 commonly used Chinese characters into the form of pictures, extract the features of the pictures, calculate the feature similarity, and find the top 10 Chinese characters most similar to a certain character. Each character has an artificially experienced glyph similarity set. The similarity set formed by the similarity is combined with the artificially experienced similarity set to form the final glyph confusion set, which can be used for approximate operations of similar Chinese characters and can efficiently generate Chinese text with glyph errors.

[0103] Optionally, after collecting the correct alternative Chinese corpus, it also includes:

[0104] For any candidate Chinese corpus, at least one word or punctuation mark is randomly deleted from the candidate Chinese corpus to obtain a third Chinese text, and the candidate Chinese corpus and the missing word error type are determined as error correction annotations for the third Chinese text.

[0105] Among them, any word, punctuation mark, etc. can be deleted therefrom, so that it can be used for missing word operation, and Chinese text with missing word error type can be generated efficiently.

[0106] Optionally, after collecting the correct alternative Chinese corpus, it also includes:

[0107] For any candidate Chinese corpus, at least one punctuation mark is randomly replaced from the candidate Chinese corpus to obtain a fourth Chinese text, and the candidate Chinese corpus and the punctuation error type are determined as error correction annotations for the fourth Chinese text.

[0108] Among them, any replacement of punctuation marks can efficiently generate Chinese text with punctuation error types.

[0109] Corresponding to the method for generating a text error correction task model in the above embodiment, Figure 7 The structure block diagram of the device for generating a text error correction task model provided in the seventh embodiment of the present application is shown. The device for generating a text error correction task model is applied to Figure 1 The server in the server, the computer device corresponding to the server is connected to the client, the database, etc., to obtain the corresponding data, and the data is processed accordingly. Of course, the model of the present application is a model for error correction tasks for Chinese text, so the model can also be deployed on the client, that is, the generation method of the present application can also be applied in the client. For ease of explanation, only the parts related to the embodiments of the present application are shown.

[0110] See also Figure 7 , the generating device for the text error correction task model comprises:

[0111] A text acquisition module 71 is used to acquire original Chinese text and corresponding error correction annotations from a preset data set, where the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0;

[0112] The first training module 72 is used to train a preset text error correction task model using the original Chinese text and the corresponding error correction annotations to obtain an initially trained text error correction task model;

[0113] A target screening module 73 is used to screen out target Chinese text and corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations;

[0114] The second training module 74 is used to use the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model to obtain a secondarily trained text error correction task model, and the secondarily trained text error correction task model is used to correct the Chinese text to be recognized.

[0115] Optionally, the error correction annotation includes the error correction target result and the error type. The first training module 72 includes:

[0116] An original splicing unit, used to splice the original Chinese text with the corresponding error correction target result and error type to obtain an original splicing result;

[0117] The first training unit is used to input the original splicing result into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

[0118] Optionally, the second training module 74 includes:

[0119] A target splicing unit is used to splice the target Chinese text with the corresponding error correction target result and error type to obtain a target splicing result;

[0120] The second training unit is used to input the target splicing result into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

[0121] Optionally, the target screening module 73 includes:

[0122] A first screening unit is used to screen out a Chinese text subset with error correction annotations of manually annotated type from the data set;

[0123] The second screening unit is used to screen out texts from users whose native language is Chinese as target Chinese texts from the Chinese text subset, and determine error correction marks corresponding to the target Chinese texts.

[0124] Optionally, the device for generating the text error correction task model further includes:

[0125] A corpus collection module is used to collect correct alternative Chinese corpora before obtaining the original Chinese text and the corresponding error correction annotations from a preset data set;

[0126] A pinyin module, for determining at least one word from any candidate Chinese corpus, and determining pinyin data of each word;

[0127] The phonetic word module is used to generate text from the pinyin data of each word using the input method interface to obtain the phonetic word corresponding to each word;

[0128] The first Chinese text generation module is used to replace the words in the alternative Chinese corpus with the phonetic words corresponding to any word for each word to obtain the first Chinese text, and determine the alternative Chinese corpus and the phonetic error type as the error correction mark corresponding to the first Chinese text.

[0129] Optionally, the device for generating the text error correction task model further includes:

[0130] A similar Chinese character module is used to determine at least one target Chinese character from the candidate Chinese corpus for any candidate Chinese corpus after collecting the correct candidate Chinese corpus, and determine a similar Chinese character with a similar shape to each target Chinese character;

[0131] The second Chinese text generation module is used to replace the target Chinese character in the alternative Chinese corpus with a similar Chinese character corresponding to the target Chinese character for any target Chinese character to obtain the second Chinese text, and determine the alternative Chinese corpus and the type of character shape error as the error correction mark of the second Chinese text.

[0132] Optionally, the device for generating the text error correction task model further includes:

[0133] The third Chinese text generation module is used to randomly delete at least one word or punctuation mark from any alternative Chinese corpus after collecting correct alternative Chinese corpus to obtain the third Chinese text, and determine the alternative Chinese corpus and the missing word error type as the error correction mark of the third Chinese text.

[0134] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules, units, and sub-units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0135] Figure 8 This is a schematic diagram of the structure of a computer device provided in Example 8 of the present application. Figure 8As shown, the computer device of this embodiment includes: at least one processor ( Figure 8 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the method for generating a text error correction task model are implemented.

[0136] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 8 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0137] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0138] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0139] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0140] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0142] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0143] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating a text error correction task model, characterized in that: include: Obtaining original Chinese texts and corresponding error correction annotations from a preset data set, wherein the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0; Using the original Chinese text and the corresponding error correction annotations, training a preset text error correction task model to obtain an initially trained text error correction task model; Filtering out target Chinese text and corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations; The target Chinese text and the corresponding error correction annotations are used to train the initially trained text error correction task model to obtain a secondary trained text error correction task model, and the secondary trained text error correction task model is used to correct the Chinese text to be recognized.

2. The method for generating a text error correction task model according to claim 1, characterized in that: The error correction annotation includes the error correction target result and the error type. The original Chinese text and the corresponding error correction annotation are used to train a preset text error correction task model to obtain an initially trained text error correction task model, including: Splicing the original Chinese text with the corresponding error correction target result and error type to obtain an original splicing result; The original splicing result is input into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

3. The method for generating a text error correction task model according to claim 2, characterized in that: The method of using the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model to obtain a secondarily trained text error correction task model includes: Splicing the target Chinese text with the corresponding error correction target result and error type to obtain a target splicing result; The target splicing result is input into a preset text error correction task model to train the preset text error correction task model to obtain an initially trained text error correction task model.

4. The method for generating a text error correction task model according to claim 1, characterized in that: The step of selecting the target Chinese text and the corresponding error correction annotation from the data set includes: Filtering out a subset of Chinese texts with manual error correction annotations from the data set; The texts from users whose native language is Chinese are selected from the Chinese text subset as target Chinese texts, and the error correction annotations corresponding to the target Chinese texts are determined.

5. The method for generating a text error correction task model according to any one of claims 1 to 4, characterized in that: Before obtaining the original Chinese text and the corresponding error correction annotation from the preset data set, the method further includes: Collect the correct alternative Chinese corpus; For any candidate Chinese corpus, determine at least one word from the candidate Chinese corpus, and determine the pinyin data of each word; Use the input method interface to generate text from the pinyin data of each word to obtain the phonetic word corresponding to each word; For any word, the words in the candidate Chinese corpus are replaced by the pronouns corresponding to the word to obtain a first Chinese text, and the candidate Chinese corpus and the pronoun error type are determined to be error correction marks corresponding to the first Chinese text.

6. The method for generating a text error correction task model according to claim 5, characterized in that: After collecting the correct alternative Chinese corpus, the following steps are also included: For any candidate Chinese corpus, determine at least one target Chinese character from the candidate Chinese corpus, and determine a similar Chinese character with a similar shape to each target Chinese character; For any target Chinese character, a similar Chinese character corresponding to the target Chinese character is used to replace the target Chinese character in the candidate Chinese corpus to obtain a second Chinese text, and the candidate Chinese corpus and the type of glyph error are determined as error correction annotations for the second Chinese text.

7. The method for generating a text error correction task model according to claim 5, characterized in that: After collecting the correct alternative Chinese corpus, the following steps are also included: For any candidate Chinese corpus, at least one word or punctuation mark is randomly deleted from the candidate Chinese corpus to obtain a third Chinese text, and the candidate Chinese corpus and the missing word error type are determined as error correction annotations for the third Chinese text.

8. A device for generating a text error correction task model, characterized in that: include: A text acquisition module, used to acquire original Chinese text and corresponding error correction annotations from a preset data set, wherein the data set includes N Chinese texts and error correction annotations corresponding to each Chinese text, where N is an integer greater than 0; A first training module is used to train a preset text error correction task model using the original Chinese text and the corresponding error correction annotations to obtain an initially trained text error correction task model; A target screening module, used to screen out target Chinese text and corresponding error correction annotations from the data set, wherein the target Chinese text is a Chinese text with accurate annotations; The second training module is used to use the target Chinese text and the corresponding error correction annotations to train the initially trained text error correction task model to obtain a secondary trained text error correction task model, and the secondary trained text error correction task model is used to correct the Chinese text to be recognized.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating a text error correction task model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a text error correction task model as described in any one of claims 1 to 7 is implemented.

Images