Method, device, computer equipment and storage medium for obtaining training samples

Abstract

The present application discloses a method, device, computer equipment and storage medium for obtaining training samples, and relates to the field of computer technology. The method comprises: obtaining a first text set and a second text set; obtaining the longest common subsequence in the correct text and the erroneous text corresponding to the correct text in the first text set; obtaining a text unit that does not belong to the longest common subsequence from the correct text as a rewritten text unit, and obtaining a text unit that does not belong to the longest common subsequence from the erroneous text as an erroneous text unit corresponding to the rewritten text unit; generating a confusion set based on the rewritten text unit and the erroneous text unit; replacing the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set, and obtaining a third text set. In this way, model training is performed based on the first text set and the third text set, which expands the text set for model training, thereby improving the generalization effect of the trained text error correction model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method, apparatus, computer device and storage medium for acquiring training samples. Background Art

[0002] Text error correction is an important research direction in computer natural language processing. Computer algorithms can be used to correct errors caused by human factors in text, such as typos, incorrect word order, etc. Taking the intelligent customer service system as an example, text error correction is applied to the intelligent customer service system. By correcting the text input by the user, the customer service can accurately judge the customer's needs.

[0003] Among them, text correction is generally implemented based on a text correction model. Since model training usually requires large-scale training data, however, in related technologies, the amount of training data that can be obtained is very limited, which leads to insufficient generalization ability of the text correction model trained based on limited training data. Summary of the invention

[0004] In view of this, the present application proposes a method, apparatus, computer device and storage medium for obtaining training samples.

[0005] In a first aspect, an embodiment of the present application provides a method for obtaining training samples, the method comprising: obtaining a first text set and a second text set, the first text set comprising annotated correct text and erroneous text, and the second text set not being annotated with correct text and erroneous text; obtaining the longest identical text unit in at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text as the longest common subsequence; obtaining a text unit that does not belong to the longest common subsequence from the correct text as a rewritten text unit, and obtaining a text unit that does not belong to the longest common subsequence from the erroneous text as an erroneous text unit corresponding to the rewritten text unit; generating a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit; replacing a specified text unit in the second text set with an erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, the specified text unit being a rewritten text unit in the confusion set and a text unit contained in the second text set, the first text set and the third text set being used together for training a text correction model, and the text correction model being used for correcting text to be corrected.

[0006] In a second aspect, an embodiment of the present application provides a training sample acquisition device, the device comprising: a text set acquisition module, a sequence acquisition module, a text unit acquisition module, a confusion set generation module and a training text acquisition module. The text set acquisition module is used to acquire a first text set and a second text set, the first text set comprising annotated correct text and erroneous text, and the second text set is not annotated with correct text and erroneous text; the sequence acquisition module is used to acquire the longest identical text unit in at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence; the text unit acquisition module is used to acquire text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and acquire text units that do not belong to the longest common subsequence from the erroneous text as The invention relates to an erroneous text unit corresponding to the rewritten text unit; a confusion set generation module, used for generating a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit; a training text acquisition module, used for replacing the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, wherein the specified text unit is a text unit included in both the rewritten text unit in the confusion set and the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used for correcting text to be corrected.

[0007] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method for acquiring training samples provided in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method for obtaining training samples provided in the first aspect.

[0009] In the scheme provided by the present application, a first text set and a second text set are obtained, wherein the first text set includes annotated correct text and erroneous text, and the second text set is not annotated with correct text and erroneous text; the longest identical text unit in at least one identical text unit contained in the correct text and the erroneous text corresponding to the correct text in the first text set is obtained as the longest common subsequence; text units that do not belong to the longest common subsequence are obtained from the correct text as rewritten text units, and text units that do not belong to the longest common subsequence are obtained from the erroneous text as erroneous text units corresponding to the rewritten text units; a confusion set is generated based on the rewritten text units and the erroneous text units corresponding to the rewritten text units; a specified text unit in the second text set is replaced by an erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, wherein the specified text unit is a text unit contained in both the rewritten text unit in the confusion set and the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used for correcting texts to be corrected. In this way, a confusion set is constructed based on the annotated first text set, and then a third text set is constructed based on the confusion set and the unannotated second text set. Then, the first text set and the third text set are used to train a text error correction model, thereby expanding the training text used to train the text error correction model and improving the generalization effect of the text error correction model. Moreover, the third text set constructed based on the confusion set can well retain the distribution of erroneous words in the first text set, so that the generated third text set has less noise, thereby making the text error correction model trained based on the first text set and the third text set more accurate in error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram showing an application scenario of a method for acquiring training samples provided in an embodiment of the present application.

[0012] Figure 2 A schematic diagram of a flow chart of a method for acquiring training samples provided in an embodiment of the present application is shown.

[0013] Figure 3 A schematic flow chart of a method for acquiring training samples provided in another embodiment of the present application is shown.

[0014] Figure 4 A schematic flow chart of a method for acquiring training samples provided in yet another embodiment of the present application is shown.

[0015] Figure 5 A schematic flow chart of a method for acquiring training samples provided in yet another embodiment of the present application is shown.

[0016] Figure 6 A schematic flow chart of a method for acquiring training samples provided in yet another embodiment of the present application is shown.

[0017] Figure 7 It is a block diagram of a training sample acquisition device provided according to an embodiment of the present application.

[0018] Figure 8 It is a block diagram of a computer device for executing the method for acquiring training samples according to an embodiment of the present application.

[0019] Fig. 9 It is a storage unit of an embodiment of the present application for storing or carrying a program code for implementing a method for acquiring training samples according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0021] Text error correction is an important research direction in computer natural language processing. Computer algorithms can be used to correct errors caused by human factors in text, such as typos, incorrect word order, etc. Taking the intelligent customer service system as an example, text error correction is applied to the intelligent customer service system. By correcting the text input by the user, the customer service can accurately judge the customer's needs.

[0022] Among them, text correction is generally implemented based on a text correction model. Since model training usually requires large-scale training data, however, in related technologies, the amount of training data that can be obtained is very limited, which leads to insufficient generalization ability of the text correction model trained based on limited training data.

[0023] In view of the above problems, the inventors propose a method, apparatus, computer device and storage medium for obtaining training samples, which can obtain a first text set and a second text set, generate a confusion set based on the first text set, and then replace the text units in the second text set based on the confusion set to obtain a third text set. The content is described in detail below.

[0024] The following introduces the application environment of the method for obtaining training samples provided in the embodiment of the present application.

[0025] Please refer to Figure 1 , Figure 1A schematic diagram of an application environment provided for an embodiment of the present application is shown. The method for obtaining training samples provided in the embodiment of the present application can be applied to Figure 1 The training sample acquisition system 10 shown. The training sample acquisition system 10 includes a computer device 101, which can be a terminal with data processing functions. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited to this. The computer device 101 can also be a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0026] In some implementations, the computer device 101 may obtain a first text set and a second text set, generate a confusion set based on the first text set, and then replace the text units in the second text set based on the confusion set to obtain a third text set, and then use the first text set and the third text set to train the text error correction model. The first text set and the second text set may be obtained by the computer device 101 from a local server, or may be obtained by the computer device 101 from other electronic devices or servers based on a wired or wireless network, and this embodiment does not limit this.

[0027] Please refer to Figure 2 , Figure 2 A method, apparatus, computer device and storage medium for obtaining training samples are provided in one embodiment of the present application. Figure 2 The method for obtaining a training sample provided in an embodiment of the present application is described in detail. The method for obtaining a training sample may include the following steps:

[0028] Step S110: obtaining a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not include annotated correct texts and erroneous texts.

[0029] In this embodiment, the first text set includes annotated correct text and erroneous text. The first text set can be manually annotated correct text and erroneous text, or annotated correct text and erroneous text downloaded from an open source database or a paid database. However, the number of texts in the first text set, whether manually annotated or downloaded from a database, is limited, and the training text for the text error correction model is not expanded. The unannotated second text set can be obtained, and a third text set that can be used for text error correction model training can be constructed based on the second text set. Among them, the unannotated second text set can be obtained by obtaining the text information input by the user in the customer service history record, and the customer service history record can also be based on the text information converted from the audio or video input by the user. .

[0030] Among them, there can be many types of incorrect text, including but not limited to homophonic errors (correct text: wear glasses - incorrect text: wear eyes), confused sound errors (correct text: Cowherd and Weaver Girl - incorrect text: Wandering Weaver Girl), reversed word order (correct text: Woody Allen - incorrect text: Allen Woody), missing words (correct text: If Love Has Destiny - incorrect text: If Love Has Destiny), errors in similar characters (correct text: sorghum - incorrect text: sorghum), full Chinese pinyin spelling (correct text: happiness - incorrect text: xingfu), Chinese pinyin abbreviations (correct text: Shenzhen - incorrect text: sz), grammatical errors (correct text: It's hard to imagine - incorrect text: It's hard to imagine), etc.

[0031] Step S120: Obtain the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence.

[0032] In this embodiment, each correct text in the first text set corresponds to an incorrect text, that is, the correct text in the first file set and its corresponding incorrect text can be regarded as a text pair. The incorrect text is generally obtained by performing transformation operations such as replacement and reordering of some text units in the corresponding correct text, that is, only some text units in the incorrect text that have undergone the aforementioned transformation operation are inconsistent with those in the correct text, and the remaining text units are consistent with those in the correct text. Therefore, by obtaining the longest identical text unit in the correct text and the incorrect text corresponding to the correct text, the partial text unit in the correct text that has undergone the aforementioned transformation operation can be determined based on the longest identical text unit.

[0033] In some embodiments, the correct text and the erroneous text can be divided according to word granularity to obtain a correct text sequence corresponding to the correct text and an erroneous text sequence corresponding to the erroneous text, wherein the correct text and the erroneous text can be divided into word granularity using a word segmentation algorithm, and the word segmentation algorithm includes but is not limited to a word segmentation algorithm based on string matching, a word segmentation algorithm based on n-gram grammar, a word segmentation algorithm based on hidden Markov model, and a word segmentation algorithm based on conditional random field, etc.

[0034] For example, the correct text is "I want to watch the TV series The Cowherd and the Weaver Girl", and the incorrect text is "I want to watch the TV series The Wandering Weaver Girl". After dividing the word granularity through the word segmentation algorithm, we can get the correct text sequence {I, want, watch, TV series, The Cowherd and the Weaver Girl} and the incorrect text sequence {I, want, watch, TV series, The Wandering Weaver Girl}.

[0035] Furthermore, based on the dynamic programming method, the longest identical subsequence contained in the correct text sequence and the incorrect text sequence is obtained as the longest common subsequence (LCS). Wherein, each element in the correct text sequence and the incorrect text sequence can be understood as a text unit, and the length of the longest common subsequence can be expressed by the following formula:

[0036]

[0037] Wherein, s1 and s2 represent any correct text sequence and the error text sequence corresponding to the correct text sequence, s[:-1] represents the subsequence of s after removing the last text unit, and s[-1] represents the last text unit of sequence s. The above formula can be understood as follows: when the correct text sequence and its corresponding error text sequence are both empty sequences, the length of the longest common subsequence obtained is 0; when the last text unit of the correct text sequence is the same as the last text unit of its corresponding error text sequence, the length of the longest common subsequence obtained is the length of the longest common subsequence of the correct text sequence with the last text unit removed and the error text sequence with the last text unit removed plus 1; in other cases except the above two cases, the longest common subsequence of the correct text sequence with the last text unit removed and the error text sequence is obtained as the first subsequence, and the longest common subsequence of the correct text sequence and the error text sequence with the last text unit removed is obtained as the second subsequence, and the length of the longest subsequence between the first subsequence and the second subsequence is obtained as the length of the aforementioned longest common subsequence.

[0038] Still taking the above example, the length of the correct text sequence is 5, the length of the wrong text sequence is also 5, and the last text unit of the correct text sequence is similar to the last text unit of the wrong text sequence, then the longest common subsequence is calculated and obtained according to the third method in the above formula; specifically, the longest common subsequence of the correct text sequence and the wrong text sequence without the last text unit is obtained as the first subsequence, that is, {I, want, watch, TV series}, whose length is 4, and the longest common subsequence of the correct text sequence and the wrong text sequence without the last text unit is obtained as the second subsequence, which is {I, want, watch, TV series}, whose length is 4, and the longest subsequence between the first subsequence and the second subsequence is obtained as the aforementioned longest common subsequence. Therefore, the longest common subsequence between the correct text sequence and the wrong text sequence can be obtained as {I, want, watch, TV series}, whose length is 4.

[0039] Step S130: obtaining text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and obtaining text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0040] Based on this, after obtaining the longest common subsequence, we can obtain the text unit that does not belong to the longest common subsequence from the correct text as the rewritten text unit, and the rewritten text unit is the form before the aforementioned transformation operation; and obtain the text unit that does not belong to the longest common subsequence from the incorrect text as the error text unit corresponding to the rewritten text unit, and the error text unit is the form of the aforementioned rewritten text unit after the aforementioned transformation operation.

[0041] Taking the above example, the text unit "Cowherd and Weaver Girl" that does not belong to the longest common subsequence is obtained from the correct text "I want to watch the TV series Cowherd and Weaver Girl" as the rewritten text unit; the text unit "Wandering Weaver Girl" that does not belong to the longest common subsequence is obtained from the incorrect text as the incorrect text unit, and the rewritten text unit "Cowherd and Weaver Girl" corresponds to the incorrect text unit "Wandering Weaver Girl".

[0042] Step S140: Generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit.

[0043] In this embodiment, the confusion set can be understood as a rewriting dictionary. After obtaining the rewritten text units in the first text set and the erroneous text units corresponding to the rewritten text units, each rewritten text unit and the erroneous text unit corresponding to each rewritten text unit can be added to the confusion set. Among them, the correspondence between the rewritten text units and the erroneous text units in the confusion set can be one-to-one, for example, the rewritten text unit "Cowherd and Weaver Girl" only corresponds to one erroneous text unit "Wandering Weaver Girl"; of course, the correspondence can also be one-to-many, for example, the rewritten text unit "Cowherd and Weaver Girl" forms a one-to-many correspondence with the erroneous text unit "Wandering Weaver Girl", the erroneous text unit "Liu Lang Weaver Girl" and the erroneous text unit "Stay Lang Weaver Girl", and this embodiment does not limit this.

[0044] Step S150: Replace the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain a third text set, wherein the designated text unit is a rewritten text unit in the confusion set and a text unit included in the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used for correcting text to be corrected.

[0045] In this embodiment, the aforementioned confusion set generated based on the first text set can be regarded as a rewriting dictionary, and the text units in the second text set are rewritten based on the rewriting dictionary to obtain the third text set.

[0046] In some embodiments, a rewritten text unit in the confusion set and a text unit included in both the second text set are obtained as designated text units; unlabeled text in the second text set including the designated text unit is obtained as unlabeled correct text in the second text set; the designated text unit in the second text set is replaced with an erroneous text unit corresponding to the designated text unit in the confusion set; the unlabeled text after the designated text unit is replaced is obtained as unlabeled erroneous text corresponding to the unlabeled correct text; and corresponding labels are respectively annotated for the unlabeled correct text and the unlabeled erroneous text to obtain a third text set.

[0047] Exemplarily, if the unannotated text in the second text set includes any rewritten text unit in the confusion set, for example, the confusion set includes the rewritten text unit "Cowherd and Weaver Girl", and its corresponding erroneous text unit is "Wandering Weaver Girl", and the unannotated text in the second text set is "I want to watch Cowherd and Weaver Girl", then it can be determined that the unannotated text also contains the rewritten text unit in the confusion set. Therefore, "Cowherd and Weaver Girl" is taken as the above-mentioned designated text unit; further, the unannotated text "I want to watch Cowherd and Weaver Girl" is taken as the unannotated correct text, and the unannotated text "I want to watch Wandering Weaver Girl" after replacing the designated text unit is taken as the unannotated erroneous text; finally, the unannotated correct text "I want to watch Cowherd and Weaver Girl" is labeled with a correct label, and the unannotated erroneous text "I want to watch Wandering Weaver Girl" is labeled with an erroneous label, to obtain the aforementioned third text set.

[0048] In this embodiment, after obtaining the third text set, the first text set and the third text set are merged to obtain a fourth text set, wherein the merge can be a random shuffle merge to make the text in the fourth text set more random; based on the fourth text set, the initial model is iteratively trained until the initial model meets the preset conditions to obtain the text error correction model.

[0049] In some embodiments, the loss value during model training can be obtained based on the cross entropy loss function, and based on the loss value, the initial model is iteratively trained until the initial model meets the preset conditions to obtain a text error correction model. Wherein, the initial model includes but is not limited to a BERT model, a transformer model, and a Sequence to Sequence model. The preset conditions may be: the loss value is less than the preset value, the loss value no longer changes, or the number of training times reaches the preset number of times, etc. It can be understood that after the initial model is iteratively trained for multiple training cycles according to the fourth text set, each training cycle includes multiple iterative trainings, and the parameters in the initial model are continuously optimized, so that the above loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above preset value. At this time, it means that the initial model has converged; of course, it can also be determined that the initial model has converged after the number of training times reaches the preset number of times. At this time, the initial model can be used as the above text error correction model. Wherein, the preset value and the preset number of times are both pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this. For example, the Sequence to Sequence model is a variant of a recurrent neural network, which includes an encoder and a decoder. The encoder inputs the erroneous text, and the decoder outputs the corrected text. In the training phase, the cross entropy loss function is calculated based on the joint probability distribution of the decoder output sequence to obtain the loss value. Based on the loss value and the aforementioned preset conditions, the initial model is iteratively trained to obtain a text error correction model.

[0050] In this embodiment, a confusion set is constructed based on the annotated first text set, and then a third text set is constructed based on the confusion set and the unannotated second text set, and then the first text set and the third text set are used to train a text error correction model, thereby expanding the training text used to train the text error correction model and improving the generalization effect of the text error correction model; and the third text set constructed based on the confusion set can well retain the distribution of erroneous words in the first text set, so that the generated third text set has less noise, thereby making the text error correction model trained based on the first text set and the third text set more accurate in error correction.

[0051] Please refer to Figure 3 , Figure 3 Another embodiment of the present application provides a method, device, computer equipment and storage medium for obtaining training samples. Figure 3 The method for obtaining a training sample provided in an embodiment of the present application is described in detail. The method for obtaining a training sample may include the following steps:

[0052] Step S210: obtaining a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not include annotated correct texts and erroneous texts.

[0053] Step S220: Obtain the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence.

[0054] Step S230: obtaining text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and obtaining text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0055] Step S240: Generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit.

[0056] In this embodiment, the specific contents of step S210 to step S250 can refer to the contents of the above-mentioned embodiment, which will not be repeated here.

[0057] Step S250: Obtain the replacement probability corresponding to the specified text unit.

[0058] In some implementations, after obtaining the rewritten text unit in the confusion set and the designated text unit included in the second text set, the computer device may randomly assign a replacement probability to the designated text unit.

[0059] In other embodiments, after obtaining the rewritten text unit of the confusion set and the designated text unit contained in the second text set, the computer device can obtain the semantic information corresponding to the unlabeled text containing the designated text unit in the second text set; according to the semantic information, the replacement probability of the designated text unit is determined. Among them, according to the semantic information of the unlabeled text, the weight (i.e., importance) corresponding to each text unit in the unlabeled text can be obtained by using the attention mechanism, and further, according to the weight of the designated text unit, its replacement probability can be determined, and the replacement probability is positively correlated with the weight. The higher the weight, the higher the replacement probability assigned by the computer device to the designated text unit, and vice versa. In this way, the replacement probability determined based on the semantic information can be used to replace the more important text units in the unlabeled text when constructing the third text set based on the confusion set, so that the trained text error correction model can accurately identify the more important words with errors in the text during the application stage, thereby improving the accuracy of the text error correction model.

[0060] Step S260: Determine whether the replacement probability is greater than a probability threshold.

[0061] Step S270: If yes, replace the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set.

[0062] In this embodiment, after obtaining the replacement probability corresponding to the specified text, it can be determined whether the replacement probability is greater than the probability threshold. If it is greater, the specified text unit in the second text set is replaced with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain the third text set. If it is not greater, the unlabeled text containing the specified text unit is not processed. The specific implementation method of replacing the third text set can refer to the content in the aforementioned embodiment, which will not be repeated here. Among them, the probability threshold can be a pre-set constant value, such as 0.7. Of course, the probability threshold can also be adjusted according to different application scenarios, and this embodiment does not limit this.

[0063] In this embodiment, the designated text unit in the second text set is replaced only when the replacement probability of the designated text unit is greater than the probability threshold. In this way, not all the designated text units in the second text set are replaced, but some of the designated text units are selectively replaced according to the replacement probability, thereby improving the randomness of the third text set, and further improving the generalization of the text error correction model trained based on the third text set and the first text set.

[0064] Please refer to Figure 4 , Figure 4 A method, apparatus, computer device and storage medium for obtaining training samples are provided in another embodiment of the present application. Figure 4 The method for obtaining a training sample provided in an embodiment of the present application is described in detail. The method for obtaining a training sample may include the following steps:

[0065] Step S310: obtaining a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not include annotated correct texts and erroneous texts.

[0066] Step S320: Obtain the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence.

[0067] Step S330: obtaining text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and obtaining text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0068] Step S340: Generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit.

[0069] In this embodiment, the specific contents of step S310 to step S340 can refer to the contents of the above-mentioned embodiments, which will not be repeated here.

[0070] Step S350: outputting the confusion set and prompt information, wherein the prompt information is used to prompt the user to check whether there is a rewritten text unit in the confusion set that does not correspond to an erroneous text unit.

[0071] In this embodiment, after generating a confusion set based on rewritten text units and erroneous text units, there may be some erroneous text units that have no corresponding rewritten text units. For example, the correct text is "If Love Has Destiny, this drama is very good", and the erroneous text is "If Love Has Destiny, this drama is very good". In this way, the rewritten text unit added to the confusion set is "If", but there is no corresponding erroneous text unit. Based on this, a prompt message can be output to prompt the user to check whether the situation in the aforementioned example exists in the confusion set, that is, to check whether there is a rewritten text unit in the confusion set that has no corresponding erroneous text unit.

[0072] Optionally, the aforementioned prompt information may also prompt the user to detect whether there are other forms of erroneous text units in the confusion set, that is, prompt the user to perform a second manual check on the confusion set to ensure the correctness of the confusion set.

[0073] Step S360: In response to a deletion operation input by the user based on the prompt information, the rewritten text unit in the confusion set that has no corresponding erroneous text unit is deleted.

[0074] Based on this, after outputting the prompt information, the user can input a deletion operation based on the prompt information to delete all rewritten text units in the confusion set that do not correspond to erroneous text units, to prevent the subsequent replacement of specified text units in the second text set due to the lack of corresponding rewritten text units, resulting in replacement failure, slowing down the process of constructing the third text set, and affecting subsequent model training.

[0075] Step S370: Replace the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain a third text set.

[0076] In this embodiment, the specific content in step S370 can refer to the content in the above-mentioned embodiment, which will not be repeated here.

[0077] In this embodiment, a prompt message can be output to prompt the user to perform a secondary check on the generated confusion set, and to delete the rewritten text units in the confusion set that do not correspond to the erroneous text units, so as to ensure that the formats of the rewritten text units and the erroneous text units contained in the confusion set are correct, thereby ensuring the smooth progress of the subsequent process of replacing the specified text units in the second text set based on the confusion set and the obtained third text set is more consistent with the distribution of the erroneous text units in the first text set, reducing text noise and improving the accuracy of the trained text error correction model.

[0078] Please refer to Figure 5 , Figure 5 A method, apparatus, computer device and storage medium for obtaining training samples are provided in another embodiment of the present application. Figure 5 The method for obtaining a training sample provided in an embodiment of the present application is described in detail. The method for obtaining a training sample may include the following steps:

[0079] Step S410: obtaining a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not include annotated correct texts and erroneous texts.

[0080] Step S420: Obtain the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence.

[0081] Step S430: obtaining text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and obtaining text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0082] In this embodiment, the specific contents of step S410 to step S430 can refer to the contents of the above-mentioned embodiments, which will not be repeated here.

[0083] Step S440: If it is detected that there is a rewritten text unit that does not correspond to the erroneous text unit, the rewritten text unit that does not correspond to the erroneous text unit is obtained as the first rewritten text unit, and the rewritten text unit that corresponds to the erroneous text unit is obtained as the second rewritten text unit.

[0084] Step S450: Add a preset operation identifier to the first rewritten text unit to obtain a third rewritten text unit, wherein the preset operation identifier is used to indicate that the text unit in the target text is transformed according to the preset change operation corresponding to the preset operation identifier, and the target text is the text in the second text set that contains the first rewritten text unit.

[0085] In this embodiment, if it is detected that there is a rewritten text unit that does not correspond to an erroneous text unit, the rewritten text unit that does not correspond to the erroneous text unit is obtained as the first rewritten text unit, and the rewritten text unit corresponding to the erroneous text unit is obtained as the second rewritten text unit; in order to prevent the subsequent failure of successful replacement due to the fact that the first rewritten text unit has no corresponding erroneous text unit, a preset operation identifier can be added to the first rewritten text unit to obtain a third rewritten text unit, wherein the preset operation identifier is used to indicate that the text unit in the target text is transformed according to the preset change operation corresponding to the preset operation identifier, and the target text is the text in the second text set that contains the first rewritten text unit.

[0086] Exemplarily, the correct text is "If Love Has Destiny, this drama is very good", and the incorrect text is "If Love Has Destiny, this drama is very good". In this way, the rewritten text unit added to the confusion set is "If", but there is no corresponding incorrect text unit. Correspondingly, a preset operation mark (such as a deletion mark) can be added to the rewritten text unit "Add". The deletion mark can directly delete the specified text unit when the specified text unit is replaced subsequently, and the unmarked text after deleting the specified text unit will be used as the incorrect text in the third text set.

[0087] Step S460: Generate the confusion set based on the second rewritten text unit, the erroneous text unit corresponding to the second rewritten text unit, and the third rewritten text unit.

[0088] Step S470: Replace the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain a third text set.

[0089] In this embodiment, the specific content in step S470 can refer to the content in the above-mentioned embodiment, which will not be repeated here.

[0090] In this embodiment, a preset operation identifier is added to the first rewritten text unit that has no corresponding erroneous text unit, so that the specified text unit in the second text set is subsequently changed according to the preset change operation corresponding to the preset operation identifier based on the confusion set, so that the distribution of erroneous text units and missing text units in the obtained third text set is more consistent with the distribution in the first text set, so that the constructed third text set has less text noise, and the error correction accuracy of the trained text correction model is higher.

[0091] Please refer to Figure 6 , Figure 6 Another embodiment of the present application provides a method, device, computer equipment and storage medium for obtaining training samples. Figure 6The method for obtaining a training sample provided in an embodiment of the present application is described in detail. The method for obtaining a training sample may include the following steps:

[0092] Step S510: Acquire a first text set, wherein the first text set includes annotated correct texts and incorrect texts.

[0093] In this embodiment, the specific content in step S510 can refer to the content in the above-mentioned embodiment, which will not be repeated here.

[0094] Step S520: obtaining text information input by the user in the customer service history records under the target application scenario as a second text set corresponding to the target application scenario, wherein the second text set is not marked with correct text and incorrect text.

[0095] In this embodiment, the intelligent customer service system can be applied to different application scenarios, such as government affairs consulting customer service, e-commerce consulting customer service, medical consulting customer service, etc. In different application scenarios, the types of text information input by users are also different. Therefore, for the unlabeled second text set in different application scenarios, the text unit can be rewritten through the confusion set to obtain the third text set in the corresponding application scenario. In this way, the content of the third text set for training the text error correction model constructed based on the second text set can be more in line with the text content of the text error correction model in different application scenarios, thereby making the recognition and error correction accuracy of the trained text error correction model higher.

[0096] Step S530: Obtain the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence.

[0097] Step S540: obtaining text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and obtaining text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0098] Step S550: Generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit.

[0099] In this embodiment, the specific contents of step S530 to step S550 can refer to the contents of the aforementioned embodiment, which will not be repeated here.

[0100] Step S560: Replace the specified text unit in the second text set corresponding to the target application scenario with the erroneous text unit corresponding to the specified text unit in the confusion set, and obtain a third text set carrying the scenario identifier of the target application scenario, wherein the scenario identifier is used to search for the third text set corresponding to the target application scenario when training a text error correction model for the target application scenario.

[0101] In this embodiment, the confusion set generated above is used to replace the specified text unit in the second text set corresponding to the target application scenario with the erroneous text unit corresponding to the specified text unit in the confusion set, so as to obtain a third text set carrying the scenario identifier of the target application scenario. In this way, the text content in the third text set is more consistent with the text content actually input by the user in the target application scenario, so that the text error correction model trained based on the third text set has a more accurate error correction effect in the target application scenario.

[0102] In some embodiments, after obtaining a third text set carrying a scene identifier, the third text set can be merged with the first text set to obtain a fourth text set carrying the same scene identifier, and then a model training is performed based on the fourth text set to obtain a text error correction model carrying a scene identifier. Optionally, the text error correction model carrying a scene identifier can be stored so that the corresponding text error correction model can be directly called through the scene identifier in the future to correct the text to be corrected in the application scenario corresponding to the scene identifier.

[0103] In other embodiments, after obtaining the third text set carrying the scene identifier, the third text set can also be stored so that the corresponding third text set can be directly obtained according to the scene identifier in the future to train the text correction model, or to facilitate subsequent search and update of the third text set in a certain application scenario, and iteratively update the text correction model based on the updated third text set.

[0104] In some other implementations, it is also possible to obtain the first text set under different application scenarios, and then construct a corresponding confusion set for each application scenario. In this way, the third text set can be more in line with the text content in the actual application scenario, thereby improving the recognition accuracy of the trained text error correction model.

[0105] In this embodiment, a third text set corresponding to different application scenarios can be generated, so that when training text correction models for different application scenarios, the corresponding third text set can be directly searched according to the scenario identifier, thereby improving the efficiency of model training. At the same time, it also makes the trained text correction model have a higher error correction accuracy in the corresponding application scenario.

[0106] Please refer to Figure 7, which shows a structural block diagram of a training sample acquisition device 600 provided in an embodiment of the present application. The device 600 may include: a text set acquisition module 610, a sequence acquisition module 620, a text unit acquisition module 630, a confusion set generation module 640 and a training text acquisition module 650.

[0107] The text set acquisition module 610 is used to acquire a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not have annotated correct texts and erroneous texts.

[0108] The sequence acquisition module 620 is used to acquire the longest identical text unit among at least one identical text unit contained in the correct text and the erroneous text corresponding to the correct text in the first text set as the longest common subsequence.

[0109] The text unit acquisition module 630 is used to acquire text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and to acquire text units that do not belong to the longest common subsequence from the incorrect text as incorrect text units corresponding to the rewritten text units.

[0110] The confusion set generating module 640 is used to generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit.

[0111] The training text acquisition module 650 is used to replace the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, wherein the specified text unit is a rewritten text unit in the confusion set and a text unit included in the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used to correct the text to be corrected.

[0112] In some embodiments, the training text acquisition module 650 includes: a probability acquisition unit, a judgment unit, and a text acquisition unit. The probability acquisition unit can be used to obtain the replacement probability corresponding to the specified text unit. The judgment unit can be used to determine whether the replacement probability is greater than a probability threshold. The text acquisition unit can be used to replace the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain the third text set if the replacement probability is greater than the probability threshold.

[0113] In this manner, the probability acquisition unit may be specifically used to: acquire semantic information corresponding to the unlabeled text containing the designated text unit in the second text set; and determine the replacement probability of the designated text unit according to the semantic information.

[0114] In some embodiments, the training text acquisition module 650 can be specifically used to: obtain the unlabeled text in the second text set that contains the specified text unit as the unlabeled correct text in the second text set; replace the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set; obtain the unlabeled text after replacing the specified text unit as the unlabeled erroneous text corresponding to the unlabeled correct text; and respectively label the unlabeled correct text and the unlabeled erroneous text with corresponding labels to obtain the third text set.

[0115] In some embodiments, the sequence acquisition module 620 may include: a text segmentation unit and a sequence acquisition unit. The text segmentation unit may be used to segment the correct text and the incorrect text according to word granularity to obtain a correct text sequence corresponding to the correct text and an incorrect text sequence corresponding to the incorrect text. The sequence acquisition unit may be used to obtain the longest identical subsequence contained in both the correct text sequence and the incorrect text sequence as the longest common subsequence based on dynamic programming.

[0116] In some embodiments, the training sample acquisition device 600 may further include: an information output module and a deletion module. The information output module may be used to output the confusion set and prompt information before replacing the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain the third text set, wherein the prompt information is used to prompt the user to check whether there is a rewritten text unit in the confusion set that does not correspond to an erroneous text unit. The deletion module may be used to delete the rewritten text unit in the confusion set that does not correspond to an erroneous text unit in response to a deletion operation input by the user based on the prompt information.

[0117] In other embodiments, the confusion set generation module 640 can be specifically used for: if it is detected that there is a rewritten text unit that does not correspond to an erroneous text unit, obtaining the rewritten text unit that does not correspond to the erroneous text unit as the first rewritten text unit, and obtaining the rewritten text unit that corresponds to the erroneous text unit as the second rewritten text unit; adding a preset operation identifier to the first rewritten text unit to obtain a third rewritten text unit, the preset operation identifier is used to indicate that the text unit in the target text is transformed according to a preset change operation corresponding to the preset operation identifier, and the target text is the text in the second text set that contains the first rewritten text unit; based on the second rewritten text unit, the erroneous text unit corresponding to the second rewritten text unit, and the third rewritten text unit, the confusion set is generated.

[0118] In some implementations, the text set acquisition module 610 may be specifically configured to acquire text information input by a user in a customer service history record as the second text set.

[0119] In other embodiments, the text set acquisition module 610 can be specifically used to acquire text information input by users in the customer service history records under the target application scenario as the second text set corresponding to the target application scenario. The training text acquisition module 650 can be specifically used to replace the specified text unit in the second text set corresponding to the target application scenario with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set carrying the scenario identifier of the target application scenario, and the scenario identifier is used to find the third text set corresponding to the target application scenario when training the text error correction model used in the target application scenario.

[0120] In some embodiments, the training sample acquisition device 600 may further include: a text merging module and a model training module. The text merging module may be used to merge the first text set with the third text set to obtain a fourth text set after replacing the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain the third text set. The model training module may be used to iteratively train the initial model based on the fourth text set until the initial model meets the preset conditions to obtain the text error correction model.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0122] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0123] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0124] In summary, in the solution provided by the embodiment of the present application, a first text set and a second text set are obtained, wherein the first text set includes annotated correct text and erroneous text, and the second text set is not annotated with correct text and erroneous text; the longest identical text unit in at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text is obtained as the longest common subsequence; text units that do not belong to the longest common subsequence are obtained from the correct text as rewritten text units, and text units that do not belong to the longest common subsequence are obtained from the erroneous text as erroneous text units corresponding to the rewritten text units; a confusion set is generated based on the rewritten text units and the erroneous text units corresponding to the rewritten text units; a specified text unit in the second text set is replaced by an erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, wherein the specified text unit is a rewritten text unit in the confusion set and a text unit contained in the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used to correct the text to be corrected. In this way, a confusion set is constructed based on the annotated first text set, and then a third text set is constructed based on the confusion set and the unannotated second text set. The first text set and the third text set are used to train the text error correction model, which expands the training text used to train the text error correction model and improves the generalization effect of the text error correction model. In addition, the third text set constructed based on the confusion set can well retain the distribution of erroneous words in the first text set, so that the generated third text set has less noise, thereby making the text error correction model trained based on the first text set and the third text set more accurate.

[0125] A computer device provided by the present application will be described below with reference to the drawings.

[0126] Reference Figure 8 , Figure 8 The structural block diagram of a computer device 700 provided in an embodiment of the present application is shown. The method for obtaining training samples provided in an embodiment of the present application can be executed by the computer device 700. The computer device 700 can be a device capable of running an application program.

[0127] The computer device 700 in the embodiment of the present application may include one or more of the following components: a processor 701, a memory 702, and one or more applications, wherein the one or more applications may be stored in the memory 702 and configured to be executed by one or more processors 701, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0128] The processor 701 may include one or more processing cores. The processor 701 uses various interfaces and lines to connect the various parts of the entire computer device 700, and executes various functions and processes data of the computer device 700 by running or executing instructions, programs, code sets or instruction sets stored in the memory 702, and calling data stored in the memory 702. Optionally, the processor 701 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 701 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 701 and implemented separately through a communication chip.

[0129] The memory 702 may include a random access memory (RAM) or a read-only memory (ROM). The memory 702 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the computer device 700 during use.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0131] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0132] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0133] Please refer to Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0134] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for obtaining training samples, characterized in that: The method comprises: Acquire a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not include annotated correct texts and erroneous texts; Obtaining the longest identical text unit among at least one identical text unit contained in the correct text and the erroneous text corresponding to the correct text in the first text set as the longest common subsequence; Acquire a text unit that does not belong to the longest common subsequence from the correct text as a rewritten text unit, and acquire a text unit that does not belong to the longest common subsequence from the erroneous text as an erroneous text unit corresponding to the rewritten text unit; generating a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit; The designated text unit in the second text set is replaced by the erroneous text unit corresponding to the designated text unit in the confusion set to obtain a third text set, wherein the designated text unit is a rewritten text unit in the confusion set and a text unit included in the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used for correcting text to be corrected.

2. The method according to claim 1, characterized in that The step of replacing the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set includes: Obtaining a replacement probability corresponding to the specified text unit; Determining whether the replacement probability is greater than a probability threshold; If so, the designated text unit in the second text set is replaced by the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set.

3. The method according to claim 2, characterized in that The obtaining the replacement probability corresponding to the specified text unit includes: Acquire semantic information corresponding to the unlabeled text containing the specified text unit in the second text set; The replacement probability of the designated text unit is determined according to the semantic information.

4. The method according to claim 1 or 2, characterized in that: The step of replacing the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set includes: Acquire the unannotated text in the second text set that contains the specified text unit as the unannotated correct text in the second text set; Replacing the designated text unit in the second text set with an erroneous text unit corresponding to the designated text unit in the confusion set; Obtaining the unmarked text after replacing the designated text unit as the unmarked error text corresponding to the unmarked correct text; The unlabeled correct text and the unlabeled incorrect text are respectively labeled with corresponding labels to obtain the third text set.

5. The method according to claim 1, characterized in that: The obtaining, as the longest common subsequence, the longest identical text contained in both the correct text and the erroneous text corresponding to the correct text, includes: Dividing the correct text and the erroneous text according to word granularity to obtain a correct text sequence corresponding to the correct text and an erroneous text sequence corresponding to the erroneous text; Based on a dynamic programming approach, the longest identical subsequence contained in both the correct text sequence and the incorrect text sequence is obtained as the longest common subsequence.

6. The method according to claim 1, characterized in that Before replacing the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set, the method further includes: Outputting the confusion set and prompt information, wherein the prompt information is used to prompt the user to check whether there is a rewritten text unit in the confusion set that does not correspond to an erroneous text unit; In response to a deletion operation input by the user based on the prompt information, the rewritten text unit in the confusion set that has no corresponding erroneous text unit is deleted.

7. The method according to claim 1, characterized in that The generating a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit comprises: If it is detected that there is a rewritten text unit that does not correspond to the erroneous text unit, the rewritten text unit that does not correspond to the erroneous text unit is obtained as the first rewritten text unit, and the rewritten text unit that corresponds to the erroneous text unit is obtained as the second rewritten text unit; Adding a preset operation identifier to the first rewritten text unit to obtain a third rewritten text unit, wherein the preset operation identifier is used to indicate that a text unit in a target text is transformed according to a preset change operation corresponding to the preset operation identifier, and the target text is the text in the second text set that includes the first rewritten text unit; The confusion set is generated based on the second rewritten text unit, the erroneous text unit corresponding to the second rewritten text unit, and the third rewritten text unit.

8. The method according to claim 1, characterized in that: The step of acquiring the second text set includes: The text information input by the user in the customer service history record is obtained as the second text set.

9. The method according to claim 8, characterized in that The step of obtaining the text information input by the user in the customer service history records as the second text set includes: Acquire text information input by the user in the customer service history records under the target application scenario as a second text set corresponding to the target application scenario; The step of replacing the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set includes: The specified text unit in the second text set corresponding to the target application scenario is replaced with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set carrying a scene identifier of the target application scenario. The scene identifier is used to find the third text set corresponding to the target application scenario when training a text error correction model for the target application scenario.

10. The method according to claim 1, characterized in that After replacing the designated text unit in the second text set with the erroneous text unit corresponding to the designated text unit in the confusion set to obtain the third text set, the method further includes: Merging the first text set with the third text set to obtain a fourth text set; Based on the fourth text set, the initial model is iteratively trained until the initial model meets preset conditions, thereby obtaining the text error correction model.

11. A device for acquiring training samples, characterized in that: The device comprises: A text set acquisition module, used to acquire a first text set and a second text set, wherein the first text set includes annotated correct texts and erroneous texts, and the second text set does not have annotated correct texts and erroneous texts; A sequence acquisition module, used for acquiring the longest identical text unit among at least one identical text unit contained in the correct text in the first text set and the erroneous text corresponding to the correct text, as the longest common subsequence; A text unit acquisition module, used to acquire text units that do not belong to the longest common subsequence from the correct text as rewritten text units, and to acquire text units that do not belong to the longest common subsequence from the erroneous text as erroneous text units corresponding to the rewritten text units; A confusion set generation module, used to generate a confusion set based on the rewritten text unit and the erroneous text unit corresponding to the rewritten text unit; A training text acquisition module is used to replace the specified text unit in the second text set with the erroneous text unit corresponding to the specified text unit in the confusion set to obtain a third text set, wherein the specified text unit is a rewritten text unit in the confusion set and a text unit included in the second text set, and the first text set and the third text set are used together for training a text correction model, and the text correction model is used for correcting the text to be corrected.

12. A computer device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.

Images