Label text generation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an annotation text generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that a labeled text template set is obtained, the labeled text template set is obtained based on an initial question text labeled with a reply text label, and the labeled text template set comprises a question template text labeled with a reply template text label; obtaining a plurality of entity keyword sample groups, and replacing the preset identifiers in the annotation text template set with the plurality of entity keyword sample groups to obtain a plurality of candidate annotation text set groups; and according to text features of a preset sample text, adjusting the plurality of candidate annotation text set groups to obtain a target annotation text set. By adopting the method, the efficiency and the accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for generating annotated text. Background Art

[0002] In the field of machine learning, a large amount of labeled data is required as training samples to train the model to ensure the performance of the trained model. Therefore, the accuracy and quantity of labeled data are crucial to the model training effect.

[0003] In the related technology, a large amount of data needs to be manually labeled. However, due to the complexity of the labeling rules and the large amount of data, this method has a large manual workload and low efficiency, and it is difficult to ensure the accuracy of the labeled data. Summary of the invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for generating annotated text that can improve accuracy and efficiency in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for generating annotated text, comprising:

[0006] Obtaining a set of annotated text templates, wherein the set of annotated text templates is obtained based on an initial question text annotated with a reply text tag, the set of annotated text templates includes a question template text annotated with a reply template text tag, the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier;

[0007] Acquire multiple entity keyword sample groups, and use the multiple entity keyword sample groups to replace the preset identifiers in the annotation text template set, so as to obtain multiple candidate annotation text set groups;

[0008] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain a target annotated text set.

[0009] In one embodiment, the method for determining the annotation text template set includes:

[0010] Get the initial question text annotated with the answer text label;

[0011] Determine entity keywords in the answer text label and entity keywords in the initial question text;

[0012] The entity keywords are replaced with preset identifiers to generate question template texts annotated with answer template text tags, thereby obtaining an annotated text template set.

[0013] In one embodiment, replacing the entity keyword with a preset identifier includes:

[0014] Determine the entity category corresponding to the entity keyword, and determine the target identifier corresponding to the entity category from the preset identifiers;

[0015] The entity keyword is replaced with the target identifier.

[0016] In one embodiment, the replacing the preset identifiers in the annotation text template set with the multiple entity keyword sample groups includes:

[0017] For each entity keyword sample group, determining the entity category corresponding to the entity keyword sample in the entity keyword sample group;

[0018] The entity keyword sample is used to replace a preset identifier that matches the entity category.

[0019] In one embodiment, adjusting the plurality of candidate annotated text sets according to text features of the preset sample texts includes:

[0020] Obtaining a preset sample text, wherein the preset sample text includes question texts of different sentence structures and corresponding answer texts;

[0021] Determining the correspondence between the sentence pattern and the text feature according to the sentence pattern and text feature of the preset sample text;

[0022] Based on the corresponding relationship, the plurality of candidate annotated text set groups are adjusted according to the sentence patterns of the candidate annotated texts and the corresponding candidate reply texts in the plurality of candidate annotated text set groups.

[0023] In one embodiment, the adjusting the plurality of candidate annotated text set groups according to the text features of the preset sample text to obtain the target annotated text set includes:

[0024] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0025] Inputting the set of annotated texts to be predicted into a preset answer model, and outputting corresponding predicted answer texts through the preset answer model;

[0026] When the difference between the predicted reply text and the to-be-predicted reply text label meets the preset requirements, a target annotated text set is obtained.

[0027] In one embodiment, the adjusting the plurality of candidate annotated text set groups according to the text features of the preset sample text to obtain the target annotated text set includes:

[0028] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0029] Acquire training text from the set of annotated texts to be predicted, and determine a category label corresponding to the training text, wherein the category label includes at least one of a correct label for the reply text and an incorrect label for the reply text;

[0030] According to the correspondence between the training text and the category label, a classification model is obtained by training;

[0031] Inputting the set of annotated texts to be predicted into the classification model, and outputting the predicted category through the classification model;

[0032] When the predicted category is that the reply text is correct, a target annotated text set is obtained.

[0033] In a second aspect, the present application also provides a device for generating annotated text, comprising:

[0034] An acquisition module, used for acquiring an annotated text template set, wherein the annotated text template set is obtained based on an initial question text annotated with a reply text tag, wherein the annotated text template set includes a question template text annotated with a reply template text tag, wherein the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier;

[0035] A replacement module, used to obtain a plurality of entity keyword sample groups, and use the plurality of entity keyword sample groups to replace the preset identifiers in the annotation text template set, so as to obtain a plurality of candidate annotation text set groups;

[0036] The adjustment module is used to adjust the plurality of candidate annotated text set groups according to the text features of the preset sample texts to obtain a target annotated text set.

[0037] In one embodiment, the device further comprises a module for determining the annotated text template set, wherein the module for determining the annotated text template set is configured to:

[0038] Get the initial question text annotated with the answer text label;

[0039] Determine entity keywords in the answer text label and entity keywords in the initial question text;

[0040] The entity keywords are replaced with preset identifiers to generate question template texts annotated with answer template text tags, thereby obtaining an annotated text template set.

[0041] In one embodiment, the module for determining the annotation text template set is further used to:

[0042] Determine the entity category corresponding to the entity keyword, and determine the target identifier corresponding to the entity category from the preset identifiers;

[0043] The entity keyword is replaced with the target identifier.

[0044] In one embodiment, the replacement module is further used for:

[0045] For each entity keyword sample group, determining the entity category corresponding to the entity keyword sample in the entity keyword sample group;

[0046] The entity keyword sample is used to replace a preset identifier that matches the entity category.

[0047] In one embodiment, the adjustment module is further used for:

[0048] Obtaining a preset sample text, wherein the preset sample text includes question texts of different sentence structures and corresponding answer texts;

[0049] Determining the correspondence between the sentence pattern and the text feature according to the sentence pattern and text feature of the preset sample text;

[0050] Based on the corresponding relationship, the plurality of candidate annotated text set groups are adjusted according to the sentence patterns of the candidate annotated texts and the corresponding candidate reply texts in the plurality of candidate annotated text set groups.

[0051] In one embodiment, the adjustment module is further used for:

[0052] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0053] Inputting the set of annotated texts to be predicted into a preset answer model, and outputting corresponding predicted answer texts through the preset answer model;

[0054] When the difference between the predicted reply text and the to-be-predicted reply text label meets the preset requirements, a target annotated text set is obtained.

[0055] In one embodiment, the adjustment module is further used for:

[0056] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0057] Acquire training text from the set of annotated texts to be predicted, and determine a category label corresponding to the training text, wherein the category label includes at least one of a correct label for the reply text and an incorrect label for the reply text;

[0058] According to the correspondence between the training text and the category label, a classification model is obtained by training;

[0059] Inputting the set of annotated texts to be predicted into the classification model, and outputting the predicted category through the classification model;

[0060] When the predicted category is that the reply text is correct, a target annotated text set is obtained.

[0061] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0062] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0063] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0064] The above-mentioned method, device, computer device, computer-readable storage medium and computer program product for generating annotated text, when generating annotated text, obtain an annotation text template set, and use the obtained entity keyword sample group to replace the preset identifier therein, obtain multiple candidate annotation text set groups, and adjust the candidate annotation text set group according to the text features of the preset sample text to obtain a target annotation text set, the annotation text template set is obtained according to the initial question text, and the annotation text template set can be obtained by replacing the entity keywords in a small amount of initial text, and then using multiple entity keyword sample groups to replace the preset identifier can obtain a large number of candidate annotation text set groups, which effectively improves the efficiency of annotation text generation, does not require manual annotation of a large amount of data, and reduces the manual workload. And the annotation text template set is reusable, and can realize the generation of a large amount of annotation text in different application scenarios, further improves the efficiency of annotation text generation, and is suitable for more application scenarios. Further, the candidate annotation text set group is adjusted according to the text features of the preset sample text to obtain the target annotation text set, and the text in the obtained target annotation text set is more fluent and natural, and conforms to the semantic expression in the actual application scenario, which effectively improves the accuracy of the generated annotation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A schematic diagram of a flow chart of a method for generating annotated text in one embodiment;

[0067] Figure 2 A schematic diagram of a flow chart of a method for determining a set of annotation text templates in one embodiment;

[0068] Figure 3 A schematic diagram of a flow chart of a method for determining a set of annotation text templates in another embodiment;

[0069] Figure 4 is a structural block diagram of a device for generating annotated text in one embodiment;

[0070] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] In one embodiment, Figure 1 As shown, a method for generating annotated text is provided. This embodiment takes the method applied to a terminal as an example for explanation. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0073] Step S110, obtaining an annotated text template set, wherein the annotated text template set is obtained based on an initial question text annotated with a reply text tag, the annotated text template set includes a question template text annotated with a reply template text tag, the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier;

[0074] Exemplarily, a marked text template set is obtained, wherein the marked text template set includes a question template text marked with a reply template text tag. In one example, the reply template text tag and the question template text include text and a preset identifier.

[0075] In one example, a set of annotated text templates is obtained based on an initial question text annotated with a reply text tag, wherein entity keywords in the reply text tag and the initial question text are replaced by a preset identifier to obtain the set of annotated text templates. In one example, entity keywords may include but are not limited to keywords such as company names, product names, person names, and place names, wherein a preset entity keyword recognition model may be used to recognize entity keywords in the reply text tag and the initial question text.

[0076] Optionally, the preset identifier may include one or more, which may be determined according to actual application scenarios. In one example, entity keywords corresponding to different entity categories may be replaced with different types of preset identifiers to distinguish them.

[0077] Step S120, obtaining a plurality of entity keyword sample groups, and using the plurality of entity keyword sample groups to replace the preset identifiers in the annotation text template set, to obtain a plurality of candidate annotation text set groups;

[0078] Exemplarily, multiple entity keyword sample groups are obtained, wherein the entity keyword sample groups can be obtained by extracting keywords from data in a database. In one example, the database can be determined based on an actual application scenario. In different application scenarios, entity keywords can be extracted from different databases to improve the relevance of the entity keyword sample groups to the actual application scenarios.

[0079] Optionally, the entity keyword sample group may be obtained by dividing according to the association relationship between entity keywords. For example, the company name of Company A and the corresponding product names and person names are divided into a group of entity keyword sample groups.

[0080] Exemplarily, by replacing preset identifiers in an annotation text template set with multiple entity keyword sample groups, multiple candidate annotation text set groups can be obtained, wherein, using the text in the annotation text template set as a template, and replacing the preset identifiers in the template with multiple entity keyword sample groups respectively, multiple different candidate annotation text set groups can be obtained.

[0081] Step S130 : adjusting the plurality of candidate annotated text set groups according to the text features of the preset sample texts to obtain a target annotated text set.

[0082] Exemplarily, after obtaining the candidate annotated text set group, further optimization and adjustment are performed to obtain the target annotated text set. In one example, the candidate annotated text set group can be adjusted according to the text features of the preset sample text, and the preset sample text can be a text determined according to the actual application scenario. For example, a text with more obvious text features can be determined according to the actual application scenario as the preset sample text, such as a standard active sentence, a standard passive sentence, etc.

[0083] The preset sample text may include multiple different sample texts. Different sample texts may correspond to different text features. These text features are used to adjust and optimize the texts in the candidate annotation text set that do not meet these text features to obtain a target annotation text set.

[0084] In the embodiment of the present disclosure, when generating the annotation text, the annotation text template set is obtained, and the preset identifier is replaced by the obtained entity keyword sample group to obtain multiple candidate annotation text set groups, and the candidate annotation text set group is adjusted according to the text features of the preset sample text to obtain the target annotation text set. The annotation text template set is obtained according to the initial question text. By replacing the entity keywords in a small amount of initial text, the annotation text template set can be obtained. Then, the preset identifier is replaced by multiple entity keyword sample groups to obtain a large number of candidate annotation text set groups, which effectively improves the efficiency of annotation text generation, does not require manual annotation of a large amount of data, and reduces the manual workload. Moreover, the annotation text template set is reusable, and can realize the generation of a large amount of annotation text in different application scenarios, further improving the efficiency of annotation text generation, and is applicable to more application scenarios. Further, the candidate annotation text set group is adjusted according to the text features of the preset sample text to obtain the target annotation text set, and the text in the obtained target annotation text set is more fluent and natural, and conforms to the semantic expression in the actual application scenario, which effectively improves the accuracy of the generated annotation data.

[0085] In one embodiment, Figure 2 As shown, the method for determining the annotation text template set includes:

[0086] Step S210, obtaining the initial question text annotated with the answer text tag;

[0087] Step S220, determining entity keywords in the reply text label and entity keywords in the initial question text;

[0088] Step S230, using a preset identifier to replace the entity keyword, generating a question template text annotated with a reply template text tag, and obtaining an annotated text template set.

[0089] Exemplarily, a set of annotated text templates is obtained according to the initial question text annotated with the reply text tag. In one example, the initial question text annotated with the reply text tag is obtained, wherein the initial question text can be a text collected in advance according to an actual application scenario, such as a question text sent by a user in historical data.

[0090] In one possible implementation, the reply text label may be obtained by annotating the reply text of the reply user in historical data; in another possible implementation, the reply text label may be obtained by manually generating the reply text label based on the initial question text and annotating it; the specific implementation may be set according to the actual application scenario.

[0091] Determine entity keywords in the reply text label and entity keywords in the initial question text. In one example, the reply text label and the initial question text can be identified by an entity keyword recognition model to determine the entity keywords therein. The reply text label and the initial question text can include one or more entity keywords, which is not limited in the present disclosure.

[0092] The determined entity keyword is replaced by a preset identifier, the replaced reply text label is determined as the reply template text label, the replaced initial question text is determined as the question template text, the question template text annotated with the reply template text label is obtained, and an annotated text template set is generated. The preset identifier can be determined according to an actual application scenario.

[0093] The disclosed embodiment obtains an initial question text annotated with a reply text tag, determines entity keywords in the reply text tag and the initial question text, and replaces the entity keywords with a preset identifier, thereby generating a question template text annotated with the reply template text tag, thereby being able to quickly and accurately generate an annotated text template set using a small amount of annotated initial question text, facilitating subsequent filling and expansion based on the annotated text template set to obtain a large amount of annotated data; and by generating a reusable annotated text template set through a small amount of annotated initial question text, annotated text can be generated directly based on the generated annotated text template set in different application scenarios, without the need to perform multiple entity keyword query replacements, further reducing the amount of data processing, and being suitable for more application scenarios.

[0094] In one embodiment, Figure 3 As shown, the replacing the entity keyword with a preset identifier includes:

[0095] Step S231, determining the entity category corresponding to the entity keyword, and determining the target identifier corresponding to the entity category from preset identifiers;

[0096] Step S232: Replace the entity keyword with the target identifier.

[0097] Exemplarily, the preset identifier may include multiple types, and when replacing, different entity categories correspond to different types of preset identifiers. In one example, the entity category corresponding to the entity keyword is determined, and the entity category can be obtained according to the actual application scenario. For example, the entity category can include but is not limited to company category, place name category, person name category, product name category, field name category, etc. In a possible implementation, the entity category corresponding to the entity keyword can be determined by an entity category recognition model.

[0098] Optionally, a target identifier corresponding to the entity category is determined from the preset identifier, and the entity keyword is replaced with the target identifier. In one example, the correspondence between the entity category and the preset identifier can be set according to the actual application scenario, and different entity categories correspond to different preset identifiers.

[0099] In the disclosed embodiment, corresponding target identifiers are used for replacement according to different entity categories, so that the positions of entity keywords of different entity categories in the text can be distinguished. In subsequent replacements, preset identifiers can be replaced based on the text category, thereby ensuring the accuracy and reliability of the text obtained after replacement.

[0100] In one embodiment, the replacing the preset identifiers in the annotation text template set with the multiple entity keyword sample groups includes:

[0101] For each entity keyword sample group, determining the entity category corresponding to the entity keyword sample in the entity keyword sample group;

[0102] The entity keyword sample is used to replace a preset identifier that matches the entity category.

[0103] Exemplarily, the preset identifier is replaced according to the entity category. In one example, the entity keyword sample group includes multiple entity keyword samples. The entity category corresponding to the entity keyword sample is determined. In one example, the entity category corresponding to the entity keyword sample can be identified by an entity category recognition model.

[0104] The entity category corresponding to the preset identifier in the annotation text template set is determined, and the preset identifier is replaced with an entity keyword sample corresponding to the entity category.

[0105] In the disclosed embodiment, when replacing a preset identifier, the corresponding preset identifier is replaced according to the entity category corresponding to the entity keyword sample in the entity keyword sample group, thereby ensuring the accuracy and reliability of the text obtained after the replacement, and being applicable to more application scenarios.

[0106] In one embodiment, the adjusting the plurality of candidate annotated text sets according to the text features of the preset sample texts includes:

[0107] Obtaining a preset sample text, wherein the preset sample text includes question texts of different sentence structures and corresponding answer texts;

[0108] Determining the correspondence between the sentence pattern and the text feature according to the sentence pattern and text feature of the preset sample text;

[0109] Based on the corresponding relationship, the plurality of candidate annotated text set groups are adjusted according to the sentence patterns of the candidate annotated texts and the corresponding candidate reply texts in the plurality of candidate annotated text set groups.

[0110] Exemplarily, the preset sample text includes question texts of different sentence structures and corresponding answer texts, wherein the different sentence structures may include but are not limited to active sentences, passive sentences, etc., which may be determined according to actual application scenarios.

[0111] Optionally, based on preset sample texts of different sentence patterns, text features corresponding to different sentence patterns are extracted, such as the order of subject, predicate, and object, etc., which can be determined based on actual application scenarios. The correspondence between sentence patterns and text features is determined, and different sentence patterns can correspond to different text features.

[0112] The candidate annotated text set group includes candidate annotated texts annotated with candidate reply text tags. Based on the correspondence between sentence patterns and text features, the corresponding text features are determined according to the sentence patterns of the candidate reply texts and the candidate annotated texts, and the candidate reply texts and the candidate annotated texts are adjusted using the determined text features so that the text features of the adjusted texts conform to the corresponding sentence patterns.

[0113] All the texts in the candidate annotated text set are adjusted according to the above adjustment method to obtain adjusted texts. In one example, if the text features of the text before adjustment are consistent with the corresponding sentence pattern, no adjustment is required.

[0114] In the disclosed embodiments, the preset sample text includes question texts of different sentence structures and corresponding answer texts. According to the sentence structure and text features of the preset sample text, the correspondence between the sentence structure and the text features can be determined, and based on the correspondence, multiple candidate annotation text set groups can be adjusted, so that according to the sentence structure of the text of the candidate annotation text set group, the text can be adjusted according to the corresponding text features, so that the adjusted text is more in line with the expression methods of different sentence structures in actual application scenarios, more fluent and natural, and the accuracy of the generated annotation text is effectively improved.

[0115] In one embodiment, the adjusting the plurality of candidate annotated text set groups according to the text features of the preset sample text to obtain the target annotated text set includes:

[0116] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0117] Inputting the set of annotated texts to be predicted into a preset answer model, and outputting corresponding predicted answer texts through the preset answer model;

[0118] When the difference between the predicted reply text and the to-be-predicted reply text label meets the preset requirements, a target annotated text set is obtained.

[0119] Exemplarily, the adjusted annotated text to be predicted is verified using a preset answer model. In one example, the set of annotated text to be predicted is obtained after adjustment based on the text features of the preset sample text, and the set of annotated text to be predicted includes the annotated text to be predicted annotated with the answer text label to be predicted.

[0120] The set of annotated texts to be predicted is input into the preset answer model, and the preset answer model outputs the corresponding predicted answer text. In one example, the preset answer model can be determined in advance according to the actual application scenario, and is mainly used to parse the question text and generate the corresponding predicted answer text.

[0121] When the difference between the predicted reply text and the reply text label to be predicted meets the preset requirements, the target annotated text set is obtained. In one example, the similarity between the predicted reply text and the corresponding reply text label to be predicted can be calculated to characterize the difference between the two. In one example, when the similarity between the two is greater than the preset similarity, it can be considered that the difference between the two is small and meets the preset requirements, and the annotated text to be predicted corresponding to the reply text label to be predicted is determined to be the target annotated text.

[0122] In one example, if the difference between the two does not meet the preset requirements, the to-be-predicted annotated text corresponding to the to-be-predicted reply text label is subjected to secondary processing, such as re-annotation, etc., to obtain a corrected to-be-predicted annotated text, which is determined as the target annotated text.

[0123] Optionally, after the to-be-predicted annotated text set is processed, a target annotated text set is obtained.

[0124] The disclosed embodiment utilizes a preset reply model to generate predicted reply texts corresponding to a set of annotated texts to be predicted, and obtains a target annotated text set when the difference between the predicted reply texts and the corresponding labels of the reply texts to be predicted meets preset requirements, thereby being able to automatically verify the replaced and optimized reply texts to be predicted through the prediction results of the preset reply model, and determining the annotated texts that pass the verification as texts in the target annotated text set, thereby ensuring the accuracy and reliability of the obtained target annotated text set.

[0125] In one embodiment, the adjusting the plurality of candidate annotated text set groups according to the text features of the preset sample text to obtain the target annotated text set includes:

[0126] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0127] Acquire training text from the set of annotated texts to be predicted, and determine a category label corresponding to the training text, wherein the category label includes at least one of a correct label for the reply text and an incorrect label for the reply text;

[0128] According to the correspondence between the training text and the category label, a classification model is obtained by training;

[0129] Inputting the set of annotated texts to be predicted into the classification model, and outputting the predicted category through the classification model;

[0130] When the predicted category is that the reply text is correct, a target annotated text set is obtained.

[0131] Exemplarily, the adjusted annotated text to be predicted is verified using a classification model. In one example, the annotated text to be predicted is adjusted according to the text features of the preset sample text to obtain a set of annotated text to be predicted, and the set of annotated text to be predicted includes the annotated text to be predicted that is annotated with a label of the answer text to be predicted.

[0132] Optionally, the classification model is trained using part of the labeled text to be predicted as training text. In one example, the training text is obtained from the set of labeled text to be predicted, wherein the training text can be part of the labeled text to be predicted that is labeled with the answer text label to be predicted.

[0133] In one example, the category label corresponding to the training text is determined based on whether the label of the reply text to be predicted of the training text is correct. If the label of the reply text to be predicted matches the annotated text to be predicted, the label of the reply text to be predicted is determined to be correct, and its corresponding category label is the correct label of the reply text. If the label of the reply text to be predicted does not match the annotated text to be predicted, the label of the reply text to be predicted is determined to be incorrect, and its corresponding category label is the incorrect label of the reply text.

[0134] According to the correspondence between the training text and the category label, a classification model is trained, and the classification model can output a prediction result of whether the answer text is wrong or correct based on the question text and the answer text.

[0135] The set of annotated texts to be predicted is input into the classification model, and the classification model outputs the predicted category. When the predicted category is that the reply text is correct, it can be considered that the accuracy of the annotated text to be predicted is high and it is determined as the target annotated text. In one example, when the predicted category is that the reply text is wrong, it can be considered that the label corresponding to the annotated text to be predicted is poor in accuracy, and the text to be predicted can be processed twice, such as secondary annotation, to obtain the target annotated text.

[0136] Optionally, all the to-be-predicted annotated texts in the to-be-predicted annotated text set are input into the classification model to obtain output results and are processed in the above manner to obtain a target annotated text set.

[0137] In a possible implementation, a method based on combining the promotion and the preset sample texts may be adopted to classify the set of annotated texts to be predicted using a classification model to determine whether the texts in the set of annotated texts to be predicted correspond.

[0138] In a possible implementation, the classification model may be obtained through STF training (Sequence-to-Sequence Training Fine-tuning, i.e., sequence-to-sequence training refinement).

[0139] In another possible implementation, a parsing model that parses questions into answers may be used to make predictions on a set of annotated texts to be predicted, and the predicted results may be compared with the labels of the answer texts to be predicted.

[0140] It is understandable that the verification process of the above different verification logics can also be performed in sequence according to a preset order to ensure the accuracy and reliability of the obtained annotated text.

[0141] In the disclosed embodiment, part of the data is obtained from the set of annotated texts to be predicted obtained after replacement and adjustment as training texts, and the category labels of the training texts are determined. A classification model is obtained through training, and the set of annotated texts to be predicted is classified using the classification model to obtain annotated texts with correct reply texts, which are determined as a target annotated text set. Thus, whether the labels in the set of annotated texts to be predicted are correct can be classified and confirmed through the classification model, and a target annotated text set with a higher probability of correct reply texts can be obtained, thereby further improving the accuracy and reliability of the target annotated text set.

[0142] In one embodiment, the necessary reply text tags are manually annotated based on the user's question text to obtain an initial question text annotated with the reply text tags. Preset identifiers are used to replace entity keywords in the initial question text and the reply text tags to create a template to obtain a set of annotated text templates. Code is written to use an entity data set, i.e., an entity keyword sample group, to replace the corresponding entity keywords in the template to obtain a candidate annotated text set group. The candidate annotated text set group is adjusted and optimized to obtain a target annotated text set.

[0143] In some possible implementations, the candidate annotated text set group may be adjusted and optimized through a large language model, and the candidate annotated text set group may be input into the large language model, and the large language model may be used for generalization and polishing.

[0144] In one example, the candidate annotated text set is generalized, polished and rewritten. For example, questions are made to fit human semantic expressions by rewriting active and passive sentences; questions are made more fluent and natural by using the capabilities of the large model; and some multi-round dialogue data are generated by means of reference replacement and subject omission. After the multi-round dialogue data is generated, the labels are optimized and adjusted using rule logic.

[0145] Optionally, when using a large model for polishing, in addition to the general prompt adjustments, you can also add some sample enhancements. By providing the model with some sample text (such as examples of active and passive paraphrases, examples of reference substitution), you can control how the model rephrases the questions.

[0146] In one example, filtering and correction may be performed on the polished and rewritten collection.

[0147] During filtering and correction, a small amount of question text is first generated by rewriting the polishing model based on the existing small amount of initial question text, and then it is manually marked whether the polished questions and answers correspond.

[0148] Based on prompts and a small number of samples, the polished annotated data is classified through a large model to distinguish whether the polished questions and answers correspond. If they correspond, the annotated data is added to the final target annotated text set, otherwise secondary processing (correction) is performed.

[0149] After generating a batch of data (labeled data to show whether questions and answers correspond) using a prompt-based and small-sample approach, and after manual review, the 7b model was used for SFT training to obtain a classification model with higher accuracy (classifying whether questions and answers correspond for the polished questions).

[0150] In one example, a parsing model that parses questions into answers is used to predict the generated questions, and the predicted answers are compared with the corresponding answers in the annotated data as a supplementary verification logic. If they are consistent, they can be added to the final target annotated text set. If they are inconsistent, they can be processed again (corrected).

[0151] In the secondary processing (correction) phase, the questions before and after rewriting are corrected. The answer predicted by the parsing model is an optional input. The model is told through a prompt that the generated answer has problems and needs to be regenerated. With reference information, the effect of the model's secondary generation can be effectively enhanced. In one example, the original questions associated with the failed samples are counted, and the questions with a high failure rate are taken and handed over to manual data supplementation.

[0152] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0153] Based on the same inventive concept, the embodiment of the present application also provides a device for generating annotated text for implementing the above-mentioned method for generating annotated text. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more devices for generating annotated text provided below can refer to the limitations of the method for generating annotated text above, and will not be repeated here.

[0154] In an exemplary embodiment, Figure 4 As shown, a device 400 for generating annotated text is provided, comprising:

[0155] The acquisition module 410 is used to acquire a set of annotated text templates, wherein the set of annotated text templates is obtained based on an initial question text annotated with a reply text tag, wherein the set of annotated text templates includes a question template text annotated with a reply template text tag, wherein the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier;

[0156] A replacement module 420 is used to obtain a plurality of entity keyword sample groups, and use the plurality of entity keyword sample groups to replace the preset identifiers in the annotation text template set, so as to obtain a plurality of candidate annotation text set groups;

[0157] The adjustment module 430 is used to adjust the plurality of candidate annotated text set groups according to the text features of the preset sample texts to obtain a target annotated text set.

[0158] In one embodiment, the device further comprises a module for determining the annotated text template set, wherein the module for determining the annotated text template set is configured to:

[0159] Get the initial question text annotated with the answer text label;

[0160] Determine entity keywords in the answer text label and entity keywords in the initial question text;

[0161] The entity keywords are replaced with preset identifiers to generate question template texts annotated with answer template text tags, thereby obtaining an annotated text template set.

[0162] In one embodiment, the module for determining the set of annotation text templates is further used to:

[0163] Determine the entity category corresponding to the entity keyword, and determine the target identifier corresponding to the entity category from the preset identifiers;

[0164] The entity keyword is replaced with the target identifier.

[0165] In one embodiment, the replacement module is further used for:

[0166] For each entity keyword sample group, determining the entity category corresponding to the entity keyword sample in the entity keyword sample group;

[0167] The entity keyword sample is used to replace a preset identifier that matches the entity category.

[0168] In one embodiment, the adjustment module is further used for:

[0169] Obtaining a preset sample text, wherein the preset sample text includes question texts of different sentence structures and corresponding answer texts;

[0170] Determining the correspondence between the sentence pattern and the text feature according to the sentence pattern and text feature of the preset sample text;

[0171] Based on the corresponding relationship, the plurality of candidate annotated text set groups are adjusted according to the sentence patterns of the candidate annotated texts and the corresponding candidate reply texts in the plurality of candidate annotated text set groups.

[0172] In one embodiment, the adjustment module is further used for:

[0173] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0174] Inputting the set of annotated texts to be predicted into a preset answer model, and outputting corresponding predicted answer texts through the preset answer model;

[0175] When the difference between the predicted reply text and the to-be-predicted reply text label meets the preset requirements, a target annotated text set is obtained.

[0176] In one embodiment, the adjustment module is further used for:

[0177] According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted;

[0178] Acquire training text from the set of annotated texts to be predicted, and determine a category label corresponding to the training text, wherein the category label includes at least one of a correct label for the reply text and an incorrect label for the reply text;

[0179] According to the correspondence between the training text and the category label, a classification model is obtained by training;

[0180] Inputting the set of annotated texts to be predicted into the classification model, and outputting the predicted category through the classification model;

[0181] When the predicted category is that the reply text is correct, a target annotated text set is obtained.

[0182] Each module in the above-mentioned device for generating annotated text can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0183] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the method described in this embodiment, such as the initial question text. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating annotated text is implemented.

[0184] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0185] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0186] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0187] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0189] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0190] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0191] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating annotated text, characterized in that: The method comprises: Obtaining a set of annotated text templates, wherein the set of annotated text templates is obtained based on an initial question text annotated with a reply text tag, the set of annotated text templates includes a question template text annotated with a reply template text tag, the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier; Acquire multiple entity keyword sample groups, and use the multiple entity keyword sample groups to replace the preset identifiers in the annotation text template set, so as to obtain multiple candidate annotation text set groups; According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain a target annotated text set.

2. The method according to claim 1, characterized in that: The method for determining the annotation text template set includes: Get the initial question text annotated with the answer text label; Determine entity keywords in the answer text label and entity keywords in the initial question text; The entity keywords are replaced with preset identifiers to generate question template texts annotated with answer template text tags, thereby obtaining an annotated text template set.

3. The method according to claim 2, characterized in that The replacing the entity keyword with a preset identifier includes: Determine the entity category corresponding to the entity keyword, and determine the target identifier corresponding to the entity category from the preset identifiers; The entity keyword is replaced with the target identifier.

4. The method according to claim 3, characterized in that: The step of replacing the preset identifiers in the annotation text template set with the multiple entity keyword sample groups includes: For each entity keyword sample group, determining the entity category corresponding to the entity keyword sample in the entity keyword sample group; The entity keyword sample is used to replace a preset identifier that matches the entity category.

5. The method according to claim 1, characterized in that: The adjusting the plurality of candidate annotated text sets according to the text features of the preset sample texts includes: Obtaining a preset sample text, wherein the preset sample text includes question texts of different sentence structures and corresponding answer texts; Determining the correspondence between the sentence pattern and the text feature according to the sentence pattern and text feature of the preset sample text; Based on the corresponding relationship, the plurality of candidate annotated text set groups are adjusted according to the sentence patterns of the candidate annotated texts and the corresponding candidate reply texts in the plurality of candidate annotated text set groups.

6. The method according to claim 1, characterized in that The step of adjusting the plurality of candidate annotated text sets according to the text features of the preset sample texts to obtain a target annotated text set includes: According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted; Inputting the set of annotated texts to be predicted into a preset answer model, and outputting corresponding predicted answer texts through the preset answer model; When the difference between the predicted reply text and the to-be-predicted reply text label meets the preset requirements, a target annotated text set is obtained.

7. The method according to claim 1, characterized in that The step of adjusting the plurality of candidate annotated text sets according to the text features of the preset sample texts to obtain a target annotated text set includes: According to the text features of the preset sample texts, the plurality of candidate annotated text set groups are adjusted to obtain an annotated text set to be predicted, wherein the annotated text set to be predicted includes the annotated text to be predicted annotated with a label of a reply text to be predicted; Acquire training text from the set of annotated texts to be predicted, and determine a category label corresponding to the training text, wherein the category label includes at least one of a correct label for the reply text and an incorrect label for the reply text; According to the correspondence between the training text and the category label, a classification model is obtained by training; Inputting the set of annotated texts to be predicted into the classification model, and outputting the predicted category through the classification model; When the predicted category is that the reply text is correct, a target annotated text set is obtained.

8. A device for generating annotated text, characterized in that: The device comprises: An acquisition module, used for acquiring an annotated text template set, wherein the annotated text template set is obtained based on an initial question text annotated with a reply text tag, wherein the annotated text template set includes a question template text annotated with a reply template text tag, wherein the reply template text tag is obtained by replacing entity keywords of the reply text tag with a preset identifier, and the initial question text is obtained by replacing entity keywords of the initial question text with the preset identifier; A replacement module, used to obtain a plurality of entity keyword sample groups, and use the plurality of entity keyword sample groups to replace the preset identifiers in the annotation text template set, so as to obtain a plurality of candidate annotation text set groups; The adjustment module is used to adjust the plurality of candidate annotated text set groups according to the text features of the preset sample texts to obtain a target annotated text set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images