Text data generation method and device, equipment and medium

By combining the annotation and verification model of structured data, the problems of manpower consumption and low text quality caused by relying on expert annotation in the prior art are solved, and efficient and accurate text data generation is achieved.

CN119003694APending Publication Date: 2024-11-22GUANGDONG ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY LAB (GUANGZHOU) +1

Patent Information

Application Number
CN202411064916.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing technology relies on experts to perform data annotation, which consumes a lot of manpower and domain knowledge. The text annotation model lacks large-scale annotation data, and the generated text quality and accuracy are low, so it is impossible to accurately represent the actual meaning.

Method used

By obtaining the pending text data of structured data, using the preset labeling model for data annotation, obtaining natural language text, and then using the preset verification model for text verification of the verification labeling data, performing data refinement based on the verification results, and obtaining the target Text data.

Benefits of technology

It improves the efficiency and accuracy of data annotation, makes up for the problem that the neural network model lacks large-scale annotation of data, and improves the quality and accuracy of text generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003694A_ABST
    Figure CN119003694A_ABST
Patent Text Reader

Abstract

The invention relates to a text data generation method and device, equipment and a medium, and relates to the technical field of text generation, and the method comprises the steps: obtaining to-be-processed text data corresponding to structured data, carrying out data labeling through a preset labeling model based on non-labeling data contained in the to-be-processed text data, obtaining a natural language text, and storing the natural language text in the structured data; performing text verification on each piece of to-be-verified annotation data contained in the natural language text through a preset verification model to obtain a verification result, and performing data refining on the natural language text according to an initial verification result and a secondary verification result contained in the verification result to obtain target text data, therefore, the problem that a large amount of manpower and domain knowledge need to be consumed due to the fact that data annotation depends on experts in the prior art is solved, the efficiency and accuracy of data annotation are improved, and then the problem that a neural network model lacks large-scale annotation data is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of text generation, and in particular to a method, device, equipment and medium for generating text data. Background Art

[0002] At present, the training methods of neural networks mainly rely on annotated text data. Therefore, generating text based on structured data (Data-to-text, D2T) is a core task of natural language processing. Text data generated using structured data can be applied to a wide range of application areas / scenarios, such as automatically generating news from statistical data, composing medical diagnosis reports from patient test results, making financial reports, and generating weather forecasts from digital inputs, live commentary of sports games, etc.

[0003] Existing technologies for converting structured data into text usually rely on templates or rules to convert data into text. Data and text usually rely on experts' domain knowledge to manually write and annotate data, involving multiple steps such as content selection, content sorting, and lexicalization, which requires a lot of manpower and domain knowledge. Summary of the invention

[0004] The present application provides a text data generation method, apparatus, device and medium, which effectively improve the efficiency and accuracy of data annotation, make up for the problem of lack of large-scale annotated data in neural network models, and thus solve the problem that the existing technology relies on experts for data annotation, which requires a lot of manpower and domain knowledge.

[0005] In a first aspect, the present application provides a method for generating text data, comprising:

[0006] Acquire text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data;

[0007] For the non-annotated data, data annotation is performed using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified;

[0008] Performing text verification on each of the labeled data to be verified through a preset verification model to obtain a verification result corresponding to the labeled data to be verified, wherein the verification result includes an initial verification result and a secondary verification result;

[0009] According to the initial verification result and the secondary verification result, data refinement is performed on the natural language text to obtain target text data.

[0010] Optionally, obtaining the text data to be processed corresponding to the structured data includes:

[0011] Acquire a structured data set, wherein the structured data set contains structured data;

[0012] Performing data conversion according to the structured data to obtain target character string data;

[0013] Text segmentation is performed based on the target character string data to obtain the text data to be processed.

[0014] Optionally, the verification model includes a first verification model and a second verification model, and the text verification is performed on each of the to-be-verified annotated data using a preset verification model to obtain a verification result corresponding to the to-be-verified annotated data, including:

[0015] Get the preset validation rules;

[0016] Based on the verification rules, the to-be-verified annotated data is verified by using the first verification model to obtain an initial verification result;

[0017] The first verification result and the labeled data to be verified are secondarily verified using the second verification model to obtain a second verification result.

[0018] Optionally, the initial verification result includes a target verification result and a non-target verification result, and the initial verification result is obtained by performing data verification on the to-be-verified labeled data through the first verification model based on the verification rule, including:

[0019] The first verification model determines whether the to-be-verified annotated data complies with the verification rule;

[0020] When the to-be-verified annotated data meets the verification rule, a target verification result is generated, wherein the target verification result includes a target verification viewpoint and a target verification instruction;

[0021] When the to-be-verified annotated data does not conform to the verification rule, a non-target verification result is generated, and the non-target verification result includes a non-target verification viewpoint and a non-target verification instruction.

[0022] Optionally, the secondary verification result includes a target discrimination result and a non-target discrimination result, and the second verification model is used to perform secondary verification on the first verification result and the labeled data to be verified to obtain the secondary verification result, including:

[0023] Performing text verification on the to-be-verified annotated data using the second verification model to obtain a text verification result;

[0024] When the initial verification result is a target verification result, determining whether the text verification result is consistent with the target verification result;

[0025] If the text verification result is consistent with the target verification result, generating a target discrimination result based on the text verification result and the target verification result;

[0026] If the text verification result does not match the target verification result, a non-target discrimination result is generated based on the text verification result.

[0027] Optionally, performing data refinement on the natural language text according to the initial verification result and the secondary verification result to obtain target text data includes:

[0028] When the initial verification result is a target verification result and the secondary verification result is a target discrimination result, generating the target text data based on the natural language text;

[0029] When the initial verification result is a non-target verification result, and / or the secondary verification result is a non-target discrimination result, the to-be-verified annotated data corresponding to the verification result is filtered from the natural language text to obtain the target text data.

[0030] Optionally, after obtaining the target text data, the method further includes:

[0031] Building a training set based on the target text data and structured data corresponding to the target text data;

[0032] Fine-tune the parameters of the annotation model based on the training set to obtain a loss function;

[0033] Perform back propagation according to the loss function to obtain gradient data;

[0034] The annotation model is updated using the gradient data.

[0035] In a second aspect, the present application provides a text data generating device, comprising:

[0036] An acquisition module, used for acquiring text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data;

[0037] A data annotation module, used for annotating the non-annotated data by using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified;

[0038] A text verification module, used to perform text verification on each of the to-be-verified annotated data through a preset verification model, and obtain a verification result corresponding to the to-be-verified annotated data, wherein the verification result includes an initial verification result and a secondary verification result;

[0039] The data refining module is used to perform data refining on the natural language text according to the initial verification result and the secondary verification result to obtain target text data.

[0040] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0041] Memory, used to store computer programs;

[0042] The processor is used to implement the steps of the text data generation method as described in any embodiment of the first aspect when executing the program stored in the memory.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the text data generation method as described in any one of the embodiments of the first aspect are implemented.

[0044] In summary, the embodiment of the present application obtains the text data to be processed corresponding to the structured data, and performs data annotation through a preset annotation model based on the non-annotated data contained in the text data to be processed to obtain a natural language text, and then performs text verification on each annotated data to be verified contained in the natural language text through a preset verification model to obtain a verification result, and based on the verification results including an initial verification result and a secondary verification result, the natural language text is refined to obtain the target text data, thereby solving the problem that the prior art relies on experts to perform data annotation, which requires a lot of manpower and domain knowledge, improves the efficiency and accuracy of data annotation, and further makes up for the problem that the neural network model lacks large-scale annotated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart of a text data generation method provided in an embodiment of the present application;

[0048] Figure 2It is a schematic diagram of the steps of a text data generation method provided by an optional embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of the multi-model self-training structured data text generation process provided by an example of this application;

[0050] Figure 4 This is a flowchart of steps for converting structured data into text data provided by an example of this application;

[0051] Figure 5 This is a flowchart of steps for evaluating and purifying text data provided by an example of this application;

[0052] Figure 6 This is a flowchart of the steps of using text data to train a model, provided as an example in this application;

[0053] Figure 7 A structural block diagram of a text data generating device provided in an embodiment of the present application;

[0054] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] In the related art, generating text based on structured data is a core task of natural language processing. In this task, structured data is converted into coherent natural language text through algorithms. By generating fluent descriptions from various data sources such as tables, graphics and databases, the technology enables users to quickly understand prominent information, identify key trend information and extract meaningful content without manually analyzing the original data. The existing technology also annotates data through algorithms or text annotation models to obtain D2T generated data. However, directly annotating structured data through algorithms lacks flexibility and scalability; annotating data using text annotation models also faces some challenges. First, the existing text annotation models lack large-scale annotated data. D2T generated data usually consists of paired data and text, which are usually manually written by experts, involving multiple steps such as content selection, content sorting, and lexicalization. Secondly, the semantic organization ability of existing text annotation models is limited. Directly using text annotation models to annotate data will result in low quality and accuracy of the generated text, and it cannot accurately represent the actual meaning.

[0057] To solve the above problems, one of the technical ideas of the embodiments of the present application is to use an annotation model to annotate the non-annotated data in the natural language text, and then use a verification model to perform text verification on each annotated data to obtain a verification result, and then based on the verification result, perform data refinement on the natural language text to obtain the target text data, thereby effectively solving: ① The problem that the existing technology relies on experts to perform data annotation, which requires a lot of manpower and domain knowledge; ② The problem that the quality and accuracy of the generated text based on the algorithm / text annotation model in text annotation is low, and the actual meaning cannot be accurately represented. The problem of improving the efficiency and accuracy of data annotation, thereby making up for the problem that the neural network model lacks large-scale annotated data.

[0058] To facilitate the understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0059] Figure 1 A flowchart of a text data generation method provided in an embodiment of the present application. Figure 1 As shown, the text data generation method provided in the embodiment of the present application may specifically include the following steps:

[0060] Step 110: Obtain text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data.

[0061] Specifically, the text data to be processed may include one or more non-annotated data. This embodiment may convert the structured data into multiple non-annotated data by performing data preprocessing on the structured data, wherein the non-annotated data may refer to a sentence, a word, an independent character, etc., and this embodiment does not limit this.

[0062] It should be noted that, in this embodiment, non-labeled data may also be referred to as unlabeled data, non-labeled data, unlabeled data, etc., specifically referring to data that has not been labeled.

[0063] Exemplarily, structured data can be acquired through data collection and other methods, and the structured data can be encoded and transcoded to convert it into text-type data. The text-type structured data can then be segmented to divide it into multiple sentences, or multiple words, or independent characters according to certain rules, thereby obtaining multiple unannotated data, and then the unannotated data can be aggregated to obtain the text data to be processed.

[0064] Step 120: annotate the unannotated data using a preset annotation model to obtain natural language text.

[0065] Wherein, the natural language text contains at least one annotated data to be verified.

[0066] Specifically, the labeled data to be verified refers to the labeled data obtained by the labeling model after the non-labeled data is labeled. Since the labeled data obtained at this time has not been verified / checked, the accuracy of its labeling cannot be determined, so the labeled data is called the labeled data to be verified, and the labeled data to be verified can also be called pseudo-labeled data. The labeling model can refer to a model with data labeling capabilities / functions, such as the labeling model can be a large language model (LargeLanguageModel, LLM for short), and illustratively, the labeling model can be a writer LLM. Of course, the labeling model can also be other neural network models with data labeling capabilities, and this embodiment does not limit this.

[0067] In a specific implementation, for each unlabeled data contained in the text data to be processed, the labeling model can label it to obtain the labeled data to be verified corresponding to the unlabeled data, and then all the labeled data to be verified can be aggregated to obtain natural language text.

[0068] Step 130 , performing text verification on each of the labeled data to be verified using a preset verification model to obtain a verification result corresponding to the labeled data to be verified.

[0069] The verification result includes an initial verification result and a secondary verification result.

[0070] In this embodiment, the process of verifying the annotation data to be verified by the verification model can be called a text verification process or an annotation data verification process, which can include an initial verification process and a secondary verification process. The initial verification result is the verification result obtained when the verification model is used to perform initial verification on the annotation data to be verified; the secondary verification result is the verification result obtained when the verification model is used to perform secondary verification on the annotation data to be verified and the initial verification result, which is not limited in this embodiment.

[0071] In a specific implementation, this embodiment can input the labeled data to be verified into the verification model. The verification model first performs an initial verification on the labeled data to be verified to obtain an initial verification result, wherein the initial verification result can be divided into an initial verification correct result and an initial verification incorrect result (or an initial verification wrong result). The initial verification correct result refers to the verification model determining that the labeled data to be verified is correct labeled data when performing the initial verification on the labeled data, that is, determining that the labeling model has accurately labeled the non-labeled data; the initial verification incorrect result refers to the verification model determining that the labeled data to be verified is incorrect labeled data or wrongly labeled data when performing the initial verification on the labeled data, that is, determining that the labeling model has not correctly labeled the non-labeled data or that the labeling model has incorrectly labeled the non-labeled data. After completing the initial verification, the initial verification result and the labeled data to be verified can be further verified for a second time to obtain a second verification result, wherein the second verification result can be divided into a correct second verification result and an incorrect second verification result (also referred to as an incorrect second verification result). The correct second verification result refers to the verification model determining that the labeled data to be verified is the correct labeled data when performing a second verification on the labeled data to be verified, and the initial verification result corresponding to the labeled data to be verified is the correct initial verification result. The correct second verification result indicates that the annotation model confirms that the non-labeled data is accurately labeled during the second verification, and agrees with the initial verification result; the incorrect second verification result refers to the fact that the labeled data to be verified corresponding to the non-labeled data is incorrectly labeled data during the second verification, and / or the initial verification result corresponding to the labeled data to be verified is an incorrect initial verification result. The correct second verification result indicates that when performing a second verification on the labeled data to be verified, the verification result obtained is different from the initial verification result, that is, it disagrees with the initial verification result.

[0072] In actual processing, the verification model can be composed of two or more models with data annotation verification functions. One or more models can be set according to actual needs for initial verification of the annotated data to be verified, and one or more models can be set according to actual needs for secondary verification of the annotated data to be verified. The embodiments of the present application do not limit this.

[0073] Step 140: Based on the initial verification result and the secondary verification result, data refinement is performed on the natural language text to obtain target text data.

[0074] In a specific implementation, for each annotated data to be verified contained in a natural language text, it can be determined whether to retain the data to be verified in the natural language text according to the initial verification result and the secondary verification result corresponding to the annotated data to be verified. Specifically, the annotated data to be verified whose initial verification result is a correct initial verification result and whose secondary verification result is a correct secondary verification result can be retained in the natural language text. The annotated data to be verified whose initial verification result is an incorrect initial verification result, and / or the annotated data to be verified whose secondary verification result is an incorrect secondary verification result, are filtered from the natural language text, thereby achieving data refinement and obtaining target text data, effectively solving the problem that the prior art relies on experts to perform data annotation, which requires a lot of manpower and domain knowledge, improves the efficiency and accuracy of data annotation, and makes up for the problem that the neural network model in the prior art lacks large-scale annotated data.

[0075] It can be seen that the embodiment of the present application obtains the text data to be processed corresponding to the structured data, and performs data annotation through a preset annotation model based on the non-annotated data contained in the text data to be processed to obtain a natural language text, and then performs text verification on each annotated data to be verified contained in the natural language text through a preset verification model to obtain a verification result, and based on the verification results including an initial verification result and a secondary verification result, the natural language text is refined to obtain the target text data, thereby solving the problem that the prior art relies on experts to perform data annotation, which requires a lot of manpower and domain knowledge, improves the efficiency and accuracy of data annotation, and further makes up for the problem of the lack of large-scale annotated data in the neural network model.

[0076] Reference Figure 2 , shows a schematic flow chart of the steps of a text data generation method provided by an optional embodiment of the present application. The method may specifically include the following steps:

[0077] Step 210: Obtain text data to be processed corresponding to the structured data.

[0078] Wherein, the text data to be processed includes unlabeled data.

[0079] For the description of step 210 , please refer to the description of step 110 above, which will not be described in detail here.

[0080] Optionally, the embodiment of the present application obtains the text data to be processed corresponding to the structured data, which may specifically include the following sub-steps:

[0081] Sub-step 2101: obtaining a structured data set, where the structured data set contains structured data.

[0082] like Figure 3 As shown, in actual processing, the embodiments of the present application can collect structured data to be converted into natural language text in a variety of ways to form a structured data set.

[0083] Sub-step 2101, performing data conversion according to the structured data to obtain target character string data.

[0084] In this embodiment, the structured data is converted into a character string corresponding to the encoding form according to a preset encoding form (such as UTF-8 encoding), thereby obtaining target character string data.

[0085] In actual processing, the text generation method provided in the embodiment of the present application mainly involves three core modules, namely a data generation module, a data filtering module and a model training module. Among them, the data generation module is used to achieve: ① converting structured data into text data to be processed; ② inputting the text data to be processed into the annotation model to generate annotation data; the data filtering module is used to filter the erroneous annotation data; the model training module is used to update the model of the correctly annotated annotation data. In the actual application stage, the actual applications corresponding to the data generation module and the data filtering module can be integrated together to obtain the text data generation stage, and the model training module can be corresponded to the model update stage in the actual application, refer to Figure 3 Among them, the text data generation stage mainly includes: text preprocessing, text generation, text checking, text meta-checking and data filtering; the model update stage mainly includes: model training and model prediction.

[0086] For example, for the text preprocessing stage, refer to Figure 4 , structured data is collected in advance through various methods. The collected structured data may contain irrelevant data that does not meet the requirements. Therefore, the structured data can be preprocessed. For example, the structured data can be cleaned and filtered first to obtain cleaned data with the data that does not meet the requirements removed, and then the cleaned data can be encoded and converted to convert the structured data into string data according to the preset encoding requirements to obtain the target string data.

[0087] Sub-step 2101, performing text segmentation based on the target character string data to obtain the text data to be processed.

[0088] In a specific implementation, a word segmentation rule or a word segmentation algorithm may be preset, and the word segmentation rule or the word segmentation algorithm may be used to perform text segmentation on the target string data, and the string text corresponding to the target string data may be divided into independent single characters, such as Figure 3Of course, in actual processing, text segmentation can also be performed according to actual needs. For example, when the text segmentation requirement is sentence segmentation, the string text corresponding to the target string data is segmented in the form of sentences according to the sentence segmentation requirement to obtain multiple independent sentences. This example does not limit this.

[0089] Step 220: annotate the non-annotated data using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified.

[0090] For example, refer to Figure 4 , using the writer LLM as the annotation model, the unannotated data can be input into the writer LLM, and the writer LLM will generate natural language text based on the unannotated data, that is, the writer LLM will annotate the unannotated data and obtain pseudo-annotated data corresponding to the unannotated data as the annotated data to be verified, where the annotated data to be verified corresponds one-to-one with the original structured data, and the annotated data to be verified is aggregated to obtain natural language text.

[0091] Step 230, obtaining preset verification rules.

[0092] The verification model includes a first verification model and a second verification model.

[0093] In a specific implementation, a verification rule for verifying whether the labeled data to be verified is correct may be preset, and the verification rule will be used as an established rule, so that the first verification model can use the verification rule to implement the verification of the labeled data.

[0094] Step 240: Based on the verification rules, the first verification model is used to perform data verification on the labeled data to be verified to obtain an initial verification result.

[0095] In a specific implementation, the initial verification result includes a target verification result and a non-target verification result, wherein the target verification result may refer to a correct initial verification result, and the non-target verification result may refer to an incorrect initial verification result, which is not limited in this embodiment.

[0096] For example, refer to Figure 4Taking the checker LLM as the first verification model as an example, the annotated data to be verified obtained by the writer LLM is input into the checker LLM, and the checker LLM verifies whether the annotated data to be verified output by the writer conforms to the established principles. When it is determined that the annotated data to be verified conforms to the established principles, an initial correct verification result (that is, the target verification result) is generated as the initial verification result corresponding to the annotated data to be verified; when it is determined that the annotated data to be verified does not conform to the established principles, an initial incorrect verification result (that is, the non-target verification result) is generated as the initial verification result corresponding to the annotated data to be verified.

[0097] In an optional embodiment, when the above-mentioned initial verification result includes a target verification result and a non-target verification result, the embodiment of the present application performs data verification on the labeled data to be verified based on the verification rules through the first verification model to obtain an initial verification result, which may specifically include: the first verification model determines whether the labeled data to be verified complies with the verification rules; when the labeled data to be verified complies with the verification rules, a target verification result is generated, and the target verification result includes a target verification viewpoint and a target verification instruction; when the labeled data to be verified does not comply with the verification rules, a non-target verification result is generated, and the non-target verification result includes a non-target verification viewpoint and a non-target verification instruction.

[0098] In actual processing, the target verification result may include a target verification viewpoint and a target verification instruction. The target verification viewpoint may carry words such as "correct" or "compliant" to indicate that the labeled data to be verified correctly labeled the non-labeled data; the non-target verification result may include a non-target verification viewpoint and a non-target verification instruction. The non-target verification viewpoint may carry words such as "incorrect" or "non-compliant" to indicate that the labeled data to be verified did not correctly label the non-labeled data. Both the target verification instruction and the non-target verification instruction can be used to trigger the second verification model to perform a secondary verification step, i.e., to execute step 250.

[0099] Step 250: perform secondary verification on the first verification result and the labeled data to be verified by using the second verification model to obtain a secondary verification result.

[0100] In a specific implementation, the secondary verification result may include a target discrimination result and a non-target discrimination result, wherein the target discrimination result may refer to a correct secondary verification result, and the non-target discrimination result may refer to an incorrect secondary verification result, which is not limited in this embodiment.

[0101] For example, refer to Figure 4Taking the meta-checker LLM as the second verification model as an example, the first verification result and the annotated data to be verified can be input into the meta-checker LLM for final verification. The meta-checker LLM can first perform annotation verification on the annotated data to be verified to determine whether the annotated data to be verified conforms to the established rules, and then the meta-checker LLM combines the initial verification result of the checker LLM to determine the secondary verification result.

[0102] Specifically, when the verification result of the meta-checker LLM for the labeled data to be verified is consistent with the initial verification result, and the initial verification result is the correct initial verification result, a secondary verification correct result (i.e., the target discrimination result) is generated, that is, when the verification result of the meta-checker LLM for the labeled data to be verified is that the labeled data to be verified is the correct labeled data, and the initial verification result of the labeled data to be verified is the correct initial verification result, the meta-checker LLM generates a secondary verification correct result as the secondary verification result.

[0103] When the meta-checker detects that the initial verification result is an incorrect initial verification result, the meta-checker LLM directly generates a secondary verification incorrect result (i.e., a non-target discrimination result) for the data to be verified as the secondary verification result; when the verification result of the meta-checker LLM for the labeled data to be verified is inconsistent with the initial verification result, a secondary verification incorrect result is generated. For example, when the verification result of the meta-checker LLM for the labeled data to be verified is that the labeled data to be verified is incorrectly labeled data, and the initial verification result is the correct initial verification result, the meta-checker LLM generates a secondary verification incorrect result as the secondary verification result. For example, when the verification result of the meta-checker LLM for the labeled data to be verified is that the labeled data to be verified is incorrectly labeled data, and the initial verification result is the correct initial verification result, the meta-checker LLM generates a secondary verification incorrect result for the labeled data to be verified as the secondary verification result.

[0104] In an optional embodiment, when the above-mentioned secondary verification result includes a target discrimination result and a non-target discrimination result, the embodiment of the present application performs secondary verification on the first verification result and the labeled data to be verified through the second verification model to obtain a secondary verification result, which may specifically include: performing text verification on the labeled data to be verified through the second verification model to obtain a text verification result; when the initial verification result is a target verification result, determining whether the text verification result is consistent with the target verification result; if the text verification result is consistent with the target verification result, generating a target discrimination result based on the text verification result and the target verification result; if the text verification result is inconsistent with the target verification result, generating a non-target discrimination result based on the text verification result.

[0105] In actual implementation, the target discrimination result may carry words such as "correct" or "agree", which is used to characterize that both the checker LLM and the meta-checker LLM determine that the labeled data to be verified are correctly labeled data; the target discrimination result may carry words such as "incorrect", "wrong" or "disagree", which is used to characterize that at least one of the checker LLM and / or the meta-checker LLM determines that the labeled data to be verified are incorrectly labeled data.

[0106] Therefore, the embodiment of the present application realizes the process of using the writer LLM to generate natural language text for a single structured data, and then using the checker LLM to generate the inspection opinion, and then using the meta-checker LLM to generate the third-party judgment result. The unlabeled data is labeled through the cascaded multi-model, and the accuracy of the labeling results is reasonably controlled. The cascade model can realize self-supervision without human supervision, avoiding the costly dependence on labeled data, and effectively solving the problem of data labeling dependence on human resources.

[0107] Step 260: Based on the initial verification result and the secondary verification result, data refinement is performed on the natural language text to obtain target text data.

[0108] For the description of step 260 , reference may be made to the description of step 140 , which will not be described in detail in this embodiment.

[0109] Optionally, the embodiment of the present application performs data refinement on the natural language text according to the initial verification result and the secondary verification result to obtain target text data, which may specifically include the following sub-steps:

[0110] Sub-step 2601, when the initial verification result is a target verification result and the secondary verification result is a target discrimination result, generating the target text data based on the natural language text.

[0111] In a specific implementation, for the labeled data to be verified, when the result output by the checker LLM model of the labeled data to be verified is the target verification result, and the result output by the checker LLM model of the labeled data to be verified is the target discrimination result, both the representation checker LLM and the meta-checker LLM believe that the labeled data to be verified correctly annotates the non-annotated data, that is, the labeled data to be verified correctly annotates the structured data. Therefore, the labeled data to be verified can be retained in the natural language text, so that the data to be verified and its corresponding structured data can be combined into training data for model training and updating.

[0112] Sub-step 2602, when the initial verification result is a non-target verification result and / or the secondary verification result is a non-target discrimination result, filtering the to-be-verified annotation data corresponding to the verification result from the natural language text to obtain the target text data.

[0113] In a specific implementation, for the annotated data to be verified, if its initial verification result is a non-target verification result, and / or, the secondary verification result is a non-target discrimination result, it indicates that the annotated data to be verified does not correctly annotate the non-annotated data, that is, the annotated data to be verified is an incorrect annotation of the structured data. Therefore, the annotated data to be verified can be filtered from the natural language text, and the remaining annotated data to be verified that represents the correct annotation in the natural language text is retained to obtain the target text data.

[0114] For example, refer to Figure 5 After obtaining the results of LLM generation of three different roles, namely, the writer LLM, the checker LLM and the meta-checker LLM, the natural language text data is filtered according to the evaluation results of the checker LLM and the meta-checker LLM. Since the generation result of the checker LLM (i.e., the initial verification result) carries the words "correct" or "wrong", these words can be used to retain the data in the checker LLM generation result that the writer LLM generation result is considered to be correct, and remove the data in the checker LLM generation result that the writer LLM generation result is considered to be wrong. Since the meta-checker LLM generation result (i.e., the secondary verification result) carries the words "agree" or "disagree", these words are used to retain the samples in the meta-checker LLM generation result that indicate agreement with the checker LLM generation conclusion, and remove the samples that indicate negation of the checker LLM generation conclusion. Among them, if the meta-checker LLM generation result contains the word "agree", it is considered to indicate agreement, and if it contains the word "disagree", it is considered to indicate disagreement. The data finally retained is the data that the checker LLM believes is correctly described and the meta-checker LLM agrees with the checker LLM's conclusion, which is used as the target text data.

[0115] Step 270: construct a training set based on the target text data and structured data corresponding to the target text data.

[0116] In a specific implementation, the embodiment of the present application can combine the target text data and its corresponding structured data together to construct a training set in a form similar to structured data-annotated data. Figure 5 , the retained data are taken as the final purified data and as the training set, the text generated by the writer's LLM in these data is regarded as the label of the training set, and the corresponding structured data is the feature of the input model.

[0117] Step 280, fine-tune the parameters of the annotation model based on the training set to obtain a loss function.

[0118] In a specific implementation, the training set can be used as training data to input into the writer LLM for parameter fine-tuning. Figure 6 , use LoRA for efficient parameter fine-tuning, only fine-tune some parameters in the model and calculate the corresponding loss function.

[0119] Step 290, performing back propagation according to the loss function to obtain gradient data.

[0120] In the specific implementation, the gradient data contains the gradient. After calculating the loss function, the loss function can be used to perform back propagation to calculate the gradient and obtain the gradient data, such as Figure 6 shown.

[0121] Step 300: Use the gradient data to update the annotation model.

[0122] In actual processing, refer to Figure 6 , perform gradient descent to update the parameters of the learnable parameters in LoRA, optimize the writer LLM, and obtain a new writer LLM after fine-tuning the parameters of the writer LLM. The objective function in the training process is defined as the commonly used negative logarithmic loss. Therefore, the embodiment of the present application uses multiple large models to interact in a cascade form to generate unlabeled structured data, that is, multiple large models are used as the basic model for text generation, and an unsupervised D2T framework is proposed. Through the automatic matching quality control flow, the single model self-inspection of LLM is expanded to the collaborative learning of multiple models, which promotes the collaboration between multiple LLMs with different roles to verify the generated text. Specifically, an LLM acts as a writer to generate candidate text based on the input data. The additional checker LLM and meta-checker LLM verify the output quality of the candidate text to screen high-quality samples for training the writer LLM, efficiently utilizing the advantages of each LLM. The cascade checking framework can strictly evaluate the factual accuracy and fluency of the language generated by the writer, and realize the use of the generated text purified after multi-model verification as a pseudo-label for structured data, thereby utilizing the language organization reasoning ability of the large model to generate constructed pseudo-labels from a large amount of unlabeled data, and use the constructed data to improve the ability of the large model to generate text for structured data. In addition, the embodiment of the present application utilizes multi-model interaction to improve the quality of text generation, further improves the semantic organization ability of the large model, and reduces the appearance of hallucinations in the generated text.

[0123] In summary, in the text data generation stage, the embodiments of the present application: by obtaining the text data to be processed corresponding to the structured data, for the non-annotated data contained in the text data to be processed, data annotation is performed through a preset annotation model to obtain natural language text, and then the preset verification rules are obtained. Based on the verification rules, each annotated data to be verified contained in the natural language text is verified through a first verification model to obtain an initial verification result, and then the first verification result and the annotated data to be verified are secondary verified through a second verification model to obtain a secondary verification result, and based on the initial verification result and the secondary verification result, the natural language text is data refined to obtain the target text data. In the model training and updating stage: a training set is constructed based on the target text data and the structured data corresponding to the target text data, and the annotation model is fine-tuned based on the training set to obtain a loss function, and then back-propagation is performed according to the loss function to obtain gradient data, and the annotation model is updated using the gradient data. The embodiment of the present application can use unlabeled unsupervised data for pseudo-label construction, which can efficiently utilize unsupervised data without manual annotation, and at the same time improve the accuracy and fluency of the semantic expression of the large model when generating text based on structured data, thereby solving various problems existing in the prior art, including: ① The problem of relying on experts for data annotation, which requires a lot of manpower and domain knowledge; ② The quality and accuracy of the generated text based on the algorithm / text annotation model in text annotation is low, and the actual meaning cannot be accurately represented.

[0124] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0125] like Figure 7 As shown, the embodiment of the present application further provides a text data generating device 700, comprising:

[0126] An acquisition module 710 is used to acquire text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data;

[0127] A data annotation module 720 is used to annotate the non-annotated data using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified;

[0128] The text verification module 730 is used to perform text verification on each of the to-be-verified annotated data through a preset verification model to obtain a verification result corresponding to the to-be-verified annotated data, wherein the verification result includes an initial verification result and a secondary verification result;

[0129] The data refining module 740 is used to perform data refining on the natural language text according to the initial verification result and the secondary verification result to obtain target text data.

[0130] Optionally, the acquisition module 710 includes:

[0131] A structured data acquisition submodule, used to acquire a structured data set, wherein the structured data set includes structured data;

[0132] A data conversion submodule, used for performing data conversion according to the structured data to obtain target character string data;

[0133] The text segmentation submodule is used to perform text segmentation based on the target character string data to obtain the text data to be processed.

[0134] Optionally, the verification model includes a first verification model and a second verification model, and the text verification is performed on each of the to-be-verified annotated data using a preset verification model. The text verification module 730 includes:

[0135] The validation rule acquisition submodule is used to obtain preset validation rules;

[0136] A data verification submodule, configured to perform data verification on the to-be-verified annotated data through the first verification model based on the verification rule to obtain an initial verification result;

[0137] The secondary verification submodule is used to perform secondary verification on the first verification result and the labeled data to be verified through the second verification model to obtain a secondary verification result.

[0138] Optionally, the initial verification result includes a target verification result and a non-target verification result, and the data verification submodule includes:

[0139] A first judgment unit, used for judging whether the to-be-verified annotated data conforms to the verification rule by using the first verification model;

[0140] A target verification result generating unit, configured to generate a target verification result when the to-be-verified annotated data meets the verification rule, wherein the target verification result includes a target verification viewpoint and a target verification instruction;

[0141] The non-target verification result generating unit is used to generate a non-target verification result when the to-be-verified marked data does not meet the verification rule, and the non-target verification result includes a non-target verification viewpoint and a non-target verification instruction.

[0142] Optionally, the secondary verification result includes a target discrimination result and a non-target discrimination result, and the secondary verification submodule includes:

[0143] Performing text verification on the to-be-verified annotated data using the second verification model to obtain a text verification result;

[0144] A second judgment unit, configured to judge whether the text verification result is consistent with the target verification result when the initial verification result is the target verification result;

[0145] a target discrimination result generating unit, configured to generate a target discrimination result based on the text verification result and the target verification result when the text verification result is consistent with the target verification result;

[0146] The non-target discrimination result generating unit is used to generate a non-target discrimination result based on the text verification result when the text verification result does not match the target verification result.

[0147] Optionally, the data refining module 740 includes:

[0148] A target text data generating unit is used to generate the target text data based on the natural language text when the initial verification result is a target verification result and the secondary verification result is a target discrimination result; when the initial verification result is a non-target verification result and / or the secondary verification result is a non-target discrimination result, filter the to-be-verified annotation data corresponding to the verification result from the natural language text to obtain the target text data.

[0149] Optionally, the text data generating device 700 further includes:

[0150] A construction module, used to construct a training set based on the target text data and structured data corresponding to the target text data;

[0151] A parameter fine-tuning module, used to fine-tune the parameters of the annotation model based on the training set to obtain a loss function;

[0152] A back propagation module, used to perform back propagation according to the loss function to obtain gradient data;

[0153] A model updating module is used to update the annotation model using the gradient data.

[0154] It should be noted that the text data generating device provided in the embodiments of the present application can execute the text data generating method provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0155] In a specific implementation, the above-mentioned text data generation device can be integrated into a device, so that the device can generate annotation data corresponding to the structured data, and verify the annotation data multiple times, as an electronic device, to improve the efficiency and accuracy of data annotation. The electronic device can be composed of two or more physical entities, or it can be composed of one physical entity, such as the electronic device can be a personal computer (PC), a computer, a server, etc., and the embodiment of the present application does not impose specific restrictions on this.

[0156] like Figure 8 As shown, the embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to implement the steps of the text data generation method provided by any of the above method embodiments when executing the program stored in the memory 113. Exemplarily, the steps of the text data generation method may include the following steps: obtaining the text data to be processed corresponding to the structured data, and the text data to be processed includes non-annotated data; for the non-annotated data, data annotation is performed by a preset annotation model to obtain a natural language text, and the natural language text includes at least one annotated data to be verified; text verification is performed on each of the annotated data to be verified by a preset verification model, and a verification result corresponding to the annotated data to be verified is obtained, and the verification result includes an initial verification result and a secondary verification result; according to the initial verification result and the secondary verification result, data refinement is performed on the natural language text to obtain target text data.

[0157] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the text data generating method provided in any of the aforementioned method embodiments are implemented.

[0158] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0159] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for generating text data, characterized in that: include: Acquire text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data; For the non-annotated data, data annotation is performed using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified; Performing text verification on each of the to-be-verified annotated data through a preset verification model to obtain a verification result corresponding to the to-be-verified annotated data, wherein the verification result includes an initial verification result and a secondary verification result; According to the initial verification result and the secondary verification result, data refinement is performed on the natural language text to obtain target text data.

2. The method according to claim 1, characterized in that The obtaining of the to-be-processed text data corresponding to the structured data includes: Acquire a structured data set, wherein the structured data set contains structured data; Performing data conversion according to the structured data to obtain target character string data; Text segmentation is performed based on the target character string data to obtain the text data to be processed.

3. The method according to claim 1, characterized in that The verification model includes a first verification model and a second verification model. The text verification is performed on each of the to-be-verified annotated data by using a preset verification model to obtain a verification result corresponding to the to-be-verified annotated data, including: Get the preset validation rules; Based on the verification rules, the to-be-verified annotated data is verified by using the first verification model to obtain an initial verification result; The first verification result and the labeled data to be verified are secondarily verified using the second verification model to obtain a second verification result.

4. The method according to claim 3, characterized in that The initial verification result includes a target verification result and a non-target verification result. Based on the verification rule, the data to be verified is verified by using the first verification model to obtain the initial verification result, including: The first verification model determines whether the to-be-verified annotated data complies with the verification rule; When the to-be-verified annotated data meets the verification rule, a target verification result is generated, wherein the target verification result includes a target verification viewpoint and a target verification instruction; When the to-be-verified annotated data does not conform to the verification rule, a non-target verification result is generated, and the non-target verification result includes a non-target verification viewpoint and a non-target verification instruction.

5. The method according to claim 4, characterized in that The secondary verification result includes a target discrimination result and a non-target discrimination result. The second verification model is used to perform secondary verification on the first verification result and the labeled data to be verified to obtain a secondary verification result, including: Performing text verification on the to-be-verified annotated data using the second verification model to obtain a text verification result; When the initial verification result is a target verification result, determining whether the text verification result is consistent with the target verification result; If the text verification result is consistent with the target verification result, generating a target discrimination result based on the text verification result and the target verification result; If the text verification result does not match the target verification result, a non-target discrimination result is generated based on the text verification result.

6. The method according to claim 5, characterized in that The step of refining the natural language text according to the initial verification result and the secondary verification result to obtain target text data includes: When the initial verification result is a target verification result and the secondary verification result is a target discrimination result, generating the target text data based on the natural language text; When the initial verification result is a non-target verification result, and / or the secondary verification result is a non-target discrimination result, the to-be-verified annotated data corresponding to the verification result is filtered from the natural language text to obtain the target text data.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the target text data, the method further includes: Building a training set based on the target text data and structured data corresponding to the target text data; Fine-tune the parameters of the annotation model based on the training set to obtain a loss function; Perform back propagation according to the loss function to obtain gradient data; The annotation model is updated using the gradient data.

8. A text data generating device, characterized in that: include: An acquisition module, used for acquiring text data to be processed corresponding to the structured data, wherein the text data to be processed includes non-annotated data; A data annotation module, used for annotating the non-annotated data by using a preset annotation model to obtain a natural language text, wherein the natural language text includes at least one annotated data to be verified; A text verification module, used to perform text verification on each of the to-be-verified annotated data through a preset verification model, and obtain a verification result corresponding to the to-be-verified annotated data, wherein the verification result includes an initial verification result and a secondary verification result; The data refining module is used to perform data refining on the natural language text according to the initial verification result and the secondary verification result to obtain target text data.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of the text data generating method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the text data generating method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Text data labeling method and device, equipment and storage medium

    CN112906375A

  • Data labeling method and device

    CN113205163A

  • Text labeling method and device, computer equipment and storage medium

    CN116257624A

  • Label-free medical table data learning method and device based on large language model

    CN117690598A

  • Method and device for processing unlabeled audio data

    CN118072736A

Cited By

  • Label data inspection method, device, equipment, medium and computer program product

    CN120067616A