Data generation method and apparatus, device, and storage medium

Abstract

The application discloses a data generation method, device and equipment and a storage medium. The method comprises the following steps: step 1, automatically processing preset unstructured data to generate structured data; step 2, generating a dialogue sample based on the structured data and a preset role prompt; step 3, respectively evaluating the structured data and the dialogue sample by a preset quality detection rule and generating a feedback instruction; step 4, optimizing a generation strategy of the structured data and / or the dialogue sample based on the feedback instruction; and iteratively executing the steps 1-4 according to the optimized structured data generation strategy and / or dialogue sample generation strategy until the evaluation of the structured data and the dialogue sample is qualified, and outputting an optimized first dialogue sample. The application replaces manual labeling with an automatic process, reduces invalid generation through real-time feedback, and guarantees data quality through iterative optimization, thereby effectively solving the problems of data scarcity, high labeling cost and quality fluctuation in the traditional method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a data generation method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large model fine-tuning technology has been widely used in various question and answer systems and AI model development in professional fields, especially in the intelligent upgrading of vertical fields such as medicine, finance, law and education. However, this technology faces the problem of scarcity of high-quality labeled data in the landing process.

[0003] At present, most AI model training relies on open source data sets or manually labeled data. However, although open source data sets have the characteristics of low acquisition cost and wide coverage, they generally have problems such as mismatch between data distribution and specific professional scene needs, insufficient annotation accuracy, and are difficult to directly adapt to fine-tuned tasks in vertical fields. Artificially labeled data can meet the needs of the scene, but there are defects such as high labeling cost, long cycle, and poor data quality stability due to differences in professional ability of labeling personnel. The limitations of the above data sources will lead to the problem of difficulty in obtaining data and low development efficiency in professional field AI model construction. SUMMARY

[0004] The main purpose of the present application is to provide a data generation method, device, equipment and storage medium, aiming to solve the technical problems of scarcity of high-quality data, high labeling cost and difficulty in ensuring data quality.

[0005] To achieve the above purpose, the present application provides a data generation method, which comprises the following steps:

[0006] Step 1: automatically processing the preset unstructured data to generate structured data;

[0007] Step 2: generating a dialogue sample based on the structured data and a preset role prompt;

[0008] Step 3: evaluating the structured data and the dialogue sample respectively by a preset quality detection rule and generating a feedback instruction;

[0009] Step 4: optimizing the generation strategy of the structured data and / or the dialogue sample based on the feedback instruction;

[0010] Iteratively executing steps 1-4 according to the optimized structured data generation strategy and / or dialogue sample generation strategy until the structured data and dialogue sample are both qualified, and outputting the first optimized dialogue sample.

[0011] In an embodiment, the outputting the dialogue sample comprises:

[0012] Step 5, inputting the first dialogue sample as a training set into a pre-trained model to perform model training through supervised fine-tuning;

[0013] Step 6, monitoring changes in indicators during training and generating feedback instructions, and returning to perform Step 4 to output an optimized second dialogue sample;

[0014] The optimized second dialogue sample is supplemented to the training set, and Steps 5-8 are iteratively performed until the indicators meet a preset threshold, and an optimized training model is output.

[0015] In an embodiment, the data generation system further comprises a coordinator module, and the step of generating a dialogue sample based on the structured data and a preset role prompt comprises:

[0016] Based on the structured data and the preset role prompt, a preset inquiry agent and a reply agent are called to automatically generate dialogue text in an adversarial manner;

[0017] The semantic correlation of the dialogue text is analyzed using the coordinator module, and when the semantic correlation is detected to be lower than a preset threshold, the inquiry agent and the reply agent are triggered to perform semantic correction;

[0018] The dialogue generation process of the inquiry agent and the reply agent is monitored, and when it is identified that the dialogue meets a preset dialogue termination rule, a termination signal is output, and the dialogue text is standardized and formatted to generate a dialogue sample that can be directly used for model training.

[0019] In an embodiment, the step of generating dialogue text based on the structured data and the preset role prompt by calling a preset inquiry agent and a reply agent in an adversarial manner comprises:

[0020] The inquiry agent generates a question text based on the structured data;

[0021] The role prompt is injected into the question text by the reply agent to form an injection text, and a preset model is called based on the injection text to perform model inference, and an individualized answer text is output.

[0022] In an embodiment, the question text comprises positive samples and negative samples, and the step of generating a question text based on the structured data using the inquiry agent comprises:

[0023] At least one semantic element is extracted from the structured data, the semantic element comprising an entity, an entity attribute, and an entity relationship;

[0024] calling a preset question template library, matching a question template corresponding to the semantic element from the preset question template library, and generating a positive sample;

[0025] adjusting the positive sample through a semantic disturbance mechanism to generate a negative sample similar in semantics to the positive sample.

[0026] In an embodiment, the step of adjusting the positive sample through a semantic disturbance mechanism to generate a negative sample similar in semantics to the positive sample comprises:

[0027] extracting a semantic element corresponding to the positive sample, and matching a disturbance mode corresponding to the semantic element from a preset semantic disturbance rule library;

[0028] adjusting the positive sample according to the disturbance mode to generate a candidate negative sample;

[0029] calculating a semantic overlap degree between the candidate negative sample and the positive sample through a semantic similarity algorithm, and determining the candidate negative sample as a negative sample if the semantic overlap degree is within a preset range.

[0030] In an embodiment, the step of automatically processing the preset unstructured data to generate structured data comprises:

[0031] adopting an analysis algorithm to extract information from the preset unstructured data, performing format cleaning on the extracted information, and mapping the information to a preset structured data template to obtain structured data.

[0032] In addition, to achieve the above-mentioned purposes, the present application also provides a data generation device, which comprises:

[0033] a case generation module configured to perform step 1: automatically processing preset unstructured data to generate structured data; and adjust a generation strategy of the structured data based on a feedback instruction;

[0034] a dialogue generation module configured to perform step 2: generating a dialogue sample based on the structured data and a preset role prompt; and adjust a generation strategy of the dialogue sample based on a feedback instruction;

[0035] a real-time feedback module configured to perform step 3: evaluating the structured data and the dialogue sample through preset quality detection rules and generating a feedback instruction;

[0036] a strategy optimization module configured to perform step 4: optimizing the generation strategy of the structured data and / or the dialogue sample based on the feedback instruction;

[0037] An iteration module iteratively performs the steps 1-4 according to the optimized structured data generation strategy and / or the dialogue sample generation strategy until the structured data and the dialogue sample are both qualified, and outputs the optimized first dialogue sample.

[0038] In addition, to achieve the above object, the present application further provides a terminal device, comprising a memory, a processor, and a data generation program stored in the memory and executable on the processor, wherein the data generation program, when executed by the processor, implements the steps of the data generation method as described above.

[0039] In addition, to achieve the above object, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a data generation program, and the data generation program, when executed by a processor, implements the steps of the data generation method as described above.

[0040] The one or more technical solutions provided by the present application have at least the following technical effects:

[0041] The present application converts unstructured data into structured data, avoids high-cost investment of manual annotation, generates dialogue samples based on structured data and role prompts, realizes rapid expansion of data scale, quantitatively evaluates the structured data and the dialogue samples according to preset quality detection rules, generates feedback instructions to optimize the generation strategy of the structured data and / or the dialogue samples, further generates structured data and / or dialogue samples according to the optimized structured data generation strategy and / or dialogue sample generation strategy, and gradually improves the qualified rate of the structured data and the dialogue samples after multiple iterations, and finally outputs dialogue samples meeting the quality standards. The present application replaces manual annotation with an automatic process, reduces invalid generation through real-time feedback, and guarantees data quality through iterative optimization, effectively solves the problems of data scarcity, high annotation cost and quality fluctuation in traditional methods, and realizes double improvement of data production efficiency and quality. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of a first embodiment of the data generation method of the present application is shown in FIG. 1;

[0043] Figure 2 A flowchart of a second embodiment of the data generation method of the present application is shown in FIG. 2;

[0044] Figure 3 A flowchart of a third embodiment of the data generation method of the present application is shown in FIG. 3;

[0045] Figure 4 A first flowchart of the third embodiment of the data generation method of the present application is shown in FIG. 4;

[0046] Figure 5A second flowchart of the third embodiment of the data generation method of the present application;

[0047] Figure 6 A third flowchart of the third embodiment of the data generation method of the present application;

[0048] Figure 7 A module structure diagram of the data generation system of the embodiment of the present application;

[0049] Figure 8 A device structure diagram of the hardware running environment involved in the data generation method in the embodiment of the present application.

[0050] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0052] The main technical solution of the present application is: step 1: automatically processing the preset unstructured data to generate structured data; step 2: generating a dialogue sample based on the structured data and a preset role prompt; step 3: respectively evaluating the structured data and the dialogue sample through a preset quality detection rule and generating a feedback instruction; step 4: optimizing the generation strategy of the structured data and / or the dialogue sample based on the feedback instruction; iteratively executing the steps 1-4 according to the optimized structured data generation strategy and / or dialogue sample generation strategy until the structured data and the dialogue sample are both qualified, and outputting the optimized first dialogue sample.

[0053] The present application actually considers that most AI model training relies on open source data sets or manually annotated data. However, open source data sets are difficult to completely adapt to specific professional scenarios; manually annotated data can meet the needs of the scene, but there are defects such as high annotation cost, long cycle, poor data quality stability caused by professional ability difference of annotators, and these limitations make the professional field AI model construction face the problems of difficult data acquisition and low development efficiency.

[0054] Based on this, the present application proposes a data generation method.

[0055] Specifically, the following are the detailed steps of the first embodiment of the data generation method of the present application:

[0056] Reference Figure 1 , Figure 1 A flowchart of the first embodiment of the data generation method of the present application. In the present embodiment, the data generation method comprises steps S10-S50:

[0057] Step S10, the preset unstructured data is automatically processed to generate structured data;

[0058] Specifically, a case generation module is called, the case generation module being a functional unit for converting unstructured data into structured data. The case generation module extracts information from the input unstructured data using a parsing algorithm, identifies the key information therein, then cleans the extracted information in format, removes redundant content, unifies the data format, maps the cleaned information into a preset structured data template, and forms JSON or Markdown format data with a fixed structure. This process does not require manual intervention, and realizes full-automatic conversion from unstructured data to structured data.

[0059] In a feasible implementation, a case generation Agent is used to automatically process the preset unstructured data to generate structured data. The case generation Agent is an intelligent agent for automatically converting unstructured data into structured data. Specifically, the case generation Agent is called, and a corresponding parsing strategy is selected according to the type of the input unstructured data: for a text file, the case generation Agent calls a text parsing model based on a Transformer architecture, extracts entity, attribute and relationship triplets in the text through sentence segmentation, word segmentation and semantic role labeling technology, and filters non-target information using a regular expression; for an image file, the case generation Agent starts an OCR text recognition engine to extract text content in the image, and then combines a target detection model to identify entity objects and their spatial positional relationships in the image, and fuses visual features and text information into a unified information unit. After completing the information extraction, the case generation Agent performs standardization processing on the extracted information, including correcting text spelling errors, unifying entity name expressions, converting numerical data formats, etc., and based on a preset structured data template, maps the cleaned information units to the corresponding fields of the template through a field matching algorithm, and finally generates structured data that meets the specifications, realizing automatic conversion of unstructured data to structured data.

[0060] By automatically processing unstructured data, the dependence on manual annotation is reduced. Traditional manual conversion of structured data requires professional personnel to parse documents one by one, which is time-consuming and costly. Automatic processing can process a large amount of data in batches, shortening the data preparation period. At the same time, the preset structured template ensures the uniformity and standardization of the data format, avoiding the format confusion problem in manual conversion.

[0061] Step S20, generating a dialogue sample based on the structured data and a preset role prompt;

[0062] It should be noted that the preset role prompt is a pre-configured dialogue role setting, including role identity, language style and interaction target and the like information; the dialogue sample is dialogue text generated by simulating a real scene, and is used for subsequent model training.

[0063] Specifically, a dialogue generation module is called, the dialogue generation module is a functional unit for simulating human-computer or human-machine interaction to generate dialogue text, at least one semantic element is extracted from the structured data by using the dialogue generation module, and a dialogue context is constructed in combination with a preset role prompt. Then, questions and answers are generated through interaction logic, for example, the dialogue generation module includes a user role and a customer service role, the user role asks questions based on the "fault phenomenon" in the structured data, and the customer service role generates answers based on the injected role prompt, and a multi-round dialogue text is automatically generated in an adversarial manner, and finally a coherent dialogue sample is formed.

[0064] A feasible implementation manner is that the dialogue generation module includes an inquiry agent and a reply agent, wherein the inquiry agent adopts a question Agent, the reply agent adopts an answer Agent, and the two automatically generate question and answer data in an adversarial manner. The question Agent is an intelligent agent for generating question text conforming to scene logic; the answer Agent is an intelligent agent responsible for answering questions, and the answer Agent activates diversified language modes and expression habits by injecting different persona prompts in the model reasoning stage, thereby simulating the differences of real users in knowledge background, personality characteristics and word style.

[0065] For example, in an education training scene, for the structured data of "mathematical function definition", the question Agent can generate the basic question "how to understand the definition domain and value domain of function?"; the answer Agent generates the answer "for example, y = 2x, x can take 1, 2, 3, which are the definition domain; the corresponding y = 2, 4, 6 is the value domain, just like the relationship between 'input' and 'output'" by injecting the persona of "novice student"; if the persona of "experienced teacher" is injected, the answer "the definition domain is the value set of the independent variable of the function, which needs to satisfy the condition of making the expression meaningful; the value domain is the corresponding value set of the dependent variable, which is determined by the definition domain and the function relationship" is generated. Through this personalized simulation, the system can generate dialogue data that is closer to the actual teaching scene, has diversity and authenticity, and effectively improves the adaptation ability of the model to different user expression habits.

[0066] In step S30, the structured data and the dialogue sample are respectively evaluated by a preset quality detection rule, and a feedback instruction is generated.

[0067] It should be noted that the preset quality detection rule is a pre-defined standard set for judging whether the data is qualified or not.

[0068] In a feasible implementation, for structured data, the detection rule is set as multi-dimensional checking logic based on Schema verification and entity relationship graph, and for dialogue samples, the detection rule is set as a joint evaluation mechanism that fuses semantic vector comparison and role feature vector. In specific implementation, a real-time feedback module is called, which is a monitoring unit responsible for data quality evaluation and optimization instruction generation. The real-time feedback module loads a pre-set structured data Schema template, performs syntax layer verification on the structured data output by the case generation module through JSON Schema Validator, such as checking whether the “entity ID” field conforms to the UUID format and whether the “timestamp” meets the ISO 8601 standard, and simultaneously calls a graph database interface to load a domain entity relationship graph, and verifies whether the entity association in the structured data conforms to the pre-set logic through SPARQL query. For dialogue samples generated by the dialogue generation module, the real-time feedback module converts the question and answer texts into semantic vectors through a pre-trained language model, calculates the cosine similarity of the two to evaluate semantic consistency, and simultaneously loads a role feature vector library to calculate the matching degree of the current dialogue sample and the target role feature vector.

[0069] In the quality evaluation process, the real-time feedback module maps the above verification results to quantitative scores, and when the score is lower than a pre-set threshold, automatically triggers feedback instruction generation logic. Specifically, for structured data, the rule engine is used to locate specific abnormal fields, and a pre-set repair strategy library is called to generate feedback instructions; for dialogue samples, an error attribution algorithm is used to identify the root cause to generate feedback instructions, and all feedback instructions are encapsulated in a standardized JSON format.

[0070] In a feasible implementation, the real-time feedback module adopts a real-time feedback Agent, which is an intelligent agent that continuously monitors data generation and outputs targeted optimization instructions. The real-time feedback module adopts the real-time feedback Agent as the core execution unit, and its core function is to monitor the quality of the outputs of the case generation Agent, the question Agent, and the answer Agent during data generation, and output dynamic feedback opinions through multi-dimensional evaluation indicators to promote the iterative optimization of the data generation strategies of the Agents, and finally form a sustainable optimization closed loop of data generation, evaluation, and feedback.

[0071] Step S40, optimizing the generation strategy of the structured data and / or dialogue samples based on the feedback instructions;

[0072] In a feasible implementation, a feedback instruction parsing engine is constructed to perform syntax parsing and semantic mapping on the standardized JSON format instructions output by the real-time feedback module, extract key control parameters, and locate specific strategy configuration nodes of the case generation module and / or the dialogue generation module. For the generation strategy adjustment of structured data, a parameter dynamic injection mechanism is adopted: when the feedback instruction points to a "field missing" type problem, the parsing engine calls the template engine API to update the structured data generation template, appends field definitions in the JSON Schema, and sends a hot update signal to the entity extractor of the case generation module to trigger the domain dictionary loading process.

[0073] For the generation strategy adjustment of dialogue samples, a double-layer control architecture is adopted: for example, at the bottom parameter layer, for instructions such as "question Agent negative sample diversity is insufficient", the parsing engine calls the Action interface of the reinforcement learning environment to dynamically adjust the disturbance factor of the negative sample generator and retrain the Q value function of the template selection strategy; at the high-level strategy layer, when the instruction requires optimizing the role adaptation degree, a new persona prompt template is generated through the Prompt Engineering tool kit.

[0074] In addition, a cross-module collaborative adjustment mechanism is adopted: when the feedback instruction involves multi-module linkage optimization, the generation strategy adjustment of the case generation module and the dialogue generation module is synchronized by the distributed coordination service, and the bias term is injected into the attention mechanism of the dialogue generation module while the associated rules are added in the case generation module, to ensure that the core elements of structured data are activated first in dialogue generation.

[0075] Step S50, according to the optimized structured data generation strategy and / or dialogue sample generation strategy, iteratively execute the steps S10~S40 until the structured data and dialogue samples are qualified, and output the optimized first dialogue sample.

[0076] A feasible implementation starts the iteration controller, sets the initial iteration number to 0 and the maximum iteration threshold. In each loop, the real-time feedback module is called to perform quality evaluation on the structured data and dialogue samples, and a quantitative score is generated through the preset indicators. If the scores meet the standards, the dialogue sample is determined to be qualified and output; otherwise, based on the feedback instruction, the case generation / dialogue generation module is driven to perform strategy adjustment. The iteration number is incremented, and it is checked whether the iteration threshold is reached. If not, the loop continues; if the threshold is reached and it is not qualified, an alarm is triggered and the current optimal sample is output.

[0077] Exemplarily, in the medical AI training data generation scenario, the case generation agent receives unstructured raw data as an outpatient medical record text containing patient complaints and examination reports. The case generation agent extracts key information and converts it into structured data through entity recognition and relationship extraction technology: patient basic information: age 45, female; symptom data: headache for 3 days, accompanied by nausea; examination data: blood pressure 140 / 90 mmHg, body temperature 37.2°C; diagnosis result: tension headache; medication recommendation: ibuprofen sustained-release capsules. Then, the question agent generates questions based on the structured data that meet the scenario logic, such as combining "headache for 3 days" and "accompanied by nausea" to generate the user question: "Have you been having headaches for the past three days, and do you have nausea and vomiting? What medicine do you need to take to relieve it?" The answer agent generates an answer by injecting the "community doctor" persona prompt to set "language is popular and easy to understand, and focuses on life advice and medication guidance combination": "Your condition is considered tension headache. You are advised to take ibuprofen sustained-release capsules, one capsule each time, twice a day. At the same time, avoid staying up late and looking at screens for a long time. If the symptoms do not improve after taking the medicine or the blood pressure continues to rise, you need to seek medical treatment as soon as possible."

[0078] In the above data generation process, the real-time feedback agent checks the structured data of the case generation agent and finds that the "medication recommendation" field does not include "contraindication explanation". Therefore, it generates a feedback instruction: "Need to supplement the extraction rules of 'ibuprofen contraindicated population' in unstructured data"; for the dialogue sample, through semantic correlation evaluation, it is found that the answer does not explicitly associate the "blood pressure 140 / 90 mmHg" in the structured data, and it is determined that there is a potential risk, and a feedback instruction is generated: "Need to combine the blood pressure value in the structured data to supplement the 'if you have a history of high blood pressure, consult a doctor before taking medicine' prompt". Based on this, the case generation agent optimizes the parsing algorithm according to the feedback and supplements the "contraindication" field; the answer agent updates the persona prompt to increase the guiding logic of "paying attention to the patient's underlying disease and medication safety", and regenerates the answer: "Your condition is considered tension headache, and your blood pressure 140 / 90 mmHg is slightly higher than normal. If you do not have a history of high blood pressure, you can take ibuprofen sustained-release capsules, one capsule each time, twice a day, and pay attention to rest; if you have high blood pressure or stomach disease, you are advised to consult a doctor before taking medicine."

[0079] Through the closed loop of real-time feedback agent driven data generation, evaluation and optimization, the system continuously improves data quality: the completeness of structured data fields is improved, the role adaptation degree and semantic safety of dialogue samples reach the preset threshold, and the final generated medical inquiry dialogue data can be directly used to train AI inquiry model, effectively improving the model's adaptation ability to real clinical scenarios.

[0080] Further, with reference toFigure 2 , Figure 2 The flowchart of the second embodiment of the data generation method of the present application is shown in the figure. Based on the first embodiment, the output of the dialogue sample includes steps S60-S80:

[0081] Step S60, input the first dialogue sample as a training set into the pre-trained model, and perform model training through supervised fine-tuning;

[0082] Step S70, monitor the index change during training and generate a feedback instruction, and return to execute step S40 to output the second dialogue sample after optimization;

[0083] Step S80, supplement the second dialogue sample after optimization to the training set as a new dialogue sample, and iteratively execute steps S60-S70 until the evaluation of the index meets the preset threshold, and output the optimized training model.

[0084] It should be noted that supervised fine-tuning refers to adjusting model parameters using labeled data, i.e., the training set, based on the pre-trained model, to adapt to specific tasks.

[0085] A feasible implementation, after outputting the first dialogue sample, first, divide the first dialogue sample into a training set and a validation set according to a preset proportion, input the training set into the pre-trained model through a distributed training framework, and perform supervised fine-tuning: load the model weight to the GPU cluster, set the learning rate to 2e-5 and the batch size to 32, update the model parameters through back propagation, and simultaneously calculate the Loss and semantic consistency score in real time on the validation set. Specifically, the matching degree between the generated text and the label can be evaluated by the cosine similarity algorithm.

[0086] Call the real-time feedback module to monitor the index change: if the Loss decrease rate is lower than the preset threshold, it is determined that the "data diversity is insufficient"; if the semantic consistency score fluctuation amplitude is greater than the preset threshold, it is marked as "sample quality is unstable". Based on this, the real-time feedback module generates a feedback instruction.

[0087] After analyzing the feedback instruction, the generation strategy of the structured data and / or dialogue sample is optimized based on the feedback instruction, and the structured data and dialogue sample are regenerated based on the adjusted strategy. The above steps are iteratively executed until the second dialogue sample after optimization is generated, and the training set is supplemented after data cleaning to form "data increment".

[0088] Subsequently, the supplemented training set is re-input into the model for the next round of supervised fine-tuning, and the real-time feedback module continues to track the index change. When the index meets the preset threshold, the iteration is terminated, and the optimized training model is output, and the final training set and strategy adjustment log are saved, forming a traceable optimization closed loop.

[0089] The second embodiment of the present application improves the quality of training data and the performance of the model by constructing a closed-loop optimization mechanism of data generation, model training and strategy adjustment. After outputting the dialogue sample, the second embodiment of the present application inputs it as a training set into a pre-trained model for supervised fine-tuning, and simultaneously generates feedback instructions by using the real-time feedback module to dynamically change the indicators in the model training process, to drive the case generation module and the dialogue generation module to optimize the generation strategy. Through iterative execution of the data generation and strategy adjustment process, the optimized dialogue sample is supplemented to the training set, forming a "data increment" to continuously benefit the model training, and finally making the model converge under a preset threshold to output a high-performance training model. The dynamic linkage of data generation and model optimization is realized, and the adaptability of the training data and the generalization ability of the model are improved.

[0090] Further, with reference to Figure 3 , Figure 3 is a flowchart of the third embodiment of the data generation method of the present application. Based on the first embodiment, the step S20 includes steps S21-S23:

[0091] Step S21, based on the structured data and the preset role prompt, calling a preset inquiry agent and a reply agent to generate dialogue text in an antagonistic manner;

[0092] Step S22, analyzing the semantic correlation degree of the dialogue text, and when detecting that the semantic correlation degree is lower than a preset threshold, triggering the inquiry agent and the reply agent to perform semantic correction;

[0093] Step S23, monitoring the dialogue generation process of the inquiry agent and the reply agent, and when identifying that a preset dialogue termination rule is met, outputting a termination signal and performing standardized format processing on the dialogue text to generate a dialogue sample that can be directly used for model training.

[0094] It should be noted that the inquiry agent is used to simulate the questioning behavior in the real scene and can actively ask questions based on the structured data. The reply agent is used to generate answer texts, and by injecting a preset role prompt, it simulates the expression habits of users with different backgrounds and personalities to generate diversified answer contents. The antagonistic manner refers to that the inquiry agent and the reply agent dynamically adjust the questioning and answering strategies to form an interactive process similar to "attack and defense", so as to generate more realistic and complex dialogue data.

[0095] Specifically, the inquiry agent extracts key semantic elements based on the structured data, and generates question texts in combination with the preset role prompt; the reply agent generates personalized answers according to the role setting and the question content, and the two interact in an antagonistic manner, the inquiry agent may intentionally ask ambiguous questions or ask for details, and the reply agent needs to respond based on professional knowledge, so as to generate dialogue texts close to real scenes.

[0096] Then, the coordinator module is called, which is used to monitor the dialogue generation process, evaluate the semantic quality, and trigger the correction mechanism. The coordinator module accesses the dialogue generation process in real time, extracts the semantic vectors of the questions and answers, and calculates the cosine similarity as the semantic correlation degree. When the semantic correlation degree is detected to be lower than the preset threshold, the coordinator module sends a correction instruction to the inquiry agent, which contains semantic deviation analysis and correction direction. After receiving the instruction, the inquiry agent generates a corrected question, and re-triggers the reply agent to generate the corresponding text. For example, the coordinator module detects the semantic correlation degree between the question "Can diabetic patients eat fruits?" and the answer "Diabetic patients should control carbohydrate intake." If the correlation degree is lower than the preset threshold, the inquiry agent and the reply agent are triggered to re-adjust the dialogue strategy, such as the inquiry agent asking "Which fruits are suitable for diabetic patients?", and the reply agent supplementing specific fruit types and eating suggestions, until the semantic correlation degree meets the standard.

[0097] The coordinator module monitors the dialogue process and determines whether the termination rule is met, such as when the core question of the inquiry agent has been fully answered, or the dialogue round reaches the preset upper limit, an output termination signal is sent, and the dialogue text is converted into a unified format to generate standardized dialogue samples that can be directly used for model training.

[0098] For example, the number of structured data fields covered by the dialogue is counted in real time, and when the coverage rate is greater than the preset threshold or the number of question and answer rounds reaches the preset upper limit, it is determined that the dialogue termination rule is met, or when the core question of the inquiry agent has been fully answered, or the number of dialogue rounds reaches the preset upper limit, it is determined that the dialogue termination rule is met, and a termination signal is immediately sent, and the inquiry agent and the reply agent terminate execution. At this time, the coordinator module calls the standardization processing tool to perform format conversion on the dialogue text, and finally outputs dialogue samples that can be directly used for model training. During the whole process, the coordinator module records the semantic evaluation results, correction trajectory, and termination rule triggering conditions through the log system.

[0099] A feasible implementation, the coordinator module adopts a coordinator agent, the coordinator agent is an intelligent agent responsible for monitoring the whole process of dialogue generation, determining the natural end time of dialogue and ensuring the efficiency of data generation. After starting the dialogue generation, the coordinator agent first loads the preset dialogue termination rule library, and accesses the interaction data stream of the inquiry intelligent agent and the reply intelligent agent in real time through the API interface. During the dialogue process, after receiving the question and answer text every round, the coordinator agent synchronously executes two judgments: one is to judge whether the proportion of the structured data field covered by the current dialogue exceeds the preset proportion, and the other is to extract the semantic vector of the question and answer, calculate the cosine similarity as the semantic correlation degree, and judge whether the semantic correlation degree is not less than the preset threshold. Based on the above judgments, the coordinator agent dynamically decides whether the dialogue is naturally ended, when the termination condition is met, the termination process is triggered immediately, and the termination signal is sent to the inquiry intelligent agent and the reply intelligent agent, so as to avoid the consumption of invalid rounds; If the termination condition is not met, the dialogue is allowed to continue iteration. In addition, the coordinator agent records the time consumption of each link, and when it is detected that the single round processing time exceeds twice the average time consumption, more computing resources are automatically allocated by calling the resource scheduling interface, so as to ensure the efficient progress of the data generation process. Through the above mechanism, the coordinator agent realizes the intelligent monitoring and dynamic regulation and control of the dialogue process, which not only guarantees the integrity and rationality of the generated data, but also avoids the process redundancy, and improves the overall generation efficiency.

[0100] In a feasible implementation, with reference to Figure 4 , the step S21 includes steps S211-S212;

[0101] Step S211, generating a question text based on the structured data using the inquiry intelligent agent;

[0102] Specifically, the inquiry agent adopts a questioning agent, which refers to an agent that generates questioning text that conforms to the logic of the scene. When generating the question text, the questioning agent first extracts at least one semantic element from the input structured data through an entity recognition model and a relation extraction model, which specifically includes entities, entity attributes, and entity relationships, and stores the extracted semantic elements as structured metadata. Next, the questioning agent calls a pre-set question template library, which contains standardized question frameworks corresponding to different combinations of semantic elements. The entities, entity attributes, and entity relationships extracted by the semantic element matching algorithm are compared with the frameworks in the template library, and the question template with the highest matching degree is selected. Then, the semantic elements are filled into the corresponding positions of the template to generate a positive sample that conforms to the scene logic and is semantically accurate. The positive sample is used as the semantic basis, and the positive sample is adjusted through a semantic perturbation mechanism to generate a negative sample similar in semantics to the positive sample. The positive sample and the negative sample are used together as the question text, which can help the model distinguish subtle semantic differences during the training process, avoid over-reliance on fixed features, and thus improve the generalization ability.

[0103] In step S212, the reply agent injects a pre-set persona prompt into the question text to form an injected text, and calls a pre-set model based on the injected text to perform model inference and output an individualized answer text.

[0104] It should be noted that the injected text is a mixed text obtained by fusing the pre-set persona prompt and the question text. The individualized answer text is a natural language answer generated by the reply agent in combination with the persona prompt and the structured data, which conforms to the style of a specific persona.

[0105] Specifically, after receiving the question text output by the inquiry agent, the reply agent first reads a pre-set persona prompt from the persona prompt library configured by the system. The prompt contains information such as the language style characteristics of the target persona, the tone and intonation parameters, and the domain knowledge background description. The persona prompt is injected into the question text to form an injected text. Subsequently, the reply agent calls a pre-set generation model, such as a domain fine-tuning model based on the LLaMA architecture, and uses the injected text as the input sequence for model inference. During the inference process, the model preferentially captures the role feature parameters in the persona prompt through the attention mechanism, activates the neuron cluster of the corresponding language mode, and for example, when the persona prompt specifies "oral expression", the model reduces the use of long sentences and increases the proportion of daily vocabulary. At the same time, the model generates initial answer content in combination with the core entities in the question text and the structured data, and then adjusts the tone and intonation of the output text through a style adaptation layer, finally outputs an individualized answer text that has both information accuracy and role characteristics.

[0106] In an implementable embodiment, the question text includes positive samples and negative samples; with reference to Figure 5 The step of generating question text based on the structured data by the inquiry agent includes steps S213-S215:

[0107] Step S213, extracting at least one semantic element from the structured data, the semantic element including entity, entity attribute and entity relationship;

[0108] Step S214, calling a preset question template library, matching a question template corresponding to the semantic element from the preset question template library, and generating a positive sample;

[0109] Step S215, taking the positive sample as a semantic basis, adjusting the positive sample through a semantic disturbance mechanism, and generating a negative sample similar in semantics to the positive sample.

[0110] It should be noted that the entity refers to a specific or abstract thing that can exist independently in the real world, which is the basic unit of semantic expression; the entity attribute refers to the nature, parameter or attribute value of the entity; the entity relationship refers to the semantic association between two or more entities, used to describe the logical contact between entities, such as interaction, subordination, space and time. The positive sample is a question text generated based on the matching of the semantic element and the question template, which conforms to the logic of the real scene and is used as the basic sample for model training. The negative sample is a question text generated based on the positive sample as the semantic basis, through the semantic disturbance mechanism, which is similar to the positive sample in surface but has differences in core semantics, and is used to enhance the discrimination ability of the model to subtle semantic differences.

[0111] Specifically, the inquiry agent extracts at least one semantic element from the structured data, the semantic element specifically including entity, entity attribute and entity relationship, and stores the extracted semantic element as structured metadata. Then, the inquiry agent calls a preset question template library, which contains standardized question frameworks corresponding to different combinations of semantic elements. The entity, entity attribute and entity relationship extracted by the semantic element matching algorithm are compared with the frameworks in the template library, the question template with the highest matching degree is selected, and then the semantic elements are filled into the corresponding positions of the template to generate a positive sample that conforms to the scene logic and is accurate in semantics.

[0112] Further, the inquiry agent generates a positive sample as a semantic basis, starts a preset semantic disturbance mechanism, and adjusts the original semantics of the positive sample by calling a large model, specifically including synonym replacement or near-synonym replacement of keywords in the positive sample, reversal processing of logical relationships between entities, or adding bias limiting conditions to the context hypothesis, thereby generating a negative sample that is highly similar in surface structure but has obvious differences in core semantic expression, realizing the adversarial generation of positive samples and negative samples, increasing the diversity of training data, and helping the model better capture semantic details in the training process to avoid overfitting.

[0113] In a feasible implementation manner, referring to Figure 6 , the step S215 includes steps S2151-S2153:

[0114] In step S2151, the semantic elements corresponding to the positive sample are extracted, and a corresponding disturbance mode is matched from a preset semantic disturbance rule library according to the semantic elements;

[0115] In step S2152, the positive sample is adjusted according to the disturbance mode to generate a candidate negative sample;

[0116] In step S2153, the semantic overlap degree of the candidate negative sample and the positive sample is calculated by a semantic similarity algorithm, and if the semantic overlap degree is within a preset range, the candidate negative sample is determined as a negative sample.

[0117] Specifically, the inquiry agent extracts core semantic elements such as entities, entity attributes, and entity relationships corresponding to the positive sample, and matches these elements with a preset semantic disturbance rule library. The rule library contains disturbance strategies designed based on different semantic element types, such as replacement rules for entities, reversal rules for logical relationships, etc., and the applicable disturbance mode is determined according to the matching result.

[0118] Then, the inquiry agent adjusts the positive sample according to the selected disturbance mode, modifies the keywords, logical conjunctions, or context limiting conditions by calling a text editing interface, and generates a preliminary candidate negative sample. Then, the semantic vectors of the candidate negative sample and the positive sample are compared using a semantic similarity algorithm to obtain the semantic overlap degree value of the two. If the value is within a preset range, the candidate negative sample is determined to meet the adversarial requirements and is formally determined as a negative sample; if it does not meet the requirements, the disturbance adjustment process of step S2152 is triggered again until a negative sample that meets the conditions is generated.

[0119] Further, in the fourth embodiment of the data generation method of the present application, on the basis of the first embodiment, the step S10 includes step S11:

[0120] In step S11, an analysis algorithm is used to extract information from the preset unstructured data, the extracted information is format cleaned, and is mapped to a preset structured data template to obtain structured data.

[0121] Specifically, a case generation agent is invoked, which is an intelligent agent for automatically converting unstructured data into structured data. Specifically, the case generation agent selects a corresponding analysis strategy according to the type of input unstructured data: for a text file, the case generation agent invokes a text analysis model based on a Transformer architecture to extract entity, attribute and relationship triples in the text through sentence segmentation, word segmentation and semantic role labeling techniques, and uses regular expressions to filter non-target information; for an image file, the case generation agent starts an OCR text recognition engine to extract text content in the image, and then combines a target detection model to identify entity objects and their spatial position relationships in the image, and fuses visual features and text information into a unified information unit. After completing information extraction, the case generation agent performs standardization processing on the extracted information, including correcting text spelling errors, unifying entity name expressions, converting numerical data formats, etc., and based on a preset structured data template, the cleaned information units are mapped to the corresponding fields of the template through a field matching algorithm, and finally structured data conforming to the specification is generated, realizing the automatic conversion of unstructured data to structured data.

[0122] By automatically processing unstructured data, the dependence on manual annotation is reduced. Traditional manual conversion of structured data requires professional personnel to analyze documents one by one, which is time-consuming and costly, while automatic processing can handle a large amount of data in batches, shortening the data preparation period. At the same time, the preset structured template ensures the uniformity and standardization of the data format, avoiding the format confusion problem in manual conversion.

[0123] In addition, the present application also proposes a data generation device, referring to Figure 7 , the data generation device comprises:

[0124] The case generation module 10 is configured to perform step 1: automatically processing the preset unstructured data to generate structured data; and adjusting the generation strategy of the structured data based on the feedback instruction.

[0125] The dialogue generation module 20 is configured to perform step 2: generating a dialogue sample based on the structured data and a preset role prompt; and adjusting the generation strategy of the dialogue sample based on the feedback instruction.

[0126] The real-time feedback module 30 is configured to perform step 3: evaluating the structured data and the dialogue sample respectively through a preset quality detection rule and generating a feedback instruction.

[0127] a policy optimization module 40 configured to perform step 4: optimizing the generation policy of the structured data and / or the dialogue sample based on the feedback instruction;

[0128] an iteration module 50 configured to perform the steps 1-4 iteratively according to the optimized generation policy of the structured data and / or the dialogue sample until the evaluation of the structured data and the dialogue sample is qualified, and output the optimized first dialogue sample.

[0129] The data generation device provided in the present application adopts the data generation method in the above embodiments, and aims to solve the technical problems of high-quality data scarcity, high labeling cost and difficult guarantee of data quality. Compared with the prior art, the data generation device provided in the present application has the same beneficial effects as the data generation method provided in the above embodiments, and other technical features in the data generation device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0130] The present application provides a data generation device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data generation method in the above embodiment one.

[0131] The data generation device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 8 The data generation device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0132] As Figure 8As shown, the data generation device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. Various programs and data required for operation of the data generation device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch panel, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the data generation device to communicate wirelessly or by wire with other devices to exchange data. Although the data generation device having various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0133] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication apparatus, or installed from the storage apparatus 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0134] The data generation device provided by the present application adopts the data generation method in the above-mentioned embodiments, aiming to solve the technical problems of scarcity of high-quality data, high labeling cost, and difficulty in guaranteeing data quality. Compared with the prior art, the data generation device provided by the present application has the same beneficial effects as the data generation method provided by the above-mentioned embodiments, and other technical features in the data generation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0135] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in one or more embodiments or examples.

[0136] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0137] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the data generation method in the above embodiments.

[0138] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0139] The above computer readable storage medium can be contained in the data generation device, or can exist separately without being assembled into the data generation device.

[0140] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0141] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0142] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0143] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned data generation method, and aims to solve the technical problems of high-quality data scarcity, high labeling cost, and difficult data quality guarantee. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the data generation method provided by the above-mentioned embodiments, and will not be described here.

[0144] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the data generation method as described above.

[0145] The computer program product provided by the application aims to solve the technical problems of scarcity of high-quality data, high labeling cost and difficulty in guaranteeing data quality. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the data generation method provided by the above-mentioned embodiments, which will not be repeated here.

[0146] Compared with the prior art, the data generation method, device, equipment, medium and computer product provided by the application embodiment extract the service feature information of the target service, perform data standardization processing on the service feature information to obtain standard feature data, perform hash processing on the standard feature data to obtain unique feature data, perform numerical processing and splicing processing on the unique feature data to obtain a first service feature value, accumulate the first service feature value of the target service to obtain a target service feature value, and finally compare the target service feature value with a feature value set to obtain a data generation result. Compared with the traditional method of generating a unique key value or a continuous serial number for each service to identify repeated services, the method is more efficient, flexible and reliable. Based on the application scheme, the service in a complex scenario is converted through a series of simple transformations, and finally converted into the comparison of two numbers, so that the comparison process is very intuitive and efficient. The system only needs to simply compare whether the two values are equal, and can quickly judge whether the two services are completely the same.

[0147] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or system that includes the element.

[0148] The above-mentioned application embodiment serial numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0149] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, controlled terminal, or network equipment, etc.) executes the method of each embodiment of the present application.

[0150] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data generating method characterized by comprising: The data generation method comprises: Step 1: automatically processing the preset unstructured data to generate structured data; Step 2: generating a dialogue sample based on the structured data and a preset role prompt; Step 3: evaluating the structured data and the dialogue sample respectively according to a preset quality detection rule and generating a feedback instruction; Step 4: optimizing the generation strategy of the structured data and / or the dialogue sample based on the feedback instruction; iteratively executing the steps 1-4 according to the optimized structured data generation strategy and / or dialogue sample generation strategy until the structured data and the dialogue sample are both qualified, and outputting an optimized first dialogue sample; wherein the step 2 comprises: based on the structured data and a preset role prompt, calling a preset inquiry agent and a reply agent to generate dialogue text in an adversarial manner, which comprises: generating a question text based on the structured data by using the inquiry agent; injecting the preset role prompt into the question text to form an injection text by using the reply agent, and calling a preset model to perform model reasoning based on the injection text, and outputting an individualized answer text; analyzing the semantic correlation degree of the dialogue text, and when detecting that the semantic correlation degree is lower than a preset threshold, triggering the inquiry agent and the reply agent to perform semantic correction; monitoring the dialogue generation process of the inquiry agent and the reply agent, and when identifying that a preset dialogue termination rule is met, outputting a termination signal, and performing standardized format processing on the dialogue text to generate a dialogue sample that can be directly used for model training.

2. The data generating method of claim 1, wherein, The output of the optimized first dialogue sample comprises: Step 5: inputting the first dialogue sample as a training set into a pre-trained model, and performing model training through supervised fine-tuning; Step 6: monitoring the index change during the training process and generating a feedback instruction, and returning to execute step 4 to output an optimized second dialogue sample; supplementing the optimized second dialogue sample to the training set, and iteratively executing the steps 5-6 until the index meets a preset threshold, and outputting an optimized training model.

3. The data generating method of claim 1, wherein, The question text comprises positive samples and negative samples, and the step of generating a question text based on the structured data by using the inquiry agent comprises: extracting at least one semantic element from the structured data, the semantic element comprising an entity, an entity attribute, and an entity relationship; calling a preset question template library, matching a question template corresponding to the semantic element from the preset question template library, and generating a positive sample; using the positive sample as a semantic basis, adjusting the positive sample through a semantic perturbation mechanism to generate a negative sample similar in semantics to the positive sample.

4. The data generating method of claim 3, wherein, The step of using the positive sample as a semantic basis, adjusting the positive sample through a semantic perturbation mechanism to generate a negative sample similar in semantics to the positive sample comprises: extracting the semantic element corresponding to the positive sample, and matching a corresponding perturbation mode from a preset semantic perturbation rule library according to the semantic element; adjusting the positive sample according to the perturbation mode to generate a candidate negative sample; The semantic overlap degree of the candidate negative sample and the positive sample is calculated by a semantic similarity algorithm, and if the semantic overlap degree is within a preset range, the candidate negative sample is determined as a negative sample.

5. The data generating method of claim 1, wherein, The step of automatically processing the preset unstructured data to generate structured data comprises: An analysis algorithm is used to extract information from the preset unstructured data, the extracted information is format-cleaned, and is mapped to a preset structured data template to obtain structured data.

6. A data generating apparatus characterized by comprising: The device comprises: A case generation module is configured to perform step 1: automatically processing the preset unstructured data to generate structured data; and adjusting the generation strategy of the structured data based on a feedback instruction. A dialogue generation module is configured to perform step 2: generating a dialogue sample based on the structured data and a preset role prompt; adjusting the generation strategy of the dialogue sample based on a feedback instruction; the dialogue generation module is further configured to generate dialogue text by calling a preset inquiry agent and a reply agent in an antagonistic manner based on the structured data and the preset role prompt; analyze the semantic correlation degree of the dialogue text, and when detecting that the semantic correlation degree is lower than a preset threshold, trigger the inquiry agent and the reply agent to perform semantic correction; monitor the dialogue generation process of the inquiry agent and the reply agent, and when identifying that a preset dialogue termination rule is met, output a termination signal, and perform standardized format processing on the dialogue text to generate a dialogue sample that can be directly used for model training; the dialogue generation module is further configured to generate a question text based on the structured data by using the inquiry agent; inject the preset role prompt into the question text to form an injection text by the reply agent, and call a preset model based on the injection text to perform model reasoning and output an individualized answer text; A real-time feedback module is configured to perform step 3: evaluating the structured data and the dialogue sample respectively by a preset quality detection rule and generating a feedback instruction; A strategy optimization module is configured to perform step 4: optimizing the generation strategy of the structured data and / or the dialogue sample based on the feedback instruction; An iteration module iteratively performs the steps 1-4 based on the optimized structured data generation strategy and / or dialogue sample generation strategy until the structured data and the dialogue sample are both qualified, and outputs an optimized first dialogue sample.

7. A data generating apparatus characterized by comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the data generation method according to any one of claims 1-5.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data generation method according to any one of claims 1-5.

Images