Text data set generation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a text data set generation method and device, equipment and a storage medium. The method comprises the following steps: acquiring a plurality of first texts from a plurality of data sources of a target scene; performing de-weighting and de-noising processing on the plurality of first texts to obtain a plurality of processed second texts; performing classification processing on the plurality of second texts to obtain a second text set under each category; according to the second text set and the category information under each category, constructing a thinking chain prompt text in a single-example sample form under each category; inputting the thinking chain prompt text under each category into a generative language model to generate a new text so as to obtain a newly generated third text under each category; and content verification is performed on the third text to obtain the target text meeting the text quality requirement, and the text data set is determined according to the second text set and the target text under each category, so that the generation efficiency and the generation quality of the text data set are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to a method, apparatus, device and storage medium for generating text datasets. Background Technology

[0002] In natural language processing research, many fundamental tasks, such as key information extraction and text summarization, heavily rely on high-quality long text datasets. Currently, the necessary text datasets are typically obtained using open-source datasets or through manual collection. However, most open-source datasets often exhibit varying degrees of long-tail distribution, with a large portion of samples concentrated in popular domains, while data in less popular domains is insufficient. This leads to biased performance in certain domains by the model, affecting training effectiveness. Manually collected datasets, on the other hand, show a direct correlation between quality and cost, and are time-consuming, reducing the efficiency of text dataset generation. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and storage medium for generating text datasets, thereby rapidly generating high-quality text datasets and improving the efficiency and quality of text dataset generation.

[0004] In a first aspect, embodiments of the present invention provide a method for generating a text dataset, including:

[0005] Retrieve multiple first texts from multiple data sources in the target scenario;

[0006] The plurality of first texts are subjected to deduplication and noise reduction processing to obtain a plurality of processed second texts;

[0007] The plurality of second texts are classified to obtain a set of second texts under each category;

[0008] Based on the second text set and category information under each category, construct mind chain prompt text in the form of a single example sample for each category;

[0009] The thought chain prompt text under each category is input into the generative language model to generate new text, so as to obtain the newly generated third text under each category;

[0010] The third text is subjected to content validation to obtain target text that meets the text quality requirements, and the text dataset is determined based on the second text set under each category and the target text.

[0011] Secondly, embodiments of the present invention also provide a text dataset generation apparatus, comprising:

[0012] The first text acquisition module is used to acquire multiple first texts from multiple data sources in the target scene;

[0013] The first text processing module is used to perform deduplication and noise reduction processing on the plurality of first texts to obtain a plurality of processed second texts.

[0014] The second text classification module is used to classify the plurality of second texts to obtain a set of second texts under each category;

[0015] The prompt text construction module is used to construct mind chain prompt text in the form of a single example sample for each category, based on the second text set under each category and the category information;

[0016] The third text generation module is used to input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category;

[0017] The third text verification module is used to perform content verification on the third text to obtain target text that meets the text quality requirements, and to determine the text dataset based on the second text set under each category and the target text.

[0018] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text dataset generation method provided in any embodiment of the present invention.

[0022] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the text dataset generation method provided in any embodiment of the present invention.

[0023] One embodiment of the above invention has the following advantages or beneficial effects:

[0024] By acquiring multiple initial first texts from multiple data sources in the target scenario and performing deduplication and noise reduction on these first texts, multiple second texts with no data source differences, regular content, and no duplicate texts are obtained. All second texts are then classified to obtain a set of second texts for each category. Based on the set of second texts and category information for each category, a single-sample thought chain prompt text is constructed for each category. This thought chain prompt text is then input into a generative language model to generate new text, ensuring the logical relevance between the contexts of the newly generated third texts. Finally, the third texts undergo content validation to obtain target texts that meet the text quality requirements. Based on the set of second texts for each category and the high-quality target texts, a text dataset is determined, thus rapidly generating a high-quality text dataset with rigorous context content and uniform category distribution, improving the generation efficiency and quality of the text dataset.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a text dataset generation method provided in one embodiment of the present invention;

[0028] Figure 2 This is an example diagram illustrating a text processing and classification process according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart of a text dataset generation method provided in one embodiment of the present invention;

[0030] Figure 4 This is a flowchart of a text dataset generation method provided in one embodiment of the present invention;

[0031] Figure 5 This is an example diagram of the network architecture of a keyword extraction model provided in one embodiment of the present invention;

[0032] Figure 6 This is a flowchart of a text dataset generation method provided in one embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of a text dataset generation device provided in one embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the text dataset generation method of this invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Figure 1 This is a flowchart illustrating a text dataset generation method according to an embodiment of the present invention. This embodiment is applicable to generating high-quality long text datasets. The method can be executed by a text dataset generation device, which can be implemented in software and / or hardware and can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0038] S110. Obtain multiple first texts from multiple data sources in the target scenario.

[0039] In this context, the target scenario refers to the business scenario where text data needs to be generated. For example, the target scenario can be, but is not limited to, a news scenario. The data source refers to the origin of the text data in the target scenario. The first text refers to the initial text directly collected from the data source. In this embodiment, the first text obtained from the data source can be long text, meaning the length of the first text is greater than a preset length, in order to construct a long text dataset. For example, in a news scenario, diverse first texts can be collected from multiple news websites, such as filtering the content of each website using regular expressions to obtain the news body and using it as the first text. During the first text collection process, there is no need to consider the text category; text collection can be based solely on popularity and time to obtain a preset number of first texts.

[0040] S120. Perform deduplication and noise reduction on multiple first texts to obtain multiple processed second texts.

[0041] Specifically, the same or similar initial text may be collected from different data sources, such as repeated reports or citations of the same event from different news websites. Therefore, it is necessary to deduplicate the collected initial text to obtain initial texts with distinct content, thus ensuring the diversity of the generated text. Since subsequent processing focuses solely on the text content, it is necessary to shield against external influences from different data sources. Furthermore, the deduplicated initial text may contain noise or incomplete content, requiring denoising processing to obtain a second text that is free from data source differences, has well-structured content, and is free of repetition, thereby ensuring the quality of subsequent text generation.

[0042] For example, step S120 may include: determining and deleting duplicate first texts based on the similarity between each pair of first texts, and re-acquiring first texts from multiple data sources in the target scenario until a preset number of first texts that are not similar to each other are obtained; and performing replacement processing on each deduplicated first text with special symbols, user information and data source identification information to obtain the processed second text.

[0043] Specifically, see Figure 2, after obtaining multiple first texts through the tweet collection module, duplicate removal processing of the first texts can be performed through the SimHash algorithm. For example, each first text is tokenized to obtain a word set corresponding to each first text, and a hash operation is performed on the word set corresponding to each first text to obtain a target hash value corresponding to each first text. For example, all words in the first text are encrypted to generate a 128-bit binary code, and only the last 16 bits are used as the target hash value of the first text. According to the target hash values corresponding to every two first texts, the similarity between every two first texts is determined. The similarity can use the Hamming distance, that is, the number of operations required for a string to become another string through replacement, deletion, and addition is denoted as the Hamming distance. Since it is represented by binary numbers, the logical operation "exclusive or" is directly used to calculate the Hamming distance, that is, the similarity. The similarity between every two first texts is compared with a similarity threshold (such as 90%), and any one of the two first texts with a similarity greater than the similarity threshold is deleted, thereby deleting duplicate first texts. In order to obtain a sufficient number of first texts and ensure the quality of subsequent text generation, first texts can be continuously obtained from multiple data sources of the target scenario until a preset number of first texts that are not similar to each other are obtained.

[0044] See Figure 2 , replacement processing of special symbols, user information, and data source identification information is performed on each deduplicated first text to obtain a processed second text. For example, special characters are filtered and replaced, such as emojis or text emojis, etc. Such content can be replaced by "escape character + Chinese description", such as "\ / happy". Use "URL" to replace the real url link, and use USER to desensitize user information. For example, the字样 "@ Xiaoming" may appear in some commentary reports. The platform logo and signature, etc. will be cleaned up to shield the external influence of different data sources. Through deduplication and denoising processing, high-quality metadata without platform differentiation, regular content, and no duplicate texts can be obtained, so as to generate new high-quality texts based on the high-quality metadata.

[0045] S130. Classify multiple second texts to obtain a set of second texts under each category.

[0046] It should be noted that the content category of the second text cannot be simply based on the currently visible category as the actual classification basis, but should be analyzed specifically for each case. Therefore, an unsupervised clustering algorithm can be used to perform unsupervised classification processing on all second texts to obtain classification results under all categories, thereby ensuring the accuracy of the classification results.

[0047] Specifically, see Figure 2After obtaining the data with the specified content, each second text is input into the encoding model for text encoding, obtaining a text vector corresponding to each second text. The encoding model can be a Longformer model suitable for encoding long texts. The Longformer model is an encoder-only structure with an encoder-only architecture, which, compared to the Transformer model using a full attention mechanism, requires O(n^2) time. 2 With a time complexity of approximately 1 / 2000, the Longformer model introduces a sparse attention mechanism, significantly accelerating the encoding process. Furthermore, due to its extended positional encoding, the Longformer model is better suited than the Transformer model for encoding long texts. Its working principle is described as V = Encoder(POS⊕([CLS]+W)), where V represents the sentence embedding vector, Encoder represents the pre-trained encoding model (here, the Longformer model), W is the second text, [CLS] marks the beginning of each sentence, POS is the absolute position sequence corresponding to W starting from "0", "⊕" indicates parallel input, and "+" indicates a horizontal concatenation operation.

[0048] See also Figure 2After obtaining the text vector corresponding to each second text (i.e., the vector database), the efficient data dimensionality reduction method UMATO (Uniform Manifold Approximation with Two-phase Optimization) can be used to reduce the dimensionality of each second text's corresponding text vector, obtaining the dimensionality-reduced target text vector. The UMATO dimensionality reduction method first constructs a global skeleton and then optimizes the local representation; this two-stage optimization results in higher computational efficiency and stronger robustness, making it suitable for scenarios that require simultaneous preservation of global and local structures. Finally, the HBSCAN (Hierarchical Desity-Based Spatial Clustering of Applications with Noise) algorithm is used to cluster based on the target text vector corresponding to each second text. HBSCAN is a density-based hierarchical clustering algorithm that can effectively identify complex data structures, adapt to clusters of different densities, and automatically identify and discard noise points. This algorithm constructs a minimum spanning tree based on data point density, starting from leaf nodes, merging child nodes according to different densities and distances, and finally generating hierarchical clustering results. For example, common news can be categorized into 15 types based on content: politics, economy, society, international affairs, science and technology, culture, sports, entertainment, education, health, finance, military, environment, local news, and in-depth reports. However, with social development, news categories are becoming increasingly diverse. Through unsupervised clustering algorithms, all secondary texts can ultimately be divided into 20 categories, obtaining a set of secondary texts for each category.

[0049] S140. Based on the second text set and category information under each category, construct the mind chain prompt text in the form of a single example sample under each category.

[0050] One-shot, or single-example, refers to guiding a generative language model to complete a task by providing only one example. This approach helps the model understand task requirements by showcasing concrete cases and is suitable for scenarios requiring explicit formatting or complex logic. Mind chains, as an improved prompting method, can guide the model to break down complex problems into step-by-step sub-problems and solve them sequentially when dealing with reasoning and multi-step thinking problems, thus significantly improving model performance and expressiveness. Compared to simply combining external knowledge with simple generated prompt text, mind chain-based generation obviously better ensures the stability of generated content, especially for long texts. Mind chain-based generation ensures logical relevance between contexts and avoids issues like "topic drift" and irrelevance due to excessive length.

[0051] Specifically, for each category after division, a thought chain prompt text in the form of a single example sample is constructed based on the second text set under that category and the category information. This thought chain prompt text is used to guide the generative language model step by step to generate high-quality new text under that category.

[0052] S150. Input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category.

[0053] The generative language model can be a pre-trained model used to generate text. For example, it can be a large language model. Specifically, based on the number of third texts to be generated in each category, the thought chain prompt text for each category is input multiple times into the generative language model to generate new text, thus obtaining multiple newly generated third texts for each category. For example, the thought chain prompt text for a certain category is repeatedly input into the generative language model 5 times to obtain 5 newly generated third texts for that category. The number of third texts to be generated in each category can be set based on actual business needs, thus ensuring a uniform distribution of text data in each category. By supplementing the sparse number of second texts, the content category distribution of the generated dataset is made as uniform as possible, avoiding a long-tailed distribution.

[0054] S160. Perform content validation on the third text to obtain target text that meets the text quality requirements, and determine the text dataset based on the second text set and target text under each category.

[0055] It should be noted that when generative language models generate new text, inaccurate output, logical errors, or even irrelevant content may occur due to insufficient contextual understanding or randomness. In order to improve text quality, the third-party text can be validated and iteratively processed. Through multiple validations or modifications, the target text that meets the text quality requirements can be obtained.

[0056] Specifically, if the generative language model is used to validate the text content it generates, even more excessive bias may occur. To address this, this embodiment employs downstream task validation and / or manual evaluation to positively influence the output of the generative language model, dynamically iterating on weaknesses in the generated dataset to gradually improve the quality of the model's output text, thereby ensuring the quality of generated text. After obtaining target text that meets the quality requirements, the second text set under each category is combined with the target text to obtain the final high-quality text for each category. This allows the construction of a high-quality text dataset with rigorous contextual content and even category distribution. If the obtained first text is a long text, the final constructed text dataset is a long text dataset.

[0057] For example, step S160, "performing content verification on the third text to obtain target text that meets text quality requirements," includes at least one of the following:

[0058] The downstream task model is trained and its performance is tested based on the second and third texts under each category. Based on the performance test results, the third texts that do not meet the text quality requirements are iteratively optimized to obtain the target texts that meet the text quality requirements under each category.

[0059] The quality of the third text under each category is evaluated, and based on the quality evaluation results, the third text that does not meet the text quality requirements is iteratively optimized to obtain the target text that meets the text quality requirements under each category.

[0060] As the first implementation method, the third text can be verified and iteratively optimized by training and testing the downstream task model, thereby obtaining the target text that meets the text quality requirements under each category.

[0061] As a second approach, the quality of the third text in each category can be assessed manually, and the content of the third text can be verified and iteratively optimized based on the quality assessment results, thereby obtaining the target text that meets the text quality requirements in each category.

[0062] As a third approach, a combination of downstream task model training and performance testing methods, along with manual quality assessment, can be used to perform content verification and iterative optimization on the third-party text, thereby obtaining higher-quality target text. For example, the third-party text can first be verified and iteratively optimized through downstream task model training and performance testing, ensuring that the third-party text in each category meets the text quality requirements. Then, manual quality assessment can be performed on the third-party text in each category, and content verification and iterative optimization can be performed based on the assessment results, resulting in higher-quality target text for each category. By combining the advantages of both content verification strategies, a higher-quality text dataset can ultimately be generated.

[0063] The technical solution of this embodiment obtains multiple initial first texts from multiple data sources in the target scene, and performs deduplication and noise reduction processing on these first texts to obtain multiple second texts with no data source differences, regular content, and no duplicate texts. All second texts are classified to obtain a set of second texts under each category. Based on the set of second texts under each category and category information, a single-example sample form of thought chain prompt text is constructed for each category. The thought chain prompt text for each category is input into a generative language model to generate new text, ensuring the logical relevance between the contexts of the newly generated third texts. Finally, the third texts are content-validated to obtain target texts that meet the text quality requirements. The text dataset is determined based on the set of second texts under each category and the high-quality target texts, thereby quickly generating a high-quality text dataset with rigorous context content and uniform category distribution, improving the generation efficiency and quality of the text dataset.

[0064] Based on the above technical solutions, step S130 may include: performing primary classification processing on multiple second texts to obtain a set of second texts under each primary category, and performing secondary classification processing on each set of second texts to obtain a set of second texts under each secondary category.

[0065] It's important to note that since subsequent thought chain prompts and new text generation are based on categories, the writing style and structural design must be consistent across all categories. This ensures that the same thought chain prompts can be used to generate new text within the same category. Therefore, if the writing style and structural design of the second text within a first-level category are inconsistent, the second text set under each first-level category needs to be further subdivided into second-level categories, and so on, until the writing style and structural design of the final-level category are consistent.

[0066] Specifically, it can be done according to Figure 2The clustering method described in the paper involves unsupervised first-level classification of all second texts to obtain a set of second texts under each first-level category. Then, supervised or unsupervised second-level classification is performed on each set of second texts to obtain a set of second texts under each second-level category. For example, in a news scenario, the text genres under each first-level category can be divided into five categories: news, feature, interview, communication, and analysis. These five genres have different writing styles and structural designs, making it impossible to use the same sample. Therefore, the second text sets under each first-level category are classified into second-level categories according to genre, resulting in a set of second texts under each genre. This leads to all second texts being divided into 20 first-level categories and 5 (maximum) second-level categories. This data provides sufficient support for subsequent text processing.

[0067] Accordingly, step S140 may include: constructing a mind chain prompt text in the form of a single example sample for each second-level category based on the second text set and the second-level category information for each second-level category.

[0068] Specifically, for each of the subdivided secondary categories, a thought chain prompt text in the form of a single example sample is constructed based on the second text set under that secondary category and the category information. This thought chain prompt text is used to guide the generative language model step by step to generate high-quality new text under that secondary category.

[0069] Accordingly, step S150 may include: inputting the thought chain prompt text under each second-level category into the generative language model to generate new text, so as to obtain the newly generated third text under each second-level category.

[0070] Specifically, generative language models generate new text by using second-level categories as units, thereby obtaining third-level texts under more detailed categories, further ensuring the quality of text generation.

[0071] Accordingly, step S160 may include: performing content verification on the third text to obtain target text that meets the text quality requirements, and determining the text dataset based on the second text set under each secondary category and the target text.

[0072] Specifically, each newly generated third text is validated through downstream task verification and / or manual evaluation. This allows for dynamic iteration to identify weaknesses in the generated dataset, gradually improving the quality of the model's output text and ensuring high-quality text generation. After obtaining target texts that meet the quality requirements, the second text sets under each secondary category are combined with the target texts to obtain the final high-quality texts for each secondary category. This constructs a high-quality text dataset with rigorous contextual content and a uniform category distribution. If the obtained first texts are long texts, the final constructed text dataset is a long text dataset; for example, a high-quality long news text dataset can be constructed using the above method.

[0073] Figure 3 This is a flowchart illustrating a text dataset generation method according to one embodiment of the present invention. Based on the aforementioned embodiments, this embodiment provides a detailed description of the process for constructing thought chain prompt text. Explanations of terms that are the same as or corresponding to those in the aforementioned embodiments are not repeated here.

[0074] See Figure 3 The text dataset generation method provided in this embodiment specifically includes the following steps:

[0075] S310. Obtain multiple first texts from multiple data sources in the target scenario.

[0076] S320. Perform deduplication and noise reduction on multiple first texts to obtain multiple processed second texts.

[0077] S330. Classify multiple second texts to obtain a set of second texts under each category.

[0078] S340. Obtain the initial text of the thought chain used to describe the text generation requirements under each category.

[0079] Specifically, the writing steps can be constrained in advance based on the uniqueness of each category and the text generation requirements to construct the initial text of the thought chain under each category. For example, in a news scenario, the writing steps can be constrained based on the uniqueness of each news genre to obtain the initial text of the thought chain required for text generation for each genre. For example, Table 1 shows the initial text of the thought chain under each genre.

[0080] Table 1 Initial Text for the Mind Chain

[0081]

[0082]

[0083] S350. For the second text set under each category, the initial text of the thought chain under that category and any second text in the second text set under that category are embedded using Lasswell's 5W model in communication studies to construct a thought chain prompt text in the form of a single example sample under that category.

[0084] Among them, Lasswell's 5W model is a highly purposeful communication model proposed by Lasswell in 1948, namely Who, through which channel, to whom, what was said, and with what effects.

[0085] It should be noted that when the text to be generated is not a simple stack of words, but a type of text with a strong purpose, such as news text, constructing a thought chain prompt text in the form of a single example sample using Lasswell's 5W model in communication studies can maximize the utilization of the language model's performance while imposing certain restrictions on the generation format, thereby improving the quality of the generated text.

[0086] Specifically, for the second text set under each category, the initial text of the thought chain under that category can be embedded as "Say What", and any second text in the second text set under that category can be used as the response content, thus forming a two-layer nested baseline thought chain paradigm and obtaining thought chain prompt texts that are more detailed and clear.

[0087] For example, a mind chain prompt text with the genre "message" would be:

[0088] Q: As a journalist, you will report news events to the general public and ultimately submit the articles to a media website. You are required to objectively state the basic facts, avoiding deliberate exaggeration, distortion, or biased language, and ensuring truthfulness, objectivity, privacy and personal rights, and accurate and standardized language. The news genre is "news," and the content requirements are:

[0089] #Define the theme: Identify the core event and extract key information;

[0090] #Element Analysis: Organize information according to the 5W1H (When, Where, Who, What Happened, Why, What Was the Result) logical order;

[0091] #Objective Presentation: State the facts in concise and objective language, avoiding subjective evaluation;

[0092] #Structure Optimization: Adopt an inverted pyramid structure, arranging information in descending order of importance to ensure that core information is conveyed first;

[0093] #Timeliness Emphasis: Highlight the latest developments in the event to reflect the timeliness of the news.

[0094] A: XXX (This is a secondary text content under the "Message" category)

[0095] Q: As a news editor, I report on the latest inventions in the technology industry and ultimately contribute to independent technology websites. I require that the basic facts be presented objectively, avoiding exaggeration, distortion, or biased writing, and ensuring truthfulness, objectivity, privacy and personal rights, and accurate and standardized language. The news genre is "news report."

[0096] In the above thought chain prompts, all content marked with "#" can be replaced based on factors such as "report subject," "news audience," "dissemination channels," "narrative effect," "news genre," and "content logic." This one-shot paradigm of thought chain guidance uses a "question-and-answer case + thought chain question" prompt format, which maximizes the language model's strong generalization and generative capabilities. Furthermore, due to the constraints of the example text, the language model will maintain consistency with similar news in terms of the length and narrative style of the generated content.

[0097] S360. Input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category.

[0098] S370. Perform content validation on the third text to obtain target text that meets the text quality requirements, and determine the text dataset based on the second text set and target text under each category.

[0099] The technical solution of this embodiment obtains the initial text of the thought chain describing the text generation requirements under each category, and embeds the initial text of the thought chain under that category and any second text in the second text set under that category using the Lasswell 5W model in communication studies. This forms a two-layer nested baseline thought chain paradigm, thereby maximizing the utilization of the language model's performance while imposing certain restrictions on the generation format, further improving the quality of text generation.

[0100] Figure 4 This is a flowchart of a text dataset generation method according to one embodiment of the present invention. Based on the above embodiments, this embodiment optimizes the step of "training and performing performance testing on the downstream task model based on the second and third texts under each category, and iteratively optimizing the third texts that do not meet the text quality requirements based on the performance test results, so as to obtain the target texts that meet the text quality requirements under each category". The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0101] See Figure 4 The text dataset generation method provided in this embodiment specifically includes the following steps:

[0102] S410. Obtain multiple first texts from multiple data sources in the target scenario.

[0103] S420. Perform deduplication and noise reduction on multiple first texts to obtain multiple processed second texts.

[0104] S430. Classify multiple second texts to obtain a set of second texts under each category.

[0105] S440. Based on the second text set and category information under each category, construct the mind chain prompt text in the form of a single example sample under each category.

[0106] S450. Input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category.

[0107] S460. Perform keyword tagging on the second and third texts under each category to obtain the keyword tags corresponding to the second text and the keyword tags corresponding to the third text under each category.

[0108] It should be noted that directly using the generated text dataset for training downstream tasks can quickly verify the overall quality of the text dataset. In this embodiment, a keyword extraction model is used to perform a supervised keyword extraction task, and the quality of the text dataset is evaluated based on the model's performance.

[0109] In order to conduct supervised training and performance testing of the keyword extraction model using the original second text and the newly generated third text, it is necessary to annotate all the second text and all the third text with keywords to obtain the keyword labels corresponding to each second text and each third text.

[0110] Specifically, prompt text can be constructed across multiple prompt dimensions. For example, prompt dimensions can include, but are not limited to, descriptive, question-and-answer, and summary types. The prompt text for descriptive prompts would be: "What keywords does this article mainly discuss? Please list no more than twenty." The prompt text for question-and-answer prompts would be: "If you were to summarize the core content of this article using no more than twenty keywords, which ones should you choose?" The prompt text for summary prompts would be: "Please extract the domain-specific terms and key entities from this article."

[0111] The prompt text for each prompt dimension and each second text are input into a generative language model for keyword extraction, obtaining a keyword set for each second text under each prompt dimension. Through a class-based voting mechanism, at least two sets of cross-keywords generated under different prompt dimensions are retained, and these cross-keyword sets are used as keyword labels for the second text, thus ensuring the accuracy of the second text annotation. Similarly, the prompt text for each prompt dimension and each third text are input into a generative language model for keyword extraction, obtaining a keyword set for each third text under each prompt dimension. Through a class-based voting mechanism, at least two sets of cross-keywords generated under different prompt dimensions are retained, and these cross-keyword sets are used as keyword labels for the third text, thus ensuring the accuracy of the third text annotation.

[0112] Since the quality of the newly generated third text is being verified, while the second text is the collected real data, it is necessary to use the labeled third text for model training and the labeled second text for performance testing of the trained model, so that the sample quality of the third text can be reflected through the model performance.

[0113] S470. Train the keyword extraction model based on the third text and the keyword tags corresponding to the third text, and perform performance testing on the trained keyword extraction model based on the keyword tags corresponding to the second text and the second text, and obtain the performance test results for each category.

[0114] The keyword extraction model is a neural network model used to extract keywords from input text. In this embodiment, the BIO rule can be used to perform temporal labeling on keywords, that is, BK, IK, and O are used to represent the beginning of the keyword, the middle part of the keyword, and a single character, respectively. Figure 5 A network architecture for a keyword extraction model is presented, such as... Figure 5As shown, the keyword extraction model includes an input layer, word segmentation, embedding layer, fully connected network, and annotation / decoding layer. To handle long text embeddings, the embedding layer can use the Longformer architecture, for example, fine-tuning and embedding using the "longformer-large-4096" model, which can handle embeddings of ultra-long input sequences exceeding 4096 tokens. For word segmentation, Longformer embedding typically uses words rather than individual characters as the smallest processing unit. However, compared to the natural advantage of English using spaces for segmentation, Chinese characters can be segmented character-by-character using the BPE (Byte Pair Encoding) strategy. The fully connected Dense network is a basic structure in neural networks, characterized by each neuron being connected to all neurons in the previous layer. This fully connected nature allows Dense layers to perform complex transformations and feature extraction on the input data. Figure 5 The keyword extraction model can use a Dense network to transform the output of the embedding layer, generating a vector representation that can be used for temporal annotation decoding, and then inputting it into the annotation decoding layer (such as a Conditional Random Field, CRF) for decoding output.

[0115] Specifically, multiple training iterations are performed using all third-party texts and their corresponding keyword tags, with the number of training epochs potentially set to 20. During each training epoch, all third-party texts are iteratively learned. Simultaneously, negative log-likelihood loss is employed as the loss function during training, aiming to minimize the difference between the model's predicted labeled sequences and the true labeled sequences.

[0116] For example, the loss function is: Where N is the number of training samples, p(y) i |x i ;θ) represents the conditional probability of the labeled sequence y given the input sequence x and the model parameters θ, and yi and xi represent the true labeled sequence and the model output sequence of the i-th sample, respectively.

[0117] After training the keyword extraction model using all third-party texts and corresponding keyword tags, the trained model is then subjected to a category-specific performance test based on the second-party text set and corresponding keyword tags for each category. The performance test results for each category are then obtained. For example, for each category, based on the formula... and Based on the number of correctly identified positive samples (TP) and the number of incorrectly identified negative samples (FN), the precision (P) and recall (R) of keyword extraction for this category are determined. Specifically, for each correctly extracted keyword, the TP value is incremented by 1; conversely, for each non-keyword character identified as a keyword, the FN value is incremented by 1. (Based on the formula...) Based on the precision P and recall R of keyword extraction in this category, determine the performance metric F for this category.

[0118] S480. Based on the performance test results, determine the first category that does not meet the text quality requirements, optimize the thought chain prompt text under the first category, and input the optimized thought chain prompt text into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

[0119] Specifically, the performance metrics for each category are compared with a threshold. Categories with metrics below the threshold are identified as the first category, failing to meet the text quality requirements. In other words, the poor quality of the third text in the first category leads to poor performance of the trained model on that category, necessitating the regeneration of the third text in the first category. To generate higher-quality third text in the first category, the thought chain prompt text in the first category needs to be optimized. For example, the text generation requirements for the first category can be incorporated into the original thought chain prompt text. For instance, if the first category is the "message" genre, the optimized thought chain prompt text would be:

[0120] (The thought process prompt text described above under the "message" genre)

[0121] #Currently generating news text in the **message** genre.

[0122] #Pay special attention to consistency in terms of thematic relevance.

[0123] #Theme Alignment: The entire article strictly requires that "the generated content closely adheres to the theme and develops related content around the theme."

[0124] By inputting the optimized thought chain prompt text into the generative language model to regenerate new text, the third text under the first category is obtained. Then, the above steps S470 and S480 are repeated until the performance index under each category is greater than or equal to the index threshold. That is, the third text under each category meets the text quality requirements. At this point, the third text that meets the text quality requirements is determined as the final target text, thus ensuring the quality of text generation.

[0125] S490. Determine the text dataset based on the second text set and the target text under each category.

[0126] The technical solution of this embodiment trains a keyword extraction model based on a third text and its corresponding keyword tags, and performs performance testing on the trained keyword extraction model based on the second text and its corresponding keyword tags to obtain performance test results for each category. Based on the performance test results, a first category that does not meet the text quality requirements is identified, and the thought chain prompt text under the first category is optimized. The optimized thought chain prompt text is then input into a generative language model to regenerate new text until a target text that meets the text quality requirements is obtained. Thus, the keyword extraction task can quickly and accurately identify the weaknesses of the generated dataset and iteratively fix them, improving the model's generalization ability and further ensuring the quality of text generation.

[0127] Figure 6 This is a flowchart of a text dataset generation method according to an embodiment of the present invention. Based on the above embodiments, this embodiment optimizes the step "to perform quality assessment on the third text under each category, and to iteratively optimize the third text that does not meet the text quality requirements based on the quality assessment results, so as to obtain the target text that meets the text quality requirements under each category". The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0128] See Figure 6 The text dataset generation method provided in this embodiment specifically includes the following steps:

[0129] S610: Obtain multiple first texts from multiple data sources in the target scenario.

[0130] S620. Perform deduplication and noise reduction on multiple first texts to obtain multiple processed second texts.

[0131] S630. Classify multiple second texts to obtain a set of second texts under each category.

[0132] S640. Based on the second text set and category information under each category, construct the mind chain prompt text in the form of a single example sample under each category.

[0133] S650. Input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category.

[0134] S660. Obtain the quality assessment results for the third text under each category. The quality assessment results are obtained from four aspects: overall fluency of the text, thematic relevance, logical rigor, and content consistency.

[0135] Specifically, this embodiment employs a scoring system for the evaluation and iteration of third-party texts. For example, due to the large number of third-party texts available for manual quality assessment, a complete sampling and inspection is not suitable. Therefore, a small number of third-party texts can be randomly selected from all texts in each category for quality assessment, providing a comprehensive overview and quickly evaluating the overall performance of the texts. Each selected third-party text is scored based on four aspects: overall fluency, thematic relevance, logical rigor, and content consistency. The scoring criteria for each aspect are shown in Table 2 below.

[0136] Table 2 Scoring Criteria and Weighting Details

[0137]

[0138]

[0139] S670. Based on the quality assessment results, identify the fourth text that does not meet the text quality requirements and the second category to which the fourth text belongs from the third text.

[0140] Specifically, the evaluation score of the third text is compared with the preset score. The third text with a score lower than the preset score is identified as the fourth text that does not meet the text quality requirements. The category to which the fourth text belongs is designated as the second category that does not meet the text quality requirements, so that a new text under the second category can be regenerated.

[0141] S680. Based on the quality assessment results of the fourth text, optimize the thought chain prompt text under the second category, and re-input the optimized thought chain prompt text into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

[0142] Specifically, based on the quality assessment results of the fourth text, a specific score for each of the above aspects can be obtained, thereby identifying in which aspects the fourth text performs poorly. For example, the theme relevance of long-form educational reports is low. Therefore, when optimizing the thought chain prompt texts under the second category, the theme relevance requirement can be added to the thought chain prompt texts under the second category. That is, the optimized thought chain prompt texts can be:

[0143] (The thought chain prompt text described above under the second category)

[0144] #Focus on consistency in terms of thematic relevance.

[0145] #Theme Alignment: The entire article strictly requires that "the generated content closely adheres to the theme and develops related content around the theme."

[0146] By inputting the optimized thought chain prompt text into the generative language model to regenerate new text, the regenerated third text under the second category is obtained. Then, the above steps S660-S680 are repeated until the evaluation scores of the third texts extracted under the second category are all greater than or equal to the preset scores. At this point, it is determined that the third texts under the second category all meet the text quality requirements, and the third texts that meet the text quality requirements are determined as the final target text, thereby ensuring the quality of text generation.

[0147] S690. Determine the text dataset based on the second text set and the target text under each category.

[0148] The technical solution of this embodiment obtains the quality assessment results of the third text under each category. The quality assessment results are obtained from four aspects: overall fluency, topic relevance, logical rigor, and content consistency. Based on the quality assessment results, the fourth text that does not meet the text quality requirements and the second category to which the fourth text belongs are determined from the third text. Based on the quality assessment results of the fourth text, the thought chain prompt text under the second category is optimized. The optimized thought chain prompt text is then re-inputted into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained. Thus, sampling and testing are performed according to category and dimension to quickly evaluate the overall text performance, improve the generalization ability of the model, and further ensure the quality of text generation.

[0149] The following are embodiments of the text dataset generation apparatus provided in this invention. This apparatus and the text dataset generation methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the text dataset generation apparatus, please refer to the embodiments of the above text dataset generation methods.

[0150] Figure 7 This is a schematic diagram of a text dataset generation device provided in an embodiment of the present invention. This embodiment is applicable to generating high-quality long text datasets. Figure 7 As shown, the device specifically includes: a first text acquisition module 710, a first text processing module 720, a second text classification module 730, a prompt text construction module 740, a third text generation module 750, and a third text verification module 760.

[0151] The system comprises the following modules: a first text acquisition module 710, used to acquire multiple first texts from multiple data sources in the target scene; a first text processing module 720, used to perform deduplication and noise reduction processing on the multiple first texts to obtain multiple processed second texts; a second text classification module 730, used to classify the multiple second texts to obtain a set of second texts under each category; a prompt text construction module 740, used to construct a single-sample form of thought chain prompt text for each category based on the set of second texts under each category and category information; a third text generation module 750, used to input the thought chain prompt text for each category into a generative language model to generate new text to obtain newly generated third text for each category; and a third text verification module 760, used to perform content verification on the third texts to obtain target texts that meet the text quality requirements, and to determine the text dataset based on the set of second texts under each category and the target texts.

[0152] The technical solution of this embodiment obtains multiple initial first texts from multiple data sources in the target scene, and performs deduplication and noise reduction processing on these first texts to obtain multiple second texts with no data source differences, regular content, and no duplicate texts. All second texts are classified to obtain a set of second texts under each category. Based on the set of second texts under each category and category information, a single-example sample form of thought chain prompt text is constructed for each category. The thought chain prompt text for each category is input into a generative language model to generate new text, ensuring the logical relevance between the contexts of the newly generated third texts. Finally, the third texts are content-validated to obtain target texts that meet the text quality requirements. The text dataset is determined based on the set of second texts under each category and the high-quality target texts, thereby quickly generating a high-quality text dataset with rigorous context content and uniform category distribution, improving the generation efficiency and quality of the text dataset.

[0153] In some optional embodiments, the first text processing module 720 is specifically used for:

[0154] Based on the similarity between any two first texts, duplicate first texts are identified and deleted, and first texts are re-acquired from multiple data sources in the target scenario until a preset number of first texts that are not similar to each other are obtained; each deduplicated first text is processed by replacing special symbols, user information and data source identification information to obtain the processed second text.

[0155] In some optional embodiments, the prompt text construction module 740 is specifically used for

[0156] Obtain the initial text of the thought chain to describe the text generation requirements under each category; for the second text set under each category, embed the initial text of the thought chain under that category and any second text in the second text set under that category using Lasswell's 5W model in communication studies to construct the thought chain prompt text in the form of a single example sample under that category.

[0157] In some optional embodiments, the third text verification module 760 includes at least one of the following:

[0158] The model training and testing submodule is used to train and test the performance of the downstream task model based on the second text and the third text under each category, and to iteratively optimize the third text that does not meet the text quality requirements based on the performance test results, so as to obtain the target text that meets the text quality requirements under each category.

[0159] The quality assessment and optimization submodule is used to perform quality assessment on the third text under each category, and to iteratively optimize the third text that does not meet the text quality requirements based on the quality assessment results, so as to obtain the target text that meets the text quality requirements under each category.

[0160] In some optional embodiments, the model training and testing submodule is specifically used for:

[0161] Keyword tagging is performed on the second text and the third text under each category to obtain the keyword tags corresponding to the second text and the third text under each category;

[0162] The keyword extraction model is trained based on the third text and the keyword tags corresponding to the third text, and the performance of the trained keyword extraction model is tested based on the second text and the keyword tags corresponding to the second text, so as to obtain the performance test results for each category.

[0163] Based on the performance test results, a first category that does not meet the text quality requirements is identified, and the thought chain prompt text under the first category is optimized. The optimized thought chain prompt text is then input into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

[0164] In some optional embodiments, the quality assessment optimization submodule is specifically used for:

[0165] Obtain the quality assessment results corresponding to the third text under each category. The quality assessment results are obtained from four dimensions: overall fluency of the text, thematic relevance, logical rigor, and content consistency.

[0166] Based on the quality assessment results, a fourth text that does not meet the text quality requirements and the second category to which the fourth text belongs are determined from the third text;

[0167] Based on the quality assessment results of the fourth text, the thought chain prompt text under the second category is optimized, and the optimized thought chain prompt text is re-input into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

[0168] In some optional embodiments, the second text classification module 730 is specifically used for:

[0169] The plurality of second texts are subjected to first-level classification processing to obtain a set of second texts under each first-level category, and each set of second texts is subjected to second-level classification processing to obtain a set of second texts under each second-level category;

[0170] The prompt text construction module 740 is specifically used to: construct the mind chain prompt text in the form of a single example sample for each second-level category based on the second text set and the second-level category information under each second-level category;

[0171] The third text generation module 750 is specifically used to: input the thought chain prompt text under each second-level category into the generative language model to generate new text, so as to obtain the newly generated third text under each second-level category.

[0172] The text dataset generation apparatus provided in this embodiment of the invention can execute the text dataset generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the text dataset generation method.

[0173] It is worth noting that in the embodiments of the above-mentioned text dataset generation device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0174] Figure 8A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0175] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0176] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various text dataset generation methods and processes described above.

[0178] In some embodiments, the text dataset generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the text dataset generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the text dataset generation method by any other suitable means (e.g., by means of firmware).

[0179] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] Computer programs for implementing the text dataset generation method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0183] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0184] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0185] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0186] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating a text dataset, characterized in that, include: Retrieve multiple first texts from multiple data sources in the target scenario; The plurality of first texts are subjected to deduplication and noise reduction processing to obtain a plurality of processed second texts; The plurality of second texts are classified to obtain a set of second texts under each category; Based on the second text set and category information under each category, construct mind chain prompt text in the form of a single example sample for each category; The thought chain prompt text under each category is input into the generative language model to generate new text, so as to obtain the newly generated third text under each category; The third text is subjected to content validation to obtain target text that meets the text quality requirements, and the text dataset is determined based on the second text set under each category and the target text.

2. The method according to claim 1, characterized in that, The plurality of first texts are subjected to deduplication and noise reduction processing to obtain a plurality of processed second texts, including: Based on the similarity between any two first texts, duplicate first texts are identified and removed, and first texts are re-acquired from multiple data sources in the target scene until a preset number of first texts that are not similar to each other are obtained. Each deduplicated first text is processed by replacing special symbols, user information, and data source identification information to obtain the processed second text.

3. The method according to claim 1, characterized in that, Based on the second text set and category information under each category, construct mind chain prompt text in the form of a single-example sample for each category, including: Obtain the initial text of the thought chain used to describe the text generation requirements under each category; For each category's second text set, the initial text of the thought chain in that category and any second text in the second text set of that category are embedded using Lasswell's 5W model in communication studies to construct a thought chain prompt text in the form of a single example sample for that category.

4. The method according to claim 1, characterized in that, The third text is subjected to content validation to obtain target text that meets the text quality requirements, including at least one of the following: The downstream task model is trained and its performance is tested based on the second and third texts under each category. Based on the performance test results, the third texts that do not meet the text quality requirements are iteratively optimized to obtain the target texts that meet the text quality requirements under each category. The quality of the third text under each category is evaluated, and based on the quality evaluation results, the third text that does not meet the text quality requirements is iteratively optimized to obtain the target text that meets the text quality requirements under each category.

5. The method according to claim 4, characterized in that, The downstream task model is trained and its performance tested based on the second and third texts under each category. Based on the performance test results, the third texts that do not meet the text quality requirements are iteratively optimized to obtain target texts that meet the text quality requirements under each category, including: Keyword tagging is performed on the second text and the third text under each category to obtain the keyword tags corresponding to the second text and the third text under each category; The keyword extraction model is trained based on the third text and the keyword tags corresponding to the third text, and the performance of the trained keyword extraction model is tested based on the second text and the keyword tags corresponding to the second text, so as to obtain the performance test results for each category. Based on the performance test results, a first category that does not meet the text quality requirements is identified, and the thought chain prompt text under the first category is optimized. The optimized thought chain prompt text is then input into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

6. The method according to claim 4, characterized in that, The quality of the third text under each category is evaluated, and based on the quality evaluation results, the third text that does not meet the text quality requirements is iteratively optimized to obtain the target text that meets the text quality requirements under each category, including: Obtain the quality assessment results corresponding to the third text under each category. The quality assessment results are obtained from four dimensions: overall fluency of the text, thematic relevance, logical rigor, and content consistency. Based on the quality assessment results, a fourth text that does not meet the text quality requirements and the second category to which the fourth text belongs are determined from the third text; Based on the quality assessment results of the fourth text, the thought chain prompt text under the second category is optimized, and the optimized thought chain prompt text is re-input into the generative language model to regenerate new text until the target text that meets the text quality requirements is obtained.

7. The method according to any one of claims 1-6, characterized in that, The classification process for the plurality of second texts to obtain a set of second texts under each category includes: The plurality of second texts are subjected to first-level classification processing to obtain a set of second texts under each first-level category, and each set of second texts is subjected to third-level classification processing to obtain a set of second texts under each second-level category. The step of constructing mind chain prompt text in the form of a single example sample for each category based on the second text set and category information for each category includes: Based on the second text set and second-level category information under each second-level category, construct the mind chain prompt text in the form of a single example sample under each second-level category; The step of inputting the thought chain prompt text under each category into a generative language model to generate new text, thereby obtaining the newly generated third text under each category, includes: The thought chain prompt text under each second-level category is input into the generative language model to generate new text, so as to obtain the newly generated third text under each second-level category.

8. A text dataset generation device, characterized in that, include: The first text acquisition module is used to acquire multiple first texts from multiple data sources in the target scene; The first text processing module is used to perform deduplication and noise reduction processing on the plurality of first texts to obtain a plurality of processed second texts. The second text classification module is used to classify the plurality of second texts to obtain a set of second texts under each category; The prompt text construction module is used to construct mind chain prompt text in the form of a single example sample for each category, based on the second text set under each category and the category information; The third text generation module is used to input the thought chain prompt text under each category into the generative language model to generate new text, so as to obtain the newly generated third text under each category; The third text verification module is used to perform content verification on the third text to obtain target text that meets the text quality requirements, and to determine the text dataset based on the second text set under each category and the target text.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the text dataset generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the text dataset generation method of any one of claims 1-7.

Images