Method, system, terminal and medium for generating training corpus based on language model

Abstract

The present invention discloses a method, system, terminal and medium for generating training corpus based on a language model, and relates to the field of artificial intelligence technology. The method obtains original corpus to be trained, obtains a target paragraph according to the original corpus to be trained; selects a target enhanced training corpus prompt from multiple types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate different types of enhanced training corpus; generates enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt; and constructs a synthetic training corpus based on the enhanced training corpus. The present invention pre-sets multiple types of enhanced training corpus prompts, and guides the language model to synthesize new text corpus based on the original training corpus through the enhanced training corpus prompts, which effectively increases the scale of the training corpus and enriches the diversity of the training corpus.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, terminal and medium for generating training corpus based on a language model. Background Art

[0002] The domain corpus training of large-scale language models has been an important task in the field of natural language processing in recent years. These language models can extract rich public knowledge from large-scale corpora through pre-training and fine-tuning, and are widely used in tasks such as knowledge question answering and text generation. However, the training of language models requires large-scale and diverse corpora, while the scale of corpora in certain fields is small and difficult to update, which limits the adaptability of language models to small-scale, domain-specific corpora.

[0003] The common solution at present is to continue pre-training to adapt the pre-trained model to a specific field. However, the expression of small-scale corpus is single and lacks diversity, which easily leads to overfitting in the continuous pre-training process of the model.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, system, terminal and medium for generating training corpus based on a language model are provided, aiming to solve the problem that the small-scale corpus in the prior art has a single expression and lacks diversity, which easily leads to overfitting in the continuous pre-training process of the model.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for generating training corpus based on a language model, the method comprising:

[0008] Obtaining an original corpus to be trained, and obtaining a target paragraph according to the original corpus to be trained;

[0009] Selecting a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate a different type of enhanced training corpus;

[0010] Generate enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt;

[0011] A synthetic training corpus is constructed based on the enhanced training corpus.

[0012] In one implementation, obtaining a target paragraph according to the original corpus to be trained includes:

[0013] Acquire an entity set according to the original corpus to be trained; wherein the entity set includes all entities of each original paragraph in the original corpus to be trained;

[0014] Generate a mapping relationship dictionary according to the entity set and the original corpus to be trained; wherein the mapping relationship dictionary is used to reflect the mapping relationship between each entity in the entity set and all the original paragraphs corresponding to each entity;

[0015] The target paragraph is obtained according to the mapping relationship dictionary.

[0016] In one implementation, obtaining an entity set according to the original corpus to be trained includes:

[0017] Get the preset entity extraction hint;

[0018] According to each of the original paragraphs and the entity extraction prompt, all the entities included in all the original paragraphs are extracted through a second language model to obtain the entity set.

[0019] In one implementation, obtaining the target paragraph according to the mapping relationship dictionary includes:

[0020] Sampling entities in the entity set to obtain an entity subset, wherein the entity subset includes the sampled target entities;

[0021] For each of the target entities in the entity subset, determining all the original paragraphs corresponding to the target entity according to the mapping relationship dictionary;

[0022] The original paragraphs of all the target entities in the entity subset are taken as the target paragraphs.

[0023] In one embodiment, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0024] The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

[0025] In one embodiment, according to the selected first type of target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset, a first enhanced training corpus is generated by the first language model, and then the following steps are further included:

[0026] If the amount of the first enhanced training corpus does not reach a preset amount threshold, repeatedly sampling the entities in the entity set to obtain an entity subset, and obtaining a target paragraph of the entity subset according to the mapping dictionary relationship;

[0027] The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

[0028] In one implementation, obtaining the target paragraph according to the mapping relationship dictionary includes:

[0029] Constructing an entity graph according to the mapping relationship dictionary;

[0030] Sampling the entity graph multiple times to obtain multiple paragraph subsets;

[0031] The paragraphs of each of the paragraph subsets are respectively used as the target paragraphs.

[0032] In one implementation, constructing an entity graph according to the mapping relationship dictionary includes:

[0033] Generate entity nodes in the entity graph based on entities in the mapping relationship dictionary;

[0034] The edges and edge attributes in the entity graph are generated according to the mapping relationship between each entity and the original paragraph in the mapping relationship dictionary, wherein the edge is used to reflect the co-occurrence relationship between two entity nodes, and the edge attribute is used to reflect the original paragraph shared by the two entity nodes.

[0035] In one embodiment, the entity graph is sampled multiple times to obtain multiple paragraph subsets, including:

[0036] Selecting an entity node of the entity graph as a first sampling entity node;

[0037] Performing a first sampling of the first sampled entity node according to the edge attribute of the entity graph to obtain a first paragraph subset, wherein the first paragraph subset includes a plurality of common original paragraphs related to the first sampled entity node;

[0038] Eliminating the first paragraph subset from the edge attributes of all entity nodes of the entity graph to obtain the remaining edge attributes of the entity graph;

[0039] Reselecting an entity node in the entity graph as a second sampling entity node;

[0040] Performing a second sampling on the second sampling entity node according to the remaining edge attribute to obtain a second paragraph subset, wherein the second paragraph subset includes a plurality of common original paragraphs related to the second sampling entity node;

[0041] The entire entity graph is traversed, and the operations of selecting a first sampling entity node in the entity graph for a first sampling to obtain a first paragraph subset and reselecting a second sampling entity node in the entity graph for a second sampling to obtain a second paragraph subset are repeatedly performed to obtain multiple paragraph subsets.

[0042] In one implementation, the first sampling entity nodes are sampled for the first time according to the edge attributes of the entity graph to obtain a first paragraph subset, including:

[0043] Obtain all neighbor nodes of the first sampling entity node;

[0044] Selecting a neighbor node from all the neighbor nodes as the next first sampling entity node, and removing the first sampling entity node from the entity graph; traversing the entire entity graph, repeatedly selecting the next first sampling entity node from all the neighbor nodes of the first sampling entity node until a stop condition is met, and obtaining a path to the first sampling entity node;

[0045] For each edge on the path of the first sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a first paragraph subset.

[0046] In one implementation, selecting a neighbor node from all the neighbor nodes as the next first sampling entity node includes:

[0047] Calculating the correlation between the first sampling entity node and each neighbor node respectively;

[0048] The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next first sampling entity node.

[0049] In one implementation, sampling the second sampled entity nodes for a second time according to the remaining edge attributes to obtain a second paragraph subset includes:

[0050] Acquire all neighbor nodes of the second sampling entity node;

[0051] Selecting a neighbor node from all the neighbor nodes as the next second sampling entity node, and removing the second sampling entity node from the entity graph; traversing the entire entity graph, repeatedly selecting the next second sampling entity node from all the neighbor nodes of the second sampling entity node until a stop condition is met, and obtaining a path to the second sampling entity node;

[0052] For each edge on the path of the second sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a second paragraph subset.

[0053] In one implementation, selecting a neighbor node from all the neighbor nodes as the next second sampling entity node includes:

[0054] Calculating the correlation between the second sampling entity node and each neighbor node respectively;

[0055] The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next second sampling entity node.

[0056] In one embodiment, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0057] According to the selected second type target enhanced training corpus prompt and each paragraph of the paragraph subset, a second enhanced training corpus is generated by the first language model.

[0058] In one embodiment, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0059] According to the selected third type target enhanced training corpus prompt and each paragraph of the paragraph subset, a third enhanced training corpus is generated by the first language model.

[0060] In one embodiment, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0061] Selecting a preset number of original paragraphs from the original corpus to be trained as target paragraphs;

[0062] According to the selected second type target enhanced training corpus prompt and the target paragraph, a fourth enhanced training corpus is generated by using the first language model.

[0063] In one embodiment, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0064] Selecting a preset number of original paragraphs from the original corpus to be trained as target paragraphs;

[0065] According to the selected third type of target enhanced training corpus prompt and the target paragraph, fifth enhanced training corpus is generated by using the first language model.

[0066] In one embodiment, constructing a synthetic training corpus based on the enhanced training corpus includes:

[0067] The synthetic training corpus is constructed according to at least one of the first enhanced training corpus, the second enhanced training corpus, the third enhanced training corpus, the fourth enhanced training corpus and the fifth enhanced training corpus, wherein the synthetic training corpus is used to pre-train the third language model.

[0068] In a second aspect, an embodiment of the present invention further provides a training corpus synthesis system based on a language model, the system comprising:

[0069] An acquisition module is used to acquire original corpus to be trained, and acquire a target paragraph according to the original corpus to be trained;

[0070] A selection module, used for selecting a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used for generating a different type of enhanced training corpus;

[0071] A generating module, configured to generate enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt;

[0072] A construction module is used to construct a synthetic training corpus based on the enhanced training corpus.

[0073] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-described training corpus generation methods based on a language model; and the processor is used to execute the program.

[0074] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-described methods for generating training corpus based on a language model.

[0075] Beneficial effects of the present invention: The embodiment of the present invention obtains the original corpus to be trained, obtains the target paragraph according to the original corpus to be trained; selects the target enhanced training corpus prompt from multiple types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate different types of enhanced training corpus; generates enhanced training corpus through the first language model according to the target paragraph and the target enhanced training corpus prompt; and constructs synthetic training corpus based on the enhanced training corpus. The present invention pre-sets multiple types of enhanced training corpus prompts, and guides the language model through the enhanced training corpus prompts to obtain the relevance and diversity of the paragraph subset of the paragraph subset enhanced sampling based on the original training corpus combined with the target paragraph based on the entity graph sampling, and then generates different types of enhanced training corpora to construct synthetic training corpora, which expands the scale of the training corpus, enriches the diversity of the training corpus, and improves the coherence and effectiveness of the synthetic training corpus, and further improves the quality and readability of the synthetic training corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0077] Figure 1 It is a flowchart of a method for generating training corpus based on a language model provided in an embodiment of the present invention.

[0078] Figure 2 It is a schematic diagram of the generation logic of different types of enhanced training corpora provided by an embodiment of the present invention.

[0079] Figure 3 It is a module diagram of a language model-based training corpus synthesis system provided in an embodiment of the present invention.

[0080] Figure 4 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The present invention discloses a method, system, terminal and medium for generating training corpus based on a language model. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0082] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0083] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0084] In view of the above-mentioned defects of the prior art, the present invention provides a method for generating training corpus based on a language model, wherein the method obtains an original corpus to be trained, obtains a target paragraph according to the original corpus to be trained; selects a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate a different type of enhanced training corpus; generates enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt; and constructs a synthetic training corpus based on the enhanced training corpus. The present invention pre-sets a plurality of types of enhanced training corpus prompts, and guides the language model to synthesize a new text corpus based on the original training corpus through the enhanced training corpus prompts, thereby effectively increasing the scale of the training corpus and enriching the diversity of the training corpus.

[0085] like Figure 1 As shown, the method specifically comprises the following steps:

[0086] Step S100, obtaining an original corpus to be trained, and obtaining a target paragraph according to the original corpus to be trained;

[0087] Step S200: selecting a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate a different type of enhanced training corpus;

[0088] Step S300, generating enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt;

[0089] Step S400: constructing synthetic training corpus based on the enhanced training corpus.

[0090] The original corpus to be trained refers to the existing corpus used for language model training. The scale of the original corpus to be trained is small and lacks diversity. Directly training with the language model of the original corpus to be trained can easily cause the continuous pre-training process of the model to fall into overfitting. In order to increase the scale of the training corpus and enrich the diversity of the corpus, this embodiment will synthesize new corpus based on the original corpus to be trained. Specifically, at least one target paragraph is obtained through the original corpus to be trained. Different types of preset enhanced training corpus prompts (prompts) are obtained. Each type of enhanced training corpus prompt is used to guide the generation of a type of enhanced training corpus. Users can select appropriate enhanced training corpus prompts according to their own needs or task requirements, and the selected enhanced training corpus prompts are used as target enhanced training corpus prompts. Taking a target enhanced training corpus prompt as an example, the target enhanced training corpus prompt is combined with the target paragraph and input into the first language model. The first language model generates a new text corpus in combination with the target paragraph according to the guidance of the target enhanced training corpus prompt, and defines the new text corpus as the enhanced training corpus.

[0091] In one implementation, obtaining a target paragraph according to the original corpus to be trained includes:

[0092] Acquire an entity set according to the original corpus to be trained; wherein the entity set includes all entities of each original paragraph in the original corpus to be trained;

[0093] Generate a mapping relationship dictionary according to the entity set and the original corpus to be trained; wherein the mapping relationship dictionary is used to reflect the mapping relationship between each entity in the entity set and all the original paragraphs corresponding to each entity;

[0094] The target paragraph is obtained according to the mapping relationship dictionary.

[0095] Specifically, Figure 2As shown, the first generation method of the enhanced training corpus provided by this embodiment is a generation method based on entity extraction, that is, the target paragraph is not an original paragraph directly selected from the original training corpus. First, for each original paragraph, all entities contained in the original paragraph are extracted, and all entities obtained through all original paragraphs are combined to obtain an entity set. For each entity in the entity set, the original paragraph source of the entity is known during the extraction process, thereby obtaining the mapping relationship between the entity and one or more original paragraphs. The mapping relationship between all entities and the original paragraphs is summarized to obtain a mapping relationship dictionary. The mapping relationship dictionary can reflect the correspondence between entities and paragraphs, the relationship between entities, and then reflect some hidden information of the original text. Selecting the target paragraph through the mapping relationship dictionary to generate enhanced training corpus can enrich the relationship description between entities.

[0096] In one implementation, obtaining an entity set according to the original corpus to be trained includes:

[0097] Get the preset entity extraction hint;

[0098] According to each of the original paragraphs and the entity extraction prompt, all the entities included in all the original paragraphs are extracted through a second language model to obtain the entity set.

[0099] Specifically, the present embodiment pre-sets entity extraction prompts to guide entity extraction operations. The entity extraction prompts can be keywords, phrases, or specific instruction formats related to the entity. Each original paragraph is selected from the original corpus to be trained in turn, and each original paragraph is spliced ​​with the entity extraction prompt and input into the second language model. The second language model outputs one or more entities contained in the original paragraph.

[0100] For example, the entity extraction prompt is: Please extract entities from the following paragraph, and the entities involved include concepts and terms such as names of people, places, events, and products. The entity extraction prompt is concatenated with an original paragraph to obtain the third concatenated data. The Bert model (a bidirectional semantic encoding model) is selected as the second language model, and the third concatenated data is input into the Bert model to obtain one or more entities contained in the original paragraph.

[0101] In one implementation, obtaining the target paragraph according to the mapping relationship dictionary includes:

[0102] Sampling entities in the entity set to obtain an entity subset, wherein the entity subset includes the sampled target entities;

[0103] For each of the target entities in the entity subset, determining all the original paragraphs corresponding to the target entity according to the mapping relationship dictionary;

[0104] The original paragraphs of all the target entities in the entity subset are taken as the target paragraphs.

[0105] Specifically, the entity set is sampled to select any number of target entities to form an entity subset. The sampling method can be set according to user needs or task requirements, such as random sampling or interval sampling. For each entity in the entity subset, the original paragraph corresponding to each entity is queried through the mapping relationship dictionary. The original paragraph where each entity appears in the queried entity subset is used as the target paragraph to generate enhanced training corpus.

[0106] In one implementation, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0107] The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

[0108] Specifically, for the first generation method of the enhanced training corpus, the original paragraph to which each target entity in the entity subset belongs is used as the target paragraph for the enhanced training corpus. If the selected target enhanced training corpus prompt is a first type of target enhanced training corpus prompt, the first type of target enhanced training corpus prompt is used to guide the operation of entity relationship analysis, and the entity subset, the target paragraph of the entity subset and the first type of target enhanced training corpus prompt are spliced ​​and input into the first language model. The first language model outputs any number of paragraphs in text format as the generated enhanced training corpus. This embodiment defines the enhanced training corpus generated according to the first generation method of the enhanced training corpus and the first type of target enhanced training corpus prompt as the first enhanced training corpus.

[0109] For example, the prompt of the first type of target enhanced training corpus is: Please analyze the specific relationship between the specified entity sets in the paragraph provided below; discuss how the interaction of entity sets affects the content of a given paragraph. Concatenate the entity subset, the target paragraph of the entity subset, and the prompt of the first type of target enhanced training corpus to obtain the fourth concatenated data. Input the fourth concatenated data into the first language model to obtain the first enhanced training corpus (containing any number of paragraphs in text format). For example, if the two entities are "neural network" and "deep learning", the text example of the first enhanced training corpus is: Based on the description of the textbook, deep learning is described as an evolved form of neural network, emphasizing the use of multi-layer structures to enhance the model's ability in image and language processing.

[0110] In one implementation, the method generates a first enhanced training corpus according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset through the first language model, and then further includes:

[0111] If the amount of the first enhanced training corpus does not reach a preset amount threshold, repeatedly sampling the entities in the entity set to obtain an entity subset, and obtaining a target paragraph of the entity subset according to the mapping dictionary relationship;

[0112] The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

[0113] Specifically, the user can set a quantity threshold according to their own needs or task requirements, indicating the upper limit of the size of the first enhanced training corpus. Whenever a new first enhanced training corpus is generated, it is determined whether the number of generated first enhanced training corpus reaches the quantity threshold. If not, it indicates that the number of generated first enhanced training corpus is insufficient, and the first enhanced training corpus is continuously generated starting from the step of sampling entities in the entity set to obtain entity subsets until the number of generated first enhanced training corpus reaches the quantity threshold.

[0114] In one implementation, obtaining the target paragraph according to the mapping relationship dictionary includes:

[0115] Constructing an entity graph according to the mapping relationship dictionary;

[0116] Sampling the entity graph multiple times to obtain multiple paragraph subsets;

[0117] The paragraphs of each of the paragraph subsets are respectively used as the target paragraphs.

[0118] Specifically, Figure 2As shown, the second generation method of the enhanced training corpus provided in this embodiment is a generation method based on the entity graph. Similar to the first generation method of the enhanced training corpus, the second generation method of the enhanced training corpus also needs to establish a mapping relationship dictionary first, and the establishment process of the mapping relationship dictionary is no longer repeated here. After obtaining the mapping relationship dictionary, an entity graph is constructed according to the mapping relationship dictionary to more intuitively characterize the entity relationship and help capture the semantic association between paragraphs. By sampling the entity graph multiple times, a paragraph subset is obtained each time, and multiple paragraph subsets are finally obtained. In this embodiment, the paragraphs of each paragraph subset are respectively used as target paragraphs to generate enhanced training corpus, which can improve the diversity of the generated corpus. In addition, the present embodiment obtains the target paragraph based on the entity graph, takes into account the entity relationship, and can also improve the coherence and relevance of the generated corpus, thereby improving the quality and readability of the generated corpus.

[0119] In one implementation, constructing an entity graph according to the mapping relationship dictionary includes:

[0120] Generate entity nodes in the entity graph based on entities in the mapping relationship dictionary;

[0121] The edges and edge attributes in the entity graph are generated according to the mapping relationship between each entity and the original paragraph in the mapping relationship dictionary, wherein the edge is used to reflect the co-occurrence relationship between two entity nodes, and the edge attribute is used to reflect the original paragraph shared by the two entity nodes.

[0122] Specifically, the mapping relationship dictionary can be used to identify whether any two entities have a co-occurrence relationship. A co-occurrence relationship refers to the existence of a common original paragraph between the two entities. The entity graph is established by taking the entity as the entity node of the entity graph, and for two entities with a co-occurrence relationship, forming an edge between the entity nodes of the two entities, and setting the edge attribute according to the original paragraph shared by the two entities.

[0123] In one implementation, the entity graph is sampled multiple times to obtain multiple paragraph subsets, including:

[0124] Selecting an entity node of the entity graph as a first sampling entity node;

[0125] Performing a first sampling of the first sampled entity node according to the edge attribute of the entity graph to obtain a first paragraph subset, wherein the first paragraph subset includes a plurality of common original paragraphs related to the first sampled entity node;

[0126] Eliminating the first paragraph subset from the edge attributes of all entity nodes of the entity graph to obtain the remaining edge attributes of the entity graph;

[0127] Reselecting an entity node in the entity graph as a second sampling entity node;

[0128] Performing a second sampling on the second sampling entity node according to the remaining edge attribute to obtain a second paragraph subset, wherein the second paragraph subset includes a plurality of common original paragraphs related to the second sampling entity node;

[0129] The entire entity graph is traversed, and the operations of selecting a first sampling entity node in the entity graph for a first sampling to obtain a first paragraph subset and reselecting a second sampling entity node in the entity graph for a second sampling to obtain a second paragraph subset are repeatedly performed to obtain multiple paragraph subsets.

[0130] Specifically, an entity node is selected as the first sampling entity node according to a specific selection method or randomly on the entity graph, and multiple common original paragraphs related to the first sampling entity node are sampled through the neighbor nodes and edge attributes corresponding to the first sampling entity node in the entity graph, and all the sampled common original paragraphs are placed in the initialized empty paragraph set, that is, the first paragraph subset is formed. Then the next round of sampling is carried out. Before sampling, in order to avoid generating duplicate corpus and ensure the diversity of corpus, the first paragraph subset needs to be removed from the edge attributes of all entity nodes in the entity graph to obtain the remaining edge attributes. An entity node is selected as the second sampling entity node again according to a specific selection method or randomly on the entity graph. Through the neighbor nodes and edge attributes corresponding to the second sampling entity node, multiple common original paragraphs related to the second sampling entity node are sampled in the remaining edge attributes of the entity graph to form the second paragraph subset. The entire entity graph is traversed, and the first sampling and second sampling of the entity nodes in the entity graph are repeated multiple times, and finally multiple paragraph subsets are obtained.

[0131] In one implementation, the first sampling entity node is sampled for the first time according to the edge attributes of the entity graph to obtain a first paragraph subset, including:

[0132] Obtain all neighbor nodes of the first sampling entity node;

[0133] Selecting a neighbor node from all the neighbor nodes as the next first sampling entity node, and removing the first sampling entity node from the entity graph; traversing the entire entity graph, repeatedly selecting the next first sampling entity node from all the neighbor nodes of the first sampling entity node until a stop condition is met, and obtaining a path to the first sampling entity node;

[0134] For each edge on the path of the first sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a first paragraph subset.

[0135] Specifically, the neighbor node of the first sampled entity node refers to the entity node that has an edge with the first sampled entity node. All neighbor nodes and all paths of the first sampled entity node are obtained, and one of these neighbor nodes is selected as the next first sampled entity node. The entire entity graph is traversed, and the selection process of the first sampled entity node is continuously performed multiple times until the stop condition is met, that is, the current next first sampled entity node is the last entity node, thereby obtaining the path of the first sampled entity node. The path consists of multiple entity nodes and multiple edges. For each edge on the path, an original paragraph is arbitrarily selected from the edge attributes of the edge, and the first paragraph subset is formed by all the selected original paragraphs.

[0136] For example, one entity node is selected from the neighbor nodes of the current first sampling entity node as the next entity node of the walk, that is, the next first sampling entity node. The edge attributes of the current first sampling entity node and the next first sampling entity node can reflect the original paragraphs shared by the two entity nodes. A shared original paragraph is selected according to the edge attributes, and the selection method can be random selection. Continue to walk until the preset condition is met, and the preset condition is used to determine the end point of the walk. If the preset condition is not met, the next first sampling entity node is used as the current first sampling entity node, and the step of selecting one from the neighbor nodes of the current first sampling entity node as the next first sampling entity node is continued, and a shared original paragraph is selected according to the edge attributes of the current first sampling entity node and the next first sampling entity node; if the preset condition is met, the walk is stopped, and all shared original paragraphs selected by a single walk are combined into a paragraph subset, that is, the first paragraph subset is obtained. The preset condition can be that the current walk depth reaches the preset depth d (the value of d is determined based on the number of entity nodes on the walk path, which will affect the size of a single paragraph subset), and based on the preset condition, a paragraph subset consisting of a path containing d+1 entity nodes and d sampling paragraphs can be obtained. Alternatively, the preset condition may be that there is no next wandering entity node, that is, except for the selected entity node, there are no other entity nodes having an edge with the current entity node.

[0137] In one implementation, each of the first paragraph subsets corresponds to a path on the entity graph, and each paragraph included in the first paragraph subset comes from a different edge of the path; and the paragraphs included in each of the first paragraph subsets do not overlap with each other.

[0138] Specifically, taking a single walk sampling as an example, continuous walk is equivalent to continuously selecting the next first sampling entity node. It is necessary to avoid repeatedly selecting the same entity node during a single walk. Then, all the original paragraphs sampled on the path of a single walk are used to form a paragraph subset, that is, the first paragraph subset is obtained. Finally, the paragraphs of each first paragraph subset are used as target paragraphs in turn to synthesize new text corpus. Repeat the walk multiple times until the paragraphs contained in all the sampled first paragraph subsets can cover all the paragraphs on the entity graph, and there are no repeated paragraphs between each two first paragraph subsets.

[0139] In one implementation, selecting a neighbor node from all the neighbor nodes as the next first sampling entity node includes:

[0140] Randomly select an entity node from neighboring nodes of the first sampling entity node as the next first sampling entity node;

[0141] Alternatively, respectively calculating the correlation between the first sampling entity node and each neighbor node;

[0142] The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next first sampling entity node.

[0143] Specifically, this embodiment provides two walking methods for the first entity node. The first walking method is to use a graph sampling algorithm using random walking: randomly select an entity node from the neighbor nodes of the current first sampling entity node as the next first sampling entity node. The second walking method is to use a graph sampling algorithm combined with context walking: calculate the correlation between the current first sampling entity node and each neighbor node, and select an entity node as the next first sampling entity node based on the correlation of each neighbor node. The second walking method takes into account the entity relationship, and uses the target paragraph obtained based on the second walking method to perform corpus synthesis, which can improve the coherence and readability of the corpus.

[0144] In one implementation, respectively calculating the correlation between the first sampling entity node and each neighbor node includes:

[0145] For each neighbor node of the first sampling entity node, determining the number of common original paragraphs between the first sampling entity node and the neighbor node according to the edge attribute between the first sampling entity node and the neighbor node;

[0146] The correlation between the first sampling entity node and the neighbor node is calculated according to the number of the common original paragraphs.

[0147] Specifically, the correlation between two entity nodes can be calculated based on the number of common original paragraphs of the corresponding two entities. The more common original paragraphs, the higher the correlation. In practical applications, the neighbor node with the greatest correlation with the current first sampled entity node can be selected as the next first sampled entity node.

[0148] In one implementation, sampling the second sampling entity node for a second time according to the remaining edge attribute to obtain a second paragraph subset includes:

[0149] Acquire all neighbor nodes of the second sampling entity node;

[0150] Selecting a neighbor node from all the neighbor nodes as the next second sampling entity node, and removing the second sampling entity node from the entity graph; traversing the entire entity graph, repeatedly selecting the next second sampling entity node from all the neighbor nodes of the second sampling entity node until a stop condition is met, and obtaining a path to the second sampling entity node;

[0151] For each edge on the path of the second sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a second paragraph subset.

[0152] Specifically, the neighbor node of the second sampling entity node refers to the entity node that has an edge with the second sampling entity node. All neighbor nodes and all paths of the second sampling entity node are obtained, and one of these neighbor nodes is selected as the next second sampling entity node. The entire entity graph is traversed, and the selection process of the second sampling entity node is continuously performed multiple times until the stop condition is met, that is, the current second sampling entity node is the last entity node, thereby obtaining the path of the second sampling entity node. The path consists of multiple entity nodes and multiple edges. For each edge on the path, an original paragraph is arbitrarily selected from the edge attributes of the edge, and the second paragraph subset is composed of all the selected original paragraphs.

[0153] For example, one entity node is selected from the neighbor nodes of the current second sampling entity node as the next entity node of the walk, that is, the next second sampling entity node. The edge attributes of the current second sampling entity node and the next two first sampling entity nodes can reflect the original paragraphs shared by the two entity nodes. A shared original paragraph is selected according to the edge attributes, and the selection method can be random selection. Continue to walk until the preset conditions are met, and the preset conditions are used to determine the end point of the walk. If the preset conditions are not met, the next second sampling entity node is used as the current second sampling entity node, and the steps of selecting one from the neighbor nodes of the current second sampling entity node as the next second sampling entity node and selecting a shared original paragraph according to the edge attributes of the current second sampling entity node and the next second sampling entity node are continued; if the preset conditions are met, the walk is stopped, and all shared original paragraphs selected in a single walk are combined into a paragraph subset, that is, the second paragraph subset is obtained.

[0154] In one implementation, each of the second paragraph subsets corresponds to a path on the entity graph, and each paragraph included in the second paragraph subset comes from a different edge of the path; and the paragraphs included in each of the second paragraph subsets do not overlap with each other.

[0155] Specifically, taking a single walk as an example, continuous walk is equivalent to continuously selecting the next second sampling entity node. It is necessary to avoid repeatedly selecting the same entity node during a single walk. Then, all the original paragraphs sampled on the path of a single walk are used to form a paragraph subset, that is, the second paragraph subset is obtained. Finally, the paragraphs of each second paragraph subset are used as target paragraphs in turn to synthesize new text corpus. Repeat the walk multiple times until the paragraphs contained in all the sampled second paragraph subsets can cover all the paragraphs on the entity graph, and there are no repeated paragraphs between each two second paragraph subsets.

[0156] In one implementation, selecting a neighbor node from all the neighbor nodes as the next second sampling entity node includes:

[0157] Randomly select an entity node from neighboring nodes of the second sampling entity node as the next second sampling entity node;

[0158] Alternatively, respectively calculating the correlation between the second sampling entity node and each neighbor node;

[0159] The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next second sampling entity node.

[0160] Specifically, the selection process of the first entity node and the second entity node is actually equivalent to two adjacent walking processes. Similar to the first entity node, this embodiment also provides two walking methods for the second entity node. The first walking method is to use a graph sampling algorithm based on random walking: randomly select an entity node from the neighboring nodes of the current second sampling entity node as the next second sampling entity node. The second walking method is to use a graph sampling algorithm combined with contextual walking: calculate the correlation between the current second sampling entity node and each neighboring node, and select an entity node as the next second sampling entity node based on the correlation of each neighboring node. The second walking method takes into account the entity relationship, and uses the target paragraph obtained based on the second walking method to synthesize the corpus, which can improve the coherence and readability of the corpus.

[0161] In one implementation, respectively calculating the correlation between the second sampling entity node and each neighbor node includes:

[0162] For each neighbor node of the second sampling entity node, determining the number of common original paragraphs between the second sampling entity node and the neighbor node according to the edge attribute between the second sampling entity node and the neighbor node;

[0163] The correlation between the second sampling entity node and the neighbor node is calculated according to the number of the common original paragraphs.

[0164] Specifically, the correlation between two entity nodes can be calculated based on the number of common original paragraphs of the corresponding two entities. The more common original paragraphs, the higher the correlation. In practical applications, the neighbor node with the greatest correlation with the current second sampling entity node can be selected as the next second sampling entity node.

[0165] In one implementation, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0166] According to the selected second type target enhanced training corpus prompt and each paragraph of the paragraph subset, a second enhanced training corpus is generated by the first language model.

[0167] Specifically, for the second generation method of the enhanced training corpus, if the selected target enhanced training corpus prompt is a second type of target enhanced training corpus prompt, the second type of target enhanced training corpus prompt is used to guide the paragraph rewriting operation, and the second type of target enhanced training corpus prompt is spliced ​​with each sampled paragraph subset and input into the first language model. The first language model outputs any number of paragraphs in text format as the generated enhanced training corpus. In this embodiment, the enhanced training corpus generated according to the second generation method of the enhanced training corpus and the second type of target enhanced training corpus prompt is defined as the second enhanced training corpus.

[0168] For example, the second type of target enhanced training corpus prompt is: Please rewrite the following paragraphs provided, and cover all the knowledge points involved in the original text. For each paragraph subset, the paragraph subset is spliced ​​with the second type of target enhanced training corpus prompt to obtain the fifth spliced ​​data. Select the GPT model as the first language model, input the fifth spliced ​​data into the GPT model, and obtain the second enhanced training corpus (including any number of paragraphs in text format).

[0169] In one implementation, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0170] According to the selected third type target enhanced training corpus prompt and each paragraph of the paragraph subset, a third enhanced training corpus is generated by the first language model.

[0171] Specifically, for the second generation method of the enhanced training corpus, if the selected target enhanced training corpus prompt is the third type of target enhanced training corpus prompt, the third type of target enhanced training corpus prompt is used to guide the operation of question-answer pair construction, and the third type of target enhanced training corpus prompt is spliced ​​with each sampled paragraph subset and input into the first language model. The first language model outputs any number of question-answer pairs in text format as the generated enhanced training corpus. In this embodiment, the enhanced training corpus generated according to the second generation method of the enhanced training corpus and the third type of target enhanced training corpus prompt is defined as the third enhanced training corpus.

[0172] For example, the third type of target enhanced training corpus prompt is: Please construct some question-answer pairs based on the following paragraphs provided. You need to construct detailed questions and answers around the knowledge points in the paragraphs, and all question-answer pairs need to cover all the knowledge points involved in the original text as much as possible. For each paragraph subset, the paragraph subset is spliced ​​with the third type of target enhanced training corpus prompt to obtain the sixth spliced ​​data. Select the GPT model as the first language model, input the sixth spliced ​​data into the GPT model, and obtain the third enhanced training corpus (containing any number of question-answer pairs in text format).

[0173] In one implementation, obtaining a target paragraph according to the original corpus to be trained includes:

[0174] Select any number of original paragraphs from the original corpus to be trained, and use the selected original paragraphs as target paragraphs.

[0175] Specifically, a predetermined number of original paragraphs can be selected from the original training data in sequence. In practical applications, in addition to sequential selection, random selection or selection at a certain interval can also be used.

[0176] In one implementation, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0177] According to the selected second type target enhanced training corpus prompt and the target paragraph, a fourth enhanced training corpus is generated by using the first language model.

[0178] like Figure 2 As shown, the third generation method of the enhanced training corpus provided in this embodiment is a generation method based on direct rewriting. Specifically, if the selected target enhanced training corpus prompt is a second type target enhanced training corpus prompt, the second type target enhanced training corpus prompt is used to guide the paragraph rewriting operation, and the second type target enhanced training corpus prompt is spliced ​​with the selected target paragraph and input into the first language model. The first language model outputs any number of paragraphs in text format as the generated enhanced training corpus. This embodiment defines the enhanced training corpus generated according to the third generation method of the enhanced training corpus and the second type target enhanced training corpus prompt as the fourth enhanced training corpus.

[0179] It should be noted that the number of generated enhanced training corpora may not be equal to the number of input target paragraphs. The first language model may generate more enhanced training corpora than the number of target paragraphs through multiple rewriting based on a small number of target paragraphs.

[0180] For example, the second type of target enhanced training corpus prompt is: Please rewrite the following paragraph provided, and cover all the knowledge points involved in the original text. The splicing method can be to splice the second type of target enhanced training corpus prompt with all target paragraphs to obtain the first splicing data; the splicing method can also be to splice the second type of target enhanced training corpus prompt with each target paragraph respectively, to obtain the first splicing data corresponding to each target paragraph. The first language model can be a GPT model (a natural language processing model based on deep learning), and the first splicing data is input into the GPT model to obtain paragraphs containing any multiple text formats, that is, the fourth enhanced training corpus.

[0181] In one implementation, generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes:

[0182] According to the selected third type of target enhanced training corpus prompt and the target paragraph, fifth enhanced training corpus is generated by using the first language model.

[0183] Specifically, for the third generation method of the enhanced training corpus, if the selected target enhanced training corpus prompt is a third type target enhanced training corpus prompt, the third type target enhanced training corpus prompt is used to guide the operation of question-answer pair construction, and the third type target enhanced training corpus prompt is spliced ​​with the selected target paragraph and input into the first language model. The first language model outputs any number of question-answer pairs in text format as the generated enhanced training corpus. This embodiment defines the enhanced training corpus generated according to the third generation method of the enhanced training corpus and the third type target enhanced training corpus prompt as the fifth enhanced training corpus.

[0184] For example, the prompt for the third type of target enhanced training corpus is: Please construct some question-answer pairs based on the following paragraphs provided. You need to construct detailed questions and answers around the knowledge points in the paragraphs, and all question-answer pairs need to cover as many knowledge points as possible in the original text. The splicing method can be to splice the third type of target enhanced training corpus prompts with all target paragraphs to obtain second spliced ​​data; or to splice the third type of target enhanced training corpus prompts with each target paragraph separately to obtain second spliced ​​data corresponding to each target paragraph. The first language model can select the GPT model, input the second spliced ​​data into the GPT model, and obtain question-answer pairs containing any multiple text formats, that is, the fifth enhanced training corpus.

[0185] In one implementation, constructing a synthetic training corpus based on the enhanced training corpus includes:

[0186] The synthetic training corpus is constructed according to at least one of the first enhanced training corpus, the second enhanced training corpus, the third enhanced training corpus, the fourth enhanced training corpus and the fifth enhanced training corpus, wherein the synthetic training corpus is used to pre-train the third language model.

[0187] Specifically, this embodiment uses one or more types of enhanced training corpora to construct synthetic training corpora to obtain a training data set for pre-training the third language model. It is also possible to combine the original corpus to be trained with one or more types of enhanced training corpora to construct a training data set for pre-training the third language model. The pre-trained training data set generated in this way has a larger scale and richer types of training corpora. By continuously pre-training the third language model (such as the LLaMA model, etc.) through the training data set and updating the language model weights, a better pre-training effect can be obtained. The third language model can learn the diversified expressions of knowledge in the original corpus through the training data set, and can directly answer questions related to the knowledge in the training corpus after deployment.

[0188] Preferably, the fourth enhanced training corpus and the fifth enhanced training corpus, and the original corpus to be trained are added to the initialized empty training database to obtain a first training database; the first enhanced training corpus is added to the first training database to obtain a second training database; the second enhanced training corpus and the third enhanced training corpus are added to the second training database to obtain a third training database; and at least one of the first training database, the second training database, and the third training database is selected to construct a training data set.

[0189] The advantages of the present invention are:

[0190] 1. Improve the utilization rate of small-scale domain data: This invention expands small-scale data into a comprehensive and diverse corpus through a high-quality corpus synthesis process, solving the problem of traditional model training relying on large-scale corpus.

[0191] 2. Solve the problem of overfitting of domain knowledge: The present invention solves the problem that the model is prone to overfitting during small-scale corpus training by generating enhanced training corpus with diversified expressions, enhances the model's ability to understand knowledge in specific domains, and makes it more versatile in reasoning.

[0192] 3. Simplify the domain adaptation process of the model: The corpus synthesis and enhancement method of the present invention has little demand for adding additional modules, can simplify the system architecture, and improve the adaptability of the model in specific fields.

[0193] 4. Reduce the cost of acquiring high-quality domain data: The present invention can reduce the reliance on a large amount of real domain corpus, reduce the cost of data collection and annotation, and provide an efficient and low-cost solution for domain model training.

[0194] Based on the above embodiments, the present invention also provides a training corpus synthesis system based on a language model, such as Figure 3 As shown, the system comprises:

[0195] An acquisition module 01 is used to acquire an original corpus to be trained, and acquire a target paragraph according to the original corpus to be trained;

[0196] A selection module 02 is used to select a target enhanced training corpus prompt from multiple types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate a different type of enhanced training corpus;

[0197] A generating module 03, configured to generate enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt;

[0198] The construction module 04 is used to construct a synthetic training corpus based on the enhanced training corpus.

[0199] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 4 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training corpus generation method based on a language model is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0200] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0201] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a method for generating training corpus based on a language model.

[0202] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0203] In summary, the present invention discloses a method, system, terminal and medium for generating training corpus based on a language model, and relates to the field of artificial intelligence technology. The method obtains the original corpus to be trained, obtains the target paragraph according to the original corpus to be trained; selects the target enhanced training corpus prompt from multiple types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate different types of enhanced training corpus; generates enhanced training corpus through the first language model according to the target paragraph and the target enhanced training corpus prompt; and constructs a synthetic training corpus based on the enhanced training corpus. The present invention pre-sets multiple types of enhanced training corpus prompts, and guides the language model to synthesize new text corpus based on the original training corpus through the enhanced training corpus prompts, which effectively increases the scale of the training corpus and enriches the diversity of the training corpus.

[0204] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating training corpus based on a language model, characterized in that: The method comprises: Obtaining an original corpus to be trained, and obtaining a target paragraph according to the original corpus to be trained, including: obtaining an entity set according to the original corpus to be trained, the entity set including all entities of each original paragraph in the original corpus to be trained; generating a mapping relationship dictionary according to the entity set and the original corpus to be trained, the mapping relationship dictionary being used to reflect the mapping relationship between each entity in the entity set and all the original paragraphs corresponding to each entity; obtaining the target paragraph according to the mapping relationship dictionary; wherein obtaining the target paragraph according to the mapping relationship dictionary includes: constructing an entity graph according to the mapping relationship dictionary; sampling the entity graph multiple times to obtain multiple paragraph subsets; and taking the paragraphs of each paragraph subset as the target paragraph respectively; Selecting a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used to generate a different type of enhanced training corpus; Generate enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt; A synthetic training corpus is constructed based on the enhanced training corpus.

2. The method for generating training corpus based on a language model according to claim 1, characterized in that: Acquiring an entity set according to the original corpus to be trained includes: Get the preset entity extraction hint; According to each of the original paragraphs and the entity extraction prompt, all the entities included in all the original paragraphs are extracted through a second language model to obtain the entity set.

3. The method for generating training corpus based on a language model according to claim 1, characterized in that: Acquiring the target paragraph according to the mapping relationship dictionary includes: Sampling entities in the entity set to obtain an entity subset, wherein the entity subset includes the sampled target entities; For each of the target entities in the entity subset, determining all the original paragraphs corresponding to the target entity according to the mapping relationship dictionary; The original paragraphs of all the target entities in the entity subset are taken as the target paragraphs.

4. The method for generating training corpus based on a language model according to claim 3, characterized in that: Generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes: The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

5. The method for generating training corpus based on a language model according to claim 4, characterized in that: According to the selected first type of target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset, a first enhanced training corpus is generated by the first language model, and then the following further comprises: If the amount of the first enhanced training corpus does not reach a preset amount threshold, repeatedly sampling the entities in the entity set to obtain an entity subset, and obtaining a target paragraph of the entity subset according to the mapping relationship dictionary; The first enhanced training corpus is generated by the first language model according to the selected first type target enhanced training corpus prompt, the entity subset and the target paragraph of the entity subset.

6. The method for generating training corpus based on a language model according to claim 1, characterized in that: Constructing an entity graph according to the mapping relationship dictionary includes: Generate entity nodes in the entity graph based on entities in the mapping relationship dictionary; The edges and edge attributes in the entity graph are generated according to the mapping relationship between each entity and the original paragraph in the mapping relationship dictionary, wherein the edge is used to reflect the co-occurrence relationship between two entity nodes, and the edge attribute is used to reflect the original paragraph shared by the two entity nodes.

7. The method for generating training corpus based on a language model according to claim 6, characterized in that: The entity graph is sampled multiple times to obtain multiple paragraph subsets, including: Selecting an entity node of the entity graph as a first sampling entity node; Performing a first sampling of the first sampled entity node according to the edge attribute of the entity graph to obtain a first paragraph subset, wherein the first paragraph subset includes a plurality of common original paragraphs related to the first sampled entity node; Eliminating the first paragraph subset from the edge attributes of all entity nodes of the entity graph to obtain the remaining edge attributes of the entity graph; Reselecting an entity node in the entity graph as a second sampling entity node; Performing a second sampling on the second sampling entity node according to the remaining edge attribute to obtain a second paragraph subset, wherein the second paragraph subset includes a plurality of common original paragraphs related to the second sampling entity node; The entire entity graph is traversed, and the operations of selecting a first sampling entity node in the entity graph for a first sampling to obtain a first paragraph subset and reselecting a second sampling entity node in the entity graph for a second sampling to obtain a second paragraph subset are repeatedly performed to obtain multiple paragraph subsets.

8. The method for generating training corpus based on a language model according to claim 7, characterized in that: The first sampling entity node is sampled for the first time according to the edge attribute of the entity graph to obtain a first paragraph subset, including: Obtain all neighbor nodes of the first sampling entity node; Selecting a neighbor node from all the neighbor nodes as the next first sampling entity node, and removing the first sampling entity node from the entity graph; Traversing the entire entity graph, repeatedly selecting the next first sampling entity node from all neighboring nodes of the first sampling entity node until a stop condition is met, and obtaining a path to the first sampling entity node; For each edge on the path of the first sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a first paragraph subset.

9. The method for generating training corpus based on a language model according to claim 8, characterized in that: Selecting a neighbor node from all the neighbor nodes as the next first sampling entity node includes: Calculating the correlation between the first sampling entity node and each neighbor node respectively; The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next first sampling entity node.

10. The method for generating training corpus based on a language model according to claim 7, characterized in that: The second sampling entity node is sampled a second time according to the remaining edge attribute to obtain a second paragraph subset, including: Acquire all neighbor nodes of the second sampling entity node; Selecting a neighbor node from all the neighbor nodes as the next second sampling entity node, and removing the second sampling entity node from the entity graph; Traversing the entire entity graph, repeatedly selecting the next second sampling entity node from all neighboring nodes of the second sampling entity node until a stop condition is met, and obtaining a path to the second sampling entity node; For each edge on the path of the second sampling entity node, an original paragraph is arbitrarily selected from the edge attributes of each edge to obtain a second paragraph subset.

11. The method for generating training corpus based on a language model according to claim 10, characterized in that: Selecting a neighbor node from all the neighbor nodes as the next second sampling entity node includes: Calculating the correlation between the second sampling entity node and each neighbor node respectively; The neighbor node with the greatest correlation is selected from among all the neighbor nodes as the next second sampling entity node.

12. The method for generating training corpus based on a language model according to claim 1, characterized in that: Generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes: According to the selected second type target enhanced training corpus prompt and each paragraph of the paragraph subset, a second enhanced training corpus is generated by the first language model.

13. The method for generating training corpus based on a language model according to claim 1, characterized in that: Generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes: According to the selected third type target enhanced training corpus prompt and each paragraph of the paragraph subset, a third enhanced training corpus is generated by the first language model.

14. The method for generating training corpus based on a language model according to claim 1, characterized in that: Generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes: Selecting a preset number of original paragraphs from the original corpus to be trained as target paragraphs; According to the selected second type target enhanced training corpus prompt and the target paragraph, a fourth enhanced training corpus is generated by using the first language model.

15. The method for generating training corpus based on a language model according to claim 1, characterized in that: Generating enhanced training corpus by using a first language model according to the target paragraph and the target enhanced training corpus prompt includes: Selecting a preset number of original paragraphs from the original corpus to be trained as target paragraphs; According to the selected third type of target enhanced training corpus prompt and the target paragraph, fifth enhanced training corpus is generated by using the first language model.

16. The method for generating training corpus based on a language model according to any one of claims 4 to 15, characterized in that: Constructing synthetic training corpus based on the enhanced training corpus includes: The synthetic training corpus is constructed according to at least one of the first enhanced training corpus, the second enhanced training corpus, the third enhanced training corpus, the fourth enhanced training corpus and the fifth enhanced training corpus, wherein the synthetic training corpus is used to pre-train the third language model.

17. A training corpus synthesis system based on a language model, characterized in that: The system comprises: An acquisition module is used to acquire original corpus to be trained, and acquire a target paragraph according to the original corpus to be trained, including: acquiring an entity set according to the original corpus to be trained, the entity set including all entities of each original paragraph in the original corpus to be trained; generating a mapping relationship dictionary according to the entity set and the original corpus to be trained, the mapping relationship dictionary being used to reflect the mapping relationship between each entity in the entity set and all the original paragraphs corresponding to each entity; acquiring the target paragraph according to the mapping relationship dictionary; wherein acquiring the target paragraph according to the mapping relationship dictionary includes: constructing an entity graph according to the mapping relationship dictionary; sampling the entity graph multiple times to obtain multiple paragraph subsets; and taking the paragraphs of each paragraph subset as the target paragraph respectively; A selection module, used for selecting a target enhanced training corpus prompt from a plurality of types of enhanced training corpus prompts, wherein each type of enhanced training corpus prompt is used for generating a different type of enhanced training corpus; A generating module, configured to generate enhanced training corpus through a first language model according to the target paragraph and the target enhanced training corpus prompt; A construction module is used to construct a synthetic training corpus based on the enhanced training corpus.

18. A computer terminal, characterized in that: The computer terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the language model-based training corpus generation method as described in any one of claims 1-16; and the processor is used to execute the program.

19. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the method for generating training corpus based on a language model as described in any one of claims 1-16.

Images