A language model training method and sample data construction method and device

Abstract

The application relates to a language model training method, which comprises the following steps: obtaining a sentence sample data set, wherein each sentence sample in the sentence sample data set comprises a text label part, and the text label is used for representing the category of the corresponding sentence sample; performing damage processing on other parts of the sentence sample except the text label in a predetermined manner; inputting the sentence sample after the damage processing into a language model to obtain a prediction result; determining whether a preset convergence condition is met based on the prediction result and the sentence sample; if the condition is not met, adjusting the parameters of the language model, and performing the next round of training based on the language model after the parameter adjustment until the preset convergence condition is met, and a target language model is obtained. The application can effectively generate the target language model by taking the specially generated sentence sample data set containing the text label as the basis in the initial language model training, and then realizes the expansion of the target category sentence sample.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a language model training method, a sample data construction method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the rapid development of Internet technology, the amount of information and the degree of application information are growing in a geometric manner, and the influence of information on the whole society is gradually increasing to an absolutely important position. As a common type in information big data, the scale of natural language data is becoming larger and larger, and the complexity is becoming higher and higher. How to efficiently realize natural language information processing has become a key problem.

[0003] As a common means in natural language processing, the language model can model the probability of natural language samples, and can be used to estimate the probability of any given sample sequence or predict the probability of a word appearing at a certain position in a sample sequence. It is a key technology for understanding and representing sample semantics. In the process of natural language data processing, the quality of the language model dataset greatly affects the model effect, and the expansion of the language model dataset is the key to improving the quality of the dataset. How to effectively expand the sentence sample dataset in a large-scale language model and further improve the accuracy of the sentence sample dataset in the language model is crucial. At present, the commonly used natural sentence dataset expansion methods in the prior art mainly include: random word replacement method, core attribute word replacement method, and model self-enhancement method. However, the above methods all have problems such as the need for manual intervention cleaning, semantic inconsistency, inability to realize sample directional expansion, and low sample expansion efficiency, and therefore, obtaining a language model capable of sample expansion based on sample categories has become a key to natural language processing. SUMMARY

[0004] The present application provides a language model training method, a sample data construction method and device, electronic equipment and a storage medium to solve the problem that the existing language model cannot expand samples according to sentence sample category information. The technical solution is as follows:

[0005] The present application provides a language model training method, which comprises:

[0006] Obtain a sentence sample dataset, wherein each sentence sample in the sentence sample dataset comprises a text label part, and the text label is used to represent the category of the corresponding sentence sample;

[0007] Destroy other parts of the sentence sample except the text label in a predetermined manner;

[0008] The corrupted sentence sample is input into the language model to obtain the prediction result;

[0009] Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met;

[0010] If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0011] Optionally, an initial statement sample is obtained, wherein the initial statement sample contains corresponding symbol labels, and the symbol labels are used to characterize the category of the corresponding initial statement sample;

[0012] The symbol tags are converted into text tags, and the text tags are embedded into preset positions in the initial statement sample to obtain the statement sample.

[0013] Optionally, embedding the text tag into a preset position in the initial statement sample to obtain the statement sample includes:

[0014] The text labels and preset symbols are combined to obtain combined labels;

[0015] The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

[0016] This application also provides a method for constructing sample data, including:

[0017] Obtain the first target statement sample;

[0018] The first tag corresponding to the first target statement sample is modified to the second tag to obtain a second target statement sample containing the second tag. The second tag is a text tag embedded in a preset position of the second target statement sample. Both the first tag and the second tag are used to characterize the category of the first target statement sample, and the category represented by the second tag is different from that represented by the first tag.

[0019] The parts of the second target statement sample other than the second tag are destroyed. The destroyed second target statement sample is input into the target language model to obtain the recovered target statement through the target language model. The target statement is used as the extended statement sample corresponding to the first target statement sample.

[0020] Optionally, the first target statement sample is a statement sample containing a first tag, and the first tag is located at a preset position in the first target statement sample.

[0021] Optionally, the remaining parts of the second target statement sample, excluding the second tag, may be corrupted, including:

[0022] For the parts of the second target sentence sample other than the second tag, the part-of-speech tagging and parsing method is used to obtain the part-of-speech tags of each word segment in the other parts of the second target sentence sample other than the second tag;

[0023] Based on the part-of-speech tags of each word in the second target sentence sample, the part-of-speech tags in the second target sentence sample that belong to the target part of speech are masked to obtain the second target sentence sample after the destruction process.

[0024] Optionally, obtain the current statement context; determine the target part of speech based on the statement context.

[0025] Optionally, the first tag corresponding to the first target statement sample is modified to a second tag to obtain a second target statement sample containing the second tag, including:

[0026] The first tag corresponding to the first target statement sample is modified to the second tag;

[0027] The second label is combined with a preset symbol to obtain a combined label;

[0028] The second combined tag is embedded in a preset position in the first target statement sample to obtain the second target statement sample.

[0029] Optionally, the second tag has the opposite part of speech to the first tag.

[0030] Optionally, the target statement is the hard-to-bear sample corresponding to the first target statement.

[0031] This application also provides a language model training device, comprising:

[0032] The acquisition unit is used to acquire a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to characterize the category of the corresponding statement sample;

[0033] The modification unit is used to destroy other parts of the statement sample except for the text tags in a predetermined manner.

[0034] The processing unit is used to input the corrupted sentence samples into the language model to obtain prediction results;

[0035] The judgment unit is used to determine whether the preset convergence condition is met based on the prediction result and the statement sample.

[0036] The post-processing unit is used to adjust the parameters of the language model if the preset convergence condition is not met, and to perform the next round of training based on the language model with adjusted parameters until the preset convergence condition is met.

[0037] The output unit is used to output the target language model.

[0038] This application also provides an apparatus for constructing sample data, including:

[0039] The acquisition unit is used to acquire the first target statement sample;

[0040] The modification unit is used to modify the first tag corresponding to the first target statement sample to a second tag to obtain a second target statement sample containing the second tag, wherein the second tag is a text tag embedded in a preset position of the second target statement sample, and both the first tag and the second tag are used to characterize the category of the first target statement sample, and the category represented by the second tag is different from that represented by the first tag;

[0041] The processing unit is used to perform destruction processing on the parts of the second target statement sample other than the second tag, input the destroyed second target statement sample into the target language model, so as to obtain the recovered target statement through the target language model, and use the target statement as the extended statement sample corresponding to the first target statement sample;

[0042] The output unit is used to output the extended statement sample corresponding to the first target statement sample.

[0043] This application also provides an electronic device, including:

[0044] Memory and processor;

[0045] The memory is used to store program instructions;

[0046] The processor is used to invoke program instructions in the memory and execute the computer-executable instructions:

[0047] Obtain a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to represent the category of the corresponding statement sample;

[0048] The portion of the statement sample other than the text tag is destroyed in a predetermined manner;

[0049] The corrupted sentence sample is input into the language model to obtain the prediction result;

[0050] Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met;

[0051] If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0052] One embodiment of this application provides a computer-readable storage medium for storing a computer program, wherein the computer program is loaded by a processor and performs the following steps:

[0053] Obtain a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to represent the category of the corresponding statement sample;

[0054] The portion of the statement sample other than the text tag is destroyed in a predetermined manner;

[0055] The corrupted sentence sample is input into the language model to obtain the prediction result;

[0056] Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met;

[0057] If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0058] Compared with existing technologies, the language model training method provided in this application obtains a sentence sample dataset, in which each sentence sample contains a text label, which represents the category of the corresponding sentence sample; the other parts of the sentence sample, excluding the text label, are corrupted in a predetermined manner; the corrupted sentence sample is input into the language model to obtain a prediction result; based on the prediction result and the sentence sample, it is determined whether a preset convergence condition is met; if the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is performed based on the parameter-adjusted language model until the preset convergence condition is met, thus obtaining the target language model. This method utilizes the language capabilities of the language model and the category information of the original sentence samples, fine-tuning the language model with sentence samples containing category information, thereby transforming the language model. The generated target language model has the function of expanding and generating new sentence samples according to category information; the target language model generated by this method only requires a small number of samples for fine-tuning and is effective even with a small sample size.

[0059] The method for constructing sample data provided in this application involves obtaining a first target sentence sample; modifying the first label corresponding to the first target sentence sample to a second label to obtain a second target sentence sample containing the second label, wherein the second label is a text label embedded in a preset position of the second target sentence sample, and both the first and second labels are used to represent the category of the first target sentence sample, and the category represented by the second label is different from that represented by the first label; destroying the other parts of the second target sentence sample except for the second label, and inputting the destroyed second target sentence sample into a target language model to obtain the recovered target sentence through the target language model, and using the target sentence as the extended sentence sample corresponding to the first target sentence sample. Since the second target sentence sample containing the second label can reflect the category information of the sample, and the category represented by the second label is different from that represented by the first label, when it is destroyed and input into the target language model, the model will naturally introduce the information reconstruction effect generated by the category of the second label during the process of restoring and reconstructing the destroyed sample, thereby establishing a "grafting" effect between the first sentence and the second label, thus generating a new form of extended sample. The method for constructing sample data provided in this application can generate target category language samples relatively accurately, and is particularly suitable for generating difficult-to-bear samples. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of a language model training method provided in one embodiment of this application;

[0062] Figure 2 This is a flowchart of a method for destroying a statement sample in a predetermined manner according to an embodiment of this application;

[0063] Figure 3 This is a flowchart of a method for constructing sample data according to another embodiment of this application;

[0064] Figure 4 This is a flowchart of a method for destroying a second target statement sample according to another embodiment of this application;

[0065] Figure 5 This is a schematic diagram of the structure of a language model training device provided in one embodiment of this application;

[0066] Figure 6This is a schematic diagram of the device structure constructed from sample data provided in another embodiment of this application;

[0067] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0068] Figure 8 This is a schematic diagram of the structure of a storage medium provided in an embodiment of this application; Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, many specific details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0070] It should be noted that the terms "first," "second," and "third," etc., used in the various parts of the embodiments and accompanying drawings of this application are used to distinguish similar objects and do not indicate any specific order or sequence among them. These terms can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a different order than that shown or described herein.

[0071] The terms “comprising” and “having”, and any variations thereof, in this application are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may include other steps or units that are not expressly listed or that are inherent to such process, method, product, or apparatus.

[0072] In this application, unless otherwise expressly stated, the term "or" includes all possible combinations except those that are impractical. For example, if expressed as a database may include A or B, then unless otherwise specified or impractical, it may include database A, or B, or A and B. As another example, if expressed as a database may include A, B, or C, then unless otherwise specified or impractical, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0073] There are various existing methods for training language models, among which the most commonly used methods are as follows:

[0074] Random word replacement method. This method primarily utilizes known antonym pairs in natural language. Words in a sentence are randomly selected and replaced with their antonyms, such as replacing "is" with "is not," or "bad" with "good." This replacement results in only a small change in the numerical representation of the sample, but a high probability of altering the semantics, thus enabling language model training and dataset expansion. However, this method is not suitable for all samples. The number of words that can be replaced by antonyms is limited, and direct replacement of some words can cause semantic incoherence, often requiring manual cleaning in practice.

[0075] Core Attribute Word Replacement Method. This method complements the methods within the random word replacement method. The random word replacement method has a limited number of antonyms and cannot adapt to all sample expansion tasks. For example, for topic domain classifications ("food," "military," "poetry," etc.), replacing the sample "This bun is really good" with "This bun is really bad" does not change the sample's category (it still belongs to food). Therefore, this method first identifies the "core attribute words" of different categories and performs inter-category replacements for these attribute words. This method overcomes some of the shortcomings of the random word replacement method. However, for each different task, it is necessary to find the core attribute words for the corresponding category based on statistics. This process may lead to misidentification of attribute words, resulting in incorrect propagation. Furthermore, attribute word replacement may also generate semantically incoherent sentences, such as replacing "I like to eat buns" with "I like to eat books."

[0076] Model self-boosting is a common method used in practical training. It leverages the model's inherent recognition capabilities to iteratively expand the sample pool. First, it trains on all available samples to obtain an initial classification model. Then, it uses this initial model to predict the training data; the incorrectly predicted categories become the expanded samples. Finally, it trains further on these expanded samples, repeating the process iteratively. Model self-boosting is a widely used method in practice. However, it essentially selects samples from an existing pool and is not suitable for situations with a small sample size.

[0077] To address the aforementioned problems in existing technologies, this application provides a method for training a language model and a method for constructing sample data. By fine-tuning the language model using sentence samples with category information, the generated target language model has the function of expanding and generating new samples based on category information. Furthermore, using a special sample data construction method, a modified second target sentence sample is input into the target language model, enabling the sentence sample to expand based on the content of the second label, thus generating target category sentence samples more accurately and producing entirely new forms of expanded samples. The methods, devices, electronic devices, and storage media described in this application will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0078] The first embodiment of this application provides a method for training a language model.

[0079] Figure 1 This is a flowchart of the language model training method provided in this embodiment. The following is in conjunction with... Figure 1 The training of the language model provided in this embodiment will be described in detail. The embodiments described below are used to explain the technical solutions of this application and are not intended to limit actual use.

[0080] Please refer to Figure 1 The figure is a flowchart of the first embodiment of this application, which includes the following steps:

[0081] Step S101: Obtain a statement sample dataset. Each statement sample in the statement sample dataset contains a text label, which is used to represent the category of the corresponding statement sample.

[0082] The purpose of this step is to obtain sentence samples from the sentence sample dataset used to train the language model.

[0083] The statement samples include all information samples presented in statement form. For example, statement samples can be: conversational statement samples in daily life, monologue statement samples presented in spoken statement form, or written statement samples presented in written or read form, or silent internal thought statement samples of thought activities.

[0084] The stated sentence sample dataset can be formed by collecting sentences from everyday chat conversations, sentences from social media messages on social networks, or sentences extracted from the text of books in a library. The source of the stated sentence sample dataset is not specifically limited here. A stated sentence sample dataset is a collection of sentence samples collected and organized from these sources and filtered and labeled by human or machine intelligence methods. For example, a stated sentence sample dataset collected from online reviews could consist of 1000 Chinese sentences collected from various sales websites, each of which has been verified as correct by human or machine, and each sentence is labeled according to different situations, including attitude labels such as positive and negative reviews; it could also include domain labels such as military, history, and literature. The ultimate goal of this application is to expand a larger number of sentence samples based on the existing sentence samples in an initial stated sentence sample dataset. For example, if the initial stated sentence sample dataset only has 1000 sentences, the method provided in this application can expand it to 10000 sentences, which can be used for further model training or other possible uses. This embodiment trains a language model for this purpose.

[0085] For example, this step specifically obtains a dataset of sample statements related to positive reviews.

[0086] For details on how to obtain the statement sample dataset, please refer to relevant technologies or subsequent embodiments; these will not be elaborated upon here.

[0087] Optionally, an initial statement sample is obtained, the initial statement sample containing corresponding symbol labels, the symbol labels being used to characterize the category of the corresponding initial statement sample;

[0088] The symbolic labels are statement sample category labels, which categorize the initial statement samples. For example: in an evaluation scenario, there are attitude category labels such as positive and negative reviews; in an emotion scenario, there are emotion category labels such as happiness and sadness; in a topic scenario, there are category labels such as item names, books, and food. The form of these statement sample category labels is diverse and not specifically limited here. In the statement sample dataset, each statement sample is assigned a symbolic label; however, this label serves as an attribute identifier for the statement sample, not part of the statement sample itself. For example, the symbolic label "1" represents a positive evaluation category in an evaluation scenario, and "0" represents a negative evaluation category in an evaluation scenario.

[0089] The symbol labels are converted into text labels, and the text labels are embedded into preset positions in the initial statement sample to obtain the statement sample. For example, the symbol label "1" is converted into the text label "positive review", the symbol label "0" is converted into the text label "negative review", and the symbol label "7*7" is converted into the text label "military", etc.

[0090] It is important to understand that this step involves converting the symbol tags in the initial statement sample into corresponding text tags, and embedding these text tags as text into the statement sample. In this way, the tag information of the statement sample becomes part of the statement sample, meaning that the statement sample includes not only its original statement information but also its category information.

[0091] The text tags are embedded into preset positions in the initial statement sample to obtain a statement sample. The embedding method includes adding the text tags of the statement sample to the beginning or end of the statement sample to obtain a statement sample containing text tags.

[0092] For example, the sentence sample "This restaurant is really delicious" includes the attitude category symbol label "1", which represents a positive evaluation. Based on this, it is converted into the text label "good review" and embedded in a preset position in the language sample ontology, thus transforming it into "Good review, this restaurant is really delicious" or "This restaurant is really delicious, good review".

[0093] Optionally, embedding the text tag into a preset position in the initial sentence sample to obtain the sentence sample includes:

[0094] The text labels and preset symbols are combined to obtain combined labels;

[0095] In this context, a preset symbol serves as the tag identifier for the text tag, and the tag identifier is combined with the text tag in a set manner. The combined tag identifier and text tag are then embedded into the statement sample body.

[0096] The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

[0097] In this step, for example, the preset symbol can be in various forms such as a single quote '', a colon :, or an exclamation mark !. This preset symbol serves as a label identifier for the text label and is used in language model training to induce the model to recognize the text label. For example: "'Good review' This restaurant is really delicious" or "This restaurant is really delicious, good review:".

[0098] A colon is used as a special marker for the text form. The colon is added to the end of the text label as a setting method for combining the preset symbol with the text label.

[0099] For example, the sentence sample "This restaurant is really delicious" in the sample dataset of positive review statements can be transformed into "Positive review: This restaurant is really delicious" or "This restaurant is really delicious positive review:".

[0100] The methods described above for embedding text labels into sentence samples can all achieve the purpose of this application. However, a simpler method that is easier to use for subsequent model training is to place the text label at the beginning of the sentence sample and add a colon after it, i.e., the pattern "Good review: This restaurant is really delicious". Different methods of embedding text labels may have a certain impact on training speed.

[0101] For example:

[0102] Original statement sample dataset format:

[0103] Context (Scene) label This restaurant is delicious! 1 Found a wire in the food 0

[0104] Improved statement sample dataset format:

[0105] Context (Scene) Positive review: This restaurant is really delicious. Negative review: Found a wire in the food.

[0106] Step S102: Destroy the parts of the statement sample other than the text tags in a predetermined manner.

[0107] The purpose of this step is to obtain the corrupted sentence sample. The purpose of corrupting the sentence sample is to allow the language model to be trained in a predetermined manner during subsequent model training. In this step, while retaining the text labels of the corrupted sentence sample, the token is randomly selected for masking, and the token is replaced with a mask to corrupt the corrupted sentence sample.

[0108] Optionally, the best results are achieved when randomly selecting 30% to 50% of the tokens for masking.

[0109] Meanwhile, when using the above-mentioned embedding method of placing the text tag at the beginning of the statement sample and followed by a colon, the masking mechanism can include the requirement that the colon at the beginning of the sample and the text before the colon should not be obscured.

[0110] This embodiment also provides an optional implementation of this step. Figure 2 This is a flowchart illustrating a method for destroying a statement sample in a predetermined manner, as provided in this embodiment. Figure 2 As shown, the specific steps for destroying the statement sample in a predetermined manner provided in this embodiment are as follows:

[0111] Step S102-1: Obtain the statement sample.

[0112] It should be understood that the above statement samples are statement samples containing text tags.

[0113] For example, the sample statements obtained in this step are "Good review: This restaurant is really delicious" and "Breakfast: Fried dough sticks and soy milk are good for breakfast".

[0114] For details on how to obtain statement samples, please refer to relevant technologies or subsequent embodiments; they will not be elaborated here.

[0115] Step S102-2: Obtain the part-of-speech tags of each word segment in the sentence sample, excluding the text tags, using part-of-speech tagging and parsing.

[0116] It's important to understand that Part-of-Speech (POS) tagging, also known as grammatical tagging or word class resolution, is a text data processing technique in corpus linguistics that labels the parts of speech of words in a corpus according to their meaning and context. POS tagging can be done manually or using specific algorithms; using machine learning methods for POS tagging is a research area in natural language processing.

[0117] Example of a POS_TAG result:

[0118] word This Restaurant real tasty Part of speech r n d a

[0119] The term "word segmentation" refers to the precise cutting of natural language text into lexical fragments based on semantics and part of speech during text processing.

[0120] Step S102-3: Based on the local part-of-speech tags of each word segment in the sentence sample, perform masking on some word segments of each word segment in the sentence sample to obtain the sentence sample after destruction processing.

[0121] For different training objectives, the key words may have different parts of speech. Therefore, the parts of speech of the words that should be masked during training will also be different. For example, in the evaluation scenario, the keywords directly related to the evaluation may be mostly adjectives. Therefore, the focus should be on masking adjectives. In the topic category scenario, nouns are more important, so the focus can be on masking nouns.

[0122] For example, in evaluation scenarios, the focus can be on masking the adjective "a" and the verb "v" and then re-predicting. In topic scenarios, the focus can be on masking the noun "n" and the verb "v".

[0123] For example, in a review scenario, the masking of "positive review: This restaurant is really delicious" is transformed into "positive review: This restaurant is really [MASK]" or "positive review: This [MASK] is really delicious"; in a topic scenario, the masking of "breakfast: breakfast is good with fried dough sticks and soy milk" is transformed into "breakfast: breakfast is good with [MASK]" or "breakfast: breakfast is good with [MASK] fried dough sticks and soy milk". Here, [MASK] is the representative symbol for masking.

[0124] Step S103: Input the corrupted sentence sample into the language model to obtain the prediction result.

[0125] The purpose of this step is to obtain prediction results using the language functions of the language model, and to train the language model in this way.

[0126] The prediction result is the prediction result of the language model based on the text labels of the damaged sentence sample, and the predicted result of the restored sentence sample is output in a targeted manner.

[0127] The language model can be any of the following: BART, BERT, or CopyNet. For example, a BART (Bidirectional and Auto-Regressive Transformers) model is a pre-trained language model that combines contextual information and autoregressive properties. BART consists of a stack of 6 transformer-encoder layers and 6 transformer-decoder layers. Each transformer encoder, from bottom to top, consists of a multi-head attention layer, a normalization layer, a fully connected layer, and another normalization layer. The encoder transforms a sentence into a corresponding vector; the decoder decodes the vector to generate new sentence samples.

[0128] For example, the defaced sentence sample "good reviews: This restaurant is really [MASK]" is input into the encoder-decoder structure of the BART model. The BART model outputs a targeted prediction result based on the text tags of the defaced sentence sample. The goal of the prediction result is to reproduce the word segmentation masked by the [MASK] mask of the sentence sample. For example, the original sample is "This restaurant is really delicious", and the sentence sample after adding the combined tags is "good reviews: This restaurant is really delicious". The defaced sentence sample is "good reviews: This restaurant is really [MASK]", then the sample received by the model is "good reviews: This restaurant is really [MASK]". The model predicts word segmentation of the "good reviews" category based on the type of text tag "good reviews" in the combined tags. It can predict many words of the "good reviews" category such as "delicious", "good", "great", "affordable", "delicious", "beautiful", "perfect" etc. Its training goal is to restore the predicted word segmentation to "delicious".

[0129] By adding the text tag "positive review:", the model gradually learns the guiding role of this tag in the reconstruction process during continuous training. After a certain training period, the model will provide predicted word segments for training samples preceded by the text tag "positive review:", such as: "delicious", "good", "great", "affordable", "delicious", "beautiful", "perfect", "excellent", etc. These predicted word segments do not achieve the goal of restoring the original word segments, but they have similar meanings. The language model outputs the reconstructed sentence sample based on the predicted word segments. The goal of training the language model is to reconstruct the sentence sample.

[0130] For example, the broken sentence sample "Breakfast: Breakfast is suitable for fried dough sticks and soy milk" is input into the BART model. Based on the text tag "breakfast", the model predicts word segments such as "fried dough sticks", "steamed buns", "noodles", "porridge", "oatmeal", and "yogurt", among many other breakfast-related words. The final output of the model is a reconstructed sentence sample, such as "Breakfast: Breakfast is suitable for fried dough sticks", "Breakfast: Breakfast is suitable for steamed buns", or "Breakfast: Breakfast is suitable for oatmeal". The training objective of the BART language model is to recover and reproduce the original sentence sample "Breakfast: Breakfast is suitable for fried dough sticks and soy milk".

[0131] Step S104: Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met.

[0132] The purpose of this step is to determine whether the trained language model meets the preset conditions.

[0133] Based on the prediction results, the model outputs a restored and reconstructed sample of the damaged statements, thus obtaining a reconstructed statement sample.

[0134] Calculate the loss entropy of the loss function of the reconstructed statement sample and the initial statement sample, and determine whether the loss entropy satisfies the preset convergence condition.

[0135] Optionally, when the loss entropy of the above loss function decreases to 0.05 to 0.1, the language model is used as the target language model.

[0136] For example, following the steps described above, the training objective of the language model is to recover the sentence sample as "Good review: This restaurant is delicious". The training effect is evaluated using the sentence sample with added text labels. The loss is calculated based on the difference between the original sentence sample and the current one, and the reconstruction effect is assessed. When the loss entropy continues to decrease until it eventually levels off, and when the loss entropy drops to 0.05–0.1, the trained BART language model is used as the target language model.

[0137] In the BART language model, based on the text label "good reviews:" of the corrupted sentence sample, the BART model predicts word segmentation results such as "delicious," "good," "great," and "affordable." The sentence samples output by the decoder structure in the BART model, such as "good reviews: This restaurant is really good," "good reviews: This restaurant is really great," and "good reviews: This restaurant is really excellent," are continuously matched with the initial sentence sample "This restaurant is really delicious." The loss entropy of the loss function between the decoder output sentence sample and the initial sentence sample is calculated. When the loss value between the decoder output sentence sample and the initial sentence sample continuously decreases until it eventually levels off, the iteratively adjusted target BART model is obtained. This target BART model uses text label categories as a guide in the sentence sample recovery and reconstruction process, and can output sentence samples that match the text category labels.

[0138] In step S105, if the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0139] The purpose of this step is to obtain a target language model that meets the preset conditions.

[0140] The parameter adjustment of the language model refers to the adjustment of the parameters of the multiple encoders during the training process of the language model.

[0141] Optionally, the parameter adjustment can also be used to adjust the proportion of tokens obscured by randomly selected masks.

[0142] It is important to understand that when the proportion of tokens obscured by the mask is too large, it can easily increase the difficulty of the initial sentence sample recovery and reconstruction process, and the convergence speed of the language model is too slow; when the proportion of tokens obscured by the mask is too small, it can easily result in the initial sentence sample recovery and reconstruction process being easier, and the language model not learning the influence of the category label text.

[0143] For example, following the steps described above, the language model training process did not achieve the preset condition that the loss entropy decreased to 0.05–0.1. Therefore, the proportion of randomly selected masked tokens was further optimized and adjusted to 33%–35% for the next round of training until the preset convergence condition was met. The trained BART language model was then used as the target language model.

[0144] The first embodiment described above provides a method for training a language model. Because text labels are used during the training process, it can provide various output results with relevant tendencies formed under the guidance of text labels during the process of recovering sample language. These results are exactly what is needed to expand the language sample dataset.

[0145] Furthermore, the second embodiment of this application provides a method for constructing sample data.

[0146] Figure 3 This is a flowchart of a method for constructing sample data provided in this embodiment.

[0147] like Figure 3 As shown, the method for constructing sample data provided in this embodiment includes the following steps:

[0148] Step S201: Obtain the first target statement sample.

[0149] The purpose of this step is to obtain a sample of the first target statement for modification.

[0150] The first target statement sample is a statement sample containing a first tag. The first tag is used to characterize the category of the corresponding statement sample and is located at a preset position in the first target statement sample.

[0151] The first label can be a text label or a symbol label, and the type of the first label is not specifically limited in this embodiment. For example, the first label can be a symbol label "label "1" representing a positive review, or a text label "negative review".

[0152] The first target statement sample includes all information samples presented in language form. For example, the first target statement sample may be: a conversational statement sample in daily life, a monologue statement sample presented in spoken language form, or a written statement sample presented in written or read text form, or a silent internal thought statement sample generated during mental activity.

[0153] Optionally, the first target statement sample can also be a positive sample to be predicted and expanded.

[0154] It's important to understand that positive samples refer to samples belonging to the target category that corresponds to the true value; negative samples refer to all other target categories that do not correspond to the true value; and hard-to-negative samples are a type of negative sample, referring to data that is literally similar to positive samples but has opposite labels. The labels mentioned above refer to the labels determined based on the current context of the statement sample. The construction of hard-to-negative samples is particularly suitable for evaluation scenarios.

[0155] The first set of target sentence samples have all been verified as correct by humans or machines, and each sentence has been assigned a primary label based on different criteria, including attitude labels such as positive or negative reviews, and domain labels such as military, history, and literature. The ultimate goal of this application is to expand a larger number of sentence samples from an initial language sample dataset. For example, if the initial language sample dataset contains only 1000 sentences, the method provided in this application can expand it to 10,000 sentences. These sentences can be used for further model training or other possible applications. This embodiment constructs sample data for this purpose, and is particularly suitable for constructing difficult-to-bear samples.

[0156] For example, the first target sentence sample obtained in this step is "label "1": This restaurant is really delicious" and "Poetry: Young people are suitable to read Li Bai".

[0157] For details on how to obtain the first target statement sample, please refer to relevant technologies or subsequent embodiments; these will not be elaborated here.

[0158] Step S202: Modify the first tag corresponding to the first target statement sample to a second tag to obtain a second target statement sample containing the second tag. The second tag is a text tag embedded in a preset position of the second target statement sample. Both the first tag and the second tag are used to characterize the category of the first target statement sample, and the category represented by the second tag is different from that represented by the first tag.

[0159] The purpose of this step is to modify the first target statement sample based on the second tag.

[0160] For example, the first label "label "1" in the first target statement sample "label "1" This restaurant is really delicious" can be modified to the second label "bad review" to obtain the second target statement sample "bad review This restaurant is really delicious" which contains the second label.

[0161] For example, the first tag "poetry" in the first target sentence sample "Poetry is suitable for young people to read Li Bai" can be modified to the second tag "military", resulting in a second target sentence sample "military is suitable for young people to read Li Bai" containing the second tag.

[0162] Optionally, the first tag corresponding to the first target statement sample is modified to a second tag to obtain a second target statement sample containing the second tag, including:

[0163] The first tag corresponding to the first target statement sample is modified to the second tag;

[0164] The second label is combined with a preset symbol to obtain a combined label;

[0165] The second combined tag is embedded in a preset position in the first target statement sample to obtain the second target statement sample.

[0166] In this context, a preset symbol serves as the tag identifier for the text tag, and the tag identifier is combined with the text tag in a set manner. The combined tag identifier and text tag are then embedded into the statement sample body.

[0167] The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

[0168] In this step, for example, the preset symbol can be in various forms such as a single quote '', a colon :, or an exclamation mark !. This preset symbol serves as a label identifier for the text label and is used in language model training to induce the model to recognize the text label. For example: "'Good review' This restaurant is really delicious" or "This restaurant is really delicious, good review:".

[0169] A colon is used as a special marker for the text form. The colon is added to the end of the text label as a setting method for combining the preset symbol with the text label.

[0170] The methods described above for embedding text labels into sentence samples can all achieve the purpose of this application. However, a simpler method that is easier to use for subsequent model training is to place the text label at the beginning of the sentence sample and add a colon after it, i.e., the pattern "Good review: This restaurant is really delicious". Different methods of embedding text labels may have a certain impact on training speed.

[0171] Optionally, the second tag has the opposite part of speech to the first tag.

[0172] It's important to understand that "opposite part of speech" in the context of the first label refers to a second label whose meaning is opposite to that of the first label. For example, if the first label is "beautiful," then the second label, based on the meaning of the first label, could be "ugly"; if the first label is "expensive," then the second label could be "cheap."

[0173] In this step, for example, the first tag "good review" in the first target statement sample "good review, this restaurant is really delicious" is modified to the second tag "bad review:", resulting in the second target statement sample "bad review: this restaurant is really delicious" which contains the second tag.

[0174] Step S203: Destroy the parts of the second target statement sample other than the second tag, input the destructed second target statement sample into the target language model, so as to obtain the recovered target statement through the target language model, and use the target statement as the extended statement sample corresponding to the first target statement sample.

[0175] The purpose of this step is to achieve the destruction of the second target statement sample.

[0176] In this step, while retaining the second tag, the token is randomly selected for masking, and the token is replaced with MASK to destroy the second target sentence sample.

[0177] The destruction process for the parts of the second target sentence sample other than the second tag includes: obtaining the part-of-speech tags of each word segment in the parts of the second target sentence sample other than the second tag by using part-of-speech tagging and parsing.

[0178] Based on the part-of-speech tags of each word in the second target sentence sample, the part-of-speech tags in the second target sentence sample that belong to the target part of speech are masked to obtain the second target sentence sample after the destruction process.

[0179] Optionally, obtain the current statement context; determine the target part of speech based on the statement context.

[0180] Optionally, the best results are achieved when randomly selecting 30% to 50% of the tokens for masking.

[0181] Optionally, the target statement is the hard-to-bear sample corresponding to the first target statement.

[0182] The extended statement sample obtained in this step also includes extensions of the difficult-to-bear sample corresponding to the first target statement.

[0183] This embodiment also provides an optional implementation of this step. Figure 4 This is a flowchart illustrating the method for corrupting a second target statement sample provided in this embodiment. Figure 4 As shown, the specific steps for destroying the second target statement sample provided in this embodiment are as follows:

[0184] Step S203-1: For the second target sentence sample, obtain the part-of-speech tags of each word in the pre-tuned language sample using part-of-speech tagging and parsing.

[0185] Example of a POS_TAG result:

[0186] word young Suitable read Li Bai Part of speech a v v n

[0187] The term "word segmentation" refers to the precise division of natural language text into lexical fragments based on semantics and part of speech during text processing.

[0188] Step S203-2: Based on the second tag of the second target sentence sample and combined with the part-of-speech tags of each word in the sentence sample, masking and destructive processing is performed on some word segments of the sentence sample to obtain the destructive processing second target sentence sample.

[0189] For example, in the evaluation scenario, the adjective "a" and verb "v" are masked and re-predicted. In the topic scenario, the noun "n" and verb "v" are masked. For example, "bad review: This restaurant is really delicious" is masked and processed into "bad review: This restaurant is really [MASK]"; "military: Young people are suitable to read Li Bai" is masked and processed into "military: [MASK] is suitable to read [MASK]" or "military: Young people are suitable for [MASK][MASK]", etc.

[0190] The target language model can be any one of the following after training: target BART language model, target BERT language model, or target CopyNet language model.

[0191] For example, the damaged second target sentence samples "Bad review: This restaurant is really [MASK]" and "Military: [MASK] is suitable for reading [MASK]" are input into the target language model of the first embodiment of this application for reconstruction. This target language model has naturally learned the guiding role of text tags in the reconstructed sentence samples due to the embedding of text tags during training. Therefore, when reconstructing the sentence samples with changed text tag categories, words more likely to be used with the changed text tags are selected; in this way, the sentence structure of the original sentence sample and the vocabulary of the changed text tags are combined, achieving a "grafting" effect between the two types of language.

[0192] For example, given the second target sentence sample as "Bad review: This restaurant is really delicious", after masking and destructing it, it becomes "Bad review: This restaurant is really [MASK]". This is then provided to the target BART language model. Since the second label of this sentence sample is "Bad review:", and the trained target BART language model has already learned the vocabulary to choose when the label is "Bad review:", it is likely to provide recovery methods such as "This restaurant is really bad" or "This restaurant is really not good", etc. In this way, a language sample of a bad review is obtained when using a positive review sentence, thus achieving the grafting of two types of language samples.

[0193] For language recognition, it is particularly difficult to recognize sentences that use a certain sentence structure but have opposite actual meanings. Language samples of such sentences are called hard negative samples. Due to the characteristics of the technical solution provided in the second embodiment of this application, it is particularly suitable for generating hard negative samples.

[0194] The extended sentence samples mentioned above can be used to expand the language model dataset, broaden its coverage, and thus improve its quality. At the same time, the extended sentence samples can also be used to train the language model, improving the accuracy of the language model's directional predictions.

[0195] The third embodiment of this application provides an apparatus embodiment corresponding to the first embodiment described above, which can be used to execute the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0196] Figure 5 This is a schematic diagram of the structure of the language model training device provided in the first embodiment of this application, as shown below. Figure 5 As shown, the language model training device 500 of this application embodiment includes: an acquisition unit 501, a modification unit 502, a processing unit 503, a judgment unit 504, a post-processing unit 505, and an output unit 506. Wherein:

[0197] The acquisition unit 501 is used to acquire a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to characterize the category of the corresponding statement sample.

[0198] Modification unit 502 is used to destroy parts of the statement sample other than the text tags in a predetermined manner.

[0199] The processing unit 503 is used to input the corrupted sentence sample into the language model to obtain the prediction result.

[0200] The judgment unit 504 is used to determine whether the preset convergence condition is met based on the prediction result and the statement sample.

[0201] The post-processing unit 505 is used to adjust the parameters of the language model if the preset convergence condition is not met, and to perform the next round of training based on the language model with adjusted parameters until the preset convergence condition is met, so as to obtain the target language model.

[0202] Output unit 506 is used to output the target language model.

[0203] Optionally, an initial statement sample is obtained, wherein the initial statement sample contains corresponding symbol labels, and the symbol labels are used to characterize the category of the corresponding initial statement sample;

[0204] The symbol tags are converted into text tags, and the text tags are embedded into preset positions in the initial statement sample to obtain the statement sample.

[0205] Optionally, embedding the text tag into a preset position in the initial statement sample to obtain the statement sample includes:

[0206] The text labels and preset symbols are combined to obtain combined labels;

[0207] The combined tags are embedded into preset positions in the initial statement sample to obtain a statement sample. The fourth embodiment of this application provides a device embodiment corresponding to the second embodiment described above. Figure 6 This is a schematic diagram of the device structure according to the fourth embodiment of this application, as shown below. Figure 6 As shown, the sample data construction apparatus 600 of this application embodiment includes: an acquisition unit 601, a modification unit 602, a processing unit 603, and an output unit 604. Wherein:

[0208] Acquisition unit 601 is used to acquire the first target statement sample.

[0209] The modification unit 602 is used to modify the first tag corresponding to the first target statement sample to a second tag to obtain a second target statement sample containing the second tag. The second tag is a text tag embedded in a preset position of the second target statement sample. Both the first tag and the second tag are used to characterize the category of the first target statement sample, and the category represented by the second tag is different from that represented by the first tag.

[0210] The processing unit 603 is used to perform destruction processing on the parts of the second target statement sample other than the second tag, input the destroyed second target statement sample into the target language model, so as to obtain the recovered target statement through the target language model, and use the target statement as the extended statement sample corresponding to the first target statement sample.

[0211] Output unit 604 is used to output the extended statement sample corresponding to the first target statement sample.

[0212] Optionally, the first target statement sample is a statement sample containing a first tag, and the first tag is located at a preset position in the first target statement sample.

[0213] Optionally, the remaining parts of the second target statement sample, excluding the second tag, may be corrupted, including:

[0214] For the parts of the second target sentence sample other than the second tag, the part-of-speech tagging and parsing method is used to obtain the part-of-speech tags of each word segment in the other parts of the second target sentence sample other than the second tag;

[0215] Based on the part-of-speech tags of each word in the second target sentence sample, the part-of-speech tags in the second target sentence sample that belong to the target part of speech are masked to obtain the second target sentence sample after the destruction process.

[0216] Optionally, obtain the current statement context; determine the target part of speech based on the statement context.

[0217] Optionally, the first tag corresponding to the first target statement sample is modified to a second tag to obtain a second target statement sample containing the second tag, including:

[0218] The first tag corresponding to the first target statement sample is modified to the second tag;

[0219] The second label is combined with a preset symbol to obtain a combined label;

[0220] The second combined tag is embedded in a preset position in the first target statement sample to obtain the second target statement sample.

[0221] Optionally, the second tag has the opposite part of speech to the first tag.

[0222] Optionally, the target statement is the hard-to-bear sample corresponding to the first target statement.

[0223] The fifth embodiment of this application provides an electronic device embodiment corresponding to the above embodiments.

[0224] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, as shown below. Figure 7 As shown, the electronic device 700 of this application includes a processor 701 and a memory 702.

[0225] The memory is used to store program instructions;

[0226] The processor is used to invoke program instructions in the memory and execute the computer-executable instructions:

[0227] Obtain a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to represent the category of the corresponding statement sample;

[0228] The portion of the statement sample other than the text tag is destroyed in a predetermined manner;

[0229] The corrupted sentence sample is input into the language model to obtain the prediction result;

[0230] Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met;

[0231] If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0232] Optionally, an initial statement sample is obtained, wherein the initial statement sample contains corresponding symbol labels, and the symbol labels are used to characterize the category of the corresponding initial statement sample;

[0233] The symbol tags are converted into text tags, and the text tags are embedded into preset positions in the initial statement sample to obtain the statement sample.

[0234] Optionally, embedding the text tag into a preset position in the initial statement sample to obtain the statement sample includes:

[0235] The text labels and preset symbols are combined to obtain combined labels;

[0236] The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

[0237] The sixth embodiment of this application provides a storage medium embodiment corresponding to the above embodiments.

[0238] Figure 8 This is a schematic diagram of the structure of a storage medium provided in an embodiment of this application, as shown below. Figure 8 As shown, the storage medium 800 of this application includes: a computer-readable storage medium 801 and a processor 802.

[0239] This application provides a storage medium storing instructions that can be loaded by a processor and executed in the following steps:

[0240] Obtain a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label portion, and the text label is used to represent the category of the corresponding statement sample;

[0241] The portion of the statement sample other than the text tag is destroyed in a predetermined manner;

[0242] The corrupted sentence sample is input into the language model to obtain the prediction result;

[0243] Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met;

[0244] If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained.

[0245] Optionally, an initial statement sample is obtained, wherein the initial statement sample contains corresponding symbol labels, and the symbol labels are used to characterize the category of the corresponding initial statement sample;

[0246] The symbol tags are converted into text tags, and the text tags are embedded into preset positions in the initial statement sample to obtain the statement sample.

[0247] Optionally, embedding the text tag into a preset position in the initial statement sample to obtain the statement sample includes:

[0248] The text labels and preset symbols are combined to obtain combined labels;

[0249] The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

[0250] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0251] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0252] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0253] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0254] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims of the present invention.

Claims

1. A method for training a language model, characterized in that, include: Obtain a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label, which is used to characterize the category of the corresponding statement sample and has a guiding function; The portion of the statement sample other than the text tag is destroyed in a predetermined manner; The corrupted sentence sample is input into the language model, and the prediction result is obtained under the guidance of the text label; Based on the prediction results and the statement samples, determine whether the preset convergence conditions are met; If the preset convergence condition is not met, the parameters of the language model are adjusted, and the next round of training is carried out based on the language model with adjusted parameters until the preset convergence condition is met, and the target language model is obtained. The target language model is expanded to generate new samples according to the category information.

2. The method according to claim 1, characterized in that, The method further includes: Obtain an initial statement sample, wherein the initial statement sample contains a corresponding symbol label, and the symbol label is used to characterize the category of the corresponding initial statement sample; The symbol tags are converted into text tags, and the text tags are embedded into preset positions in the initial statement sample to obtain the statement sample.

3. The method according to claim 2, characterized in that, The step of embedding the text tag into a preset position in the initial statement sample to obtain the statement sample includes: The text labels and preset symbols are combined to obtain combined labels; The combined tags are embedded into a preset position in the initial statement sample to obtain the statement sample.

4. A method for constructing sample data, characterized in that, By combining the sentence structure of the first target statement sample with the vocabulary of the second tag, a bridge is achieved between the first tag type language and the second tag type language, including: Obtain the first target statement sample; The first tag corresponding to the first target statement sample is modified to the second tag to obtain a second target statement sample containing the second tag. The second tag is a text tag embedded in a preset position of the second target statement sample. Both the first tag and the second tag are used to characterize the category of the first target statement sample and have a guiding role. The category represented by the second tag is different from that represented by the first tag. The second target statement sample, excluding the second tag, is destructed. The destructed second target statement sample is then input into the target language model. Guided by the second tag, the target statement is recovered through the target language model and used as the extended statement sample corresponding to the first target statement sample.

5. The method according to claim 4, characterized in that, The first target statement sample is a statement sample containing a first tag, and the first tag is located at a preset position in the first target statement sample.

6. The method according to claim 4, characterized in that, The remaining parts of the second target statement sample, excluding the second tag, are subjected to destructive processing, including: For the parts of the second target sentence sample other than the second tag, the part-of-speech tagging and parsing method is used to obtain the part-of-speech tags of each word segment in the other parts of the second target sentence sample other than the second tag; Based on the part-of-speech tags of each word in the second target sentence sample, the part-of-speech tags in the second target sentence sample that belong to the target part of speech are masked to obtain the second target sentence sample after the destruction process.

7. The method according to claim 6, characterized in that, The method further includes: Get the current statement context; The target part of speech is determined based on the context of the statement.

8. The method according to claim 4, characterized in that, Modify the first tag corresponding to the first target statement sample to the second tag to obtain a second target statement sample containing the second tag, including: The first tag corresponding to the first target statement sample is modified to the second tag; The second label is combined with a preset symbol to obtain a combined label; The combined tags are embedded in a preset position in the first target statement sample to obtain the second target statement sample.

9. The method according to claim 4, characterized in that, The second tag has the opposite part of speech to the first tag.

10. The method according to claim 9, characterized in that, The target statement is the hard-to-bear sample corresponding to the first target statement.

11. A training device for a language model, characterized in that, include: The acquisition unit is used to acquire a statement sample dataset, wherein each statement sample in the statement sample dataset contains a text label, and the text label is used to characterize the category of the corresponding statement sample and has a guiding function. The modification unit is used to destroy other parts of the statement sample except for the text tags in a predetermined manner. The processing unit is used to input the corrupted sentence samples into the language model and obtain prediction results under the guidance of text labels; The judgment unit is used to determine whether the preset convergence condition is met based on the prediction result and the statement sample. The post-processing unit is used to adjust the parameters of the language model if the preset convergence condition is not met, and to perform the next round of training based on the language model with adjusted parameters until the preset convergence condition is met. The output unit is used to output the target language model, which expands and generates new samples according to the category information.

12. An apparatus for constructing sample data, characterized in that, By combining the sentence structure of the first target statement sample with the vocabulary of the second tag, a bridge is achieved between the first tag type language and the second tag type language, including: The acquisition unit is used to acquire the first target statement sample; The modification unit is used to modify the first tag corresponding to the first target statement sample to a second tag to obtain a second target statement sample containing the second tag. The second tag is a text tag embedded in a preset position of the second target statement sample. Both the first tag and the second tag are used to characterize the category of the first target statement sample and have a guiding function. The category represented by the second tag is different from that represented by the first tag. The processing unit is used to perform destruction processing on the parts of the second target statement sample other than the second tag, input the destroyed second target statement sample into the target language model, and obtain the recovered target statement through the target language model under the guidance of the second tag, and use the target statement as the extended statement sample corresponding to the first target statement sample; The output unit is used to output the extended statement sample corresponding to the first target statement sample.

13. An electronic device, characterized in that, include: Memory and processor; The memory is used to store program instructions; The processor is used to invoke program instructions in the memory to execute the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 10.

Images