Training method and device for improving generalization ability of text analysis model

Abstract

The embodiments of this specification provide a training method and device for improving the generalization ability of a text analysis model, wherein the training method for improving the generalization ability of a text analysis model includes: obtaining a training sample set, wherein the training sample set includes training texts; generating a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes environment labels corresponding to each training text; training a text analysis model according to the training texts and the environment labels corresponding to the training texts, wherein the text analysis model includes a classifier corresponding to the preset environment classification information. Through this method, the prediction accuracy of the text analysis model is improved when there is a distribution difference between the prediction distribution of the text analysis model in the prediction stage and the training distribution in the training stage.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a training method for improving the generalization ability of a text analysis model. Background Art

[0002] Natural Language Processing is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. Among them, natural language understanding tasks have always attracted the attention of practitioners, but most of the current methods focus on the scenario where the test data is the same as the training data, which conceals the fragility of the model. Therefore, how to improve the effect of the model in natural language understanding tasks has become a problem that technicians need to solve urgently. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a training method for improving the generalization ability of a text analysis model, a text analysis method, a training method for improving the generalization ability of a data processing model, and a data processing method. One or more embodiments of this specification also involve a training device for improving the generalization ability of a text analysis model, a text analysis device, a training device for improving the generalization ability of a data processing model, a data processing device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a training method for improving the generalization ability of a text analysis model is provided, comprising:

[0005] Acquire a training sample set, wherein the training sample set includes training text;

[0006] Generate a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes an environment label corresponding to each training text;

[0007] A text analysis model is trained according to the training text and the environment label corresponding to the training text, wherein the text analysis model includes a classifier corresponding to the preset environment classification information.

[0008] According to a second aspect of an embodiment of this specification, a training method for improving the generalization ability of a data processing model is provided, comprising:

[0009] Acquire a training sample set, wherein the training sample set includes training project data;

[0010] Generate a target environment matrix based on the preset environment classification information of the training sample set, wherein the target environment matrix includes an environment label corresponding to each training item data;

[0011] A data processing model is trained according to the training project data and the environment labels corresponding to the training project data, wherein the data processing model includes a classifier corresponding to the preset environment classification information.

[0012] According to a third aspect of the embodiments of this specification, a text analysis method is provided, including:

[0013] receiving text to be analyzed;

[0014] Inputting the text to be analyzed into a pre-trained text analysis model, wherein the text analysis model is obtained by training using the above training method;

[0015] Obtain a target classification label output by the text analysis model based on the text to be analyzed.

[0016] According to a fourth aspect of the embodiments of this specification, a data processing method is provided, including:

[0017] Receive project data to be processed;

[0018] Inputting the project data to be processed into a pre-trained data processing model, wherein the data processing model is obtained by training using the method provided in the fourth aspect;

[0019] Obtain a target classification label output by the data processing model based on the project data to be processed.

[0020] According to a fifth aspect of the embodiments of this specification, a training device for improving the generalization ability of a text analysis model is provided, comprising:

[0021] An acquisition module is configured to acquire a training sample set, wherein the training sample set includes training text;

[0022] A generating module is configured to generate a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes an environment label corresponding to each training text;

[0023] The training module is configured to train a text analysis model based on the training text and the environment label corresponding to the training text, wherein the text analysis model includes a classifier corresponding to the preset environment classification information.

[0024] According to a sixth aspect of the embodiments of this specification, a text analysis device is provided, including:

[0025] A receiving module, configured to receive a text to be analyzed;

[0026] An input module, configured to input the text to be analyzed into a pre-trained text analysis model, wherein the text analysis model is obtained by training using the above training method;

[0027] The acquisition module is configured to obtain the target classification label output by the text analysis model according to the text to be analyzed.

[0028] According to a seventh aspect of an embodiment of this specification, there is provided a training device for improving the generalization ability of a data processing model, comprising:

[0029] An acquisition module is configured to acquire a training sample set, wherein the training sample set includes training project data;

[0030] A generating module is configured to generate a target environment matrix based on the preset environment classification information of the training sample set, wherein the target environment matrix includes an environment label corresponding to each training item data;

[0031] The training module is configured to train a data processing model based on the training project data and the environment labels corresponding to the training project data, wherein the data processing model includes a classifier corresponding to the preset environment classification information.

[0032] According to an eighth aspect of the embodiments of this specification, there is provided a data processing device, including:

[0033] A receiving module, configured to receive project data to be processed;

[0034] An input module, configured to input the project data to be processed into a pre-trained data processing model, wherein the data processing model is obtained by training using the method provided in the fourth aspect;

[0035] The acquisition module is configured to obtain the target classification label output by the data processing model according to the project data to be processed.

[0036] According to a ninth aspect of the embodiments of this specification, a computing device is provided, including:

[0037] Memory and processor;

[0038] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the training method of the above-mentioned text analysis model or the steps of the text analysis method are implemented.

[0039] According to the tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned training method for improving the generalization ability of a text analysis model or a text analysis method or a training method for improving the generalization ability of a data processing model or a data processing method.

[0040] According to the eleventh aspect of the embodiments of the present specification, a computer program is provided, wherein, when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned training method for improving the generalization ability of a text analysis model or the text analysis method or the training method for improving the generalization ability of a data processing model or the data processing method.

[0041] An embodiment of the present specification implements a training method for improving the generalization ability of a text analysis model, including obtaining a training sample set, wherein the training sample set includes training texts; generating a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes environment labels corresponding to each training text; training a text analysis model according to the training texts and the environment labels corresponding to the training texts, wherein the text analysis model includes a classifier corresponding to the preset environment classification information. Through the training method of the text analysis model provided by the embodiment of the present specification, the training texts in the training sample set are first environmentally classified, and the training texts are automatically divided into corresponding environment classification labels, so as to eliminate the influence of hidden confounding factors on the training texts, and then the text analysis model is trained according to the training texts and the environment classification labels, so that when there is a distribution difference between the predicted distribution of the text analysis model in the prediction stage and the training distribution in the training stage, the prediction accuracy of the text analysis model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flowchart of a training method for improving the generalization ability of a text analysis model provided by an embodiment of this specification;

[0043] Figure 2 It is a schematic diagram of the architecture of a training method for a text analysis model provided by an embodiment of this specification;

[0044] Figure 3 is a processing flow chart of a training method for improving the generalization ability of a text analysis model provided by an embodiment of this specification;

[0045] Figure 4 is a flow chart of a text analysis method used in one embodiment of the present specification;

[0046] Figure 5 It is a structural schematic diagram of a training device for improving the generalization ability of a text analysis model provided by an embodiment of this specification;

[0047] Figure 6 is a structural schematic diagram of a text analysis device provided by an embodiment of this specification;

[0048] Figure 7 is a flow chart of a training method for improving the generalization ability of a data processing model provided by an embodiment of this specification;

[0049] Figure 8 is a flow chart of a data processing method provided by an embodiment of this specification;

[0050] Fig. 9 It is a structural schematic diagram of a training device for improving the generalization ability of a data processing model provided by an embodiment of the present specification;

[0051] Fig.10 A schematic diagram of the structure of a data processing device provided by an embodiment of this specification;

[0052] Fig.11 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0053] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0054] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0055] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0056] First, the terms involved in one or more embodiments of this specification are explained.

[0057] Independent and Identical Distribution (IID): Independent and identically distributed, the data belongs to the same distribution and are independent of each other.

[0058] Out-Of-Distribution (OOD): Out-of-distribution, the distribution of test data is different from the training data (IID).

[0059] Natural Language Understanding: Usually refers to tasks in natural language processing that require model understanding other than generation tasks, such as natural language inference.

[0060] Causal Inference: Determine and calculate the causal relationship and causal effect between different variables.

[0061] Confounder: A variable that affects both the input variable X and the output variable Y is called a confounder. Confounders introduce backdoor paths that affect the calculation of the causal effect of X on Y.

[0062] Intervention, Stratification: Intervention is an operation to remove the influence of confounding factors. Its core is to use data stratification to cut off the influence of confounding factors on input variables X, so that the causal effect of X on Y is only caused by X.

[0063] Empirical Risk Minimization, Invariant Risk Minimization: ERM is the optimization goal of most machine learning and deep learning algorithms: directly minimizing the loss function of the entire data, and its premise is that the data is independent and identically distributed. IRM minimizes the loss function that makes the prediction invariant. Optimizing according to IRM will improve the generalization ability of the model.

[0064] The task of natural language understanding has always attracted the attention of researchers, but most of the current methods focus on the scenario where the test data is the same as the training data, which masks the fragility of the model because neural network models are very easy to fit non-robust features during training. For example, in the training data of the natural language inference task, the probability that two sentences express the same meaning is usually proportional to the degree of word overlap in the two sentences. Therefore, the model will believe during training that if the degree of word overlap in two sentences is very high, the two sentences will express the same meaning. However, in fact, whether two sentences express the same meaning is determined by semantics. The effect of this natural language inference model will be greatly reduced on test data with different distributions.

[0065] Based on this, this specification provides a training method for improving the generalization ability of a text analysis model, a text analysis method, a training method for improving the generalization ability of a data processing model, and a data processing method. This specification also involves a training device for improving the generalization ability of a text analysis model, a text analysis device, a training device for improving the generalization ability of a data processing model, a data processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0066] See also Figure 1 , Figure 1 A flowchart of a training method for improving the generalization ability of a text analysis model provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0067] Step 102: Obtain a training sample set, wherein the training sample set includes training text.

[0068] Among them, the training sample set specifically refers to a collection of training samples used to train the text analysis model. In practical applications, the training samples include training texts. Specifically, the specific content of the training texts is related to the specific tasks of the text analysis model. For example, if the text analysis model is used for semantic understanding, the training text can be an article; if the text analysis model is used for natural language inference tasks, the training text can be a sentence pair, etc. In the embodiments provided in this specification, the specific content of the training text in the training sample set is not limited and is subject to practical applications.

[0069] In a specific implementation provided in this specification, taking the natural language inference task as an example, the text analysis model is used to analyze the probability that two sentences express the same meaning. At this time, each training text in the training sample set includes a sentence pair consisting of two sentences.

[0070] Step 104: Generate a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes an environment label corresponding to each training text.

[0071] The preset environment classification information is specifically used to indicate the number of environment divisions. For example, if there are 5 preset environment classification information, it means that 5 environment classifications are divided; if there are 2 preset environment classification information, it means that 2 environment classifications are divided.

[0072] The target environment matrix specifically refers to a matrix generated by the training sample set and the preset environment classification information for storing the probability of each training text under a certain environment label. For example, there are 3 environment classification information in total, and the probability of a certain text in the 3 environment classification information is 0.2, 0.5, and 0.3 respectively. Then the probability distribution of the text under each environment label can be expressed as (0.2, 0.5, 0.3). In practical applications, the target environment matrix can be expressed as M*N, where M is the number of training texts in the training sample set and N is the number of environment classification information.

[0073] In practical applications, a target environment matrix is ​​generated based on the training sample set and preset environment classification information, including S1042-S1044:

[0074] S1042: Generate an initial environment matrix based on the training sample set and preset environment classification information.

[0075] The initial environment matrix specifically refers to an initialized matrix generated based on the training sample information and the preset environment classification information, and the information in the initial environment matrix is ​​randomly generated. The initial environment matrix is ​​generated to store the probability information of each text information corresponding to each environment classification in the subsequent processing process.

[0076] Specifically, generating an initial environment matrix based on the training sample set and preset environment classification information includes:

[0077] Obtaining the number of training texts in the training sample set and the number of environments corresponding to the preset environment classification information;

[0078] An initial environment matrix is ​​generated according to the number of the training texts and the number of the environments.

[0079] Among them, the number of training texts in the training sample set is the total number of training texts in the training sample set, and the number of environments corresponding to the preset environment classification information is the number of pre-set environments. As shown in the above embodiment, if there are M training texts in the training sample set, the number of training texts is M; if there are N preset environment classification information, the number of environments is N.

[0080] After determining the number of training texts and the number of environments, an initial environment matrix can be generated. The initial environment matrix can be expressed as M*N, where the rows of the initial environment matrix represent the number M of training texts, and the list of the initial environment matrix presets the environment classification information.

[0081] In practical applications, different environmental classification information can be set from different dimensions. When there is more environmental classification information, it can be considered as a fine-grained environmental classification, such as 5 environmental classifications; when there is less environmental classification information, it can be considered as a coarse-grained environmental classification, such as 2 environmental classifications.

[0082] In this application context, the preset environment classification information includes first environment classification information and second environment classification information.

[0083] Get the number of environments corresponding to the preset environment classification information, including:

[0084] Obtaining the number of first environments corresponding to the first environment classification information, and obtaining the number of second environments corresponding to the second environment classification information, wherein the number of the first environments is greater than the number of the second environments,

[0085] Accordingly, an initial environment matrix is ​​generated according to the number of the training texts and the number of the environments, including:

[0086] A first initial environment matrix is ​​generated according to the number of the training texts and the first environment number, and a second initial environment matrix is ​​generated according to the number of the training texts and the second environment number.

[0087] In practical applications, the preset environment classification information can be divided into first environment classification information and second environment classification information. In the embodiments provided in this specification, the specific contents of the first environment classification information and the second environment classification information are not limited. It is only used to indicate that the environment classification of the first environment classification information is finer than that of the second environment classification information. For example, the number of environments in the first environment classification information is larger, and the number of environments in the second environment classification information is smaller. By distinguishing the environment classification information of different dimensions, it is convenient to mark the environment labels for the training text from different dimensions in the subsequent processing of the training text.

[0088] Based on this, to obtain the number of environments corresponding to the preset environment classification information, it is necessary to obtain the number of environments for each preset environment classification information respectively, and then generate the initial environment matrix corresponding to each preset environment classification information respectively according to the number of environments for each preset environment classification information.

[0089] For example, the number of first environments corresponding to the first environment classification information is 5, the number of second environments corresponding to the second environment classification information is 2, and the number of training texts is 500. Then, a first initial environment matrix can be generated according to the number of training texts 500 and the number of first environments 5, and the size of the first initial environment matrix is ​​500*5; a second initial environment matrix can be generated according to the number of training texts 500 and the number of second environments 2, and the size of the second initial environment matrix is ​​500*2.

[0090] S1044: Update the initial environment matrix according to the preset reference semantic model and the training sample set to generate a target environment matrix.

[0091] Among them, the preset reference semantic model refers to a neural network model used to determine the environmental classification probability corresponding to the training text. The preset reference semantic model is a pre-trained neural network model. The input of the preset reference semantic model is the training text. The environmental classification probability of the training text is determined according to the optimality of the training text on the preset reference semantic model, thereby updating the parameters in the initial environment matrix to obtain the updated target environment matrix.

[0092] Specifically, updating the initial environment matrix according to the preset reference semantic model and the training sample set includes:

[0093] Calculate the IRM loss function according to the training sample set and the preset reference semantic model;

[0094] The initial environment matrix is ​​updated according to the IRM loss function.

[0095] In practical applications, a BERT model can be pre-trained (BERT: Bidirectional Encoder Representation from Transformers), a deep bidirectional, unsupervised language representation model that is pre-trained using only a plain text corpus, and is used for pre-training technology for natural language processing (NLP). After the BERT model is trained, the training text is input into the reference semantic model, and the parameter value corresponding to each training text in the initial environment matrix is ​​updated according to the IRM loss function of each training text.

[0096] The IRM loss function (Invariant Risk Minimization) is a loss function that can make the prediction invariant. Optimizing according to IRM will improve the generalization ability of the model. It should be noted that in the embodiment of this specification, in the process of updating the initial environment matrix according to the IRM loss function, the maximization of the IRM loss function is used.

[0097] According to the IRM loss value of the IRM loss function corresponding to each training text, the matrix element corresponding to the training text in the initial environment matrix is ​​updated. After each training text in the training sample set is processed through the above steps, the target environment matrix can be obtained. In the target environment matrix, each element represents the probability of the corresponding training text in the corresponding environment classification.

[0098] It can be learned from the above embodiments of the present specification that the preset environment classification information may further include first environment classification information and second environment classification information. In this scenario, the initial environment matrix is ​​updated according to the preset reference semantic model and the training sample set to generate a target environment matrix, including:

[0099] Update the first initial environment matrix according to the preset reference semantic model and the training sample set to generate a first target environment matrix;

[0100] The second initial environment matrix is ​​updated according to the preset reference semantic model and the training sample set to generate a second target environment matrix.

[0101] Specifically, the first initial environment matrix and the second initial environment matrix may be updated respectively according to the preset reference semantic model and the training sample set, so as to obtain the first target environment matrix and the second target environment matrix.

[0102] In practical applications, in order to accurately classify the training text into different environmental categories, the same preset reference semantic model needs to be used. Only when the same reference semantic model is used to extract the same training text can the consistency of the extracted feature information be guaranteed. Specifically, after pre-training a reference semantic model, the model parameters of the reference semantic model are fixed. The reference semantic model is only used to generate the target environment matrix. After obtaining the target environment matrix, the reference semantic model can no longer be used.

[0103] For example, the pre-trained reference semantic model is a BERT model, which includes an embedding layer, an encoder, and an output layer. The BERT model is pre-trained in advance to obtain a reference semantic model based on the BERT model. The first initial environment matrix and the second initial environment matrix are updated according to the reference semantic model and the training sample set to obtain a first target environment matrix and a second target environment matrix.

[0104] Step 106: training a text analysis model according to the training text and the environment label corresponding to the training text, wherein the text analysis model includes a classifier corresponding to the preset environment classification information.

[0105] Among them, the text analysis model specifically refers to an artificial intelligence model used to analyze text. After obtaining the environmental labels corresponding to the training text, the text analysis model can be trained according to the training text and the environmental labels corresponding to the training text, so that the text classification model can extract the feature information of the training text according to the environmental labels.

[0106] The text analysis model provided in the embodiments of this specification includes a feature extraction module and a classification module. The classification module includes a classifier corresponding to each environmental classification information. The training texts in the training sample set share the same feature extraction module. For each environmental classification, the feature extraction module and the classifier corresponding to the environmental classification are the text analysis model of the environmental classification.

[0107] In practical applications, training a text analysis model according to the training text and the environment label corresponding to the training text includes:

[0108] Inputting the training text into the text analysis model to obtain a predicted classification label output by the text analysis model;

[0109] Calculate a model loss value according to the predicted classification label and the environment label;

[0110] The model parameters of the text analysis model are adjusted according to the model loss value, and the text analysis model is continuously trained until a training stop condition is reached.

[0111] Among them, the training text is input into the text classification model for text recognition. The text classification model will extract the feature information of the training text and generate a category vector based on the feature information. By analyzing the category vector of the training text, the predicted category label of the training text will be output. The difference between the predicted category label and the environment label is calculated and the model loss value is determined. The model parameters of the text analysis model are adjusted through back propagation, and the new text analysis model is used to continue training the text analysis model until the model training stop condition is reached.

[0112] During the training process of the text classification model, the CLS label is added to the training text and input into the text classification model. The feature extraction module of the text classification model extracts the CLS vector of the training text. The CLS vector is used for downstream classification tasks. The CLS vector serves as the semantic identifier of the training text and can represent the semantic type of the training text. The predicted classification label of the training text is output based on the CLS vector.

[0113] The model loss value is calculated according to the predicted classification label output by the model and the environmental label corresponding to the training text. The loss function for calculating the model loss value can be a cross entropy loss function, a square loss function, a 0-1 loss function, etc. in practical applications. In the embodiments provided in this specification, there is no limitation on the choice of loss function. Since the data sets in the embodiments of this specification are all classified data sets, the loss function preferably uses a cross entropy loss function.

[0114] After obtaining the model loss value, the model parameters of the text analysis model are adjusted by back propagation according to the model loss value, and the text analysis model is continuously trained until the model training stop condition is reached.

[0115] Specifically, in actual applications, the model training stopping conditions specifically include that the model loss value is less than a preset threshold and / or the number of training rounds reaches a preset number of rounds. The model training stopping conditions are not limited in this specification and are subject to actual application.

[0116] In the above embodiments provided in this specification, it is mentioned that by distinguishing the environmental classification information of different dimensions, it is convenient to mark the environmental labels for the training text from different dimensions in the subsequent processing of the training text. Therefore, the first target environment matrix and the second target environment matrix are obtained. The environmental labels corresponding to the first target environment matrix and the second target environment matrix are different. When there are at least two target environment matrices, the text analysis model needs to be trained according to the training text and the environmental labels in each target environment matrix.

[0117] Specifically, the text analysis model includes an encoder and a classifier, wherein the number of the classifiers corresponds to the number of the environments;

[0118] Accordingly, training a text analysis model according to the training text and the environment label corresponding to the training text includes:

[0119] Training a text analysis model to be trained according to the training text and the environment labels corresponding to the first target environment matrix, updating encoder parameters and first classifier parameters of the text analysis model to be trained, and obtaining an initial text analysis model, wherein the number of the first classifiers corresponds to the number of the first environments;

[0120] The initial text analysis model is trained according to the environment labels corresponding to the training text and the second target environment matrix, and the second classifier parameters of the initial text analysis model are updated to obtain a text analysis model, wherein the number of the second classifiers corresponds to the number of the second environments.

[0121] In practical applications, the encoder of the text analysis model is the feature extraction module, and the classifier is the classifier corresponding to the environmental classification information. There are as many classifiers as there are environments in the environmental classification information. For example, if there are 5 environmental classification information in the first target environment matrix, there are 5 first classifiers; if there are 2 environmental classification information in the second target environment matrix, there are 2 second classifiers.

[0122] Based on this, the text analysis model can be first trained with the first target environment matrix, preferably the text analysis model to be trained is obtained, and the text analysis model to be trained is preliminarily trained according to the training text and the environment label corresponding to the first target environment matrix. During the preliminary training process, the encoder and the first classifier parameters in the text analysis model to be trained are updated, and after the model training stop condition is reached, the initial text analysis model is obtained. The initial text analysis model specifically refers to the text analysis model generated by training according to the training text and the environment label corresponding to the first target environment matrix.

[0123] The initial text analysis model only has the ability to analyze the first target environment, but not the second target environment. Therefore, the initial text analysis model needs to be further trained. When further training the initial text analysis model, the training text in the training sample set is still used, combined with the environment label corresponding to the second target environment matrix to further train the initial text analysis model.

[0124] In practical applications, in order to prevent the initial text analysis model from forgetting the feature extraction capabilities learned previously during further training of the initial text analysis model based on the environment labels corresponding to the training text and the second target environment matrix, it is necessary to fix the encoder parameters and the first classifier parameters in the initial text analysis model, and only update the second classifier parameters based on the environment labels corresponding to the training text and the second target environment matrix. Until the model training stop condition is reached, a trained text analysis model is obtained.

[0125] It should be noted that in order to reduce the large gap between classifier parameters corresponding to different environments, regularization terms are added during the model training process to reduce the variance of the weights of different classifier parameters.

[0126] The training method for improving the generalization ability of the text analysis model provided in this specification is to construct a target environment matrix to store different environment labels for each training text in the training sample set, and automatically stratify the training text into different environments, thereby solving the adverse effects of confounding factors on data stratification. The text analysis model is then trained based on the environment labels and training texts in the target environment matrix, and the training process of the text analysis model is intervened by the environment labels, so that when there is a distribution difference between the predicted distribution of the text analysis model in the prediction stage and the training distribution in the training stage, the prediction accuracy of the text analysis model is improved.

[0127] The following combination Figure 2 and attached Figure 3 Taking the application of the training method for improving the generalization ability of the text analysis model provided in this specification in two environmental classifications as an example, the training method for improving the generalization ability of the text analysis model is further explained. Figure 2 A schematic diagram of the architecture of a training method for improving the generalization ability of a text analysis model provided by an embodiment of this specification is shown.

[0128] like Figure 2 As shown in a, the first initial environment matrix is ​​updated according to the training text and the reference sentence model in the training set to obtain the first target environment matrix, wherein the first target environment matrix has 5 environment classifications; the second initial environment matrix is ​​updated according to the training text and the reference sentence model in the training set to obtain the second target environment matrix, wherein the second target environment matrix has 2 environment classifications.

[0129] like Figure 2 As shown in b, the text analysis model is preliminarily trained according to the training texts in the training set and the first target environment matrix. Specifically, the parameters of the encoder and the first classifier in the text analysis model are updated, that is, the parameters of the part in the dotted box in the text analysis model are updated.

[0130] After initial training is complete. Figure 2 c in the training combination, and then further train the text classification model according to the training text and the second target environment matrix. Specifically, the encoder parameters and the model parameters of the first classifier 1-first classifier 5 need to be fixed, and the training text is feature extracted by the encoder. Then, the parameters of the second classifier 1 and the second classifier 2 are updated through the environment labels of the second target environment matrix, that is, the parameters of the virtual box part in c are updated. At this point, the text analysis model training is completed.

[0131] Figure 3 A processing flow chart of a training method for improving the generalization ability of a text analysis model provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0132] Step 302: Obtain the number of first environments corresponding to the first environment classification information, and obtain the number of second environments corresponding to the second environment classification information.

[0133] Step 304: Generate a first initial environment matrix according to the number of the training texts and the first environment number, and generate a second initial environment matrix according to the number of the training texts and the second environment number.

[0134] Step 306: Update the first initial environment matrix according to the preset reference semantic model and the training sample set to generate a first target environment matrix.

[0135] Step 308: Update the second initial environment matrix according to the preset reference semantic model and the training sample set to generate a second target environment matrix.

[0136] Step 310: Train a text analysis model according to the training sample set and the environment labels corresponding to the first target environment matrix, and update encoder parameters and first classifier parameters in the text analysis model.

[0137] Step 312: Continue to train the text analysis model according to the training sample set and the environment label corresponding to the second target environment matrix, and update the second classifier parameters in the text analysis model.

[0138] See also Figure 4 , Figure 4 A flowchart of a text analysis method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0139] Step 402: Receive text to be analyzed.

[0140] Step 404: input the text to be analyzed into a pre-trained text analysis model.

[0141] The text analysis model is obtained by training using the training method provided in the above embodiment.

[0142] Step 406: Obtain a target classification label output by the text analysis model based on the text to be analyzed.

[0143] Through the text analysis method provided in this specification, the text to be analyzed is analyzed by a trained text analysis model. The text analysis model trained by the training method provided in the above embodiment solves the problems of invisibility and multi-factor nature of confounding factors in natural language understanding tasks, and improves the accuracy of the text analysis model.

[0144] Corresponding to the above-mentioned training method embodiment for improving the generalization ability of a text analysis model, this specification also provides a training device embodiment for improving the generalization ability of a text analysis model. Figure 5 FIG. 1 is a schematic diagram showing a structure of a training device for improving the generalization ability of a text analysis model provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0145] An acquisition module 502 is configured to acquire a training sample set, wherein the training sample set includes training text;

[0146] A generating module 504 is configured to generate a target environment matrix based on the training sample set and preset environment classification information, wherein the target environment matrix includes an environment label corresponding to each training text;

[0147] The training module 506 is configured to train a text analysis model according to the training text and the environment label corresponding to the training text, wherein the text analysis model includes a classifier corresponding to the preset environment classification information.

[0148] Optionally, the generating module 504 is further configured to:

[0149] Generate an initial environment matrix based on the training sample set and preset environment classification information;

[0150] The initial environment matrix is ​​updated according to the preset reference semantic model and the training sample set to generate a target environment matrix.

[0151] Optionally, the generating module 504 is further configured to:

[0152] Obtaining the number of training texts in the training sample set and the number of environments corresponding to the preset environment classification information;

[0153] An initial environment matrix is ​​generated according to the number of the training texts and the number of the environments.

[0154] Optionally, the preset environment classification information includes first environment classification information and second environment classification information;

[0155] The generating module 504 is further configured to:

[0156] Obtaining the number of first environments corresponding to the first environment classification information, and obtaining the number of second environments corresponding to the second environment classification information, wherein the number of the first environments is greater than the number of the second environments;

[0157] A first initial environment matrix is ​​generated according to the number of the training texts and the first environment number, and a second initial environment matrix is ​​generated according to the number of the training texts and the second environment number.

[0158] Optionally, the generating module 504 is further configured to:

[0159] Update the first initial environment matrix according to the preset reference semantic model and the training sample set to generate a first target environment matrix;

[0160] The second initial environment matrix is ​​updated according to the preset reference semantic model and the training sample set to generate a second target environment matrix.

[0161] Optionally, the text analysis model includes an encoder and a classifier, wherein the number of the classifiers corresponds to the number of the environments;

[0162] The training module 506 is further configured to:

[0163] Training a text analysis model to be trained according to the training text and the environment labels corresponding to the first target environment matrix, updating encoder parameters and first classifier parameters of the text analysis model to be trained, and obtaining an initial text analysis model, wherein the number of the first classifiers corresponds to the number of the first environments;

[0164] The initial text analysis model is trained according to the environment labels corresponding to the training text and the second target environment matrix, and the second classifier parameters of the initial text analysis model are updated to obtain a text analysis model, wherein the number of the second classifiers corresponds to the number of the second environments.

[0165] Optionally, the generating module 504 is further configured to:

[0166] Calculate the IRM loss function according to the training sample set and the preset reference semantic model;

[0167] The initial environment matrix is ​​updated according to the IRM loss function.

[0168] Optionally, the training module 506 is further configured to:

[0169] Inputting the training text into the text analysis model to obtain a predicted classification label output by the text analysis model;

[0170] Calculate a model loss value according to the predicted classification label and the environment label;

[0171] The model parameters of the text analysis model are adjusted according to the model loss value, and the text analysis model is continuously trained until a training stop condition is reached.

[0172] Optionally, the training module 506 is further configured to:

[0173] The model loss value is less than a preset threshold; and / or,

[0174] The training rounds reach the preset rounds.

[0175] The training device for improving the generalization ability of the text analysis model provided in this specification constructs a target environment matrix to store different environment labels for each training text in the training sample set, automatically stratifies the training text into different environments, and solves the adverse effects of confounding factors on data stratification. The text analysis model is then trained based on the environment labels and training texts in the target environment matrix, and the training process of the text analysis model is intervened by the environment labels, thereby improving the prediction accuracy of the text analysis model.

[0176] The above is a schematic scheme of a training device for improving the generalization ability of a text analysis model in this embodiment. It should be noted that the technical scheme of the training device for improving the generalization ability of a text analysis model and the technical scheme of the training method for improving the generalization ability of a text analysis model belong to the same concept, and the details of the technical scheme of the training device for improving the generalization ability of a text analysis model that are not described in detail can all be referred to the description of the technical scheme of the training method for the text analysis model.

[0177] Corresponding to the above text analysis method embodiment, this specification also provides a text analysis device embodiment, Figure 6 FIG. 1 shows a schematic diagram of a text analysis device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0178] A receiving module 602 is configured to receive a text to be analyzed;

[0179] An input module 604 is configured to input the text to be analyzed into a pre-trained text analysis model, wherein the text analysis model is obtained by training using the above training method;

[0180] The acquisition module 606 is configured to acquire the target classification label output by the text analysis model according to the text to be analyzed.

[0181] Through the text analysis device provided in this specification, the text to be analyzed is analyzed by using a trained text analysis model. The text analysis model trained by the training method provided in the above embodiment solves the problems of invisibility and multi-factor nature of confounding factors in natural language understanding tasks, and improves the accuracy of the text analysis model.

[0182] The above is a schematic scheme of a text analysis device of this embodiment. It should be noted that the technical scheme of the text analysis device and the technical scheme of the above text analysis method belong to the same concept, and the details not described in detail in the technical scheme of the text analysis device can be referred to the description of the technical scheme of the above text analysis method.

[0183] See also Figure 7 , Figure 7 A flowchart of a training method for improving the generalization ability of a data processing model provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0184] Step 702: Acquire a training sample set, wherein the training sample set includes training item data.

[0185] Among them, the training project data includes text data, image data, voice data, etc.

[0186] Step 704: Generate a target environment matrix based on the preset environment classification information of the training sample set, wherein the target environment matrix includes an environment label corresponding to each training item data.

[0187] Step 706: Training a data processing model based on the training project data and the environment labels corresponding to the training project data, wherein the data processing model includes a classifier corresponding to the preset environment classification information.

[0188] The training method for improving the generalization ability of the data processing model provided in this specification, by constructing a target environment matrix, stores different environmental labels for each training project data in the training sample set, and automatically stratifies the training project data into different environments, thereby solving the adverse effects of confounding factors on data stratification. The data processing model is then trained based on the environmental labels and training project data in the target environment matrix, and the training process of the data processing model is intervened through the environmental labels, so that when there is a distribution difference between the predicted distribution of the data processing model in the prediction stage and the training distribution in the training stage, the prediction accuracy of the data processing model is improved.

[0189] See also Figure 8 , Figure 8 A flow chart of a data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0190] Step 802: Receive project data to be processed.

[0191] The project data to be processed include text data, image data, voice data, etc.

[0192] Step 804: Input the project data to be processed into a pre-trained data processing model.

[0193] Wherein, the data processing model is obtained by training using the training method provided in the above embodiment.

[0194] Through the data processing method provided in this specification, the project data to be processed is analyzed by a trained data processing model. The data processing model obtained by training with the training method provided in the above embodiment solves the problems of invisibility and multi-factor nature of confounding factors in natural language understanding tasks, and improves the accuracy of the data processing model.

[0195] Step 806: Obtain the target classification label output by the data processing model based on the project data to be processed.

[0196] Corresponding to the above-mentioned training method embodiment for improving the generalization ability of a data processing model, this specification also provides a training device embodiment for improving the generalization ability of a data processing model. Fig. 9 FIG. 1 is a schematic diagram showing a structure of a training device for improving the generalization ability of a data processing model provided by an embodiment of the present specification. Fig. 9 As shown, the device comprises:

[0197] An acquisition module 902 is configured to acquire a training sample set, wherein the training sample set includes training project data;

[0198] A generating module 904 is configured to generate a target environment matrix based on the preset environment classification information of the training sample set, wherein the target environment matrix includes an environment label corresponding to each training item data;

[0199] The training module 906 is configured to train a data processing model based on the training project data and the environment labels corresponding to the training project data, wherein the data processing model includes a classifier corresponding to the preset environment classification information.

[0200] The training device for improving the generalization ability of the data processing model provided in this specification stores different environmental labels for each training project data in the training sample set by constructing a target environment matrix, and automatically stratifies the training project data into different environments, thereby solving the adverse effects of confounding factors on data stratification. The data processing model is then trained based on the environmental labels and training project data in the target environment matrix, and the training process of the data processing model is intervened by the environmental labels, thereby improving the prediction accuracy of the data processing model.

[0201] Corresponding to the above data processing method embodiment, this specification also provides a data processing device embodiment. Fig.10 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Fig.10As shown, the device comprises:

[0202] The receiving module 1002 is configured to receive the project data to be processed;

[0203] An input module 1004 is configured to input the project data to be processed into a pre-trained data processing model;

[0204] Wherein, the data processing model is obtained by training using the training method provided in the above embodiment;

[0205] The acquisition module 1006 is configured to acquire the target classification label output by the data processing model according to the project data to be processed.

[0206] Fig.11 The block diagram of a computing device 1100 according to one embodiment of the present specification is shown. The components of the computing device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0207] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0208] In one embodiment of the present specification, the above components of the computing device 1100 and Fig.11 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.11 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0209] The computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 1100 may also be a mobile or stationary server.

[0210] Among them, the processor 1120 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned data processing method when executed by the processor. The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device belongs to the same concept as the above-mentioned training method of the text analysis model or the technical scheme of the text analysis method. The details not described in detail in the technical scheme of the computing device can be found in the description of the above-mentioned training method of the text analysis model or the technical scheme of the text analysis method.

[0211] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the training method of the above-mentioned text analysis model or the steps of the text analysis method.

[0212] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the training method of the text analysis model or the text analysis method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the training method or text analysis method for improving the generalization ability of the text analysis model or the training method or data processing method for improving the generalization ability of the data processing model.

[0213] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned training method for improving the generalization ability of a text analysis model or the text analysis method or the training method for improving the generalization ability of a data processing model or the data processing method.

[0214] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the training method for improving the generalization ability of the text analysis model or the text analysis method or the training method for improving the generalization ability of the data processing model or the data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the training method for improving the generalization ability of the text analysis model or the text analysis method or the training method for improving the generalization ability of the data processing model or the data processing method.

[0215] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0216] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0217] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0218] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0219] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A training method for improving the generalization ability of a text analysis model, comprising: Acquire a training sample set, wherein the training sample set includes training text; Generate an initial environment matrix based on the training sample set and the preset environment classification information; update the initial environment matrix according to the preset reference semantic model and the training sample set to generate a target environment matrix, wherein the target environment matrix includes an environment label corresponding to each training text, the preset environment classification information includes first environment classification information and second environment classification information, the initial environment matrix includes a first initial environment matrix and a second initial environment matrix, the first initial environment matrix is ​​generated according to the number of the training texts and the number of first environments corresponding to the first environment classification information, and the second initial environment matrix is ​​generated according to the number of the training texts and the number of second environments corresponding to the second environment classification information; A text analysis model is trained according to the training text and the environment label corresponding to the training text, wherein the text analysis model includes a classifier corresponding to the preset environment classification information.

2. The method according to claim 1, generating an initial environment matrix based on the training sample set and preset environment classification information, comprising: Obtaining the number of training texts in the training sample set and the number of environments corresponding to the preset environment classification information; An initial environment matrix is ​​generated according to the number of the training texts and the number of the environments.

3. The method according to claim 2, obtaining the number of environments corresponding to the preset environment classification information, comprises: Obtaining the number of first environments corresponding to the first environment classification information, and obtaining the number of second environments corresponding to the second environment classification information, wherein the number of the first environments is greater than the number of the second environments; Accordingly, an initial environment matrix is ​​generated according to the number of the training texts and the number of environments, including: A first initial environment matrix is ​​generated according to the number of the training texts and the first environment number, and a second initial environment matrix is ​​generated according to the number of the training texts and the second environment number.

4. The method according to claim 3, updating the initial environment matrix according to the preset reference semantic model and the training sample set to generate the target environment matrix, comprises: Update the first initial environment matrix according to the preset reference semantic model and the training sample set to generate a first target environment matrix; The second initial environment matrix is ​​updated according to the preset reference semantic model and the training sample set to generate a second target environment matrix.

5. The method of claim 4, wherein the text analysis model comprises an encoder and a classifier, wherein: The number of the classifiers corresponds to the number of the environments; Training a text analysis model according to the training text and the environment label corresponding to the training text includes: Training a text analysis model to be trained according to the training text and the environment labels corresponding to the first target environment matrix, updating encoder parameters and first classifier parameters of the text analysis model to be trained, and obtaining an initial text analysis model, wherein the number of the first classifiers corresponds to the number of the first environments; The initial text analysis model is trained according to the environment labels corresponding to the training text and the second target environment matrix, and the second classifier parameters of the initial text analysis model are updated to obtain a text analysis model, wherein the number of the second classifiers corresponds to the number of the second environments.

6. The method according to claim 1, updating the initial environment matrix according to the preset reference semantic model and the training sample set, comprising: Calculate the IRM loss function according to the training sample set and the preset reference semantic model; The initial environment matrix is ​​updated according to the IRM loss function.

7. The method according to claim 1, training a text analysis model according to the training text and the environment label corresponding to the training text, comprising: Inputting the training text into the text analysis model to obtain a predicted classification label output by the text analysis model; Calculate a model loss value according to the predicted classification label and the environment label; The model parameters of the text analysis model are adjusted according to the model loss value, and the text analysis model is continuously trained until a training stop condition is reached.

8. A training method for improving the generalization ability of a data processing model, comprising: Acquire a training sample set, wherein the training sample set includes training project data; Generate an initial environment matrix based on the training sample set and the preset environment classification information; update the initial environment matrix according to the preset reference semantic model and the training sample set to generate a target environment matrix, wherein the target environment matrix includes an environment label corresponding to each training project data, the preset environment classification information includes first environment classification information and second environment classification information, the initial environment matrix includes a first initial environment matrix and a second initial environment matrix, the first initial environment matrix is ​​generated according to the number of the training project data and the number of first environments corresponding to the first environment classification information, and the second initial environment matrix is ​​generated according to the number of the training project data and the number of second environments corresponding to the second environment classification information; A data processing model is trained according to the training project data and the environment labels corresponding to the training project data, wherein the data processing model includes a classifier corresponding to the preset environment classification information.

9. A text analysis method, comprising: receiving text to be analyzed; Inputting the text to be analyzed into a pre-trained text analysis model, wherein the text analysis model is trained by the method according to any one of claims 1 to 7; Obtain a target classification label output by the text analysis model based on the text to be analyzed.

10. A data processing method, comprising: Receive project data to be processed; Inputting the project data to be processed into a pre-trained data processing model, wherein the data processing model is obtained by training according to the method of claim 8; Obtain a target classification label output by the data processing model based on the project data to be processed.

11. A text analysis device, comprising: A receiving module, configured to receive a text to be analyzed; An input module, configured to input the text to be analyzed into a pre-trained text analysis model, wherein the text analysis model is obtained by training using the training method according to any one of claims 1 to 7; The acquisition module is configured to obtain the target classification label output by the text analysis model according to the text to be analyzed.

12. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1-7 or 8 or 9 or 10 are implemented.

13. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 7 or 8 or 9 or 10.

Images