A training method for text comprehension model in the power field based on knowledge transfer

Abstract

The invention discloses a training method for a text understanding model in the electric power field based on knowledge transfer, and belongs to the technical field of electric power knowledge text understanding. The training method for the text understanding model in the electric power field based on knowledge transfer comprises the following steps: obtaining text data related to the electric power field, constructing a data set D1 and a data set D2, obtaining a first model fine-tuned by D1 and a second model fine-tuned by D2, initializing a knowledge transfer model KTM using the second model fine-tuned by D2, inputting training samples in the data set D1 into the knowledge transfer model KTM and the first model fine-tuned by D1 for training respectively, outputting predicted values, soft labels and true labels of the samples, calculating loss values ​​according to the predicted values, soft labels and true labels of the samples, and then allowing gradients to be back-propagated back to the knowledge transfer model KTM, thereby avoiding the problems inherent in large models such as many parameters, large memory usage, long training time, slow prediction speed, etc., and achieving a win-win situation in terms of resource consumption and performance.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power knowledge text comprehension, and in particular to a training method for an electric power field text comprehension model based on knowledge transfer. Background Art

[0002] The texts in the power field have the following three characteristics: first, they are highly professional and usually contain a large number of professional terms and concepts; second, structured and unstructured data coexist, including not only structured tabular data, but also unstructured text reports, fault descriptions, operation logs, etc.; third, multi-source data fusion, the data comes from different sources, such as equipment monitoring data, maintenance records, accident reports, etc. These data types are diverse and require comprehensive analysis.

[0003] There are the following problems when using general large language models (LLMs) directly for text understanding in the power field:

[0004] 1. Many parameters, large memory usage, long training time and slow prediction speed;

[0005] 2. Directly using a general large language model to train actual power production data poses the risk of sensitive data being leaked when uploaded to the cloud;

[0006] 3. General large-scale language models lack specific knowledge in the power sector and are not effective;

[0007] 4. General large-scale language models require large computing resources and are costly.

[0008] Therefore, it is necessary to provide a training method for a text comprehension model in the power field based on knowledge transfer, aiming to solve the above problems. Summary of the invention

[0009] In view of the deficiencies in the prior art, an embodiment of the present invention aims to provide a training method for a text comprehension model in the electric power field based on knowledge transfer, so as to solve the problems in the above-mentioned background technology.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] A training method for a text understanding model in the electric power field based on knowledge transfer includes the following steps:

[0012] Step S1, obtaining text data related to the power field, constructing a data set D1 and a data set D2, fine-tuning the first model according to the data set D1 to obtain the first model fine-tuned by D1, and fine-tuning the second model according to the data set D2 to obtain the second model fine-tuned by D2;

[0013] Step S2, using the second model fine-tuned by D2 to initialize the knowledge transfer model KTM;

[0014] Step S3, input the training samples in the data set D1 into the knowledge transfer model KTM and the first model fine-tuned by D1 for training, and output the predicted value, soft label and true label of the sample;

[0015] Step S4: Calculate the loss value based on the predicted value, soft label and true label of the sample, and then let the gradient back propagate back to the knowledge transfer model KTM.

[0016] As a further solution of the present invention, the data set D1 in step S1 is obtained by traversing multiple data sets and randomly dividing and merging data, and the data set D2 is a professional data set in the power field.

[0017] As a further solution of the present invention, the training in step S3 includes: given an m-classification data set in the form of samples (X, Y), by minimizing the cross entropy loss function To train the classifier,

[0018] ,

[0019] ,

[0020] Among them, CE is the first model fine-tuned by D1, is the class probability distribution, is the logits of the model, F is the true label, and k represents the training batch.

[0021] As a further solution of the present invention, the predicted value of the sample in step S3 includes a soft predicted value and hard prediction value , the soft prediction value The calculation formula is:

[0022] ,

[0023] The hard prediction value The calculation formula is:

[0024] ,

[0025] Among them, the and are the logits of the first model fine-tuned by D1 and the second model fine-tuned by D2, is a hyperparameter.

[0026] As a further solution of the present invention, the loss function when the second model fine-tuned by D2 learns the output of the first model fine-tuned by D1 is Defined as:

[0027] ,

[0028] Wherein, KD is the second model fine-tuned by D2, q k is the hard prediction value for the kth batch of training, u k is the soft prediction value of the kth training batch.

[0029] As a further solution of the present invention, the knowledge transfer model KTM The loss function is:

[0030] ,

[0031] in, is the sum of the parameters of KTM, |·| represents the number of elements in the set, KD is the second model fine-tuned by D2, q is the hard prediction value, u is the soft prediction value, D1 is the data set to be trained, v is the starting position of the knowledge text, l is the ending position of the knowledge text, j represents the jth word in the knowledge text, L is the text length, N is the number of categories, a is the subscript of the category array N, and w is a value of the category array N, that is, w=N[a].

[0032] As a further solution of the present invention, the optimization goal of the knowledge transfer model KTM is and Weighted sum:

[0033] ,

[0034] in, , is the weight.

[0035] In summary, compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0036] The present invention proposes a pre-trained model with a smaller scale that is suitable for text understanding in the power field. The method avoids the problems inherent in large models, such as large number of parameters, large memory usage, long training time and slow prediction speed. It avoids the risk of sensitive data being leaked when directly using a general large-scale language model to train actual power production data. The application makes use of knowledge distillation, joint learning of similar data sets and fine-tuning of similar tasks to make the performance of the small model comparable to or surpass that of the large model, thereby achieving a win-win situation in resource consumption and performance.

[0037] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of an embodiment of the invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0041] In one embodiment of the present invention, see Figure 1 The training method of the electric power field text understanding model based on knowledge transfer comprises the following steps:

[0042] Step S1, obtaining text data related to the power field, constructing a data set D1 and a data set D2, fine-tuning the first model according to the data set D1 to obtain the first model fine-tuned by D1, and fine-tuning the second model according to the data set D2 to obtain the second model fine-tuned by D2;

[0043] Step S2, using the second model fine-tuned by D2 to initialize the knowledge transfer model KTM;

[0044] Step S3, input the training samples in the data set D1 into the knowledge transfer model KTM and the first model fine-tuned by D1 for training, and output the predicted value, soft label and true label of the sample;

[0045] Step S4: Calculate the loss value based on the predicted value, soft label and true label of the sample, and then let the gradient back propagate back to the knowledge transfer model KTM.

[0046] In this embodiment, the data set D1 in step S1 is a data set that traverses multiple data sets and randomly divides and merges data, and the data set D2 is a professional data set in the power field;

[0047] The training in step S3 includes: given an m-classification data set in the form of samples (X, Y), by minimizing the cross entropy loss function To train the classifier,

[0048] ,

[0049] ,

[0050] Among them, CE is the first model fine-tuned by D1, is the class probability distribution, is the logits of the model, F is the true label, and k represents the training batch. If the above training process is the process of the classifier learning according to the true label Y, then knowledge distillation also includes the student model according to the category probability distribution output by the teacher model The learning process; similar to Y, in many cases, the probability value of Y in the correct category will be very high, close to 1, and the probability of other categories will be 0;

[0051] The predicted value of the sample in step S3 includes a soft prediction value and hard prediction value , the soft prediction value The calculation formula is:

[0052] ,

[0053] The hard prediction value The calculation formula is:

[0054] ,

[0055] Among them, the and are the logits of the first model fine-tuned by D1 and the second model fine-tuned by D2, is a hyperparameter. The larger the value of T, The softer it is, the more information the teacher model provides; therefore, is called a soft label, and the corresponding true label Y is called a hard label; u is called a soft prediction value, It is called hard prediction value;

[0056] The loss function when the second model fine-tuned by D2 learns the output of the first model fine-tuned by D1 Defined as:

[0057] ,

[0058] Wherein, KD is the second model fine-tuned by D2, q k is the hard prediction value for the kth batch of training, u k is the soft prediction value for training batch k;

[0059] Through the above formula, we can get the D1 soft label q * D1 ,D2 soft labelq * D2 , and the knowledge transfer model KTM soft prediction value ,by For example:

[0060] ,

[0061] Where T is temperature, W is weight matrix, b1 is weight deviation value, is the real temperature;

[0062] After replacing the above soft labels and soft prediction values, calculate the true value and probability distribution , we can get the knowledge transfer model KTM Loss function:

[0063] ,

[0064] in, is the sum of the parameters of KTM, |·| represents the number of elements in the set, KD is the second model fine-tuned by D2, q is the hard prediction value, u is the soft prediction value, D1 is the data set to be trained, v is the starting position of the knowledge text, l is the ending position of the knowledge text, j represents the jth word in the knowledge text, L is the text length, N is the number of categories, a is the subscript of the category array N, and w is a value of the category array N, that is, w=N[a].

[0065] In summary, the optimization goal of the knowledge transfer model KTM is and Weighted sum:

[0066] ;

[0067] in, , The training process iterates epochmax rounds. Before each round, the training set is shuffled. , to ensure the randomness of the training samples, and then, from the disordered training set Select a mini-batch from: , whose samples come from the same dataset ;Will Enter the dataset separately The fine-tuned large model and the knowledge transfer model KTM output the soft labels and soft and hard prediction values ​​required to calculate the loss value; then, the loss is calculated according to different data sets. The loss value ; Finally, the gradients of each parameter are calculated by the loss value and the knowledge transfer model KTM is updated.

[0068] In summary, this application proposes a small-scale pre-trained model suitable for text understanding in the power field, which avoids the problems inherent in large models such as large number of parameters, large memory usage, long training time, slow prediction speed, etc.; it makes the performance of the small model comparable to or even exceeds that of the large model by comprehensively using knowledge distillation, joint learning of similar data sets, and fine-tuning of similar tasks, achieving a win-win situation in resource consumption and performance.

[0069] The knowledge transfer method combines the characteristics of text understanding, fully utilizes the common characteristics between data sets to expand training samples, and combines similar text understanding tasks to allow the model to learn how to understand the text in advance and maximize the performance of the small model.

[0070] Compared with large models, the knowledge transfer model KTM has fewer parameters, lower memory usage, significantly shorter training time, and faster prediction speed. At the same time, through customized training for specific tasks in the power field, its text comprehension ability performs better than general large models.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for a text comprehension model in the electric power field based on knowledge transfer, characterized in that: The following steps are involved: Step S1, obtaining text data related to the power field, constructing a data set D1 and a data set D2, fine-tuning the first model according to the data set D1 to obtain the first model fine-tuned by D1, and fine-tuning the second model according to the data set D2 to obtain the second model fine-tuned by D2; Step S2, using the second model fine-tuned by D2 to initialize the knowledge transfer model KTM; Step S3, input the training samples in the data set D1 into the knowledge transfer model KTM and the first model fine-tuned by D1 for training, and output the predicted value, soft label and true label of the sample; Step S4: Calculate the loss value based on the predicted value, soft label and true label of the sample, and then let the gradient back propagate back to the knowledge transfer model KTM; In step S1, the data set D1 is a data set that traverses multiple data sets and randomly divides and merges data, and the data set D2 is a professional data set in the power field; The training in step S3 includes: given an m-classification data set in the form of samples (X, Y), by minimizing the cross entropy loss function To train the classifier, P = Softmax(z), ， Among them, CE is the first model fine-tuned by D1, P is the category probability distribution, is the logits of the model, F is the true label, and k represents the training batch; The prediction value of the sample in step S3 includes a soft prediction value u and a hard prediction value q. The calculation formula of the soft prediction value u is: u=softmax(z s ;T=τ), The calculation formula of the hard prediction value q is: q=softmax(z t ;T=τ), Among them, the z t and z s are the logits of the first model fine-tuned by D1 and the second model fine-tuned by D2, respectively, and τ is a hyperparameter; The loss function when the second model fine-tuned by D2 learns the output of the first model fine-tuned by D1 Defined as: ， Wherein, KD is the second model fine-tuned by D2, q k is the hard prediction value for the kth batch of training, u k is the soft prediction value for training batch k; The knowledge transfer model KTM The loss function is: ， Where Θ is the sum of the parameters of KTM, |·| represents the number of set elements, KD is the second model fine-tuned by D2, q is the hard prediction value, u is the soft prediction value, D1 is the data set to be trained, v is the starting position of the knowledge text, l is the ending position of the knowledge text, j represents the jth word in the knowledge text, L is the text length, N is the number of categories, a is the subscript of the category array N, and w is a value of the category array N, that is, w = N[a]; The optimization goal of the knowledge transfer model KTM is and Weighted sum: ， Among them, λ and γ are weights.

Images