Multitask electric power data entity identification method based on knowledge distillation
Through knowledge distillation technology, the knowledge of large models is compressed into small models, combined with multi-task learning, and the problems of insufficient generalization ability and high resource consumption of named entity recognition in the power system are solved, and efficient and accurate entity recognition of power data is achieved. Suitable for small equipment and systems.
Patent Information
- Application Number
- CN202410947638.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional naming entity technology based on large language models has weak generalization ability to synonyms, polysynonyms and irregular terms of proper nouns in power systems, and has high requirements for large model computing resources, making it difficult to apply to small devices and systems.
Using multi-task learning method based on knowledge distillation, the knowledge in large models is compressed into small models, improve the accuracy and generalization ability of small models, and use multi-task learning of the mutual relationship between different tasks to improve the transfer ability of the model and Generalization performance.
It realizes efficient and accurate recognition of the proper nouns of the power system, reduces the computing resources and memory requirements, is suitable for small devices and systems, enhances the ability to generalize synonyms, polysynonyms and irregular terms, and improves the efficiency of naming entity recognition and applicability.
Smart Images

Figure CN118940757A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an entity recognition method and device, and in particular to a multi-task power data entity recognition method and device based on knowledge distillation. Background Art
[0002] With the rapid development of the power system and the continuous increase in demand for power resources, the user data of the power system is becoming increasingly large, and the grid fault patterns are becoming increasingly complex. How to accurately and efficiently manage the huge and changeable power user data and the grid fault entities containing a large number of relevant problem descriptions, power policies, and power equipment is an important issue currently being studied by scientific researchers in the power field.
[0003] Although the traditional named entity technology based on large language models has produced certain effects, its generalization ability for synonyms, polysemous words and non-standard terms in the power system's proprietary terms is relatively weak. It is difficult to meet the needs of efficient and accurate identification and management of the above-mentioned large-scale power data through established rules or manual labeling methods; and the large model calculation of large language models has high requirements for device resources and memory, which is a waste of resources and difficult to apply to some small devices and systems in the power system field; although the method of compressing large model selection parameters by pruning or low-rank approximation or quantization or knowledge selection can effectively reduce the size of the model and the complexity of calculation, it still requires the model to be retrained and affects the accuracy of the model.
[0004] In order to solve the above problems, the present invention proposes a multi-task power data entity recognition method based on knowledge distillation. Through knowledge distillation combined with multi-task learning, the knowledge in the large model is compressed into the small model to improve the accuracy and generalization ability of the small model; the relationship between different tasks in multi-task learning is used to improve the migration ability and generalization performance of the model, effectively improve the named entity recognition effect in the field of power systems, greatly reduce the requirements for computing resources and memory, and is suitable for promotion and application in practical applications. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a method for compressing the knowledge in a large model into a small model through knowledge distillation to improve the accuracy and generalization ability of the small model; combining multi-task learning technology and utilizing the relationships between different tasks to improve the migration ability and generalization performance of the model, so as to achieve good named entity recognition effect, effectively reduce the requirements for computing resources and memory, adapt to the working requirements of small equipment and systems, greatly enhance the generalization ability of synonyms, polysemous words and non-standard terms in the power system proper nouns, and manage accurate, efficient and easy-to-operate multi-task power data entity recognition method based on knowledge distillation.
[0006] The present invention achieves the above-mentioned object through the following technical solutions: In a first aspect, a multi-task power data entity recognition method based on knowledge distillation includes obtaining multiple model indicator requirements and system configurations required by a user; characterized in that it is implemented by the following method: 1.1) Determine a target entity recognition model from a plurality of models to be selected according to the plurality of model indicator requirements and system configuration; wherein the model to be selected is obtained by: Acquire a power user data set; the power user data set includes: a problem description data set, a power policy data set, and a power equipment data set; A RoBERTa-large model is trained according to the problem description dataset, the power policy dataset, and the power equipment dataset respectively to obtain multiple pre-trained teacher models.
[0007] 1.2) Construct multiple initial student models with different structures; For each initial student model, a model to be selected is trained using a knowledge distillation algorithm based on the power user data set and the multiple pre-trained teacher models.
[0008] 1.3) The training to obtain a model to be selected includes: For each power user data in the power user data, input the power user data into the initial student model to obtain predicted hidden features and predicted entity recognition results; Inputting the power user data into the plurality of pre-trained teacher models respectively to obtain probability distribution; The step of calculating a knowledge distillation loss function according to the predicted hidden features and the multiple probability distributions; Calculating a true value loss function according to the entity recognition result and the label corresponding to the power user data; The total loss function is formed based on the knowledge distillation loss function and the true value loss function, as shown in the following formula: In the formula, L represents the loss function, D S Indicates the label corresponding to the power user data, is the parameter learned by the pre-trained teacher model on the corresponding single-task dataset, θ S Indicates the predicted entity recognition result; L KD is the knowledge distillation loss function, L GT is the true value loss function, λ is the weight parameter; The total loss function is minimized through back propagation, the initial student model is trained, and the model to be selected is obtained.
[0009] 1.4) The step of calculating the knowledge distillation loss function according to the predicted hidden features and the multiple probability distributions includes: According to the predicted hidden features and the multiple probability distributions, the knowledge distillation loss function is calculated based on the following formula: Where, L KD is the knowledge distillation loss function, D S represents the electricity user dataset, are the parameters learned by the pre-trained teacher model on the corresponding single-task dataset, x represents the power user data, θ S represents the predicted entity recognition result, y represents the label corresponding to the power user data, H represents the maximum sequence length of the input data, k∈{B,I,O}, BIO represents the IOB2 annotation system.
[0010] 1.5) The method of calculating the true value loss function according to the entity recognition result and the label corresponding to the power user data includes: According to the entity recognition result and the label corresponding to the power user data, the true value loss function is calculated based on the following formula: Where, L GT represents the loss function, D S represents the electricity user dataset, are the parameters learned by the pre-trained teacher model on the corresponding single-task dataset, x represents the power user data, θ S represents the predicted entity recognition result, y represents the label corresponding to the power user data, S represents the probability distribution of the student model, 1 represents the one-hot encoding in the label, H represents the maximum sequence length of the input data, k∈{B,I,O}, BIO represents the IOB2 annotation system.
[0011] According to 1.1) of claim 1, a RoBERTa-large model is trained respectively according to the problem description dataset, the power policy dataset, and the power equipment dataset to obtain multiple pre-trained teacher models, including: Train a RoBERTa-large model based on the problem description dataset to obtain a pre-trained teacher model; Train a RoBERTa-large model based on the electricity policy dataset to obtain a pre-trained teacher model; A RoBERTa-large model is trained according to the power equipment dataset to obtain a pre-trained teacher model.
[0012] In a second aspect, a multi-task power data entity recognition device based on knowledge distillation is characterized by comprising: a first acquisition module, a return model module, a second acquisition module, a first training module, a first construction module, a second training module, a computer-readable storage medium, a computer device, a computer program product including instructions, and a chip; The first acquisition module is used to obtain multiple model indicator requirements and system configurations required by the user; Returning to the model module, determining the target entity recognition model from multiple models to be selected according to the multiple model indicator requirements and system configuration; wherein the model to be selected is obtained in the following manner: Acquire a power user data set through a second acquisition module; the power user data set includes a problem description data set, a power policy data set, and a power equipment data set; Through the first training module, a RoBERTa-large model is trained according to the problem description dataset, the power policy dataset, and the power equipment dataset to obtain multiple pre-trained teacher models; Through the first building module, multiple initial student models with different structures are constructed; Through the second training module, for each initial student model, based on the power user data set and the multiple pre-trained teacher models, a knowledge distillation algorithm is used to train a model to be selected.
[0013] A multi-task power data entity recognition device based on knowledge distillation, characterized in that instructions are stored in a computer-readable storage medium, and when the instructions are executed on a terminal, the terminal executes the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
[0014] A multi-task power data entity recognition device based on knowledge distillation, characterized in that the computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5 is implemented.
[0015] A multi-task power data entity recognition device based on knowledge distillation, characterized in that the computer program product containing instructions, when the computer program product is run on a computer device, enables the computer device to execute the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
[0016] A multi-task power data entity recognition device based on knowledge distillation, characterized in that the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
[0017] The technical solution of the present invention has at least the following beneficial effects compared with the prior art: The multi-task power data entity recognition method based on knowledge distillation compresses the knowledge in the large model into the small model through knowledge distillation to improve the accuracy and generalization ability of the small model; utilizes the mutual relationship between different tasks of multi-task learning to improve the migration ability and generalization performance of the model, achieves good named entity recognition effect, effectively reduces the requirements for computing resources and memory, adapts to the working requirements of small equipment and systems, greatly enhances the generalization ability of synonyms, polysemous words and non-standard terms in the power system proper nouns, manages accurately, efficiently and is easy to operate; the method of the present invention allows the system to select different versions of student models according to specific needs to optimize and balance indicators such as running time and accuracy. This adaptability enables the final model to better serve the actual application scenarios of different power systems, improve the efficiency and applicability of entity recognition in the power field, and provide more accurate information support for power system management and optimization. It has important practical significance for promoting the intelligent development of the power field and improving the system operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings constitute a part of the specification and are used to explain the present invention together with the embodiments of the specification of the present invention, but do not constitute a limitation of the present invention.
[0019] Figure 1 A schematic diagram of a flow chart of a multi-task power data entity recognition method based on knowledge distillation according to the present invention; Figure 2 A schematic diagram of a process for training multiple power data entity recognition models of the present invention; Figure 3 Schematic diagram of the student model training framework of the Multi-Task Net of the present invention; Figure 4 This is a structural schematic diagram of a multi-task power data entity recognition device based on knowledge distillation in the present invention. DETAILED DESCRIPTION
[0020] The present invention finds that in the prior art, for complex power system user data, power fault-related entities include a large number of entities related to problem descriptions, power policies, and power equipment. Although the traditional named entity technology based on large language models is used to manage them, it has a certain effect, but the generalization ability for synonyms, polysemous words, and non-standard terms in the power system proper nouns is weak; the traditional rule-based or manual annotation methods are difficult to meet the needs of large-scale power data. In addition, the calculation of large models has high requirements for equipment resources and memory. Some small devices and systems in the power system field are difficult to adapt to large models. The traditional methods of compressing large models to achieve model lightweighting include parameter pruning, low-rank approximation, quantization, and knowledge selection. These methods can effectively reduce the model size and reduce the computational complexity, and are suitable for undertaking large-scale named entity recognition tasks, but there is a problem that each small model needs to be retrained and affects the accuracy of the model.
[0021] The present invention proposes a method for multi-task power data entity recognition based on knowledge distillation, which provides a power system entity recognition model according to system requirements and provides multiple different models at one time for different system requirements, such as running time, memory requirements, model accuracy, etc.
[0022] The following is a technical solution of a method for multi-task power data entity recognition based on knowledge distillation in the present invention, which is further described in detail through the accompanying drawings and embodiments (see Figure 1-4 ): It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0023] Embodiment 1 of the present invention provides a multi-task power data entity recognition method based on knowledge distillation (see Figure 1 ),like Figure 1 As shown, the method includes: First, obtain multiple model indicator requirements and system configurations required by users; In the first embodiment of the present invention, there are multiple entity recognition model indicators in the power system model configuration interface. These indicators are intended to provide users with a wide range of choices and in-depth understanding of the performance and characteristics of each model. Through clear and intuitive visual display, users can fully understand the performance of different models in various performance indicators, providing sufficient information support for their subsequent selection. The optional model indicators include: memory requirements, CPU requirements, running speed, model accuracy, among which the running speed is subject to the hardware configuration, so the user needs to fill in the current hardware configuration; The optional model indicators include multiple aspects. Users can select the following key performance indicators based on actual needs and system configuration to better meet their personalized model needs: Users can select the model's memory requirements, CPU requirements, running speed, and model accuracy: Memory requirements ensure that the selected model can run efficiently in a specific hardware environment while meeting the availability of system resources. Selecting the memory requirement indicator helps optimize the overall system performance and avoid insufficient system resources due to excessive memory usage; Select CPU requirements to effectively match the hardware environment and ensure that the model's calculation process fully utilizes the hardware's processing power to improve the model's operating efficiency; Select running speed to specify the model's actual execution efficiency under a specific hardware configuration. Since the running speed is affected by the hardware configuration, users need to provide the current hardware configuration when filling in the running speed requirements in order to more accurately evaluate the model's performance in the actual environment; Select the model's accuracy based on the specific requirements of the task to ensure that the selected model can achieve the expected performance level in actual applications, which helps the system achieve more accurate and reliable entity recognition in different application scenarios.
[0024] Second, according to multiple model indicator requirements and system configuration, a target entity recognition model is determined from multiple models to be selected; In the first embodiment of the present invention, based on the model indicator requirements filled in by the user and the system configuration, the system selects an alternative model that meets the user's requirements from a plurality of preset models to be selected as the target entity recognition model.
[0025] In the first embodiment of the present invention, the multiple models to be selected are obtained in the following manner: (see Figure 2 ),like Figure 2 As shown: The first step is to obtain the power user data set; The basic information of power users collected by the power grid electricity question and answer service system and 95,598 work order data screened by the customer service information system were obtained. The customer service information system accumulates all records of users' calls to the power company and the original data of the corresponding call details, including the user's home address, phone information, consultation content, problem description, complaint reason and other power-related text data.
[0026] After obtaining the power user data set, the power user data set is preprocessed and labeled, which specifically includes the following steps: Perform data cleaning and text segmentation on the power user dataset.
[0027] The IOB2 annotation system is used to annotate the three types of entities in the power user dataset, namely problem description, power policy, and power equipment, to obtain three datasets. The training dataset, validation dataset, and test dataset are set in the ratio of 7:1.5:1.5.
[0028] A RoBERTa-large model is trained according to the problem description dataset, the power policy dataset, and the power equipment dataset respectively to obtain multiple pre-trained teacher models.
[0029] A RoBERTa-large model is trained according to the problem description dataset to obtain a pre-trained teacher model; a RoBERTa-large model is trained according to the power policy dataset to obtain a pre-trained teacher model; a RoBERTa-large model is trained according to the power equipment dataset to obtain a pre-trained teacher model.
[0030] Let E be the set of target entity types, where e i ∈E represents the i-th entity type (where i∈{1,...,|E|}). For each entity type, there is a labeled database D i , and D i ={(x,y)∈X i ×Y i}, where X i is a set of input data x (labeled sequence x j ∈x, where j∈{1,...,H}, H represents the maximum sequence length of the input data), Y i In the first embodiment of the present invention, the IOB2 tagging system is referred to, and the B tag is assigned to the beginning of the entity mention, the I tag is assigned to the inside of the entity, and the O tag is a word that does not belong to any entity.
[0031] Based on such a dataset, the goal is to learn a model f that can transform each character x in the input data x j Mapped to its label y j ∈Y multi On, where Y multi As shown in formula (2): The second step is to construct multiple initial student models with different structures.
[0032] In this specific embodiment, the initial student model can provide the following four: 1) RoBERTa-large: Same as the teacher model, a large model suitable for scenarios that require processing large-scale text data and complex tasks; 2) RoBERTa-base: A relatively small RoBERTa model suitable for general text processing tasks; 3) BERT-Distil: A simplified version of BERT; 4) BERT-Tiny: A more streamlined version of BERT.
[0033] The third step is to train each initial student model to obtain a model to be selected based on the electricity user dataset and the multiple pre-trained teacher models using a knowledge distillation algorithm.
[0034] The training obtains a model to be selected, which specifically includes the following steps: 3.1) For each power user data in the power user data, the power user data is input into the initial student model to obtain the predicted hidden features and the predicted entity recognition results.
[0035] The entity type set in the first embodiment of the present invention is E = {e1, e2, e3} = {problem description, power policy, power equipment}. Therefore, Multi-Task Net will learn how to predict each entity of the input sample as one of the labels shown in formula (3): Y multi = {B-Problem, I-Problem, B-Policy,I-Policy, B-Equipment,B-Equipment,O}(3) This study aims to train multiple single-task named entity recognition models using a dataset of power system users with different entity annotations; Figure 3 Demonstrates the training framework of Multi-Task Net.
[0036] (See Figure 3 ),like Figure 3 As shown, the method of the present invention roughly divides the training framework of Multi-Task Net into four stages: 1) First, the data sets of single entities are merged to establish a database DS containing multiple entities as the true value when the student model is trained; 2) Then, each teacher model outputs a prediction result for each sample in the multi-entity database DS; 3) The output distributions of multiple teacher models are aggregated to establish a database A for extracting information from the collective knowledge of the teacher models; 4) Finally, the student model is trained to better understand and generalize the characteristics of each entity by simultaneously minimizing the knowledge distillation loss function and the true value loss function.
[0037] Dataset merging: Based on the power system user datasets D1, D2, ..., DE labeled with different entities, an aggregate dataset DS is established. The expression is as follows: D S ={(x,y)∈X S ′×Y S ′} (4) Among them, X′ S It is obtained by concatenating the sample data in each single-task dataset, that is, X′ S =X1+X2+...+X E , similarly, label Y S ′ is also obtained by concatenation, the only difference is that the entity types corresponding to the labels are different.
[0038] 3.2) Inputting the power user data into multiple pre-trained teacher models to obtain probability distribution; The aggregated dataset DS will be used as a distribution to obtain the teacher model’s knowledge, i.e., the data source for knowledge distillation, while providing real labeled data during student model training.
[0039] Get the prediction results of the teacher model: is the parameter learned by the teacher model on the corresponding single-task dataset. For each sample data character xj∈x, the i-th teacher model provides the probability distribution T i j , the expression is as follows: Distribution aggregation: Through knowledge distillation, the student model will learn to imitate the output distribution of the teacher model. Unlike traditional knowledge distillation methods, the student model in the embodiment of the present invention needs to learn from a set of heterogeneous teacher models, each of which can recognize different entity types. Therefore, an aggregation stage is required, in which the distributions of the teacher models are merged into a single distribution for use in the knowledge distillation framework.
[0040] Let x j ∈x is the character of a sample data that needs to be aggregated; represents the probability distribution of the i-th teacher model for label k, where k∈{B,I,O}; for character x j Assigned to the label Be i ,Ie i The probabilities of and O are calculated as the intersection probabilities of the following events, respectively, and the expressions are as follows: P(O)=P(∩ i (T i assigns O)) (8).
[0041] In view of the independence between teacher models and the mutually exclusive characteristics of the distribution of each teacher model, the aggregate probability distribution A is calculated according to the following formula, which is expressed as follows:
[0042] In summary, given a sample data character x j ∈x, where j∈{1,...,H}, the output of the distribution aggregation stage is the aggregate distribution of Y′ entities, expressed as follows:
[0043] 3.3) Based on the predicted hidden features and multiple probability distributions, the knowledge distillation loss function is calculated.
[0044] 3.4) Based on the entity recognition results and the labels corresponding to the power user data, the true value loss function is calculated.
[0045] 3.5) The total loss function is constructed based on the knowledge distillation loss function and the true value loss function.
[0046] 3.6) Minimize the total loss function through back propagation, train the initial student model, and obtain the model to be selected.
[0047] Student model training: using parameters θ S represents the student model, and the output distribution of the student is expressed as S{y t = k|x;θ S}, where k∈Y′.
[0048] The goal of the Multi-Task Net training framework is to minimize the total loss function consisting of the knowledge distillation loss function and the true value loss function: the knowledge distillation loss function measures the distance between the student model distribution and the teacher model distribution, and the true value loss function calculates the error between the target student model and the true value. The expression of the total loss function is as follows: Among them, L KD and L GT They are the knowledge distillation loss function and the true value loss function respectively, and λ is a hyperparameter that controls the weights of the two losses in the total loss function L.
[0049] Knowledge distillation loss function L KD And the true value loss function L GT The expression is as follows: Where S represents the student model probability distribution and 1 represents the one-hot encoding in the true annotation.
[0050] The method of the present invention uses the Adam optimizer with an initial learning rate of 0.5×e -4 , β1=0.9, β2=0.999. Batchsize is set to 8, and the maximum sequence length is 128. The student model currently provides the following four types: 1) RoBERTa-large: Same as the teacher model, a large model suitable for scenarios that require processing large-scale text data and complex tasks; 2) RoBERTa-base: A relatively small RoBERTa model suitable for general text processing tasks; 3) BERT-Distil: A simplified version of BERT. Compared with BERT, it reduces the number of parameters by 40% and runs 60% faster; 4) BERT-Tiny: A more streamlined version of BERT, with a model size 7.5 times smaller than BERT and a prediction speed that is 9.4 times faster.
[0051] In the first embodiment of the present invention, a multi-task power data entity recognition method based on knowledge distillation is implemented to construct an entity recognition training framework Multi-Task Net based on knowledge distillation. Teacher models are trained separately for various entities in the power system user data to capture the semantic information of different entities. Through the knowledge distillation technology, the student model Multi-Task Net fully learns the semantic information of each entity and the differences and correlations between different entity types, so that different Multi-Task Net student models can be selected according to different scenario requirements. This has important practical value for power system user data management. Embodiment 2
[0052] Based on the same inventive concept, the second embodiment of the present invention also provides a power user data entity identification device based on knowledge distillation (see Figure 4 ),like Figure 4 As shown, the device comprises: The first acquisition module 101 is used to acquire multiple model indicator requirements and system configurations required by the user; Returning to the model module 102, a target entity recognition model is determined from a plurality of models to be selected according to the plurality of model indicator requirements and system configuration; wherein the model to be selected is obtained in the following manner: The second acquisition module 103 is used to acquire a power user data set; the power user data set includes a problem description data set, a power policy data set, and a power equipment data set; A first training module 104 is used to train a RoBERTa-large model according to the problem description dataset, the power policy dataset, and the power equipment dataset to obtain multiple pre-trained teacher models; A first construction module 105 is used to construct a plurality of initial student models with different structures; The second training module 106 is used to train each initial student model to obtain a model to be selected based on the power user data set and the multiple pre-trained teacher models using a knowledge distillation algorithm. Embodiment 3
[0053] Based on the same inventive concept, the second embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the multi-task power data entity recognition method based on knowledge distillation as described in the above-mentioned first embodiment is implemented. Embodiment 4
[0054] Based on the same inventive concept, embodiment 2 of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-task power data entity recognition method based on knowledge distillation as described in the above embodiment 1 is implemented. Embodiment 5
[0055] Based on the same inventive concept, the second embodiment of the present invention also provides a computer program product comprising instructions. When the computer program product is executed on a computer device, the computer device executes the multi-task power data entity recognition method based on knowledge distillation as described in the above-mentioned first embodiment. Embodiment 6
[0056] Based on the same inventive concept, embodiment 2 of the present invention also provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the multi-task power data entity recognition method based on knowledge distillation as described in the above embodiment 1.
[0057] Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects; moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0058] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A multi-task power data entity recognition method based on knowledge distillation, comprising: Obtain multiple model indicator requirements and system configurations required by users; characterized in that it is achieved through the following methods: Determining a target entity recognition model from a plurality of models to be selected according to the plurality of model indicator requirements and system configuration; The model to be selected is obtained in the following way: Acquire a power user data set; the power user data set includes a problem description data set, a power policy data set, and a power equipment data set; According to the problem description dataset, the power policy dataset, and the power equipment dataset, a RoBERTa-large model is trained respectively to obtain multiple pre-trained teacher models; Construct multiple initial student models with different structures; For each initial student model, a model to be selected is trained using a knowledge distillation algorithm based on the power user data set and the multiple pre-trained teacher models.
2. The method according to claim 1, for each initial student model, based on the power user data set and the plurality of pre-trained teacher models, using a knowledge distillation algorithm, training a model to be selected, comprising: For each power user data in the power user data, input the power user data into the initial student model to obtain predicted hidden features and predicted entity recognition results; Inputting the power user data into the plurality of pre-trained teacher models respectively to obtain probability distribution; Calculating a knowledge distillation loss function according to the predicted hidden features and the multiple probability distributions; Calculating a true value loss function according to the entity recognition result and the label corresponding to the power user data; The total loss function is formed based on the knowledge distillation loss function and the true value loss function, as shown in the following formula: In the formula, L represents the loss function, D S Indicates the label corresponding to the power user data, is the parameter learned by the pre-trained teacher model on the corresponding single-task dataset, θ S Indicates the predicted entity recognition result; L KD is the knowledge distillation loss function, L GT is the true value loss function, and λ is the weight parameter. The total loss function is minimized through back propagation, the initial student model is trained, and the model to be selected is obtained.
3. The method according to claim 2, characterized in that The calculating of the knowledge distillation loss function according to the predicted hidden features and the multiple probability distributions includes: According to the predicted hidden features and the multiple probability distributions, the knowledge distillation loss function is calculated based on the following formula: Where, L KD is the knowledge distillation loss function, D S represents the electricity user dataset, are the parameters learned by the pre-trained teacher model on the corresponding single-task dataset, x represents the power user data, θ S represents the predicted entity recognition result, y represents the label corresponding to the power user data, H represents the maximum sequence length of the input data, k∈{B,I,O}, BIO represents the IOB2 annotation system.
4. The method according to claim 2, characterized in that The calculating of the true value loss function according to the entity recognition result and the label corresponding to the power user data includes: According to the entity recognition result and the label corresponding to the power user data, the true value loss function is calculated based on the following formula: Where, L GT represents the loss function, D S represents the electricity user dataset, are the parameters learned by the pre-trained teacher model on the corresponding single-task dataset, x represents the power user data, θ S represents the predicted entity recognition result, y represents the label corresponding to the power user data, S represents the probability distribution of the student model, 1 represents the one-hot encoding in the label, H represents the maximum sequence length of the input data, k∈{B,I,O}, BIO represents the IOB2 annotation system.
5. The method according to claim 1, characterized in that The method of training a RoBERTa-large model according to the problem description dataset, the power policy dataset, and the power equipment dataset to obtain multiple pre-trained teacher models includes: training a RoBERTa-large model according to the problem description dataset to obtain a pre-trained teacher model; Train a RoBERTa-large model based on the electricity policy dataset to obtain a pre-trained teacher model; A RoBERTa-large model is trained according to the power equipment dataset to obtain a pre-trained teacher model.
6. A multi-task power data entity recognition device based on knowledge distillation, characterized in that: include: A first acquisition module, a return model module, a second acquisition module, a first training module, a first construction module, a second training module, a computer-readable storage medium, a computer device, a computer program product including instructions, and a chip; The first acquisition module is used to obtain multiple model indicator requirements and system configurations required by the user; The model module is returned, and is used to determine the target entity recognition model from multiple models to be selected according to the multiple model indicator requirements and system configuration; wherein the model to be selected is obtained in the following manner: The second acquisition module is used to acquire a power user data set; the power user data set includes a problem description data set, a power policy data set, and a power equipment data set; The first training module is used to train a RoBERTa-large model according to the problem description dataset, the power policy dataset, and the power equipment dataset to obtain multiple pre-trained teacher models; The first building module is used to build multiple initial student models with different structures; The second training module is used to train each initial student model to obtain a model to be selected based on the power user data set and the multiple pre-trained teacher models using a knowledge distillation algorithm.
7. The device according to claim 6, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the terminal, the terminal executes the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
8. The device according to claim 6, characterized in that The computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1 to 5 is implemented.
9. The device according to claim 6, characterized in that The computer program product comprising instructions, when executed on a computer device, enables the computer device to execute the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
10. The device according to claim 6, characterized in that The chip comprises a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the multi-task power data entity recognition method based on knowledge distillation as described in any one of claims 1-5.
Citation Information
Cited By
Adjusting and optimizing method and system based on large power dispatching model
CN119558488A
Model training method and device based on knowledge distillation, electronic equipment and readable storage medium
CN119761456A
Lightweight information extraction method and system based on knowledge distillation and thinking chain
CN120011533A
