Power marketing data query method and system of JointBERT model based on knowledge graph

Abstract

The invention is suitable for the technical field of data query, and provides a power marketing data query method and system based on a JointBERT model of a knowledge graph, and the method comprises the steps: obtaining field background knowledge related to the power industry, and constructing the knowledge graph; a JointBERT model is constructed and trained, and the model comprises an input layer used for processing input text data; the coding layer is used for carrying out fusion coding on rich general semantic knowledge in a corpus and domain knowledge in the knowledge graph; the decoding layer is used for generating field and intention corresponding to the input text and slot labels corresponding to words; compressing and optimizing the model by adopting an early leaving mechanism and a knowledge distillation method to obtain a target model; and the power demand problem of the user, which is received in real time, is used as the input parameter and is input into the target model for data query and analysis, so that the enterprise information query efficiency of related workers is improved.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a power marketing data query method and system based on a JointBERT model of a knowledge graph. Background Art

[0002] With the continuous reform of the power market and the construction of smart grids, the transformation of the power industry requires power companies to continuously upgrade their marketing services and serve the society in a more efficient, environmentally friendly and safe way. At the same time, with the increasing diversification and personalization of power customer needs, traditional power marketing methods have problems such as inefficiency and irregular data management, and can no longer meet the needs of the modern market. With the development of technologies such as deep learning and natural language processing, intelligent dialogue systems are also applied to power marketing services. The intelligent dialogue system and users use natural language to realize the process of asking and answering. Unlike traditional query methods, the intelligent question-answering system based on natural language question-answering is more flexible and more in line with human thinking habits. However, due to the large number of professional terms and complex business scenarios in the power field, the intelligent dialogue system sometimes finds it difficult to understand users' vague or industry-specific questions. In addition, the system's knowledge update is lagging, which makes the power marketing service have some potential negative effects and worsens the user experience.

[0003] In view of this, a power marketing data query method and system based on the JointBERT model of knowledge graph is needed. Summary of the invention

[0004] The embodiments of the present application provide a power marketing data query method and system based on a JointBERT model of a knowledge graph, which is used to solve the problem of poor user experience.

[0005] The first aspect of the embodiment of the present application provides a power marketing data query method based on the JointBERT model of the knowledge graph, including:

[0006] Acquire background knowledge related to the power industry and build a knowledge graph;

[0007] Construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to words corresponding to the input text.

[0008] The JointBERT model is compressed and optimized by using an early exit mechanism and a knowledge distillation method to obtain a target JointBERT model;

[0009] The power demand problems of users received in real time are used as input parameters and input into the target JointBERT model for data query and analysis.

[0010] Furthermore, the acquisition of field background knowledge related to the power industry and the construction of a knowledge graph include:

[0011] Extracting entities, relations and attributes based on text and data in the domain background knowledge;

[0012] Knowledge representation and knowledge fusion are performed based on the extracted entities, relationships and attributes to construct a knowledge graph, and the constructed knowledge graph is stored in a preset database.

[0013] Furthermore, the input layer is used to process input text data, including:

[0014] The beginning of the input text is marked as a classification, the end of the input text is marked as a segmentation, and the length of the input text is truncated and padded.

[0015] Furthermore, the encoding layer is used to fuse the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph, including:

[0016] By encoding fusion, the rich general semantic knowledge in the massive corpus and the domain knowledge in the power domain knowledge graph are encoded into a unified vector representation;

[0017] After calculating the attention scores of the feature vectors, the attention distribution between the tokens in the input text is obtained;

[0018] Normalize the attention scores of each input vector and use them as weights for feature fusion;

[0019] By stacking multiple Encoder blocks, high-level feature representations of the input text are gradually extracted.

[0020] Furthermore, the decoding layer is used to generate a domain label, an intent label, and a slot label corresponding to the input text, including:

[0021] The encoded features are input into the BERT Model layer for processing to obtain the corresponding hidden state sequence;

[0022] Inputting the hidden layer vectors corresponding to the characters in the hidden state sequence into the domain classification layer and the intention recognition layer respectively;

[0023] The number of neurons of the characters in the hidden state sequence is linearly converted into the number of predefined domain and intent label categories by using the linear layer of the neural network, and the corresponding domain label, intent label and slot label corresponding to the word are determined respectively.

[0024] Furthermore, the expressions of the domain label, intent label, and slot label corresponding to the word are as follows:

[0025] Probability distribution of the field:

[0026] p(D│x)=softmax(W d ·h [CLS] +b d );

[0027] Where: D is the label category of the field, x is the original input sentence, h [CLS] is the hidden layer vector corresponding to “[CLS]”, W d is the domain weight matrix, b d is the domain offset vector;

[0028] Probability distribution of intent:

[0029] p(I│x)=softmax(W i ·h [CLS] +b i );

[0030] Where: I is the label category of the intent, W i is the intention weight matrix, b i is the intention offset vector;

[0031] Probability distribution of slot labels corresponding to words:

[0032] p(S│x)=softmax(W s ·h [CLS] +b s );

[0033] Where: S is the label category of the slot corresponding to the word, W s is the slot weight matrix, b s is the slot offset vector.

[0034] Furthermore, the early exit mechanism and knowledge distillation method are used to compress and optimize the JointBERT model to obtain a target JointBERT model, including:

[0035] Insert a classification layer between each Transformer layer in the student model to perform domain classification, intent recognition, and slot filling tasks, and judge whether the model meets the patience-based early retirement criteria based on the results;

[0036] The Self-attention mechanism of the JointBERT model is replaced with the ProbSpare self-attention mechanism, and a distillation mechanism is introduced when multiple attention layers are superimposed.

[0037] A second aspect of the embodiment of the present application provides a power marketing data query system based on a JointBERT model of a knowledge graph, including:

[0038] The knowledge graph construction unit is used to obtain the background knowledge of the power industry and construct the knowledge graph;

[0039] A JointBERT model construction and training unit, which is used to construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to words corresponding to the input text.

[0040] A target JointBERT model determination unit, used to compress and optimize the JointBERT model by using an early withdrawal mechanism and a knowledge distillation method to obtain a target JointBERT model;

[0041] The data query and analysis unit is used to input the power demand problem of the user received in real time as an input parameter into the target JointBERT model for data query and analysis.

[0042] A third aspect of the embodiments of the present application provides a computer device, including:

[0043] memories, transceivers, processors, and bus systems;

[0044] Wherein, the memory is used to store programs;

[0045] The processor is used to execute the program in the memory, including executing the power marketing data query method based on the JointBERT model of the knowledge graph as described above;

[0046] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0047] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the power marketing data query method based on the JointBERT model based on the knowledge graph as described above.

[0048] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0049] 1. By constructing a concept graph to intuitively represent and analyze the complex relationships between entities, the efficiency of knowledge retrieval, reasoning and analysis can be improved. The information of the knowledge graph can assist in setting the corresponding domains, intentions and slots for natural language understanding tasks, provide additional knowledge representation for the natural language understanding model, and improve the efficiency of relevant staff in conducting enterprise information queries.

[0050] 2. Use JointBERT to quickly learn and update language knowledge, process large-scale natural language data, and achieve fast data loading and query. Since it is trained on large-scale general corpus, it can fully learn the context-related semantic knowledge of text data in this task.

[0051] 3. By combining the early exit mechanism and knowledge distillation, the natural language understanding model is compressed and optimized to ensure the ability of fast reasoning in actual application scenarios while meeting the needs of deployment cost control. While keeping the performance degradation small, the complexity of the model and the computing resource requirements are effectively reduced.

[0052] Other advantages, objectives, and features of the present invention will be set forth in part in the following description, and in part will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of an embodiment of a method for querying power marketing data based on a JointBERT model of a knowledge graph in the present invention;

[0054] Figure 2 It is a panoramic view of the concept map of the power marketing field in the present invention;

[0055] Figure 3 This is the overall architecture diagram of the JointBERT model in the present invention. DETAILED DESCRIPTION

[0056] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Embodiment 1

[0058] The implementation method in this embodiment can be implemented in the system, can be implemented in the server, and can also be implemented in the terminal, without specific limitation. The following will introduce the power marketing data query method based on the JointBERT model of the knowledge graph in this application from the perspective of system implementation. Figure 1 , the method provided in the embodiment of the present application comprises the following steps:

[0059] S11. Obtain background knowledge related to the power industry and build a knowledge graph;

[0060] In this embodiment, the field background knowledge related to the power industry includes power package data of each platform, basic marketing data, and power industry document data, including power industry professional knowledge, specifications, work reports, etc. Building a knowledge graph also includes:

[0061] 1. Extract entities, relationships and attributes based on text and data in domain background knowledge;

[0062] 2. Perform knowledge representation and knowledge fusion based on the extracted entities, relationships and attributes, construct a knowledge graph, and store the constructed knowledge graph in a preset database.

[0063] Specifically, the conceptual entities, relationships and attributes in the filter domain are selected. The basic structure of the graph includes nodes (entities), edges (relationships) and attribute boxes (attributes). According to the power package, twelve categories of concepts are defined in the concept graph, including: lines, units, substations, users, metering points, transformers, industries, user pricing strategies, catalog prices, transformers, time periods, and energy meters. Please refer to Figure 2 , Figure 2 The following table lists the concepts and their key attributes:

[0064] Table 1

[0065]

[0066] Based on the current electricity retail platform business and electricity market demand, the relationship between basic data is sorted out to identify the core concepts in the field. The relationship model between concepts is determined, one-to-one correspondence, one-to-many inclusion relationship or many-to-many specific connection, and the consistency of the entity fields of the conceptual model is judged. Specific instances and relationships between instances are extracted from the structured data corresponding to the conceptual model, and converted into instance nodes and relationship edges in the knowledge graph. Finally, the graph database Neo 4j is used as the storage database of the knowledge graph, and the knowledge graph modeling tool is used to parse and analyze the database file to extract relevant data; the Echarts library is used to dynamically display the concept graph in a visual form for maintenance.

[0067] In addition, the specific extraction of knowledge graph instances is as follows: first, based on the pre-built concept graph, the original data is obtained. Then, the data analysis package pandas is used to clean the data in the database table, mainly including: ① using regular expressions to match strings; ② removing duplicate and redundant data; ③ missing value processing; ④ outlier detection processing; ⑤ link replacement for data lacking semantic information. According to the relationship type defined in the concept graph, rule matching and text similarity calculation methods are used to further classify and divide potential connections.

[0068] S12. Construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to the input text.

[0069] Traditional RNN-based language models learn contextual semantic information in a time-step manner, and can perform well for tasks with sequence characteristics in natural language processing. However, since the state transition of the current time step depends on the previous time step, the training time cost is high, and there will be a forgetting problem when processing longer sequences. The BERT model is trained on a large-scale general corpus through masked language model tasks and next sentence prediction tasks, and fully learns the context-related semantic knowledge of text data.

[0070] Here, by encoding and fusing the rich general semantic knowledge in the pre-trained model and the power field-specific knowledge in the knowledge graph, the knowledge is unified and the applicability and accuracy of the model are improved. The model can analyze the user input request and convert the natural language input into predefined fields, intents and slots to achieve efficient understanding and analysis of power marketing problems. The overall architecture of the Joint BERT model is as follows: Figure 3 The JointBERT model architecture includes the input layer, encoding layer, and decoding layer, as shown below:

[0071] 1. The input layer is used to process input text data, including:

[0072] Classify and mark the beginning of the input text: insert a specific classification tag "[CLS]" at the beginning of each sentence to summarize the feature information of the sequence;

[0073] Mark the end of the input text: Add a segmentation marker "[SEP]" at the end of each sentence to separate different sentences;

[0074] The length of the input text is truncated and padded: the length of the input text is unified and fixed to length n. When the input text exceeds n, it is truncated to length n; when the input text is less than n, a special "[PAD]" mark is used to padded to length n.

[0075] 2. The encoding layer is used to fuse the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph, including:

[0076] By encoding fusion, the rich general semantic knowledge in the massive corpus and the domain knowledge in the power domain knowledge graph are encoded into a unified vector representation;

[0077] After calculating the attention scores of the feature vectors, the attention distribution between the tokens in the input text is obtained;

[0078] Normalize the attention scores of each input vector and use them as weights for feature fusion;

[0079] By stacking multiple Encoder blocks, high-level feature representations of the input text are gradually extracted.

[0080] 3. The decoding layer is used to generate the domain label, intent label and slot label corresponding to the input text, including:

[0081] The encoded features are input into the BERT Model layer for processing to obtain the corresponding hidden state sequence h;

[0082] The hidden layer vector h corresponding to the character "[CLS]" in the hidden state sequence is input into the domain classification layer and the intent recognition layer respectively;

[0083] Use the linear layer of the neural network to transform the hidden state sequence h [CLS] The number of neurons in the characters is linearly converted into the number of predefined domain and intent label categories to determine the corresponding domain label, intent label, and slot label corresponding to the word.

[0084] The probability distribution of a domain is calculated as follows:

[0085] p(D│x)=softmax(W d ·h [CLS] +b d );

[0086] Where: D is the label category of the field, x is the original input sentence, h [CLS] is the hidden layer vector corresponding to “[CLS]”, W d is the domain weight matrix, b d is the domain offset vector.

[0087] The probability distribution of intent is calculated as follows:

[0088] p(I│x)=softmax(W i ·h [CLS] +b i );

[0089] Where: I is the label category of the intent, W i is the intention weight matrix, b i is the intention offset vector;

[0090] The calculation formula for the probability distribution of the slot label corresponding to the word is as follows:

[0091] p(S│x)=softmax(W s ·h [CLS] +b s );

[0092] Where: S is the label category of the slot corresponding to the word, W s is the slot weight matrix, b s is the slot offset vector.

[0093] 4. Use the joint loss function to associate the three subtasks of domain classification, intent recognition, and slot filling to train the natural language understanding model. The process is as follows:

[0094] The formula for calculating the joint loss is as follows:

[0095] L=L D +LI +L S ;

[0096] Where: L D Represents the loss function of the domain classification task, Table L I The loss function for the schematic recognition task, L S Represents the loss function for the question text classification task.

[0097] Calculate the domain discrimination loss. The domain discrimination task adopts the improved loss function EMO based on the optimal transmission idea, which is derived from the cross entropy loss. It solves some limitations of the traditional cross entropy loss function by introducing the idea of ​​optimal transmission. The cost function formula in the EMO loss function is as follows:

[0098]

[0099] Where: e d and e t For the Token Embedding obtained in advance, the final EMO loss function is as follows:

[0100]

[0101] Where: p d is the predicted probability that the observed sample belongs to category d, c d,t is the corresponding cost function.

[0102] The loss function used for the computational graph classification task is cross entropy, as shown in the following formula:

[0103]

[0104] Where: p i is the predicted probability that the observed sample belongs to category i, c i,t is the corresponding cost function.

[0105] Calculate the slot filling loss. The cross entropy loss function calculation formula for the slot filling task is as follows:

[0106]

[0107] Where: p s is the predicted probability that the observed sample belongs to slot s, c s,t is the corresponding cost function.

[0108] S13. Use the early exit mechanism and knowledge distillation method to compress and optimize the JointBERT model to obtain the target JointBERT model;

[0109] In natural language processing models, large models are often used to extract valuable information from a large amount of computing resources to improve the performance of the model. However, as the model size increases, the model's inference time is also increasing, and the demand for computing resources and storage space is also high. The goal of model lightweighting is to reduce the model's computing and storage requirements without affecting the model's performance. Knowledge distillation is used here for quantization. Knowledge distillation aims to use large models to improve performance during the training phase, and use small models to improve speed and save resources during the deployment phase.

[0110] The framework introduces an early exit mechanism in the process of training the student model by knowledge distillation. The knowledge distillation module aims to build a lightweight student model that can approach the performance of a large-scale teacher model. Model early exit is to calculate the performance of the model on the validation set during training and stop training when the accuracy starts to drop, thereby avoiding overfitting. The model can also dynamically adjust the number of execution layers to adapt to samples of different complexities.

[0111] In knowledge distillation, the Self-attention mechanism of the Joint BERT model is mainly replaced by the newly proposed ProbSpare self-attention mechanism. The distillation mechanism is introduced when multiple attention layers are superimposed. ProbSpare self-attention: The self-attention in the Transformer model is based on three inputs (Query, Key, value), and Dot-Product is used to calculate the input attention matrix, as shown below:

[0112]

[0113] Where: Q∈R m×d , K∈R n×d , Q represents the Query evaluated by M, and the probability of the attention coefficient of the i-th Query is as follows:

[0114]

[0115] However, the attention mechanism is sparse, that is, fewer Dot-Products contribute most of the attention score. Therefore, KL divergence is used to evaluate the sparsity of each query. The evaluation formula is as follows:

[0116]

[0117] The ProbSpare self-attention algorithm process is as follows:

[0118] 1. Randomly sample a certain amount of keys for each query;

[0119] 2. Calculate M(q) for each query i ,K);

[0120] 3. Select M(q i ,K) the highest uquery;

[0121] 4. Only calculate the dot product of u queries and all keys to get the attention score;

[0122] 5. The scores corresponding to other queries are the average of the input of the self-attention layer to ensure that the input and output sequence lengths are both L.

[0123] Self-attention distillation mechanism: Since each position output obtains information from other positions after each layer of ProbSpare self-attention, in order to reduce the amount of calculation, improve network efficiency, and avoid redundant calculations, it is crucial for deep networks to appropriately reduce the length of the output sequence. The essence of the self-attention distillation mechanism is to perform a one-dimensional convolution and maximum pooling operation after each layer of ProbSpare self-attention.

[0124] In the model early exit mechanism, a patience-based early exit algorithm is adopted to avoid overfitting of the model during training by terminating the training in advance at an appropriate time. Therefore, a classification layer is inserted between each Transformer layer in the student model to perform domain classification, intent recognition, and slot filling tasks, and the results are used to determine whether the model meets the patience-based early exit criteria.

[0125] 1. Patience-based early exit algorithm PABEE is a model early exit strategy, which is mainly implemented through a "patience index" f initialized to 0. The basic principle is: at the end of each training epoch, the validation set accuracy is calculated from the bottom layer of the model. If the current layer is consistent with the prediction result of the previous layer, the patience index increases by 1. When the prediction result is inconsistent, the patience index is reset to 0. When the patience index reaches the preset threshold, the model will terminate early at the current layer to prevent overfitting, otherwise the prediction result of the last layer will be output as the final result. The algorithm flow is as follows:

[0126] (1) Input validation set K, model layer number hidden_layers, and patience index reaching the preset threshold P;

[0127] (2) Initialize patience index patience = 0;

[0128] (3) Insert a classification layer between each Transformer layer and perform fine-tuning, starting from the bottom layer for verification;

[0129] (4) Output the final prediction result.

[0130] 2. The main function of the teacher model is to provide guidance and supervision to guide the student model to learn and optimize. The main goal of the student model is to use the knowledge provided by the teacher model for learning and optimization, so as to achieve similar performance and accuracy as the teacher model. Specifically, knowledge distillation realizes the process of knowledge learning by migrating the target distribution through a loss function with temperature smoothing labels. Since the probability distribution entropy output by the teacher model is transformed through the logical unit Softmax activation function, the probability of negative labels will be very close to zero under the action of the function, resulting in the loss of some information about negative labels. More information can be provided through the loss function with temperature smoothing labels. Therefore, the temperature variable T is required. The calculation method of the Softmax function after adding the temperature T is as follows:

[0131]

[0132] As T decreases, the probability gap between the output categories becomes larger (steeper), resulting in a smaller loss. The original Softmax function can be regarded as the case of T = 1. When T increases, the probability gap between the output categories will become smaller and smaller, resulting in a larger loss. Compared with the true value output or the probability output of the original Softmax, the softened probability value can provide more information.

[0133] 3. Since the student network mainly learns the inter-class similarity information between the real label and the teacher network, the overall loss of the student model consists of two parts: the loss between the predicted value of the student model output and the real label and the distillation loss generated when the student model and the teacher model are distilled. The calculation method is as follows:

[0134] L total =λ·L stu +(1+λ)·L dis

[0135] Where: λ is the loss adjustment coefficient, which is used to control the ratio of student loss to distillation loss.

[0136] Specifically, the following experiments are carried out to verify:

[0137] In the experiment, the model parameters were loaded and three modules were developed, including: domain classification, intent classification and slot filling. The model structure includes a 12-layer Transformer structure, a word vector with 768 dimensions, a maximum input sequence length of 512, and a learning rate of 5e^-5. The model training is optimized using the Adam optimizer, and the training process is repeated 10 times to ensure the stability of the model. The specific model parameter settings for distillation temperature, patience index, number of student model Transformer layers, number of teacher model Transformer layers, loss adjustment coefficient, and epoch are as follows:

[0138] Table 2

[0139]

[0140] When evaluating the performance of the model, multiple indicators are considered comprehensively, including semantic accuracy (SA), inference time (Time), and parameter quantity (Param). The evaluation results use semantic accuracy SA as the evaluation criterion, that is, the ratio of the number of correctly predicted domains and intents in the sentence and the number of all slots and their values ​​to the total number of sentences. The inference time of the model is calculated as the time it takes for the model to process one input text sample. It starts from the input being passed to the model and ends when the model generates output.

[0141] S14. The power demand problem of the user received in real time is used as an input parameter and input into the target JointBERT model for data query and analysis.

[0142] Finally, each inference model is encapsulated using the JAVA backend interface service microframework Spring Cloud to form an online service interface, which is integrated into the existing power retail system to meet production requirements. First, the model is instantiated, the weight parameters are loaded, and the user's natural language text is received as the input parameter of the model and passed into the interface. Then, the structured parsing results are obtained through the model's predictive reasoning and returned in Json format. By encapsulating the model as an interface and providing services to the outside, a logical closed loop is formed from tasks, data, models to services.

[0143] Embodiment 2

[0144] An embodiment of a power marketing data query system based on a JointBERT model of a knowledge graph in the present invention includes the following steps:

[0145] The knowledge graph construction unit is used to obtain the background knowledge of the power industry and construct the knowledge graph;

[0146] A JointBERT model construction and training unit, which is used to construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to words corresponding to the input text.

[0147] A target JointBERT model determination unit is used to compress and optimize the JointBERT model by using an early withdrawal mechanism and a knowledge distillation method to obtain a target JointBERT model;

[0148] The data query and analysis unit is used to input the power demand problems of users received in real time as input parameters into the target JointBERT model for data query and analysis.

[0149] Embodiment 3

[0150] The present invention provides a computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the above method are implemented.

[0151] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0152] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored. In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0154] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A power marketing data query method based on the JointBERT model of knowledge graph, characterized in that: include: Acquire background knowledge related to the power industry and build a knowledge graph; Construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to words corresponding to the input text. The JointBERT model is compressed and optimized by using an early exit mechanism and a knowledge distillation method to obtain a target JointBERT model; The power demand problems of users received in real time are used as input parameters and input into the target JointBERT model for data query and analysis.

2. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 1 is characterized in that: The acquisition of field background knowledge related to the power industry and the construction of a knowledge graph include: Extracting entities, relations and attributes based on text and data in the domain background knowledge; Knowledge representation and knowledge fusion are performed based on the extracted entities, relationships and attributes to construct a knowledge graph, and the constructed knowledge graph is stored in a preset database.

3. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 1 is characterized in that: The input layer is used to process input text data, including: The beginning of the input text is marked as a classification, the end of the input text is marked as a segmentation, and the length of the input text is truncated and padded.

4. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 1 is characterized in that: The encoding layer is used to fuse the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph, including: By encoding fusion, the rich general semantic knowledge in the massive corpus and the domain knowledge in the power domain knowledge graph are encoded into a unified vector representation; After calculating the attention scores of the feature vectors, the attention distribution between the tokens in the input text is obtained; Normalize the attention scores of each input vector and use them as weights for feature fusion; By stacking multiple Encoder blocks, high-level feature representations of the input text are gradually extracted.

5. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 1 is characterized in that: The decoding layer is used to generate the domain label, intent label and slot label corresponding to the input text, including: The encoded features are input into the BERT Model layer for processing to obtain the corresponding hidden state sequence; Inputting the hidden layer vectors corresponding to the characters in the hidden state sequence into the domain classification layer and the intention recognition layer respectively; The number of neurons of the characters in the hidden state sequence is linearly converted into the number of predefined domain and intent label categories by using the linear layer of the neural network, and the corresponding domain label, intent label and slot label corresponding to the word are determined respectively.

6. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 5 is characterized in that: The expressions of the domain label, intent label, and slot label corresponding to the word are as follows: Probability distribution of the field: p(D|x)=softmax(W d ·h [CLS] +b d ); Where: D is the label category of the field, x is the original input sentence, h [CLS] is the hidden layer vector corresponding to "[CLS]", W d is the domain weight matrix, b d is the domain offset vector; Probability distribution of intent: p(I|x)=softmax(W i ·h [CLS] +b i ); Where: I is the label category of the intent, W i is the intention weight matrix, b i is the intention offset vector; Probability distribution of slot labels corresponding to words: p(S|x)=softmax(W s ·h [CLS] +b s ); Where: S is the label category of the slot corresponding to the word, W s is the slot weight matrix, b s is the slot offset vector.

7. The power marketing data query method based on the JointBERT model of the knowledge graph according to claim 1 is characterized in that: The early exit mechanism and the knowledge distillation method are used to compress and optimize the JointBERT model to obtain a target JointBERT model, including: Insert a classification layer between each Transformer layer in the student model to perform domain classification, intent recognition, and slot filling tasks, and judge whether the model meets the patience-based early retirement criteria based on the results; The Self-attention mechanism of the JointBERT model is replaced with the ProbSpare self-attention mechanism, and a distillation mechanism is introduced when multiple attention layers are superimposed.

8. A power marketing data query system based on the JointBERT model of knowledge graph, characterized in that: include: The knowledge graph construction unit is used to obtain the background knowledge of the power industry and construct the knowledge graph; A JointBERT model construction and training unit, which is used to construct a JointBERT model and perform model training. The JointBERT model includes an input layer, an encoding layer, and a decoding layer. The input layer is used to process input text data. The encoding layer is used to fuse and encode the rich general semantic knowledge in the corpus with the domain knowledge in the knowledge graph. The decoding layer is used to generate domain labels, intent labels, and slot labels corresponding to words corresponding to the input text. A target JointBERT model determination unit, used to compress and optimize the JointBERT model by using an early withdrawal mechanism and a knowledge distillation method to obtain a target JointBERT model; The data query and analysis unit is used to input the power demand problem of the user received in real time as an input parameter into the target JointBERT model for data query and analysis.

9. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including executing the power marketing data query method based on the JointBERT model of the knowledge graph according to any one of claims 1 to 7; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

10. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, enable the computer to execute the power marketing data query method based on the JointBERT model of the knowledge graph as described in any one of claims 1 to 7.

Images