BERT model and graph neural network-based power grid cost project information extraction and anomaly detection method
By combining the BERT model with graph neural networks, the problems of accuracy and intelligence in power grid project information extraction and anomaly detection have been solved, achieving efficient project management and resource optimization.
Patent Information
- Application Number
- CN202511805971.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-01-16
AI Technical Summary
Existing power grid companies have problems in special cost management, such as inaccurate information extraction, inability to effectively process complex texts, and lack of dynamic modeling of complex relationships between projects, which makes it difficult to identify potential risks and abnormal resource consumption during project implementation.
The BERT model is used for contextual semantic modeling, and graph neural networks and T5 models are used for information cleaning and preprocessing. Graph neural networks are used to capture complex dependencies between projects, and Gaussian mixture models and Wasserstein distance are used for anomaly detection.
It improves the accuracy and efficiency of information extraction, enhances the intelligence level of project document analysis, and can accurately identify abnormal resource consumption, thereby optimizing resource allocation and management.
Smart Images
Figure CN121350252A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information processing and management technology, specifically to a method for extracting and detecting anomalies in power grid cost items based on the BERT model and graph neural network. Background Technology
[0002] With the continuous development of the power industry, power grid companies face enormous challenges in ensuring power supply, improving grid stability, and adapting to the integration of new energy sources. Against this backdrop, power grid-specific cost management, as a core element in ensuring the company's operational efficiency and rational resource allocation, has become increasingly important. Power grid companies are often involved in numerous construction, maintenance, and renovation projects, each of which requires detailed cost estimates and effective expense control.
[0003] Currently, the power grid industry has begun to introduce automation technologies and intelligent analysis methods into its special cost management. Some enterprises have started using rule-based automated information extraction methods to process key information in project documents and utilize data cleaning and standardization techniques to improve data quality. In addition, traditional clustering algorithms and anomaly detection methods are also being applied to project classification and risk identification. While these methods have improved the efficiency of project management to some extent, they have significant limitations. First, rule-based information extraction methods cannot effectively handle complex text, especially when encountering polysemous words, synonyms, and contextual differences, where the accuracy of extraction is greatly reduced. Second, existing clustering and anomaly detection technologies mainly focus on static data analysis and lack dynamic modeling of complex relationships between projects, making it difficult to identify potential risks and abnormal resource consumption during project implementation. Therefore, existing technologies cannot meet the needs of power grid enterprises for efficient, accurate, and intelligent special cost management. Summary of the Invention
[0004] The purpose of this invention is to provide a method for extracting power grid cost item information and detecting anomalies based on the BERT model and graph neural network, in order to solve the technical problems pointed out in the background art.
[0005] The objective of this invention can be achieved through the following technical solutions:
[0006] A method for extracting power grid cost item information and detecting anomalies based on BERT model and graph neural network includes:
[0007] S1: By automatically extracting key project information, we use the contextual semantic modeling technology of the BERT model to extract structured data from textual materials. By defining and selecting key information fields, we combine the BERT model to perform deep semantic modeling, ensuring accurate parsing and unified standardization of project description information.
[0008] S2: Perform information cleaning and preprocessing, use the T5 model for multi-task learning, and combine it with a generative adversarial network;
[0009] S3: Use graph neural networks to learn the multi-dimensional features of projects, capture the complex dependencies between projects, obtain high-quality embedding representations for each project, use the SimCLR self-supervised learning method to further optimize the embedding representations of projects, and use spectral clustering to identify the structured relationships between projects and group similar projects into the same cluster.
[0010] S4: Fit historical project data of the same type using Gaussian mixture models, statistically analyze the resource consumption of each project, establish normal consumption ranges, and use Z-score and Wasserstein distance to detect and determine anomalies in the resource consumption of the project to be implemented.
[0011] Preferably, in contextual semantic encoding based on the BERT model, BERT is used for deep semantic modeling after the text preprocessing stage; the formula for the BERT model is:
[0012]
[0013] in, The input is a sequence of text tokens; This is the corresponding contextual semantic representation; This is the parameter set for the BERT model.
[0014] Preferably, when performing semantic normalization and synonym fusion, the domain ontology lexicon and semantic mapping rules are used to solve the problem of diverse expressions in power grid project documents. This is achieved through cosine similarity calculation of word vectors. The formula for cosine similarity of word vectors is:
[0015] ;
[0016] in, and Each word and Word vectors; Cosine similarity is the similarity between two words, used to determine the similarity between words.
[0017] Preferably, the T5 model processes multiple tasks in parallel, involving the following expressions:
[0018]
[0019] in, 1 represents the original text data input, i.e., the project document fragment to be corrected; This is the corrected text output by T5; These are the parameters for the T5 model.
[0020] Preferably, a generative adversarial network (GAN) is introduced. The GAN is trained adversarially between a generator and a discriminator. The generator formula is as follows:
[0021] ;
[0022] Discriminator formula:
[0023] ;
[0024] in, The T5 corrected text or its vector representation; Further refine the text output by the generator; For generator networks, Its parameters; For the discriminator network, Its parameters; This is the probability score given by the discriminator for whether the input text is real and conforms to the specifications.
[0025] Preferably, the Skip-gram Word2Vec model is used when training domain word vectors:
[0026]
[0027] in, As the central word, For context words; The corresponding word vectors; The vocabulary is derived from the classic probabilistic form of neural language models; domain-specific word vectors are trained on a corpus of power grid projects; probability. It is dimensionless, and both the numerator and denominator are in exponential form.
[0028] Preferably, through multi-layer graph convolution operations, the high-dimensional feature representation of the item will be gradually compressed into a low-dimensional space, ultimately obtaining the embedded representation of each item.
[0029] Preferably, after obtaining the optimized item embedding representation, spectral clustering is used for cluster analysis; the similarity between items is calculated using Euclidean distance or cosine similarity, a Laplacian matrix is constructed, and the eigenvalues and eigenvectors of the Laplacian matrix are calculated. The smallest eigenvalue is selected. The eigenvectors corresponding to each eigenvalue are used to form a low-dimensional representation of the project;
[0030] K-means clustering is performed on the obtained low-dimensional embeddings to assign similar items to the same cluster; based on the spectral clustering and variational inference results, an association network of items is constructed to show the similarity between items and the clustering results; finally, each item is assigned to a cluster and the clustering results are output.
[0031] Preferably, when determining outliers, the Z-score is calculated and analyzed. If the calculated Z-score exceeds a preset threshold, the resource consumption of the project is determined to be abnormal.
[0032] Preferably, the resource consumption data of historical projects are distributed and modeled, and the Wasserstein distance between the resource consumption distribution of the project to be implemented and the distribution of historical projects is calculated. If the Wasserstein distance exceeds a preset threshold, the project is determined to be an abnormal project.
[0033] Compared to existing solutions, the beneficial effects achieved by this invention are:
[0034] This invention combines advanced natural language processing technology with graph neural networks to effectively improve the information extraction and analysis capabilities in power grid project management. By using the BERT model for contextual semantic modeling, it can accurately extract key information such as project name, budget, and cost details from project documents, significantly improving the accuracy and efficiency of information extraction. The BERT model, through its bidirectional context awareness capability, greatly enhances the intelligence level of project document analysis.
[0035] This invention makes significant improvements in information cleaning and data standardization. It adopts the T5 model for multi-task learning, automatically correcting spelling errors, grammatical problems, and standardizing units, which greatly improves the automation and accuracy of data processing. The T5 model can automatically handle different text correction tasks through multi-task learning, and combined with generative adversarial networks, it further optimizes the fluency and naturalness of the text, thereby improving the effectiveness and efficiency of text correction, ensuring the consistency, standardization, and high quality of the data, providing reliable input data for subsequent analysis and decision-making, and comprehensively improving the intelligence level of the power grid project management system.
[0036] This invention combines graph neural networks with self-supervised learning, overcoming the limitations of traditional clustering methods in project classification and grouping. GNN can perform deep learning based on the multidimensional feature relationships between projects, effectively identifying dependencies and structured relationships between projects. Furthermore, it optimizes the embedding representation through self-supervised learning, making the clustering results more accurate.
[0037] This invention detects abnormal resource consumption in a multi-dimensional way by combining Gaussian Mixture Model (GMM), Z-score, and Wasserstein distance. By fitting historical project data with GMM and combining Z-score and Wasserstein distance, it can accurately identify whether resource consumption exceeds the normal range on a dynamic basis, thereby improving the sensitivity to project resource consumption. It can also dynamically adjust the judgment criteria according to the characteristics of different projects, effectively preventing resource waste and optimizing resource allocation. Attached Figure Description
[0038] The present invention will now be further described with reference to the accompanying drawings.
[0039] Figure 1 This is a flowchart illustrating the operation of a method for extracting power grid cost item information and detecting anomalies based on the BERT model and graph neural network according to the present invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] like Figure 1 As shown, this invention is a method for extracting power grid cost item information and detecting anomalies based on the BERT model and graph neural network, comprising:
[0042] S1: By automatically extracting key project information, we use the contextual semantic modeling technology of the BERT model to extract structured data from textual materials. By defining and selecting key information fields, we combine the BERT model to perform deep semantic modeling, ensuring accurate parsing and unified standardization of project description information.
[0043] S2: Perform information cleaning and preprocessing, use the T5 model for multi-task learning, and combine it with a generative adversarial network;
[0044] S3: Use graph neural networks to learn the multi-dimensional features of projects, capture the complex dependencies between projects, obtain high-quality embedding representations for each project, use the SimCLR self-supervised learning method to further optimize the embedding representations of projects, and use spectral clustering to identify the structured relationships between projects and group similar projects into the same cluster.
[0045] S4: Fit historical project data of the same type using Gaussian mixture models, statistically analyze the resource consumption of each project, establish normal consumption ranges, and use Z-score and Wasserstein distance to detect and determine anomalies in the resource consumption of the project to be implemented.
[0046] In step S1, information collection and extraction of cost items are carried out. During the information collection and extraction stage, key information is automatically extracted from textual materials such as feasibility study reports and project proposals, mainly including project name, budget amount, implementing unit, and cost details. Through contextual semantic modeling of the BERT model, it is ensured that the input text can be transformed into high-quality structured data, providing accurate support for subsequent analysis and decision-making.
[0047] When selecting fields and collecting data, key information fields that are crucial to the analysis of power grid projects are defined and selected through expert systems or manual input, such as project name, implementing unit, budget amount, project content, and cost details. These fields provide a basic framework for subsequent text analysis, enabling the extracted text information to accurately correspond to different dimensions of the project.
[0048] In contextual semantic encoding based on the BERT model, deep semantic modeling is performed using BERT after the text preprocessing stage. BERT, based on the Transformer architecture, possesses bidirectional context awareness capabilities, enabling it to handle the contextual relationships of the input text and provide high-precision semantic representations for subsequent tasks. The formula for the BERT model is:
[0049]
[0050] in, The input is a sequence of text tokens, such as project description text obtained through word segmentation; This is the corresponding contextual semantic representation; This refers to the parameter set of the BERT model.
[0051] This formula originates from a contextual semantic modeling method based on the Transformer encoder structure, a classic model in the field of natural language processing. Building upon this, this invention introduces a power grid cost project corpus during training, uniformly incorporating tokens corresponding to fields such as project name, budget amount, cost details, and implementing unit. Parameters adapted to the power grid field are obtained through fine-tuning. and output This serves as the input feature for subsequent sequence labeling and synonym normalization.
[0052] Both sides of the formula are dimensionless vector sequences, representing the numerical representation of the text in the semantic space, with consistent dimensions; subsequent calculations such as BiLSTM, CRF, cosine similarity, and SimCSE are all performed using these vector sequences. Based on this;
[0053] For example, when processing power grid project descriptions, the BERT model can associate "power grid renovation project" with "XX Power Company" through contextual relationships, and provide a deep semantic representation for subsequent relationship extraction tasks.
[0054] When performing sequence labeling and semantic segment recognition, BiLSTM-CRF is the core technology in information extraction. It uses a bidirectional long short-term memory network to capture the contextual dependencies of the text, and a conditional random field is used for global label optimization to ensure that the dependencies between labels do not violate the actual scenario.
[0055] BiLSTM is a bidirectional version of the Long Short-Term Memory network. While LSTM is suitable for capturing long-term dependencies in sequence data, BiLSTM further enhances the understanding of context by using both forward and backward methods. Its main structure can be described as a combination of two LSTM layers, one that processes information from the forward direction and the other that processes information from the backward direction.
[0056] For each moment The core computation process of LSTM is as follows:
[0057] The gating and state update process of a unidirectional LSTM is as follows:
[0058]
[0059] in, For the first The input vector at each time step is taken from the token representation of BERT output in this embodiment of the invention; This represents the hidden state at the current moment; This is the unit state; These are the input gate, forget gate, and output gate, respectively. These are the trainable parameters of the LSTM; For the Sigmoid function; This is element-wise multiplication;
[0060] Based on this, a bidirectional LSTM introduces two LSTMs, a forward LSTM and a backward LSTM, and the outputs are concatenated as follows:
[0061]
[0062] The formula originates from the classic structures of recurrent neural networks and long short-term memory networks; input The output vector of BERT is further captured by the bidirectional structure of BiLSTM to capture the contextual dependencies in the text of power grid cost items, providing more stable contextual features for determining labels such as cost type, work content, and time information;
[0063] Each variable is a dimensionless real vector, representing semantics or state, and does not involve physical units; gating value The range of values is Output It can be directly used as input to the CRF conditional probability model;
[0064] CRF is used to optimize the dependencies between labels in a labeled sequence. CRF avoids irrational label combinations by maximizing the conditional probability of sequence labels; for each time step Its goal is to optimize the joint distribution of labels, and the relevant expression is:
[0065]
[0066] in: For the first Tags for each token, such as B-FT, B-WC, etc.; Let be the transition matrix in a CRF, representing the transition probabilities between labels; The conditional probability of the label is obtained through BiLSTM, and the relevant expression is:
[0067] in, Features provided by BERT–BiLSTM for the observed sequence; The tag sequence, in this embodiment of the invention, includes B-FT (fee type), B-WC (work content), B-TI (time information), B-DO (object unit), etc.; The elements of the label transition matrix represent the transition from label to label. Transfer to label The score; For time step Label The local score is obtained by mapping the BiLSTM output; The partition function is used to normalize all possible label sequences.
[0068] This formula belongs to the standard form of Conditional Random Fields (CRFs). It designs a BIO tagging system and tag transition constraints for power grid cost projects, enabling the CRF to automatically learn business rules such as cost types not being directly followed by certain time tags, thereby improving the accuracy of field extraction; probability. Dimensionless, matrix elements Local score Score the real numbers;
[0069] When setting tags, the BIO tagging system is used to label which category each word belongs to:
[0070] B-FT: Fee type, such as "Inspection fee"; B-WC: Work content, such as "Organizing training"; B-TI: Time information, such as "May 2024"; B-DO: Target unit, such as "XX Power Company";
[0071] For the input text: "Organize expert training (work content), May 2024 (time information), testing fee and labor cost (fee type)";
[0072] The model will output: Organization (B-WC), Experts (I-WC), Training (I-WC), May 2024 (B-TI), Testing Fee (B-FT), Labor Cost (B-FT).
[0073] Using this method, the model can accurately identify key information fields in the text.
[0074] When performing semantic normalization and synonym fusion, the domain ontology lexicon and semantic mapping rules are used to address the diverse expressions in power grid project documents. The standard expression of synonyms is unified through cosine similarity calculation of word vectors. The formula for cosine similarity of word vectors is as follows:
[0075]
[0076] in, and Each word and The word vectors can be derived from embeddings trained by BERT, Word2Vec, or SimCSE. Cosine similarity is the similarity between two words, used to determine the similarity between words; Represents the dot product of vectors; It is the Euclidean norm;
[0077] Synonym classification and standardization are performed. Through cosine similarity calculation, words with similarity scores exceeding a set threshold are grouped and merged into standardized terms. The relevant expression is:
[0078]
[0079] in, It is a set of synonyms to be standardized, such as "testing fee", "experiment fee", and "experimental testing expenditure". These are standardized terms, such as "testing fees".
[0080] "Testing fees" and "experiment fees" are now categorized under "testing-related fees" to eliminate the difference between the two synonyms.
[0081] Finally, after BERT modeling, BiLSTM-CRF sequence labeling, synonym processing and semantic standardization, the extracted information will be transformed into structured data. Through the high-dimensional semantic vectors extracted by BERT and the annotation information provided by BiLSTM-CRF, information such as project name, budget amount, and implementing unit is accurately identified and extracted, and the final output is standardized structured data.
[0082] In this embodiment of the invention, by combining natural language processing technology with graph neural networks, the information extraction and analysis capabilities in power grid project management can be effectively improved. Traditional power grid project management relies on manual methods for data extraction, which is inefficient and prone to errors, especially when processing large-scale unstructured text. By employing the BERT model for contextual semantic modeling, this invention can accurately extract key information such as project name, budget, and cost details from project documents, significantly improving the accuracy and efficiency of information extraction. Furthermore, traditional methods typically rely solely on rule matching and cannot handle polysemous words, synonyms, and complex contextual information in text. The BERT model solves this problem through its bidirectional contextual awareness capability, greatly enhancing the intelligence level of project document analysis.
[0083] In step S2, information cleaning and preprocessing are performed. The automatically extracted information undergoes deep cleaning and standardization to ensure data consistency, accuracy, and operability. Methods such as spell correction, grammar correction, unit standardization, and synonym processing are used to eliminate noise in the text, standardize terminology and format, and provide high-quality input data for subsequent analysis tasks.
[0084] Spelling errors and grammatical problems are very common in text, especially when extracting information from unstructured text data. These errors often affect the quality of information extraction, interfering with subsequent model learning. Therefore, spelling correction and grammatical correction are essential steps in the information cleaning stage. To this end, the T5 model is used for multi-task learning, training it to perform spelling correction, grammatical correction, and grammar recognition tasks, ensuring more efficient text correction.
[0085] The T5 model is trained through multi-task learning to perform spelling correction, grammar correction, and grammar recognition tasks. It can efficiently correct spelling errors and grammar problems in text by processing multiple tasks in parallel. The relevant expressions are:
[0086]
[0087] in, The input is the raw text data, i.e., the fragment of the project document to be corrected; This is the corrected text output by T5; For T5 model parameters;
[0088] The T5 model employs a Transformer architecture to achieve multi-task learning of the input text. In this embodiment, the training objective of T5 targets spelling errors, grammatical errors, and non-standard expressions in power grid project texts. It simultaneously learns error correction and grammatical correction through a multi-task loss function, resulting in a model adapted for the power grid field. .
[0089] All variables are dimensionless text or their embeddings, belonging to a symbol-vector mapping relationship. In terms of workflow, the output content enters the generative adversarial network (GAN) and BERT encoding module, providing more standardized text input for subsequent information extraction and feature modeling. During training, T5 can handle multiple tasks simultaneously within a single model framework. The shared learning approach enables the model to more efficiently correct various errors in the text. Furthermore, to further improve the text correction effect, a GAN is introduced. Through adversarial training between the generator and discriminator, the GAN continuously optimizes the quality of text correction, generating more natural and fluent text.
[0090] Generative Adversarial Networks (GANs) are trained adversarially using a generator and a discriminator to correct spelling errors and grammatical problems in text. The generator formula is as follows:
[0091]
[0092] Discriminator formula:
[0093]
[0094] in, The T5 corrected text or its vector representation; Further refine the text output by the generator; For generator networks, Its parameters; For the discriminator network, Its parameters; This is the probability score given by the discriminator for whether the input text is real and conforms to the specifications;
[0095] Derived from the Generative Adversarial Network (GAN) framework, this approach continuously improves the consistency between generated and real samples through adversarial training between the generator and discriminator. It uses manually revised text as real samples and T5 outputs and their rewrites as generated samples. By optimizing the corresponding adversarial loss, it guides the generator to produce text that better matches the style of real project documents.
[0096] The process of combat training:
[0097] The generator and discriminator are optimized through adversarial training:
[0098] Generator: Generates corrected text from the original text;
[0099] Discriminator: The discriminator evaluates the generated text to determine whether it matches the characteristics of real text;
[0100] Adversarial training: Through adversarial training, the generator continuously improves the text correction effect, while the discriminator guides the generator to optimize the quality of the corrected text by continuously evaluating the generated text.
[0101] When standardizing numbers and units, power grid project data often includes numerical data such as budget amounts, quantities, and times. The formats and units of this data are frequently inconsistent, which can affect subsequent analysis and decision-making. Standardization of numbers and units is particularly crucial when processing budget data. Therefore, meta-learning and generative adversarial networks are chosen to automatically select appropriate standardization strategies, ensuring consistent numerical formats and unified units. Meta-learning automatically selects the most suitable standardization strategy based on the data type; it also automatically selects the appropriate standardization method through feature learning of the data.
[0102] Among them, the meta-learning framework selects the optimal standardization strategy:
[0103]
[0104] in, This is a set of candidate standardization strategies, such as unifying the monetary unit to yuan, unifying the monetary unit to ten thousand yuan, and unifying the date to YYYY-MM-DD format, etc. This is a dataset of historical projects used for evaluation; To adopt a strategy After standardization, the loss function for data quality or downstream task performance can comprehensively consider indicators such as unit conflict rate, field missing rate, and downstream model error. The strategy that minimizes loss;
[0105] This approach originates from the model selection concept in optimization theory. It automatically selects appropriate unit conversions and format standardization methods for different fields, ensuring that resource consumption data used in GMM, Z-score, and Wasserstein distance calculations have consistent and reasonable physical dimensions.
[0106] When handling synonyms and ensuring semantic consistency, the same term or concept may have multiple expressions in different documents, leading to semantic inconsistencies. To ensure data uniformity and consistency, BERT and Word2Vec technologies are used to unify synonyms and eliminate semantic differences in expression. BERT and Word2Vec technologies are used for synonym processing to eliminate semantic differences in expression and ensure semantic consistency of data; for example, "budget amount" and "total cost" are uniformly encoded using BERT.
[0107] When training domain word vectors, the Skip-gram Word2Vec model is used:
[0108]
[0109] in, As the central word, For context words; The corresponding word vectors; The vocabulary is derived from the classic probabilistic form of neural language models. Domain word vectors are trained on the power grid project corpus to give terms such as budget amount, total cost, and detection fee a more semantically consistent distribution representation. This is then combined with cosine similarity for terminology unification and synonym unification.
[0110] probability It is dimensionless, and both the numerator and denominator are in exponential form;
[0111] SimCSE is a self-supervised learning method for generating vector representations with high semantic consistency, playing a crucial role, particularly in synonym processing and terminology unification. To further optimize item embeddings and bring similar items closer together in the vector space, a contrastive learning loss based on the SimCLR framework is introduced. For the ... For each sample, the loss can be defined as:
[0112]
[0113] in, and The embedding vector for positive sample pairs can be obtained from the same item under different data augmentation methods in this embodiment of the invention; For embedding in other projects; For the similarity function, cosine similarity can be used; Temperature coefficient; The total number of samples;
[0114] This formula is derived from the InfoNCE loss form of self-supervised contrastive learning. By combining the application of the formula, the embedding vectors of the same or similar items are more clustered, which enhances the clustering effect of spectral clustering and makes the item classification results more in line with business logic.
[0115] SimCSE, through contrastive learning, can generate semantically consistent vector representations, effectively eliminating differences between synonyms. A domain ontology library is used to standardize terminology in power grid projects, ensuring consistency of project terminology across different documents. By establishing an ontology library, commonly used terms and definitions in the power grid industry are unified, avoiding terminology confusion.
[0116] In this embodiment of the invention, the information cleaning and preprocessing uses advanced deep learning technologies such as T5, GAN, BERT, Word2Vec, and SimCSE for spell correction, grammar correction, unit standardization, and synonym processing. Through multi-task learning and generative adversarial training, it can efficiently correct spelling and grammar errors in the text, ensure data format consistency and semantic consistency, and provide high-quality input data for subsequent project analysis.
[0117] In step S3, the cleaned project data includes project content, cost type, and cost composition. Cluster analysis and project association modeling are performed using a joint optimization method of graph neural network and self-supervised learning. By utilizing the similarity between projects, the quality of the embedding representation is improved through self-supervised learning. The complex relationships between projects are captured by graph neural network, providing data support for subsequent optimization decisions and resource allocation.
[0118] In project clustering analysis, the similarity between projects is not only reflected in the text content but also in the implicit relationships between them. These relationships cannot be represented by traditional Euclidean distance calculations but are better modeled using graph structures. Graph neural networks, as a deep learning model, can capture complex dependencies between nodes by propagating node information through adjacency matrices.
[0119] Graph Neural Networks (GNNs) can effectively capture complex dependencies between projects. Projects are viewed as nodes in a graph, with project features (project content, cost type, cost composition) acting as node attributes, and project similarities as edges. By learning multi-dimensional features of projects through GNNs, high-quality embedding representations of each project are obtained.
[0120] Step a: Graph construction, including node representation, edge representation, and adjacency matrix construction;
[0121] Node representation: Each project is considered as a node in the graph. The characteristics of a node consist of the project's multi-dimensional features, including project description, cost type, and cost composition.
[0122] Edge representation: The similarity between projects, for example, based on content similarity, fee type, cost composition, etc., as the weight of the edge; similarity calculation can be defined by cosine similarity or Gaussian kernel;
[0123] Gaussian kernel calculation formula:
[0124]
[0125] in, The first , No. The feature vectors of each project include multi-dimensional features such as text embedding, cost composition, and resource structure; This refers to the Gaussian kernel bandwidth parameter; For the project With the project The similarity between them;
[0126] Derived from kernel methods and radial basis function kernels, similarity is a commonly used similarity measurement tool in machine learning. This step uses kernel methods to construct a similarity matrix. The features are then converted into an adjacency matrix of the graph for subsequent graph convolution and spectral clustering; the feature vectors are normalized to be dimensionless, and the distance... It is also dimensionless, and the output of the exponential function is... Falling , which is a dimensionless similarity value;
[0127] Adjacency Matrix Construction: The calculated similarity matrix is transformed into an adjacency matrix of the graph, representing the connection relationships between items.
[0128]
[0129] in, This is the adjacency matrix of the project graph, where the elements are the similarity between projects, which can be obtained from cosine similarity. This is a degree matrix, with the diagonal elements representing the degree values of each node; For a standardized adjacency matrix;
[0130] Step b: Graph convolution operation;
[0131] Graph convolution operations can weighted aggregate node features with the features of neighboring nodes to obtain a new embedding representation for each node; this process integrates multi-layer information and captures more complex relationships; Graph convolution operation formula:
[0132] The update of each layer in a graph convolutional network can be written as:
[0133]
[0134] in, For the first The node feature matrix of the layer, Features of the initial project; , For the first Layer weights and biases; It is a non-linear activation function;
[0135] Derived from the standard derivation of graph convolutional networks, this is a first-order approximate convolution of graph signals based on the graph Laplacian. Combining the formula's principles, by stacking multiple layers of graph convolution, the multidimensional features of items are fused with information from neighboring items to obtain an embedded representation containing inter-item dependencies, providing foundational features for subsequent self-supervised contrastive learning and spectral clustering; matrix All of them are dimensionless numerical matrices;
[0136] Step c: Training and node embedding representation of the graph neural network;
[0137] Through multi-layer graph convolution operations, the high-dimensional feature representation of each item is gradually compressed into a low-dimensional space, ultimately yielding the embedding representation of each item. This representation can effectively reflect the similarities and connections between projects;
[0138] When performing self-supervised learning to optimize embedding representations, SimCLR is a self-supervised learning method based on contrastive learning, which is widely used in embedding learning and feature optimization. In the absence of explicit labels, SimCLR optimizes the embedding representation of items by maximizing the similarity of similar samples and minimizing the similarity of different samples, making the representation space of items more suitable for cluster analysis. SimCLR can be used to further optimize the embedding representation of items, ensuring that the embedding vectors of similar items are closer together.
[0139] SimCLR is used to optimize the item embedding representation, minimizing the distance between similar samples and maximizing the distance between negative samples. SimCLR uses contrastive loss to ensure that similar items in the embedding space are close together and different items are dispersed. The relevant expression is:
[0140] ;
[0141] in, and It is the first and the Embedded representation of each item; It is a temperature parameter that controls the degree of similarity adjustment; It is the number of negative samples, which defines the number of negative samples in contrastive learning;
[0142] By optimizing the SimCLR loss, the quality of item embeddings is further improved, making similar items closer in the vector space and different items farther apart.
[0143] After obtaining the optimized item embedding representation, cluster analysis is performed using spectral clustering: spectral clustering can effectively identify structured relationships between items in low-dimensional data and is suitable for handling complex, non-spherical datasets;
[0144] The similarity between items is calculated using Euclidean distance or cosine similarity. Similarity calculation is the foundation of cluster analysis and can be obtained using the following formula:
[0145] ;
[0146] Constructing the Laplace matrix Used for graph cutting in spectral clustering:
[0147] ;
[0148] in, It is a degree matrix, defined as a diagonal matrix. A is the similarity matrix;
[0149] The spectral decomposition process can be represented as follows:
[0150]
[0151] in, For eigenvalues; For the corresponding feature vector;
[0152] The formula is derived from spectral theory and spectral clustering methods. By selecting a number of smaller eigenvalues and their eigenvectors, the project is mapped to a low-dimensional space. Then, the K-means algorithm is used to achieve clustering. This method automatically classifies projects into categories such as construction, research and consulting, parts procurement, testing, information operation and maintenance, and publicity and promotion, creating conditions for subsequent GMM modeling and anomaly detection within similar project sets.
[0153] Calculate the eigenvalues and eigenvectors of the Laplacian matrix, and select the smallest one. The eigenvectors corresponding to each eigenvalue are used to form a low-dimensional representation of the project:
[0154] ;
[0155] in, It is the first 1 eigenvector; These are the corresponding eigenvalues;
[0156] K-means clustering is performed on the obtained low-dimensional embeddings to assign similar items to the same cluster; based on the spectral clustering and variational inference results, an association network of items is constructed to show the similarity between items and the clustering results;
[0157] Finally, each item is assigned to a cluster, and the clustering results are output:
[0158] ;
[0159] in, For clusters of all projects, arrive A set for each cluster; The total number of clusters.
[0160] By combining graph neural networks and self-supervised learning, information can be effectively extracted from the multimodal features of projects, and cluster analysis is performed using spectral clustering. Ultimately, this innovative method efficiently captures the complex relationships between projects and provides a strong data foundation for subsequent decision support. The final classification results are shown in the table below:
[0161] Serial Number Primary Classification Project investment direction 1 Construction 1. Substation hazard mitigation; 2. Transmission line hazard mitigation; 3. Cable duct and fiber optic cable management; 4. Construction of star-rated power supply stations. 2 Research and Consulting 1. Research-based 2. Consulting-based 3 Parts purchase 1. Purchase of electricity metering devices 2. Purchase of smart electricity devices 4 Detection 1. Substation hazard mitigation 2. Transmission line hazard mitigation 3. Distribution equipment hazard mitigation 4. Feeder hazard mitigation 5. Drone inspection 5 Information Operations and Maintenance Hardware, software and infrastructure maintenance 6 Promotional and advertising 1. Brand enhancement 2. Corporate promotion 3. Business environment promotion
[0162] In this embodiment of the invention, a combination of graph neural networks and self-supervised learning is employed to overcome the limitations of traditional clustering methods in project classification and grouping. Traditional clustering algorithms often ignore the complex relationships between projects, resulting in poor classification performance and an inability to fully capture the subtle relationships between projects. By treating projects as nodes in a graph, GNNs can perform deep learning based on the multidimensional feature relationships between projects, effectively identifying dependencies and structured relationships between projects. Furthermore, self-supervised learning optimizes the embedding representation, making the clustering effect more accurate. This innovation provides a more scientific basis for resource allocation, risk assessment, and other decisions in power grid projects, improving the decision-making efficiency of power grid project management.
[0163] In step S4, the resource consumption of each project, including labor, machine shifts and material consumption, is statistically analyzed using historical project data. The normal range of consumption is calculated using a Gaussian mixture model (GMM), and then it is determined whether the resource consumption of the project to be implemented is abnormal.
[0164] When conducting statistical analysis of resource consumption based on the Gaussian mixture model, historical project data is used to calculate the mean and standard deviation of labor, machine hours, and material consumption. This provides a foundation for subsequent calculations of normal consumption ranges.
[0165] Gaussian Mixture Models (GMMs) are used to fit historical project data. GMMs can model multimodal data through multiple Gaussian distributions and reflect the resource consumption patterns under different resource consumption modes. The GMM is trained using the Expectation-Maximization (EM) algorithm to obtain the distribution of each resource consumption type. The mean ± 1.96 standard deviations are used to calculate the normal range for each resource consumption, thereby establishing a normal range and providing a benchmark for subsequent outlier detection.
[0166] Gaussian Mixture Models (GMMs) are powerful generative models that can fit multimodal data distributions. Since resource consumption typically exhibits multimodal characteristics, modeling with GMMs can effectively identify consumption patterns in different projects and support subsequent outlier detection.
[0167] Step A: Gaussian Mixture Model (GMM) Modeling
[0168] Gaussian distribution modeling: assuming the resource consumption of each project It is composed of a mixture of multiple Gaussian distributions. Gaussian Mixture Models (GMMs) can learn the multimodal characteristics of data through multiple Gaussian distributions and help us derive the normal range of resource consumption. The formula for a Gaussian Mixture Model is:
[0169]
[0170] in, As a resource consumption vector, it can include dimensions such as labor quantity, man-day, machine shift quantity, machine shift, material quantity, and yuan; For the first The mixing weights of the Gaussian components satisfy... ; For the first The mean vector of Gaussian components; It is the covariance matrix; The number of Gaussian components;
[0171] Training the GMM: The Expectation-Maximization (EM) algorithm is used to train the GMM and optimize it. , and This allows us to learn the multimodal distribution of resource consumption, using the log-likelihood function formula:
[0172]
[0173] in, For the set of model parameters; For the first Resource consumption data for historical projects; This represents the number of samples.
[0174] The principle of this method is based on Gaussian mixture model and maximum likelihood estimation. A Gaussian mixture model (GMM) is fitted individually to each item category obtained from spectral clustering, and the maximization problem is solved using the EM algorithm to obtain the distribution of this item category in terms of resource consumption dimensions such as labor, machine hours, and material consumption. Furthermore, the normal consumption range is calculated from the mean and standard deviation of each component, providing a parameter basis for subsequent Z-score and Wasserstein distance calculations.
[0175] Calculation of the normal range for resource consumption: After training, calculate the normal range for resource consumption for each item; use the mean ± 1.96 standard deviations to calculate the 95% confidence interval, and determine whether it exceeds this interval. The interval calculation formula is as follows:
[0176]
[0177] in, This is the average value for this resource dimension from similar projects in the past; The standard deviation represents the resource consumption value for a given project to be implemented. Its Z-score:
[0178]
[0179] when When the consumption exceeds a preset threshold, such as 2 or 3, it can be considered that the resource consumption is abnormal in this dimension.
[0180] Range output: Through the above steps, define a reasonable normal range for each resource consumption dimension, such as labor quantity, machine shift quantity, and material quantity, and determine whether the consumption of the project to be implemented exceeds this normal range.
[0181] When performing outlier detection, the Z-score is calculated to quantify the difference in resource consumption between the project to be implemented and historical projects. The Z-score is a standardized method used to measure the degree of deviation between the resource consumption of a new project and the historical average. Its calculation formula is the difference between the resource consumption of the project to be implemented and the mean and standard deviation of historical projects, divided by the standard deviation. By calculating the Z-score and comparing it with a preset threshold, it is determined whether the consumption of the project to be implemented is within the normal range. If the Z-score exceeds the threshold, the resource consumption of the project is considered abnormal and further analysis is required.
[0182] When calculating the resource consumption of projects to be implemented, the projects to be implemented The resource consumption vector of the project to be implemented includes: (Calculate the labor, machine hours, and material consumption, and compare them with the normal consumption range of historical project data.)
[0183] Manual measurement, machine shift measurement, material quantity: ;
[0184] Z-score calculation formula:
[0185]
[0186] ;
[0187] in, The average amount of manpower in historical projects; The standard deviation representing the historical project's labor volume;
[0188] When identifying outliers, the Z-score is calculated and analyzed. If the calculated Z-score exceeds a preset threshold, such as 2 or 3, the resource consumption of the project is determined to be abnormal.
[0189] Anomaly detection formula: ;
[0190] When identifying outliers in the Wasserstein distance test, the Wasserstein distance (Earth transport distance) measures the difference between two distributions, and is particularly suitable for measuring the difference between resource consumption distributions. By calculating the Wasserstein distance, we can quantify the difference between the resource consumption of the project to be implemented and the consumption of historical projects, thereby determining whether the project is an outlier.
[0191] A distribution model is performed on the resource consumption data of historical projects, and the Wasserstein distance between the resource consumption distribution of the project to be implemented and the distribution of historical projects is calculated. The Wasserstein distance formula is as follows:
[0192]
[0193] in, The historical distribution of resource consumption in projects; The distribution of resource consumption for projects to be implemented or a batch of projects; This corresponds to the cumulative distribution function; The endpoints of the discretized resource consumption range; The interval width;
[0194] This formula originates from the first-order Wasserstein distance in optimal transport theory, also known as the Earth transport distance. Resource consumption is discretized into intervals to construct an empirical CDF, calculating the difference between historical and current distributions. A threshold is set based on the Wasserstein distance statistics of normal project samples. ,when When this occurs, it is determined that the overall resource consumption pattern of the project or collection of projects is abnormal;
[0195] If the Wasserstein distance exceeds a preset threshold, it is considered an abnormal item.
[0196] In this embodiment of the invention, by combining Gaussian Mixture Model (GMM), Z-score, and Wasserstein distance, abnormal resource consumption is detected in a multi-dimensional way. Traditional anomaly detection methods usually rely on a single statistical indicator or fixed threshold, which cannot fully consider the diversity and differences of projects, resulting in inaccurate anomaly identification. By fitting historical project data with GMM and combining Z-score and Wasserstein distance, it is possible to accurately identify whether resource consumption exceeds the normal range on a dynamic adjustment basis. This not only improves the sensitivity to project resource consumption, but also allows for dynamic adjustment of judgment criteria according to the characteristics of different projects, effectively preventing resource waste and optimizing resource allocation.
[0197] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.
[0198] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0199] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in the form of hardware plus software functional modules.
[0200] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.
[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A power grid cost item information extraction and anomaly detection method based on a BERT model and a graph neural network, characterized in that, Comprise: S1: By automatically extracting project key information, using the context semantic modeling technology of BERT model to extract structured data from text materials, by defining and selecting key information fields, combining BERT model for deep semantic modeling, to ensure accurate analysis and unified standardization of project description information; S2: Information cleaning and pretreatment, using T5 model for multi-task learning, combined with generative adversarial network; S3: Use graph neural network to learn the multi-dimensional characteristics of the project, capture the complex dependency relationship between projects, obtain high-quality embedding representation of each project, use SimCLR self-supervised learning method to further optimize the embedding representation of the project, identify the structured relationship between projects through spectral clustering method, and cluster similar projects in the same cluster; S4: Through Gaussian mixture model fitting of historical project data of the same type, statistical analysis of resource consumption of each project, establishment of normal consumption interval, and abnormal detection and judgment of resource consumption of the project to be implemented through Z-score and Wasserstein distance.
2. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 1, characterized in that, When encoding the context semantics based on the BERT model, after text preprocessing, BERT is used for deep semantic modeling; The formula of BERT model is: wherein, is an input sequence of text tokens; is a corresponding contextual semantic representation; is a set of parameters of a BERT model.
3. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 2, characterized in that, When performing semantic normalization and synonym fusion, the domain ontology vocabulary and semantic mapping rules are used to solve the diversification of power grid project documents, and the cosine similarity of word vectors is calculated, the cosine similarity formula of word vectors: ； wherein, and are word vectors of the words and respectively; is the cosine similarity between two words, which is used to determine the similarity between words.
4. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 1, characterized in that, T5 model parallel processing multiple tasks, involving expressions: wherein, is the input raw text data, i.e., the project document fragment to be revised; is the revised text output by T5; are the T5 model parameters.
5. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 4, characterized in that, The generative adversarial network is introduced, and the generator and discriminator are trained through the generative adversarial network, the generator formula: ； The discriminator formula is: ； wherein, is the revised text for T5 or its vector representation; is the further revised text output by the generator; is the generator network, are its parameters; is the discriminator network, are its parameters; is the probability score given by the discriminator for the input text being real, or conforming to the norm.
6. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 5, characterized in that, When training the field word vector, the Skip-gram Word2Vec model is adopted: ， wherein, is the center word, is the context word; is the corresponding word vector; is the vocabulary, derived from the classical probability form of the neural language model; the domain word vector is trained on the power grid project corpus; the probability is dimensionless, both numerator and denominator are in exponential form.
7. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 6, characterized in that, Through multi-layer graph convolution operation, the high-dimensional feature representation of the project will be gradually compressed to low-dimensional space, and finally the embedding representation of each project is obtained.
8. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 7, characterized in that, After obtaining the optimized project embedding representation, a spectral clustering method is used for clustering analysis; the similarity between projects is calculated using Euclidean distance or cosine similarity, a Laplacian matrix is constructed, the eigenvalues and eigenvectors of the Laplacian matrix are calculated, and the eigenvector corresponding to the smallest eigenvalue is selected to form a low-dimensional representation of the project K-means clustering is performed on the obtained low-dimensional embedding, and similar projects are assigned to the same cluster; Based on the results of spectral clustering and variational inference, the relevance network of the project is constructed, and the similarity and clustering results between projects are displayed; Finally, each project is assigned to a cluster, and the clustering results are output.
9. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 8, characterized in that, When performing abnormal value judgment, by calculating the Z-score value and analyzing, if the calculated Z-score value exceeds the preset threshold, it is determined that the resource consumption of the project is abnormal.
10. The power grid cost item information extraction and anomaly detection method based on the BERT model and the graph neural network according to claim 9, characterized in that, The resource consumption data of the historical project is distributed modeling, and the Wasserstein distance between the resource consumption distribution of the project to be implemented and the historical project distribution is calculated, if the Wasserstein distance exceeds the preset threshold, it is determined as an abnormal project.