Power grid data intelligent processing method based on NLP and dynamic blood relationship

Abstract

The application discloses a power grid data intelligent processing method based on NLP and dynamic blood relationship, and relates to the field of data processing.The method comprises the following steps: determining a target power grid, and constructing a power grid multi-source heterogeneous platform; in the power grid multi-source heterogeneous platform, a standard structure business set is acquired; a knowledge graph is constructed, and a data blood relationship is generated; a power grid large model is built; power grid data processing rules are collected, input into the power grid large model, and SQL rules are output; and the target power grid is processed based on the SQL rules.The technical problem that due to the diversity and complexity of data sources, power grid data is often difficult to effectively integrate and manage, and it is difficult to meet the demand of intelligent power grid for efficient and accurate data processing is solved, efficient intelligent master data management operation platform is constructed, efficient construction and integration of data assets are realized, data quality protection and repair are automatically performed, and the quality of power grid data and the effect of data management are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method for intelligent processing of power grid data based on NLP and dynamic lineage. Background Technology

[0002] With the rapid development of smart grids, the power industry's reliance on data is increasing. Smart grids need to cover all users, collect all electricity consumption information, comprehensively control electricity costs, and monitor line loss consumption in real time. Furthermore, the power industry needs to develop personalized electricity consumption products and services based on customer needs. Data sources in the power industry are extensive, including not only data from the grid's own business operations and management, but also IoT data collected from various sensors such as voltage, current, and signal processing sensors, as well as a large amount of external data related to distributed power sources and residential users. Due to the diversity and complexity of data sources, grid data is often difficult to integrate and manage effectively. This restricts the development of information sharing and intelligent decision-making within power companies. Although NLP technology has broad application prospects in smart grids, its practical application still faces many challenges due to high technical barriers, large data volumes, and complex processing. Existing grid data management methods are still insufficient in terms of intelligence and automation, making it difficult to meet the demands of smart grids for efficient and accurate data processing.

[0003] In summary, due to the diversity and complexity of data sources, power grid data is often difficult to integrate and manage effectively, making it difficult to meet the technical challenges of smart grids in terms of efficient and accurate data processing. Summary of the Invention

[0004] Therefore, it is necessary to provide a smart power grid data processing method based on NLP and dynamic lineage to address the aforementioned technical issues. This method can solve the problem that power grid data is often difficult to effectively integrate and govern due to the diversity and complexity of data sources, and thus fails to meet the needs of smart grids for efficient and accurate data processing. By efficiently building an intelligent master data management and operation platform, the method can achieve efficient construction and integration of data assets, and improve the quality of power grid data and the effectiveness of data governance through automated data quality protection and repair.

[0005] Based on this, an intelligent power grid data processing method based on NLP and dynamic lineage is provided. The method includes: identifying a target power grid; determining equipment structure and related services based on the target power grid; constructing a multi-source heterogeneous power grid platform according to the equipment structure and related services; performing standardized description and structured processing of the related services based on service metadata within the multi-source heterogeneous power grid platform to obtain a standard structured service set; constructing a knowledge graph to capture the semantic relationships and logical connections of the standard structured service set, generating data lineage relationships; introducing a large-scale model tool to build a large power grid model by combining the standard structured service set and the data lineage relationships, the large power grid model including an association recognition model and a statement transformation model; collecting power grid data processing rules; inputting the power grid data processing rules into the large power grid model and outputting SQL rules; and processing the target power grid based on the SQL rules.

[0006] The aforementioned NLP-based and dynamic lineage-based intelligent power grid data processing method addresses the technical challenges of effectively integrating and governing power grid data due to the diversity and complexity of data sources, thus failing to meet the demands of smart grids for efficient and accurate data processing. By efficiently constructing an intelligent master data management and operation platform, it enables the efficient construction and integration of data assets, and through automated data quality protection and repair, it improves the quality of power grid data and the effectiveness of data governance.

[0007] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a method for intelligent processing of power grid data based on NLP and dynamic lineage in one embodiment.

[0009] Figure 2 This is a flowchart illustrating the multi-dimensional information filling process of a power grid data intelligent processing method based on NLP and dynamic lineage in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] like Figure 1 As shown, this application provides a method for intelligent processing of power grid data based on NLP and dynamic lineage, the method comprising:

[0012] Determine the target power grid, determine the equipment structure and related services based on the target power grid, and construct a multi-source heterogeneous power grid platform according to the equipment structure and the related services.

[0013] NLP, or Natural Language Processing, is an interdisciplinary field that integrates linguistics, computer science, and mathematics. Its aim is to enable computers to understand and process human language. NLP's goals include language understanding, language generation, and making decisions or taking actions based on language. Dynamic lineage refers to the technology or method of tracking and recording the flow and changes of data throughout its lifecycle. It can capture data changes and flows in real time, ensuring the accuracy and up-to-dateness of data lineage information. Intelligent data processing methods for power grids based on NLP and dynamic lineage can significantly improve the intelligence level of power grid operation and maintenance, enhance the efficiency and quality of data governance, and provide strong support for the sustainable development of the power industry.

[0014] The target power grid to be processed and analyzed must be clearly defined. This is usually determined based on specific business needs, regional scope, or power grid characteristics. For example, a power company may want to optimize the operation of its distribution network in a certain region, then that region's distribution network becomes the target power grid. Equipment structure refers to the internal composition, components, and connection methods and relationships of various devices in the power grid (such as transformers, switches, lines, etc.). Equipment information of the target power grid is collected, including equipment type, quantity, location, parameters, etc. Through topology analysis, the connection relationships and hierarchical structure between devices are understood. Based on the function and attributes of the equipment, it is classified and corresponding equipment models are constructed. Related business refers to various business activities related to the operation and management of the target power grid, such as equipment monitoring, fault handling, planned maintenance, and electricity metering. The business needs of power grid operation and management are clearly defined, understanding which business activities need to be supported. The business processes of power grid operation and management are streamlined, understanding the correlations and dependencies between each link, mapping business activities to power grid equipment, and determining which equipment is related to which business activities. A multi-source heterogeneous platform refers to a system platform capable of integrating data and information from different sources and of different types, and performing unified management and processing. In the power grid sector, multi-source heterogeneous platforms can integrate power grid equipment data, operational data, and business data to support intelligent management and operation of the power grid. Identifying the target power grid, determining the equipment structure and related business based on the target power grid, and constructing a multi-source heterogeneous power grid platform based on this information are crucial steps in smart grid construction. Through this process, the integration and sharing of power grid data can be achieved, improving the power grid's intelligence level and management efficiency.

[0015] The process involves: identifying the target power grid; acquiring multiple sets of equipment based on the target power grid's design blueprint, including equipment composition, connection methods, and equipment functions, with the connection methods including physical and logical connections; constructing a device network topology diagram; obtaining business requirements; performing power grid business activity analysis based on the business requirements to obtain business scenarios and business processes; determining the association between the business requirements and the multiple sets of equipment based on the business scenarios and business processes to obtain associated businesses; and constructing a multi-source heterogeneous power grid platform based on the equipment structure and the associated businesses.

[0016] Identify the target power grid that needs to be analyzed and managed. This could be a specific regional power grid, a substation, or a complex inter-regional power grid system. Based on the target power grid's design blueprint, list all equipment types, quantities, and their locations within the grid. Record the physical connections between equipment, such as the list and paths of connected devices like cables and busbars, and the logical relationships between devices, such as control signals and communication protocols. Describe the role and function of each device within the power grid. Based on the acquired equipment information, use graphical tools to construct a device network topology diagram. This topology diagram should clearly show the physical and logical connections between devices, as well as their hierarchical structure within the power grid. Obtain business requirements related to the target power grid from power grid operators or other stakeholders. Based on these business requirements, analyze and identify key business scenarios in power grid operation, such as equipment monitoring, fault handling, and load management. For each business scenario, describe its business process in detail, including participating roles, activity sequence, inputs, and outputs. Map each business requirement to specific equipment or equipment groups, clarifying which equipment or equipment groups are responsible for fulfilling which business requirements. Based on the above mapping relationships, obtain the business activities associated with each equipment or equipment group, forming a list of related business activities. Design a data platform capable of integrating data from different data sources. Based on the device structure and related business processes, construct corresponding data and business models. These models should accurately describe the relationships between devices, the processes of business activities, and their interactions. Based on the data and business models, develop various functional modules and applications. These functions may include device monitoring, fault warning, data analysis, and report generation. Deploy the developed functional modules and applications to the target environment, and conduct testing and optimization to ensure they run stably and efficiently.

[0017] For example, suppose the target power grid is a city distribution network, containing multiple substations and a large number of distribution equipment. By analyzing the grid's design blueprint and business requirements, the following information can be obtained: Equipment information, including the type, quantity, location, and interconnections of equipment such as transformers, switches, and lines; Business requirements, including real-time monitoring of grid operation status, rapid response to fault events, and optimized load distribution; Business scenarios, such as equipment fault handling scenarios, where a fault occurs and the fault point needs to be quickly located, the fault area isolated, and power restored; Related services, which may include fault detection, fault location, fault isolation, and power restoration for equipment fault handling scenarios. Based on this information, a device network topology diagram can be constructed, and a multi-source heterogeneous power grid platform can be developed. This platform can integrate data from different data sources and provide functions such as equipment monitoring, fault early warning, and data analysis to meet the business needs of power grid operators.

[0018] In the aforementioned power grid multi-source heterogeneous platform, the associated services are standardized and structured based on service metadata to obtain a standard structured service set.

[0019] It is necessary to identify all business metadata related to the associated business, such as business data definitions (describing the structure, type, and scope of business data), business rules (specifying the processing methods and constraints of business data), and business process definitions (detailing the processes, steps, and roles of business activities). Based on the identified business metadata, a business metadata model is established. This model should clearly represent the relationships and dependencies between business data, business rules, and business processes. Through the model, the overall picture of the associated business can be intuitively understood. Using the business metadata model, the associated business is described in a standardized manner. This includes using unified terminology and definitions to describe business data, business rules, and business processes, assigning unique identifiers and descriptive information to each associated business, and clearly representing the dependencies and interaction methods between associated businesses. Based on the standardized description, the associated business is structured. This includes converting business data into structured formats (such as XML and JSON) for easy storage, transmission, and processing; converting business rules and business processes into executable forms (such as scripts and workflows) for automated execution; and building business data models and business logic models to support the rapid development and deployment of associated businesses. By standardizing descriptions and structuring data, a standardized business set can be obtained. This business set contains standardized descriptions and structured data for all related businesses, enabling the platform to manage and process these businesses uniformly. At the same time, the standardized business set also provides a foundation for subsequent business analysis, optimization, and expansion.

[0020] Define the business metadata requirements; identify power grid business metadata, classify and identify it, and obtain classified business metadata; define a data description standard template and a business metadata data structure, standardize and structure the classified business metadata, obtain preprocessed business metadata, and integrate the preprocessed business metadata to obtain a standard structured business set.

[0021] Define the scope of business metadata required to be supported by the multi-source heterogeneous power grid platform. This typically includes data requirements from key business areas such as power grid equipment monitoring, fault handling, load management, and electricity metering. Collect power grid business-related metadata from existing power grid systems, documents, databases, etc., and perform preliminary classification and organization of the collected business metadata to identify data items related to power grid operation, management, and monitoring. Define appropriate classification standards based on the characteristics and requirements of power grid business, such as equipment, fault, load, and electricity categories. Classify and label the power grid business metadata according to the defined classification standards to ensure that each data item can be accurately categorized. To uniformly describe business metadata, a standard data description template needs to be defined, including key information such as data name, data type, data range, unit, and source. Define appropriate data structures based on the characteristics and requirements of business metadata, such as relational database table structures or JSON object structures. Use the defined data description standard template to standardize the description of the classified business metadata, ensuring that the description of each data item conforms to a unified standard. Based on the defined business metadata data structure, perform structured processing on the standardized business metadata, converting the data into a storable, transmissible, and processable format. After standardization and structuring, the resulting business metadata is called preprocessed business metadata. This metadata already possesses a unified standard and structure, laying the foundation for subsequent data integration and platform construction. The preprocessed business metadata is then integrated according to business logic and relationships to form a complete business dataset. This integrated business dataset is the standard structured business set, which contains all the business metadata required by the multi-source heterogeneous power grid platform and is organized according to a unified standard and structure.

[0022] Construct a knowledge graph to capture the semantic relationships and logical connections of the standard structure business set and generate data lineage relationships.

[0023] The construction goals of the knowledge graph should be clearly defined, namely, which semantic relationships and logical connections to capture, and how this knowledge will be used to support the functions of the multi-source heterogeneous platform of the power grid. In the operations of the multi-source heterogeneous platform of the power grid, entities may include equipment, faults, personnel, operations, locations, etc. Clear identifiers and attributes need to be defined for these entities. Relationships define the connections between entities; for example, the "equipment-fault" relationship represents a possible fault of equipment, and the "personnel-operation" relationship represents an operation performed by personnel. The data structure of the standard structured business set should be analyzed to understand the data fields and their relationships, extracting entities, entity attributes, and relationships between entities from the business set. Based on the extracted entities and attributes, corresponding entity nodes and attribute edges should be created in the knowledge graph. Relationship edges should be created in the knowledge graph according to the extracted entity relationships, connecting related entity nodes. Existing ontology libraries and vocabularies, such as professional terms and concepts in the power grid field, should be used to provide semantic definitions and explanations for entities and relationships in the knowledge graph. Semantic reasoning techniques (such as rule-based reasoning and graph reasoning) should be used to automatically capture and derive the implicit semantic relationships in the knowledge graph. Based on business scenarios and processes, analyze the logical connections between different entities and relationships, such as the logical relationships between equipment, personnel, and operations in a fault handling process. Identify dependencies between data, such as the impact of a change in one data item on other data items. Trace the source and flow of data, recording the entire process from data generation to use. Explicitly represent data lineage in the knowledge graph, such as using special edges or attributes to label the source, generation time, modification history, etc.

[0024] like Figure 2 As shown, the goal and scope of the knowledge graph are defined, multi-dimensional knowledge extraction is performed on the standard structure business set to obtain multi-dimensional information; entity alignment is performed on the multi-dimensional information to obtain aligned multi-dimensional information; attribute value filling is performed based on the aligned multi-dimensional information to obtain filled multi-dimensional information.

[0025] The goal of a knowledge graph is to provide a structured, queryable database containing power grid business knowledge to support various application scenarios such as decision analysis, question-answering systems, and recommendation systems. Its scope includes: business domains (key business areas such as power grid equipment monitoring, fault management, operation optimization, and asset management); data types (entities such as equipment, personnel, operations, time, and geographical location, and the relationships between them); and information dimensions (multi-dimensional information such as basic information, operating status, historical data, and related events). Understanding the data structure, field meanings, and relationships of the business set is crucial. Based on the requirements of the knowledge graph and the characteristics of the business set, rules for extracting knowledge from the business set are defined. Rules or machine learning models are used to extract entities, attributes, and relationships from the business set to obtain multi-dimensional information. Knowledge extraction includes entity extraction, automatically discovering specific entities such as equipment, business processes, and data items from a standard structured business set; relationship extraction, using linguistic and statistical methods to extract semantic relationships between entities from text or structured data, such as connections between equipment and dependencies between businesses; and attribute extraction, extracting relevant attribute information for each entity, such as equipment model, location, and status, to obtain multi-dimensional information. This process involves entity alignment of multi-dimensional information, identifying records representing the same entity from different sources or formats, handling entities with the same name but different meanings, ensuring each entity has a unique identifier, and using methods such as string matching and machine learning models to align entities from different sources. It also aligns "Transformer T1" from device information from different systems to ensure they refer to the same device, handling devices with the same name but different models, and ensuring they can be correctly distinguished in the knowledge graph. Furthermore, it identifies instances of missing or incomplete entity attribute values ​​in the knowledge graph, determines attribute value filling strategies based on data availability and business needs, and uses methods such as rules, machine learning models, or external data sources to fill in missing attribute values. Through these methods, multi-dimensional information is extracted from a standard structured business dataset, and a complete and accurate knowledge graph is constructed to support various knowledge-based applications.

[0026] Integrate and populate multi-dimensional information, perform data preprocessing, and obtain processed multi-dimensional information; store the processed multi-dimensional information in the data layer for conceptual abstraction, and construct a pattern layer based on the ontology; perform quality assessment on the pattern layer and obtain assessment results; set a confidence threshold, and judge the assessment results based on the confidence threshold. When the assessment results meet the confidence threshold, they are retained in the knowledge graph.

[0027] The previously populated multi-dimensional information is integrated to ensure data integrity and consistency. This may involve handling data redundancy, conflicts, or inconsistencies. The integrated multi-dimensional information undergoes data preprocessing to eliminate noise, outliers, duplicates, etc., and may also require data transformation, standardization, or normalization for subsequent processing and analysis. After data preprocessing, the processed multi-dimensional information is obtained, which will serve as the foundational data for building the knowledge graph. This processed multi-dimensional information is stored in the data layer. Above the data layer, conceptual abstraction is performed, mapping specific business data to higher-level concepts or categories. This helps reduce data complexity and enhances the understandability and queryability of the knowledge graph. An ontology is a model in the knowledge graph used to describe concepts, entities, attributes, and the relationships between them. Based on the ontology, a schema layer is built, defining entity types, attribute types, relationship types, and the constraints and rules between them in the knowledge graph. The schema layer provides a clear structure and semantics for the knowledge graph, making it easy to understand and query. The constructed schema layer undergoes quality evaluation to ensure its accuracy, completeness, consistency, and usability. Quality assessment may include checking the accuracy of entity and relationship definitions, the presence of redundant or inconsistent definitions, and whether they meet business requirements. Based on the quality assessment results, an assessment report or score for the schema layer quality is obtained. This assessment result will serve as the basis for subsequent decisions. A confidence threshold is set based on business requirements and data quality. This threshold will be used to determine whether the schema layer quality meets the requirements. The assessment results are compared with the confidence threshold. If the assessment result meets or exceeds the confidence threshold, the schema layer quality is considered acceptable, and it is retained in the knowledge graph. If the assessment result is below the confidence threshold, the schema layer needs to be reviewed again to identify problems and make corrections or adjustments. When the schema layer passes the quality assessment and meets the confidence threshold, it is officially retained in the knowledge graph. At this point, the knowledge graph contains the validated and optimized schema layer, which can support various knowledge-based applications and analyses. Through these steps, the accuracy and reliability of the knowledge graph can be ensured, providing a solid foundation for subsequent applications and analyses.

[0028] The semantic relationships and logical connections of the standard structure business set are captured based on the knowledge graph; bloodline relationship types are defined, and bloodline relationship models are established based on the bloodline relationship types; algorithm tracking data is imported into the bloodline relationship model to generate data bloodline relationships.

[0029] A detailed analysis of the constructed knowledge graph is required to understand its entities, attributes, and the relationships between them. Different types of semantic relationships can be identified within the knowledge graph, such as "equipment-fault" relationships and "personnel-operation" relationships. These relationships are typically represented by edges between entities. Beyond semantic relationships, the logical connections between entities also need to be understood. This usually involves business rules, processes, or event sequences. These logical connections can be derived by analyzing the entities and relationships in the knowledge graph. Based on business needs and information in the knowledge graph, key data lineage types should be identified. For example, information such as the data source, transformation process, and users should be considered. Corresponding attributes should be defined for each lineage type. For example, for data source relationships, attributes such as "source system," "source table," and "source field" need to be defined. Based on the defined lineage types, a model structure capable of representing these relationships should be designed. This typically includes defining entities, relationships, and their hierarchical structure. Appropriate tools or technologies (such as databases, graph databases, etc.) should be used to implement the lineage model. The model should be ensured to be able to store and query lineage data. Use algorithms or tools to track the flow of data within business systems. This data may include information such as data sources, transformation processes, and users. Clean and transform the collected tracking data to ensure it conforms to the requirements of a lineage model. Import the cleaned and transformed data into the lineage model. Use the lineage model to query and visualize data lineage relationships. This helps in understanding the data's source, flow path, and scope of impact.

[0030] A large-scale model tool is introduced, and a large-scale power grid model is built by combining the standard structure business set and the data lineage relationship. The large-scale power grid model includes an association identification model and a statement transformation model.

[0031] Considering the complexity and requirements of large-scale power grid models, it is necessary to select large-scale modeling tools capable of handling massive amounts of data, possessing powerful computing capabilities and efficient algorithms. These tools typically support advanced technologies such as deep learning and natural language processing (NLP) and have efficient data processing and model training capabilities. Data related to the power grid, including equipment information, operation records, and fault data, is extracted from a standard structured business dataset. The extracted data is cleaned to remove duplicates, errors, or invalid data, and then integrated according to a unified standard. Within the data lineage, the types of lineage relationships related to key business operations such as power grid operation and equipment maintenance are identified. Based on the identified lineage relationship types, a data lineage relationship network is constructed to clarify the flow and dependencies between data. According to the actual needs of power grid business, an association identification model is designed to identify the relationships and dependencies in the power grid data. Relevant features are extracted from the integrated standard structured business dataset and used as input to the association identification model. The large-scale modeling tool is then used to train the association identification model, enabling it to accurately identify the relationships in the power grid data. This paper analyzes the natural language commands and query requirements in power grid operations, determines the functional requirements of the statement conversion model, designs the statement conversion model, and converts natural language commands into query or operation statements that can be executed by the power grid big model. The statement conversion model is then trained using big model tools to enable it to accurately understand and convert natural language commands and obtain SQL statements.

[0032] Analyze the data lineage and the standard structure business set to establish a sample set; select a large model architecture, determine initial parameters based on task requirements and data scale, and construct an initial model; perform supervised training on the initial model based on the sample set; preset the model accuracy, and make a judgment based on the model accuracy when the initial model converges; if the model accuracy is satisfied, the association recognition model is obtained.

[0033] A deep understanding of data lineage—that is, the source, flow, transformation, and usage of data in power grid operations—is crucial for identifying correlations and dependencies between data. A detailed analysis of standard structured business sets is essential to understand the data structure, field meanings, and relationships between them. This helps extract features and labels relevant to correlation identification. Based on the analysis results of data lineage and standard structured business sets, the data is labeled. The labeled data should reflect the correlations and dependencies between data. The labeled data is designated as a sample set and divided into training, validation, and test sets. The training set is used for model training, the validation set is used to adjust model parameters and prevent overfitting, and the test set is used to evaluate model performance. A suitable large-scale model architecture is selected based on the specific requirements of the correlation identification task. A suitable large-scale model architecture is also chosen based on the data size. For large-scale datasets, a model architecture with more parameters and stronger performance can be selected. Initial model parameters, such as learning rate, batch size, and number of iterations, are determined based on task requirements and data size. These parameters will affect the model's training speed and performance. An initial model is constructed based on the selected large-scale model architecture and the determined initial parameters. The training set is loaded into the initial model to begin supervised training. Through multiple iterations, the model's parameters and weights are adjusted, allowing the model to gradually learn the relationships and dependencies between data points. After each iteration, the model is validated using a validation set, and its parameters and structure are adjusted based on the validation results. This helps prevent overfitting and improves generalization ability. A reasonable model accuracy threshold is preset based on business needs and data characteristics. This threshold should reflect the model's performance requirements in practical applications. When the model converges (i.e., when the training loss no longer decreases significantly), the model is evaluated using a test set, and its accuracy is calculated. If the model's accuracy meets the preset threshold, the model is considered successfully trained. Through these steps, a sample set can be established based on data lineage and a standard structure business set, and a suitable large-scale model architecture can be selected for the construction and training of the correlation identification model. This helps to more accurately identify the relationships and dependencies between power grid data, providing strong support for subsequent power grid business analysis and decision-making.

[0034] Based on historical power grid data processing rules collected from the power grid field; text cleaning, part-of-speech tagging, and entity recognition are performed on the historical power grid data processing rules to obtain preprocessed rules; the preprocessed rules are judged, and if a corresponding SQL statement exists, it is judged and added to the training dataset; if no corresponding SQL statement exists, it is sent to experts for manual writing and added to the training dataset; the conversion model is trained in a supervised manner using the training dataset to generate a statement conversion model.

[0035] Historical power grid data processing rules were collected from various documents, databases, and log files within the power grid sector. These rules included detailed descriptions of power grid processing. The collected historical power grid data processing rules underwent text cleaning to remove noise, irrelevant information, and special characters. The text was converted to a unified format and encoding for subsequent processing. Natural Language Processing (NLP) techniques were used to perform part-of-speech tagging on the cleaned text, marking the part of speech of each word. Entity recognition was also performed to identify key entities in the text, such as table names, field names, and operation types. Based on the results of part-of-speech tagging and entity recognition, preprocessing rules related to SQL statement transformation were extracted, including data transformation logic, conditional judgments, and field mapping. The preprocessing rules were traversed to determine if a corresponding SQL statement already existed for each rule. This could be done by querying an existing SQL statement database or historical records. For preprocessing rules without a matching SQL statement, they were sent to experts for manual SQL statement creation. The experts wrote the corresponding SQL statement based on the rule content. The matched SQL statements and the expert-written SQL statements were added to the training dataset, which should include the pairing relationship between preprocessing rules and their corresponding SQL statements. The conversion model is trained in a supervised manner using a training dataset. This model can be rule-matching based, and during training, it learns how to convert preprocessing rules into SQL statements. After sufficient training, a statement conversion model is generated. This model should be able to accept new preprocessing rules as input and output the corresponding SQL statements. The optimized statement conversion model is then deployed to the platform. When new power grid data processing rules need to be converted into SQL statements, this model can be used for automated conversion, improving efficiency and accuracy.

[0036] Collect power grid data processing rules, input the power grid data processing rules into the power grid big model, and output SQL rules.

[0037] Power grid data processing rules are collected from sources such as power grid system documents, user manuals, operation guides, and database logs. These pre-processed rules are then input into a previously constructed large-scale power grid model. Within this model, the input rules are parsed. During parsing, the model may need to identify key information such as rule type, operation object, and operation conditions based on keywords, sentence structure, and context. This key information is then converted into a representation that the model can understand, such as vectors or tensors. Based on the parsed and converted results, the large-scale power grid model uses its internal statement transformation model to generate corresponding SQL rules. These generated SQL rules should conform to SQL syntax standards and accurately implement the data operations described by the original power grid data processing rules. The generated SQL rules are then output in text form for use by users or other systems. In this way, power grid data processing rules can be input into the large-scale power grid model, and corresponding SQL rules can be automatically output, thus achieving automated conversion and execution of power grid data processing rules.

[0038] The target power grid is processed based on the SQL rules.

[0039] Identify the target power grid's database or data storage system and prepare the data tables or datasets to be processed, ensuring the normal operation of the target power grid's database system and the integrity and security of the data. Before practical application, validate the generated SQL rules. This can be done by executing the SQL rules in a test environment to ensure their logical correctness and absence of syntax errors. The validation process should focus on the performance, execution time, and impact on the target power grid data of the SQL rules. Apply the validated SQL rules to the target power grid's database system. This is typically achieved through programming or database management tools. When executing the SQL rules, ensure sufficient permissions and appropriate resources to avoid unnecessary interference or damage to the target power grid. During SQL rule execution, monitor the processing to ensure it proceeds as expected. This can be done by reviewing database logs, using query monitoring functions in database management tools, or writing dedicated monitoring scripts. After the SQL rules have completed execution, check the processing results to ensure they meet expectations. This can be done by querying the data in the target power grid's database to ensure the data has been properly cleaned, transformed, or aggregated.

[0040] For specific embodiments of the intelligent power grid data processing method based on NLP and dynamic lineage, please refer to the above. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0041] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for intelligent processing of power grid data based on NLP and dynamic lineage, characterized in that, The method includes: Identify the target power grid, determine the equipment structure and related services based on the target power grid, and construct a multi-source heterogeneous power grid platform according to the equipment structure and related services; In the aforementioned power grid multi-source heterogeneous platform, the associated services are standardized and structured based on service metadata to obtain a standard structured service set; Construct a knowledge graph to capture the semantic relationships and logical connections of the standard structure business set and generate data lineage relationships; A large model tool is introduced to build a power grid large model by combining the standard structure business set and the data lineage relationship. The power grid large model includes an association identification model and a sentence transformation model. Collect power grid data processing rules, input the power grid data processing rules into the power grid big model, and output SQL rules; The target power grid is processed based on the SQL rules.

2. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 1, characterized in that, The method includes: determining a target power grid, determining equipment structure and related services based on the target power grid, and constructing a multi-source heterogeneous power grid platform according to the equipment structure and related services. Determine the target power grid, obtain multiple sets of equipment for the target power grid based on the design blueprint of the target power grid, including equipment composition, connection method and equipment function, the connection method includes physical connection and logical connection, and construct the equipment network topology diagram; Obtain business requirements, and analyze power grid business activities based on the business requirements to obtain business scenarios and business processes; Based on the business scenario and the business process, determine the association between the business requirements and the multiple groups of devices, and obtain the associated business; A multi-source heterogeneous power grid platform is constructed based on the equipment structure and the associated services.

3. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 1, characterized in that, The method involves standardizing and structuring the associated services based on business metadata to obtain a standard structured service set. Define business metadata requirements; Identify power grid business metadata, classify and label it based on the power grid business metadata, and obtain classified business metadata; Define a standard template for data description and a data structure for business metadata. Standardize and structure the categorized business metadata to obtain preprocessed business metadata. Integrate the preprocessed business metadata to obtain a standard structured business set.

4. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 1, characterized in that, The method for constructing a knowledge graph, capturing the semantic relationships and logical connections of the standard structure business set, and generating data lineage relationships includes: Define the goals and scope of the knowledge graph, and perform multi-dimensional knowledge extraction on the standard structure business set to obtain multi-dimensional information; Perform entity alignment on the multi-dimensional information to obtain aligned multi-dimensional information; The attribute values ​​are filled based on the alignment multi-dimensional information to obtain the filled multi-dimensional information.

5. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 4, characterized in that, The method includes: Integrate and populate multi-dimensional information, perform data preprocessing, and obtain and process multi-dimensional information; The processed multi-dimensional information is stored in the data layer for conceptual abstraction, and a pattern layer is constructed based on the ontology. Perform a quality assessment on the mode layer and obtain the assessment results; Set a confidence threshold, and judge the evaluation result based on the confidence threshold. When the evaluation result meets the confidence threshold, it is retained in the knowledge graph.

6. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 5, characterized in that, The method includes: Based on the knowledge graph, the semantic relationships and logical connections of the standard structure business set are captured; Define blood relationship types and establish blood relationship models based on the defined blood relationship types; The algorithm tracks the data and imports it into the bloodline model to generate data bloodlines.

7. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 1, characterized in that, A large-scale model tool is introduced to construct a large-scale power grid model by combining the standard structure business set and the data lineage relationship. The large-scale power grid model includes an association identification model and a statement transformation model. The method includes: Analyze the data lineage and the standard structure business set to establish a sample set; Choose a large model architecture, determine the initial parameters based on task requirements and data scale, and build the initial model; The initial model is trained under supervision based on the sample set. A preset model accuracy is used to determine when the initial model converges. If the accuracy of the model is met, then the association recognition model is obtained.

8. The intelligent power grid data processing method based on NLP and dynamic lineage as described in claim 7, characterized in that, The method includes: According to the rules for collecting and processing historical power grid data in the power grid sector; The historical power grid data processing rules are subjected to text cleaning, part-of-speech tagging, and entity recognition to obtain preprocessed rules; The preprocessing rules are evaluated, and if a corresponding SQL statement exists, it is evaluated and added to the training dataset. If the corresponding SQL statement does not exist, it will be sent to an expert to write it manually and add it to the training dataset. The conversion model is trained in a supervised manner using the training dataset to generate a sentence conversion model.

Images