Power log data processing method and device based on distributed architecture

Abstract

The invention relates to a power log data processing method and device based on a distributed architecture. The method comprises the following steps: extracting power log data of different log types from a plurality of data sources of a power data circulation platform; the power log data are stored in a distributed storage engine in a centralized mode, and original power log data stored in a classified mode are obtained through data cleaning processing; performing information extraction on the original power log data of each log type by adopting a data model correspondingly configured based on each log type through a log analysis rule in the data model, and performing normalization processing on the extracted log information through a log normalization rule in the data model to obtain analyzed power log data; and storing the analyzed power log data to a log storage engine based on distributed full-text retrieval. By adopting the method, efficient centralized management can be performed on the multi-source power log data, and the processing efficiency and quality of the power log data are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing power log data based on a distributed architecture. Background Technology

[0002] With the development of information technology in the power industry, the log data generated by various power systems has increased dramatically. Power log data covers all aspects of power generation, transmission, distribution, and consumption, and is an important basis for equipment status monitoring, fault diagnosis, and business analysis. It is characterized by its massive volume, heterogeneity, and high real-time requirements.

[0003] Traditional power log processing methods typically employ a centralized architecture, which lacks elastic scalability and dynamic resource adjustment capabilities when dealing with sudden surges in logs, affecting the timeliness of power operation and maintenance and making it difficult to support the power industry's needs for real-time data processing and large-scale data analysis. Furthermore, traditional methods usually use a single database or local file system for storage, resulting in low storage space utilization, slow query response, and poor scalability. Summary of the Invention

[0004] Therefore, it is necessary to provide a power log data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on a distributed architecture that can improve the efficiency of power log data processing, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for processing power log data based on a distributed architecture, the method comprising:

[0006] Extract power log data of different log types from multiple data sources of the power data circulation platform;

[0007] The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning.

[0008] A data model based on the configuration corresponding to each log type is adopted. The log parsing rules in the data model are used to extract information from the original power log data of each log type. The log normalization rules in the data model are used to normalize the extracted log information to obtain the parsed power log data.

[0009] The parsed power log data is stored in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0010] In one embodiment, the method further includes:

[0011] Based on the log characteristics of each log type, construct the corresponding data model for each log type;

[0012] Configure log parsing rules and log normalization rules in each of the data models using a data dictionary;

[0013] The log features include any one or more of the following:

[0014] Features of equipment operation logs, power grid monitoring logs, load control logs, fault and alarm logs, power network security logs, and new energy-related logs.

[0015] In one embodiment, configuring the log parsing rules in each of the data models via a data dictionary includes:

[0016] In the data dictionary, information extraction fields for corresponding data models are configured according to each log feature, and information decomposition rules for corresponding data models are configured according to the log data structure of each log type; the information decomposition rules include regular expressions, delimiters, or specified format parsing.

[0017] The information extraction fields and the information decomposition rules are used as the log parsing rules; the log parsing rules are used to establish the parsing mapping relationship between the original power log data and different data models.

[0018] In one embodiment, configuring log normalization rules in each of the data models via a data dictionary includes:

[0019] In the data dictionary, the enumeration range information and normalization format information of each data model are configured as the log normalization rules.

[0020] In one embodiment, the method further includes:

[0021] In response to a log attribute field adjustment operation, a new log attribute field is added to the data model specified by the log attribute field adjustment operation.

[0022] In one embodiment, storing the parsed power log data in a log storage engine based on distributed full-text search includes:

[0023] The parsed power log data is stored in the log storage engine using preset log storage configuration information; the log storage configuration information is used to indicate the storage location and / or storage format.

[0024] The method further includes:

[0025] In response to a log query request, a full-text search is performed on the parsed power log data in the distributed storage engine.

[0026] Secondly, this application also provides a power log data processing device based on a distributed architecture, the device comprising:

[0027] The power log data extraction module is used to extract power log data of different log types from multiple data sources of the power data circulation platform.

[0028] The log distributed storage module is used to centrally store the power log data into the distributed storage engine and obtain the original power log data for classified storage through data cleaning processing;

[0029] The log parsing and normalization module is used to extract information from the original power log data of each log type using a data model configured according to the log type, and to normalize the extracted log information using the log normalization rules in the data model to obtain parsed power log data.

[0030] The parsed log storage module is used to store the parsed power log data in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0032] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0034] The aforementioned power log data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on a distributed architecture extract power log data of different log types from multiple data sources of a power data circulation platform, centrally store the power log data in a distributed storage engine, and obtain classified and stored raw power log data through data cleaning. Then, using a data model configured according to each log type, information is extracted from the raw power log data of each log type through log parsing rules in the data model, and the extracted log information is normalized through log normalization rules in the data model to obtain parsed power log data. The parsed power log data is then stored in a log storage engine based on distributed full-text retrieval. This log storage engine is used to perform log data analysis for various power business systems in the power data circulation platform, realizing efficient centralized management of multi-source power log data. By utilizing configurable data models and normalization processing, heterogeneous logs can be transformed into standardized data, and the log storage engine can provide fast and unified log analysis services to various business systems, thereby improving the processing efficiency and quality of power log data and enhancing power log data analysis capabilities. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a power log data processing method based on a distributed architecture in one embodiment.

[0037] Figure 2 This is a schematic diagram of a power log data preprocessing service in one embodiment;

[0038] Figure 3 This is a flowchart illustrating a power log data processing method based on a distributed architecture, as described in another embodiment.

[0039] Figure 4 This is a structural block diagram of a power log data processing device based on a distributed architecture in one embodiment;

[0040] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] In one exemplary embodiment, such as Figure 1 As shown, a power log data processing method based on a distributed architecture is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 101 to 104. Wherein:

[0043] Step 101: Extract power log data of different log types from multiple data sources of the power data circulation platform.

[0044] As an example, power log data may include, but is not limited to, equipment operation logs, power grid monitoring logs, load control logs, fault and alarm logs, power network security logs, and data sharing records; the log type may be the power business type to which the corresponding power log data belongs, such as equipment operation type, fault and alarm type, and sharing record type.

[0045] Among them, by taking power data and energy data as the main body, a reliable data circulation platform based on power data (i.e., power data circulation platform) can be established by adopting a secure and reliable data circulation mechanism. This platform can realize the usability but not visibility of power data based on privacy computing technology, and form standards and protocols for data exchange and sharing. Thus, while protecting the privacy of power data, it can enable the secure and efficient exchange and utilization of data between different power-related systems.

[0046] Step 102: The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning processing.

[0047] As an example, a distributed storage engine can be a distributed storage system, such as HDFS (Hadoop Distributed File System, a highly fault-tolerant distributed file system).

[0048] In practical applications, the power data circulation platform can extract power log data collected from different data sources into a distributed storage engine. After data cleaning, the power log data can be classified and stored according to the log type, so as to carry out subsequent log preprocessing work based on data integration. For example, power log data of different log types can be extracted from multiple heterogeneous data sources and centrally stored in a scalable and highly available distributed storage system. The extracted log source files can also be backed up.

[0049] In one example, data cleaning can include data imputation, such as imputing empty or missing data, and marking data that cannot be processed; it can also include data replacement, such as replacing invalid data. Thus, data cleaning can analyze data quality in terms of accuracy, consistency, and redundancy, identify potential problems such as inconsistent data formats, data errors, and incomplete data, and address them accordingly.

[0050] In one alternative embodiment, HDFS serves as the foundation for data storage management in distributed computing. It supports streaming data access and processing of extremely large files, featuring high fault tolerance, high reliability, high scalability, high availability, and high throughput. It provides fault-free storage for massive amounts of data and is suitable for processing ultra-large datasets. By automatically saving data as multiple replicas, fault tolerance is improved and automatic recovery is achieved through adding replicas. It is suitable for processing large volumes of data and supports streaming file access. Once written, HDFS files cannot be modified, only appended; a single write allows for multiple reads, ensuring data consistency.

[0051] Step 103: Using a data model configured according to each of the log types, information is extracted from the original power log data of each log type through the log parsing rules in the data model, and the extracted log information is normalized through the log normalization rules in the data model to obtain parsed power log data.

[0052] For example, log parsing rules may include information extraction fields and information decomposition rules to perform information extraction processing on the raw power log data; log normalization rules may include enumeration range information and normalization format information to perform normalization processing on the extracted log information.

[0053] Step 104: Store the parsed power log data into a log storage engine based on distributed full-text retrieval.

[0054] The log storage engine can be used to perform log data analysis for various power business systems within the power data circulation platform.

[0055] In practical implementation, by configuring log parsing rules and log normalization rules, raw power log data of multiple log types can be automatically identified and parsed. For example, by using a data model and following the configured rules, raw power log data can be parsed into standard format data and stored in a log storage engine, such as the distributed search and analysis engine ElasticSearch. This engine can be used in cloud computing to achieve real-time search, stability, reliability, and fast processing. Furthermore, preprocessed log data (i.e., parsed power log data) can be stored based on the distributed full-text search engine ElasticSearch, enabling second-level queries of massive amounts of data.

[0056] Compared to traditional methods, the technical solution in this embodiment can be based on a horizontally scalable distributed architecture, such as using two different distributed systems, HDFS and Elasticsearch, which can work together in a big data architecture. Figure 2 As shown, the collected power log data undergoes preprocessing operations such as extraction, cleaning, log parsing, and generalization (normalization). After preprocessing, the parsed data can be stored in the log storage engine for log analysis by the upper-layer power business system. It can achieve full storage of raw log data based on HDFS and supports compression of raw logs. At the same time, the storage engine can support data backup and recovery functions, has the capacity for storing large amounts of data, and supports horizontal scaling, such as expanding storage nodes according to data storage needs. When the data volume increases, the storage space of the storage nodes can be expanded, and when the data volume decreases, the data storage space can be reduced, thereby meeting the requirements of high reliability, high scalability, and real-time analysis of power log data.

[0057] The aforementioned power log data processing method based on a distributed architecture extracts power log data of different log types from multiple data sources on the power data circulation platform, centrally stores the power log data in a distributed storage engine, and obtains categorized raw power log data through data cleaning. Then, using a data model configured according to each log type, information is extracted from the raw power log data of each log type through log parsing rules in the data model, and the extracted log information is normalized through log normalization rules in the data model to obtain parsed power log data. Finally, the parsed power log data is stored in a log storage engine based on distributed full-text retrieval, achieving efficient centralized management of multi-source power log data. The configurable data model and normalization processing can transform heterogeneous logs into standardized data, and the log storage engine can provide fast and unified log analysis services for various business systems, improving the processing efficiency and quality of power log data and enhancing power log data analysis capabilities.

[0058] In one exemplary embodiment, the following steps may also be included:

[0059] Based on the log characteristics of each log type, construct a data model corresponding to each log type; configure the log parsing rules and log normalization rules in each data model through a data dictionary.

[0060] The log features may include any one or more of the following: equipment operation log features, power grid monitoring log features, load control log features, fault and alarm log features, power network security log features, and new energy-related log features.

[0061] In practical applications, building data models is the foundation of log parsing and generalization (normalization). By building data models, the output fields after log parsing and processing can be standardized. The basic fields of all data models can be configured through a data dictionary, and the required data models can be built according to the log characteristics of different log types to achieve model classification and management.

[0062] In this embodiment, by constructing a data model corresponding to each log type based on the log characteristics of each log type, and configuring the log parsing rules and log normalization rules in each data model through a data dictionary, the readability and consistency of power log data can be improved, providing data support for the standardized parsing and unified processing of multi-source heterogeneous logs.

[0063] In an exemplary embodiment, configuring the log parsing rules in each of the data models through the data dictionary may include the following steps:

[0064] In the data dictionary, information extraction fields for corresponding data models are configured according to each log feature, and information decomposition rules for corresponding data models are configured according to the log data structure of each log type. The information decomposition rules include regular expressions, delimiters, or specified format parsing. The information extraction fields and the information decomposition rules are used as the log parsing rules. The log parsing rules are used to establish the parsing mapping relationship between the original power log data and different data models.

[0065] Specifically, for the data model corresponding to each log type, corresponding information extraction fields can be configured based on the characteristics of each log to facilitate information extraction from power log data of different log types. In addition, combined with information decomposition rules, such as supporting rule syntax based on regular expressions, delimiters, and JSON parsing (i.e., specified format parsing), log parsing rules can be customized to build a mapping relationship between the original power log data and the data model, thereby enabling the extraction of key information from the original messy data.

[0066] In this embodiment, by configuring information extraction fields for corresponding data models based on the characteristics of each log in the data dictionary, and configuring information decomposition rules for corresponding data models based on the log data structure of each log type, the information extraction fields and information decomposition rules are used as log parsing rules. This enables structured parsing and accurate extraction of key information from various types of power log data, improving the parsing accuracy of log processing and facilitating subsequent log data analysis and application.

[0067] In an exemplary embodiment, configuring the log normalization rules in each of the data models through a data dictionary may include the following steps:

[0068] In the data dictionary, the enumeration range information and normalization format information of each data model are configured as the log normalization rules.

[0069] In one example, for enumeration type fields in a data model, the range of enumeration data can be defined through a data dictionary, and custom extensions can be added to the data dictionary to support flexible configuration of the data enumeration range. The normalization format field of the data model can also be edited in the data dictionary for custom configuration. In log data processing, enumeration is the process of mapping discrete values ​​(such as status codes, error types, operation types, etc.) in the raw logs to a predefined, standardized, finite set of values. It can transform messy, concrete values ​​into categories with unified business semantics, facilitating subsequent statistical analysis and visualization.

[0070] Optionally, the data dictionary can also provide some fields by default for the data model, such as the original log generation time, log reception time, operation type, log name, log summary, log level, terminal name, terminal type, and original log.

[0071] In this embodiment, by configuring the enumeration range information and normalization format information of each data model in the data dictionary as log normalization rules, the standardization and consistency of log data can be achieved, and the maintenance of log normalization rules can be simplified, which helps to improve the automation and efficiency of log processing.

[0072] In one exemplary embodiment, the following steps may also be included:

[0073] In response to a log attribute field adjustment operation, a new log attribute field is added to the data model specified by the log attribute field adjustment operation.

[0074] In practical applications, the data model also supports the dynamic addition of log attribute fields. Besides the default fields, additional attribute fields can be added based on log processing needs. All newly added attribute fields can participate in subsequent log queries, log correlation analysis, and statistical analysis. This allows for the dynamic expansion of fields to address situations where the system's built-in fields cannot meet the requirements, thereby enhancing the system's analytical capabilities.

[0075] In this embodiment, by responding to the log attribute field adjustment operation, a new log attribute field is added to the data model specified by the log attribute field adjustment operation, which enables flexible expansion of the log attribute field without manually modifying the model structure.

[0076] In an exemplary embodiment, storing the parsed power log data in a log storage engine based on distributed full-text search may include the following steps:

[0077] The parsed power log data is stored in the log storage engine using preset log storage configuration information; the log storage configuration information is used to indicate the storage location and / or storage format.

[0078] For example, log storage configuration information can be defined to specify how parsed power log data is stored, such as storage location, format, and other storage configuration information.

[0079] Also includes:

[0080] In response to a log query request, a full-text search is performed on the parsed power log data in the distributed storage engine.

[0081] In its implementation, the distributed search and analysis engine ElasticSearch (i.e., the log storage engine) can support real-time search. By forming a peer-to-peer network structure with nodes, it can automatically allocate other nodes to take over when some nodes fail. Based on the log storage engine, it can respond to log query requests and perform full-text search on the parsed power log data in the distributed storage.

[0082] In this embodiment, by using preset log storage configuration information, the parsed power log data is stored in the log storage engine. Then, in response to log query requests, the parsed power log data in the distributed storage is fully searched in the log storage engine. This enables centralized management and fast access to log data. Furthermore, based on the full-text search capability, the log data in the distributed storage can be queried in real time and accurately, improving the efficiency and response speed of power log retrieval.

[0083] In one exemplary embodiment, such as Figure 3The diagram illustrates another method for processing power log data based on a distributed architecture. In this embodiment, the method includes the following steps:

[0084] In step 301, power log data of different log types is extracted from multiple data sources of the power data circulation platform. In step 302, the power log data is centrally stored in a distributed storage engine, and the raw power log data is categorized and stored through data cleaning. In step 303, based on the log characteristics of each log type, a data model corresponding to each log type is constructed, and log parsing rules and log normalization rules in each data model are configured using a data dictionary. In step 304, using the data model configured for each log type, information is extracted from the raw power log data of each log type through the log parsing rules in the data model, and the extracted log information is normalized through the log normalization rules in the data model to obtain parsed power log data. In step 305, the parsed power log data is stored in the log storage engine using preset log storage configuration information. In step 306, in response to a log query request, a full-text search is performed on the parsed power log data in the distributed storage engine.

[0085] It should be noted that the specific limitations of the above steps can be found in the above description of the specific limitations of a power log data processing method based on a distributed architecture, and will not be repeated here.

[0086] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0087] Based on the same inventive concept, this application also provides a distributed architecture-based power log data processing device for implementing the aforementioned distributed architecture-based power log data processing method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more distributed architecture-based power log data processing device embodiments provided below can be found in the limitations of the distributed architecture-based power log data processing method described above, and will not be repeated here.

[0088] In one exemplary embodiment, such as Figure 4 As shown, a power log data processing device based on a distributed architecture is provided, comprising:

[0089] The power log data extraction module 401 is used to extract power log data of different log types from multiple data sources of the power data circulation platform.

[0090] The log distributed storage module 402 is used to centrally store the power log data to the distributed storage engine and obtain the original power log data for classified storage through data cleaning processing;

[0091] The log parsing and normalization module 403 is used to extract information from the original power log data of each log type using a data model configured according to the log type, and to normalize the extracted log information using the log normalization rules in the data model to obtain parsed power log data.

[0092] The parsed log storage module 404 is used to store the parsed power log data in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0093] In one embodiment, the device further includes:

[0094] The data model building module is used to build a data model corresponding to each log type based on the log characteristics of each log type.

[0095] Configure log parsing rules and log normalization rules in each of the data models using a data dictionary;

[0096] The log features include any one or more of the following:

[0097] Features of equipment operation logs, power grid monitoring logs, load control logs, fault and alarm logs, power network security logs, and new energy-related logs.

[0098] In one embodiment, the data model construction module is specifically used to configure information extraction fields for corresponding data models in the data dictionary according to the log characteristics of each log, and to configure information decomposition rules for corresponding data models according to the log data structure of each log type; the information decomposition rules include regular expressions, delimiters, or specified format parsing; the information extraction fields and the information decomposition rules are used as the log parsing rules; the log parsing rules are used to establish the parsing mapping relationship between the original power log data and different data models.

[0099] In one embodiment, the data model construction module is specifically used to configure the enumeration range information and normalization format information of each data model in the data dictionary as the log normalization rule.

[0100] In one embodiment, the device further includes:

[0101] The field dynamic adjustment module is used to add new log attribute fields to the data model specified by the log attribute field adjustment operation in response to the log attribute field adjustment operation.

[0102] In one embodiment, the parsed log storage module 404 is specifically used to store the parsed power log data to the log storage engine using preset log storage configuration information; the log storage configuration information is used to indicate the storage location and / or storage format.

[0103] The device further includes:

[0104] The log retrieval module is used to perform full-text retrieval of the parsed power log data in the distributed storage engine in response to a log query request.

[0105] The modules in the aforementioned distributed architecture-based power log data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0106] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a power log data processing method based on a distributed architecture.

[0107] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0108] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0109] Extract power log data of different log types from multiple data sources of the power data circulation platform;

[0110] The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning.

[0111] A data model based on the configuration corresponding to each log type is adopted. The log parsing rules in the data model are used to extract information from the original power log data of each log type. The log normalization rules in the data model are used to normalize the extracted log information to obtain the parsed power log data.

[0112] The parsed power log data is stored in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0113] In one embodiment, the processor, when executing the computer program, also implements the steps of the power log data processing method based on a distributed architecture as described in the other embodiments above.

[0114] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0115] Extract power log data of different log types from multiple data sources of the power data circulation platform;

[0116] The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning.

[0117] A data model based on the configuration corresponding to each log type is adopted. The log parsing rules in the data model are used to extract information from the original power log data of each log type. The log normalization rules in the data model are used to normalize the extracted log information to obtain the parsed power log data.

[0118] The parsed power log data is stored in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0119] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the power log data processing method based on a distributed architecture as described in the other embodiments above.

[0120] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0121] Extract power log data of different log types from multiple data sources of the power data circulation platform;

[0122] The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning.

[0123] A data model based on the configuration corresponding to each log type is adopted. The log parsing rules in the data model are used to extract information from the original power log data of each log type. The log normalization rules in the data model are used to normalize the extracted log information to obtain the parsed power log data.

[0124] The parsed power log data is stored in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

[0125] In one embodiment, when the computer program is executed by a processor, it also implements the steps of the power log data processing method based on a distributed architecture as described in the other embodiments above.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing power log data based on a distributed architecture, characterized in that, The method includes: Extract power log data of different log types from multiple data sources of the power data circulation platform; The power log data is centrally stored in a distributed storage engine, and the raw power log data is classified and stored through data cleaning. A data model based on the configuration corresponding to each log type is adopted. The log parsing rules in the data model are used to extract information from the original power log data of each log type. The log normalization rules in the data model are used to normalize the extracted log information to obtain the parsed power log data. The parsed power log data is stored in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

2. The method according to claim 1, characterized in that, The method further includes: Based on the log characteristics of each log type, construct the corresponding data model for each log type; Configure log parsing rules and log normalization rules in each of the data models using a data dictionary; The log features include any one or more of the following: Features of equipment operation logs, power grid monitoring logs, load control logs, fault and alarm logs, power network security logs, and new energy-related logs.

3. The method according to claim 2, characterized in that, The step of configuring log parsing rules in each of the data models through a data dictionary includes: In the data dictionary, information extraction fields for corresponding data models are configured according to each log feature, and information decomposition rules for corresponding data models are configured according to the log data structure of each log type; the information decomposition rules include regular expressions, delimiters, or specified format parsing. The information extraction fields and the information decomposition rules are used as the log parsing rules; the log parsing rules are used to establish the parsing mapping relationship between the original power log data and different data models.

4. The method according to claim 2, characterized in that, The step of configuring log normalization rules in each of the data models through a data dictionary includes: In the data dictionary, the enumeration range information and normalization format information of each data model are configured as the log normalization rules.

5. The method according to claim 1, characterized in that, The method further includes: In response to a log attribute field adjustment operation, a new log attribute field is added to the data model specified by the log attribute field adjustment operation.

6. The method according to any one of claims 1 to 5, characterized in that, The step of storing the parsed power log data into a log storage engine based on distributed full-text search includes: The parsed power log data is stored in the log storage engine using preset log storage configuration information; the log storage configuration information is used to indicate the storage location and / or storage format. The method further includes: In response to a log query request, a full-text search is performed on the parsed power log data in the distributed storage engine.

7. A power log data processing device based on a distributed architecture, characterized in that, The device includes: The power log data extraction module is used to extract power log data of different log types from multiple data sources of the power data circulation platform. The log distributed storage module is used to centrally store the power log data into the distributed storage engine and obtain the original power log data for classified storage through data cleaning processing; The log parsing and normalization module is used to extract information from the original power log data of each log type using a data model configured according to the log type, and to normalize the extracted log information using the log normalization rules in the data model to obtain parsed power log data. The parsed log storage module is used to store the parsed power log data in a log storage engine based on distributed full-text retrieval; the log storage engine is used for log data analysis by various power business systems in the power data circulation platform.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images