Normalization method for power enterprise data resources

Abstract

The invention relates to the technical field of power data management, and discloses a power enterprise data resource standardization method comprising the following steps: collecting multi-source heterogeneous data from different systems, devices and departments of a power enterprise; performing identification and fusion processing on the multi-source heterogeneous data by using a data fusion technology to generate multi-source heterogeneous fusion data; performing data cleaning on the multi-source heterogeneous fusion data, and performing format conversion on the multi-source heterogeneous fusion data in combination with a format conversion algorithm to generate text data; performing text analysis on the text data by using a natural language processing technology, and generating a label and a classification result in combination with the classification model; and the labels and the classified data are stored in a standardized database, so that quick retrieval and application are facilitated. According to the method, multi-source heterogeneous data can be efficiently collected, recognized and fused, analysis and classification of text data are achieved through a natural language processing technology, the standardized data are stored in a standardized database, and rapid retrieval and application are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of power data management, and in particular to a method for standardizing data resources of power enterprises. Background Art

[0002] With the rapid development of the power industry, the amount and types of data in power companies are constantly increasing. Multi-source heterogeneous data such as power system operation data, equipment status data, environmental monitoring data, electricity consumption data, economic operation data, and safety monitoring data have gradually become important resources for the daily operation and management of power companies. Early power data management mainly relied on manual processing and simple database technology, which could not meet the high requirements of modern power companies for data real-time, accuracy, and consistency.

[0003] In recent years, with the rapid development of big data technology, Internet of Things technology and artificial intelligence technology, power companies have gradually introduced technologies such as data fusion, data cleaning, format conversion and natural language processing in order to achieve standardized processing of data resources and improve data management efficiency and decision support capabilities; however, the existing power data resource management technology still has many shortcomings, as follows:

[0004] 1. The complexity and diversity of multi-source heterogeneous data make data identification and fusion processing a major challenge. Existing technical means still need to be improved in terms of accuracy and efficiency of data fusion;

[0005] 2. Although data cleaning and format conversion technologies have been developed, it is often difficult to achieve adaptive and efficient processing when faced with the complex data structure and variable data quality of power companies;

[0006] 3. Use natural language processing technology to analyze and classify text data. Although it has been widely used in other fields, its application in the power industry is still immature, and there are problems such as insufficient accuracy of classification models and low processing efficiency.

[0007] Therefore, the existing technologies still have much room for improvement in data recognition, fusion processing, data cleaning and format conversion, text data analysis and classification, etc. Summary of the invention

[0008] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract of the specification and the title of the invention of this application to avoid blurring the purpose of this section, the abstract of the specification and the title of the invention, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0009] In view of the above existing problems, the present invention is proposed.

[0010] Therefore, the technical problem solved by the present invention is that the existing power data resource management technology has the problems of low efficiency in identifying and integrating complex data, insufficient adaptive capabilities in data cleaning and format conversion, and low accuracy in text data analysis and classification.

[0011] To solve the above technical problems, the present invention provides the following technical solutions: collecting multi-source heterogeneous data from different systems, equipment and departments of the power enterprise;

[0012] Using data fusion technology, the multi-source heterogeneous data are identified and fused to generate multi-source heterogeneous fused data;

[0013] After cleaning the multi-source heterogeneous fusion data, convert the format of the data using a format conversion algorithm to generate text data;

[0014] Using natural language processing technology to perform text analysis on the text data, and combining the classification model to generate labels and classification results;

[0015] The tags and classified data are stored in a standardized database for quick retrieval and application.

[0016] As a preferred solution of the standardization method for power enterprise data resources described in the present invention, the collected multi-source heterogeneous data includes but is not limited to power system operation data, power equipment status data, environmental monitoring data, electricity consumption data, economic operation data, safety monitoring data, energy consumption and efficiency data, planning and scheduling data, user feedback and demand data, policy and regulatory data.

[0017] As a preferred solution of the method for standardizing data resources of electric power enterprises described in the present invention, the identification and merging process includes:

[0018] Read the collected multi-source heterogeneous data from the temporary database and analyze its data format and content;

[0019] Identify the multi-source heterogeneous data to determine the source, type and format of the data;

[0020] According to the recognition results, the same data items are fused to ensure data consistency;

[0021] Verify the fused data to ensure its accuracy and completeness;

[0022] Store the verified data in the intermediate database.

[0023] As a preferred solution of the standardization method for power enterprise data resources described in the present invention, a fusion model is established by using data fusion technology to perform fusion processing. The mathematical expression formula is as follows:

[0024]

[0025] Among them, F ij represents the fused data item, w k is the weight parameter, is the data item of the kth source.

[0026] As a preferred solution of the standardization method for power enterprise data resources described in the present invention, the fusion model is trained using a random forest algorithm to optimize the data fusion process and ensure the consistency and accuracy of the data. The final mathematical expression of the fusion model is:

[0027]

[0028] Among them, T i Represents the i-th decision tree, x is the input data, and y is the output result.

[0029] As a preferred solution of the method for standardizing data resources of electric power enterprises described in the present invention, the data cleaning includes removing noise data, completing missing data and correcting erroneous data, wherein:

[0030] Use statistical methods to detect noise data in data;

[0031] Interpolation method was used to fill in missing data;

[0032] Use business rules and historical data to correct detected erroneous data.

[0033] As a preferred solution of the standardization method for power enterprise data resources described in the present invention, regular expressions are used to implement format conversion:

[0034] formatted_data=re.sub(pattern,replacement,data)

[0035] Among them, pattern is the matching pattern, replacement is the replacement content, and data is the data to be converted.

[0036] As a preferred solution of the method for standardizing data resources of electric power enterprises described in the present invention, natural language processing technology is used to perform text analysis on the text data, and a label and classification result are generated in combination with a classification model, including:

[0037] Using natural language processing technology to perform word segmentation and part-of-speech tagging on the text data;

[0038] Generate corresponding labels for the text data after part-of-speech tagging according to data content and business requirements;

[0039] A classification model is established by using a support vector machine classifier to classify the text data with corresponding labels, and a classification result is output, that is, the data is divided into different categories.

[0040] As a preferred solution of the standardization method for power enterprise data resources described in the present invention, the classification process of the classification model can be expressed by the following formula:

[0041]

[0042] Among them, Class is the classification result, T i is the i-th text data segment, λ i is the weight parameter of text data, F i is the text data segment after the i-th part-of-speech tag, μ i is the weight parameter of part-of-speech tagging, N i Generate a data segment for the i-th label, W i The weight parameter generated for the label, R j is the jth classification model data segment, ν j is the weight parameter of the classification model, Q j is the jth support vector data segment, ρ j is the weight parameter of the support vector, is the part-of-speech tagging function of the text data segment, ψ(F i ,μ i ) is the label generation function of the data after part-of-speech tagging, γ(N i ) is the data filtering function generated by the label, θ(W i ,σ i ) is the data weight adjustment function, δ(R j ,ν j ) is the operation function of the classification model, ξ(Q j ,ρ j ) is the operation function of the support vector, a, b, c, d, p, q, r, s are the integration intervals, n is the number of text data segments, and m is the number of classification model data segments.

[0043] As a preferred solution of the method for standardizing data resources of electric power enterprises described in the present invention, the standardized database is a distributed database, wherein:

[0044] Importing the cleaned, format-converted and classified data into the distributed database;

[0045] Create indexes for stored data to ensure fast retrieval and access to data;

[0046] Back up data regularly to ensure data security and recoverability;

[0047] Power companies manage stored data through data management platforms, including but not limited to data access control, data authority management and data auditing.

[0048] The beneficial effects of the present invention are as follows: the present invention can efficiently collect, identify and fuse multi-source heterogeneous data, perform data cleaning and format conversion, and realize analysis and classification of text data through natural language processing technology, and store normalized data in a standardized database for rapid retrieval and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0050] Figure 1 It is a flowchart of the method for standardizing data resources of electric power enterprises shown in the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0052] Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without making any creative work should fall within the scope of protection of the present invention.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] Example 1

[0055] According to an embodiment of the present invention, Figure 1 The flowchart shown is a method for standardizing data resources of electric power enterprises, which specifically includes the following steps:

[0056] S1. Collect multi-source heterogeneous data from different systems, equipment and departments of the power enterprise. Among them, the following points need to be explained in this step:

[0057] The collected multi-source heterogeneous data include but are not limited to power system operation data, power equipment status data, environmental monitoring data, power consumption data, economic operation data, safety monitoring data, energy consumption and efficiency data, planning and scheduling data, user feedback and demand data, policy and regulatory data;

[0058] The power system operation data at least includes the real-time operation parameters of voltage, current, power and frequency;

[0059] The power equipment status data at least includes the equipment operating status (on / off), fault status, and maintenance records;

[0060] Environmental monitoring data shall at least include environmental parameters such as temperature, humidity, wind speed, wind direction, and rainfall;

[0061] Electricity consumption data at least includes electricity consumption, peak and valley electricity consumption, and user classification (residential, industrial, commercial);

[0062] Economic operation data at least includes electricity price, market demand, cost and revenue;

[0063] Security monitoring data at least includes security monitoring videos, intrusion detection alarms, and fire alarms;

[0064] Energy consumption and efficiency data shall at least include energy consumption rate, energy conversion efficiency and energy efficiency ratio;

[0065] Planning and dispatching data at least include power grid dispatching plan, load forecast, and equipment maintenance plan;

[0066] User feedback and demand data at least include user satisfaction surveys, fault repair records, and service requirements;

[0067] Policy and regulatory data include at least national and local electricity policies, industry standards, and regulatory requirements.

[0068] As an example, when collecting data, the reliability and consistency of data transmission can be ensured through data interface standards (such as IEC 61850).

[0069] In an optional implementation manner, the data collection method in the embodiment of the present invention can be implemented by the following steps:

[0070] When the system (such as the power enterprise data management system) is started, the configuration file is loaded, which includes information such as the type of data source, connection method, and collection frequency;

[0071] According to the configuration file, the system establishes connections with various data sources;

[0072] The system obtains real-time data from the data source according to the set collection frequency;

[0073] Perform preliminary processing on the collected data, including data error correction and outlier detection.

[0074] It should be noted that the collection of multi-source heterogeneous data ensures the diversity and comprehensiveness of the data, making the subsequent data fusion, processing and analysis more extensive and representative, thereby improving the overall quality and coverage of data processing.

[0075] S2. Using data fusion technology, identify and fuse multi-source heterogeneous data to generate multi-source heterogeneous fused data. It should be noted that this step includes:

[0076] (1) Data analysis: Read the collected multi-source heterogeneous data from the temporary database and analyze its data format and content;

[0077] (2) Data identification: Identify multi-source heterogeneous data and determine the source, type, and format of the data;

[0078] (3) Data merging: Based on the recognition results, the same data items are merged to ensure data consistency;

[0079] (4) Data verification: Verify the fused data to ensure its accuracy and completeness;

[0080] (5) Data storage: The verified data is stored in an intermediate database.

[0081] As an example, a fusion model is established using data fusion technology to perform fusion processing. The mathematical expression formula is as follows:

[0082]

[0083] Among them, F ij represents the fused data item, w k is the weight parameter, is the data item of the kth source.

[0084] As an example, the random forest algorithm is used to train the fusion model, optimize the data fusion process, and ensure the consistency and accuracy of the data. The final mathematical expression of the fusion model is:

[0085]

[0086] Among them, T i Represents the i-th decision tree, x is the input data, and y is the output result.

[0087] It should also be noted that in this embodiment, through data fusion technology, multi-source heterogeneous data is identified, its data format and content are analyzed, the source, type and format of the data are determined, and based on the identification results, the same data items are fused to ensure data consistency, and the fused data is verified to ensure data accuracy and completeness.

[0088] Preferably, data from different sources are processed uniformly through fusion processing, eliminating data silos, improving data consistency and integrity, and thus providing a high-quality data foundation for subsequent data analysis.

[0089] S3. After cleaning the multi-source heterogeneous fusion data, convert the format of the data using the format conversion algorithm to generate text data. It should be noted that data cleaning includes removing noise data, completing missing data, and correcting erroneous data, among which:

[0090] Noise data detection: Use statistical methods to detect noise data in data;

[0091] Missing data completion: interpolation method is used to complete missing data;

[0092] Error data correction: Use business rules and historical data to correct detected error data.

[0093] As an example, noise data detection can be expressed by the following mathematical expression:

[0094]

[0095] Among them, σ represents the standard deviation, xi is the data item, μ is the mean, and the data items exceeding three times the standard deviation are regarded as noise data.

[0096] Furthermore, the format conversion algorithm is combined to convert the format, and the implementation process is as follows:

[0097] Data feature analysis: Analyze the features of cleaned data and determine the format and structure of the data;

[0098] Format conversion rule generation: Generate corresponding format conversion rules based on data characteristics and business requirements;

[0099] Data format conversion: Use conversion rules to convert data format to ensure that the data meets standardization requirements.

[0100] As an example, use regular expressions to implement format conversion:

[0101] formatted_data=re.sub(pattern,replacement,data)

[0102] Among them, pattern is the matching pattern, replacement is the replacement content, and data is the data to be converted.

[0103] Preferably, the method of the present invention further improves the quality and consistency of data through data cleaning and format conversion, ensures the accuracy and standardization of data, and lays a solid foundation for the analysis of text data.

[0104] S4. Use natural language processing technology to analyze text data and generate labels and classification results in combination with classification models. The following steps need to be explained in detail:

[0105] Use natural language processing technology to segment and tag text data;

[0106] Generate corresponding labels for the text data after part-of-speech tagging based on data content and business requirements;

[0107] A classification model is established through the support vector machine classifier to classify the text data with corresponding labels and output the classification results, that is, to classify the data into different categories.

[0108] As an example, the classification process of the classification model can be expressed by the following formula:

[0109]

[0110] Among them, Class is the classification result, T i is the i-th text data segment, λ i is the weight parameter of text data, F i is the text data segment after the i-th part-of-speech tag, μ i is the weight parameter of part-of-speech tagging, N i Generate a data segment for the i-th label, W i The weight parameter generated for the label, R j is the jth classification model data segment, ν j is the weight parameter of the classification model, Q j is the jth support vector data segment, ρ j is the weight parameter of the support vector, is the part-of-speech tagging function of the text data segment, ψ(F i ,μ i ) is the label generation function of the data after part-of-speech tagging, γ(N i ) is the data filtering function generated by the label, θ(W i ,σ i ) is the data weight adjustment function, δ(R j ,ν j) is the operation function of the classification model, ξ(Q j ,ρ j ) is the operation function of the support vector, a, b, c, d, p, q, r, s are the integration intervals, n is the number of text data segments, and m is the number of classification model data segments.

[0111] Preferably, the method of the present invention realizes the automatic processing and classification of large-scale text data through text analysis and classification models, greatly improving the efficiency and accuracy of data management.

[0112] S5. Store the labeled and classified data in a standardized database for quick retrieval and application. It should also be noted that the standardized database is a distributed database, in which:

[0113] Import the cleaned, format-converted and classified data into the distributed database;

[0114] Create indexes for stored data to ensure fast retrieval and access to data;

[0115] Back up data regularly to ensure data security and recoverability;

[0116] Power companies manage stored data through data management platforms, including but not limited to data access control, data authority management and data auditing.

[0117] Preferably, through the storage and management of standardized databases, an efficient data storage and retrieval mechanism is provided, which ensures the security, reliability and availability of data and meets the high standards of power companies for data management and application.

[0118] The statistical method, interpolation method and data identification method in the aforementioned data preprocessing can be performed using methods and means in the prior art and will not be described in detail in this example.

[0119] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for standardizing data resources of electric power enterprises, characterized in that: include: Collect multi-source heterogeneous data from different systems, equipment and departments of power enterprises; Using data fusion technology, the multi-source heterogeneous data are identified and fused to generate multi-source heterogeneous fused data; After cleaning the multi-source heterogeneous fusion data, convert the format of the data using a format conversion algorithm to generate text data; Using natural language processing technology to perform text analysis on the text data, and combining the classification model to generate labels and classification results; The tags and classified data are stored in a standardized database for quick retrieval and application.

2. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: The multi-source heterogeneous data collected include but are not limited to power system operation data, power equipment status data, environmental monitoring data, electricity consumption data, economic operation data, safety monitoring data, energy consumption and efficiency data, planning and scheduling data, user feedback and demand data, and policy and regulatory data.

3. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: The identification and merging process is performed, including: Read the collected multi-source heterogeneous data from the temporary database and analyze its data format and content; Identify the multi-source heterogeneous data to determine the source, type and format of the data; According to the recognition results, the same data items are fused to ensure data consistency; Verify the fused data to ensure its accuracy and completeness; Store the verified data in the intermediate database.

4. The method for standardizing data resources of electric power enterprises according to claim 1 or 3, characterized in that: The fusion model is established by using data fusion technology to perform fusion processing. The mathematical expression formula is as follows: Among them, F ij represents the fused data item, w k is the weight parameter, is the data item of the kth source.

5. The method for standardizing data resources of electric power enterprises according to claim 4, characterized in that: The fusion model is trained using the random forest algorithm to optimize the data fusion process and ensure the consistency and accuracy of the data. The final mathematical expression of the fusion model is: Among them, T i Represents the i-th decision tree, x is the input data, and y is the output result.

6. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: The data cleaning includes removing noise data, filling missing data and correcting erroneous data, wherein: Use statistical methods to detect noise data in data; Interpolation method was used to fill in missing data; Use business rules and historical data to correct detected erroneous data.

7. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: Use regular expressions to achieve format conversion: formatted_data=re.sub(pattern,replacement,data) Among them, pattern is the matching pattern, replacement is the replacement content, and data is the data to be converted.

8. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: The text data is analyzed using natural language processing technology, and labels and classification results are generated in combination with a classification model, including: Using natural language processing technology to perform word segmentation and part-of-speech tagging on the text data; Generate corresponding labels for the text data after part-of-speech tagging according to data content and business requirements; A classification model is established by using a support vector machine classifier to classify the text data with corresponding labels, and a classification result is output, that is, the data is divided into different categories.

9. The method for standardizing data resources of electric power enterprises according to claim 8, characterized in that: The classification process performed by the classification model can be expressed by the following formula: Among them, Class is the classification result, T i is the i-th text data segment, λ i is the weight parameter of text data, F i is the text data segment after the i-th part-of-speech tag, μ i is the weight parameter of part-of-speech tagging, N i Generate a data segment for the i-th label, W i The weight parameter generated for the label, R j is the jth classification model data segment, ν j is the weight parameter of the classification model, Q j is the jth support vector data segment, ρ j is the weight parameter of the support vector, is the part-of-speech tagging function of the text data segment, ψ(F i ,μ i ) is the label generation function of the data after part-of-speech tagging, γ(N i ) is the data filtering function generated by the label, θ(W i ,σ i ) is the data weight adjustment function, δ(R j ,ν j ) is the operation function of the classification model, ξ(Q j ,ρ j ) is the operation function of the support vector, a, b, c, d, p, q, r, s are the integration intervals, n is the number of text data segments, and m is the number of classification model data segments.

10. The method for standardizing data resources of electric power enterprises according to claim 1, characterized in that: The standardized database is a distributed database, wherein: Importing the cleaned, format-converted and classified data into the distributed database; Create indexes for stored data to ensure fast retrieval and access to data; Back up data regularly to ensure data security and recoverability; Power companies manage stored data through data management platforms, including but not limited to data access control, data authority management and data auditing.

Images