Multi-source data tracing method for power data service

Abstract

The invention discloses an electric power data service-oriented multi-source data tracing method, which is characterized by comprising the following steps of automatic analysis, acquisition of electric power data tracing information, system tracking, mark tracking of various electric power data, machine learning and improvement of the analysis processing capability of the electric power data. Therefore, the related power traceability information can be accurately obtained by analyzing various power basic data, and then the identifier corresponding to the traceability information of the data is added to the data by adding the corresponding log or annotation in the data processing program, so that the tracking of the data flow direction is realized; and a machine can quickly identify the data and the traceability information thereof in subsequent machine learning, and the similarity and dependency relationship among the data are calculated according to the identifiers of the data, so that the traceability speed and accuracy of various power data can be continuously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power data tracing, and in particular to a multi-source data tracing method for power data services. Background Art

[0002] Data is an important resource in today's Internet era. Through the analysis, interpretation, and induction of various data, sufficient and accurate information can be obtained, thereby providing strong support for social management, business operations, and future trend forecasts in specific fields.

[0003] In the process of providing power services, the power sector involves many fields such as power production, transmission and distribution, power sales, and energy management. In order to optimize the various fields involved and ultimately achieve the goals of improving the efficiency of the entire power service process, reducing power service costs, and streamlining power service processes, it is necessary to effectively trace the data in various fields involved, so as to provide an investigation basis for discovering problems in various fields and processes.

[0004] With the development of technology, data sources are constantly increasing and diversifying, thus giving rise to the concept of multi-source heterogeneous data. Multi-source heterogeneous data refers to data from different sources and of different types. These data may have different structures, formats, semantics and uses, and these differences bring challenges to data traceability. In the process of power services, a large amount of multi-source heterogeneous data will also be generated, and the existing multi-source data traceability methods are difficult to accurately and effectively trace these data, resulting in poor quality and low accuracy of the traced data, which cannot be used to optimize various fields of power services. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-source data tracing method for power data services, which solves the problem that the data obtained by the existing tracing methods are of poor quality and low accuracy and cannot be used to optimize various fields of power services.

[0006] In order to achieve the above objectives, the following technical solutions are proposed: A multi-source data tracing method for power data services includes the following steps: S1, automatic analysis, obtaining power data traceability information; S2, system tracking, marking and tracking various types of power data; S3, machine learning, improves the ability to analyze and process power data; S4, manually collect, supplement and verify data traceability information.

[0007] The present invention can accurately obtain relevant power traceability information by parsing various types of basic power data, and then add corresponding logs or annotations in the data processing program to add identifiers corresponding to the traceability information of the data, thereby realizing the tracking of data flow, and can enable the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method.

[0008] Preferably, the automatic parsing includes parsing SQL statement files, storing procedure files and ETL process files to obtain data traceability information in various fields of power production services. The automatic parsing includes parsing SQL statement files, storing procedure files and ETL process files to obtain data traceability information in various fields of power production services. The present invention can fully and accurately obtain the traceability information of various types of data in various fields involved in power services, thereby improving the accuracy and reliability of data traceability.

[0009] Preferably, the traceability information of the power data includes one or more of the source or type or mode or processing process or dependency. The traceability information of the power data includes source, type, mode, processing process, dependency, etc. The present invention can trace the power data more accurately and efficiently, thereby providing a more reliable basis for investigation of subsequent problems in various fields and processes, thereby promoting the generation of more effective optimization decisions, thereby improving the level of power production, sales and services.

[0010] Preferably, the multi-source data is processed first in the automatic parsing process, especially dynamic data, heterogeneous data and uncertain data. In the automatic parsing process, it is necessary to process the multi-source data first in order to efficiently obtain the power data traceability information, specifically, to process heterogeneous data, dynamic data and uncertain data. The present invention can improve the efficiency of power data traceability, reduce data deviations, errors or duplications in the traceability process, and thus improve the effect and reliability of the traceability work.

[0011] Preferably, processing the heterogeneous data requires integrating the heterogeneous data. Processing the heterogeneous data requires integrating, fusing and integrating the heterogeneous data. The present invention can achieve the purposes of data unification, standardization, cleaning and conversion, thereby improving the consistency and availability of data.

[0012] Preferably, one or more of streaming computing processing, real-time analysis processing, or incremental update processing is performed on the dynamic data. Processing the dynamic data requires the use of technologies such as streaming computing, real-time analysis, and incremental update. The present invention can achieve rapid processing, response, and feedback of data, thereby effectively improving the efficiency of multi-source data tracing.

[0013] Preferably, one or more of data quality assessment processing, data repair processing, data completion processing, or data fusion processing is performed on the uncertain data. Data quality assessment, data repair, data completion, data fusion and other technologies need to be used to process the uncertain data. The present invention can realize error correction, optimization and improvement of data, so as to ensure the accuracy, authenticity and reliability of the data source, thereby laying a good foundation for subsequent data tracing work, ensuring that reliable data tracing results can be obtained in the future to support relevant personnel to draw more accurate and reliable conclusions, thereby effectively improving the correctness of relevant decisions.

[0014] Preferably, the system tracking includes adding an identifier corresponding to the traceability information of the already parsed basic power information in the data processing program, and the identifier includes a log or an annotation. The system tracking is to add an identifier corresponding to the traceability information of the already parsed basic power information by adding a corresponding log or annotation in the data processing program. The present invention can add identifiers to the relevant basic power data, so that the information can be clearly identified in the subsequent machine learning process, and then the similarity or dependency between the data can be calculated by these identifiers, thereby forming a virtuous circle, accelerating the speed of machine learning, and ultimately more and more effectively improving the speed and accuracy of data traceability, and can provide a basis for subsequent data traceability and further improve the speed of data traceability by tracking the corresponding data status in the entire power production service process.

[0015] Preferably, the machine learning includes inferring data traceability information by calculating the dependency or similarity between the data. The specific method of the machine learning is to infer data traceability information by calculating the similarity or dependency between the data. The present invention can realize data traceability from point to surface through the similarity or dependency between the data, thereby significantly improving the efficiency of data traceability.

[0016] Preferably, the manual collection includes manually labeling the data and manually collecting and collating the data. The manual collection specifically includes manually collecting and collating the data and manually labeling the data. The present invention can check for omissions and fill in gaps in the basic power data, further improving the accuracy and reliability of the power data tracing method.

[0017] The beneficial effects of the present invention are as follows: by parsing various types of basic power data, relevant power traceability information can be accurately obtained, and then corresponding logs or annotations are added to the data processing program to add identifiers corresponding to the traceability information, thereby realizing the tracking of data flow, and enabling the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of a multi-source data tracing method of the present invention.

[0019] Figure 2 It is another flow chart of the multi-source data tracing method of the present invention. DETAILED DESCRIPTION

[0020] Embodiment 1: This embodiment proposes a multi-source data tracing method for power data services. Figure 1 , including the following steps: S1, automatic parsing: used to obtain power data traceability information; the power data traceability information includes source, type, mode, processing process, dependency, etc. The automatic parsing includes parsing SQL statement files, stored procedure files and ETL process files to obtain data traceability information in various fields of power production services.

[0021] The present invention can fully and accurately obtain the traceability information of various types of data in various fields involved in power services, thereby improving the accuracy and reliability of data traceability.

[0022] The present invention can trace the power data more accurately and efficiently, thereby providing a more reliable basis for investigating subsequent problems in various fields and processes, and further promoting the generation of more effective optimization decisions, thereby improving the level of power production, sales and services.

[0023] In the automatic analysis process, it is necessary to first process the multi-source data in order to efficiently obtain the power data traceability information, specifically, to process heterogeneous data, dynamic data and uncertain data.

[0024] The present invention can improve the efficiency of power data tracing, reduce data deviations, errors or duplications in the tracing process, and thus improve the effectiveness and reliability of the tracing work.

[0025] Processing the heterogeneous data requires integrating, merging and integrating the heterogeneous data.

[0026] The present invention can achieve the purposes of data unification, standardization, cleaning, conversion, etc., thereby improving the consistency and availability of data.

[0027] Processing the dynamic data requires the use of technologies such as streaming computing, real-time analysis, and incremental updates. The present invention can achieve rapid processing, response, and feedback of data, thereby effectively improving the efficiency of multi-source data tracing.

[0028] Processing the uncertain data requires the use of data quality assessment, data repair, data completion, data fusion and other technologies. The present invention can achieve error correction, optimization and improvement of data, thereby ensuring the accuracy, authenticity and reliability of the data source, thus laying a good foundation for subsequent data tracing work, ensuring that reliable data tracing results can be obtained in the future to support relevant personnel to draw more accurate and reliable conclusions, thereby effectively improving the correctness of relevant decisions.

[0029] S2, system tracking: used to mark and track various types of electricity data; specifically, various types of basic electricity data include electricity production, electricity sales price, electricity production and sales cost, and electricity utilization rate, etc.

[0030] S3, machine learning: used to improve the ability to analyze and process power data; S4, manual collection: used to supplement and verify data traceability information.

[0031] The present invention can accurately obtain relevant power traceability information by parsing various types of basic power data, and then add corresponding logs or annotations in the data processing program to add identifiers corresponding to the traceability information of the data, thereby realizing the tracking of data flow, and can enable the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method.

[0032] Embodiment 2: This embodiment optimizes S2 on the basis of embodiment 1 and proposes a multi-source data tracing method for power data services. Figure 2 , including the following steps: S1, automatic parsing: used to obtain power data traceability information; the power data traceability information includes source, type, mode, processing process, dependency, etc. The automatic parsing includes parsing SQL statement files, stored procedure files and ETL process files to obtain data traceability information in various fields of power production services.

[0033] The present invention can fully and accurately obtain the traceability information of various types of data in various fields involved in power services, thereby improving the accuracy and reliability of data traceability.

[0034] The present invention can trace the power data more accurately and efficiently, thereby providing a more reliable basis for investigating subsequent problems in various fields and processes, and further promoting the generation of more effective optimization decisions, thereby improving the level of power production, sales and services.

[0035] In the automatic analysis process, it is necessary to first process the multi-source data in order to efficiently obtain the power data traceability information, specifically, to process heterogeneous data, dynamic data and uncertain data.

[0036] The present invention can improve the efficiency of power data tracing, reduce data deviations, errors or duplications in the tracing process, and thus improve the effectiveness and reliability of the tracing work.

[0037] Processing the heterogeneous data requires integrating, merging and integrating the heterogeneous data.

[0038] The present invention can achieve the purposes of data unification, standardization, cleaning, conversion, etc., thereby improving the consistency and availability of data.

[0039] Processing the dynamic data requires the use of technologies such as streaming computing, real-time analysis, and incremental updates. The present invention can achieve rapid processing, response, and feedback of data, thereby effectively improving the efficiency of multi-source data tracing.

[0040] Processing the uncertain data requires the use of data quality assessment, data repair, data completion, data fusion and other technologies. The present invention can achieve error correction, optimization and improvement of data, thereby ensuring the accuracy, authenticity and reliability of the data source, thus laying a good foundation for subsequent data tracing work, ensuring that reliable data tracing results can be obtained in the future to support relevant personnel to draw more accurate and reliable conclusions, thereby effectively improving the correctness of relevant decisions.

[0041] S2, system tracking: used to mark and track various types of electricity data; specifically, various types of basic electricity data include electricity production, electricity sales price, electricity production and sales cost, and electricity utilization rate, etc.

[0042] The system tracking is to add an identifier corresponding to the traceability information of the already parsed basic power information by adding corresponding logs or annotations in the data processing program. The present invention can add identifiers to the relevant basic power data, so as to facilitate the clear identification of the information in the subsequent machine learning process, and then calculate the similarity or dependency between the various data through these identifiers, thereby forming a virtuous circle, accelerating the speed of machine learning, and ultimately more and more effectively improving the speed and accuracy of data traceability, and can provide a basis for subsequent data traceability and further improve the speed of data traceability by tracking the corresponding data status in the entire power production service process through the identifier.

[0043] S3, machine learning: used to improve the ability to analyze and process power data; S4, manual collection: used to supplement and verify data traceability information.

[0044] The present invention can accurately obtain relevant power traceability information by parsing various types of basic power data, and then add corresponding logs or annotations in the data processing program to add identifiers corresponding to the traceability information of the data, thereby realizing the tracking of data flow, and can enable the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method.

[0045] Embodiment 3: This embodiment optimizes S3 on the basis of embodiment 2 and proposes a multi-source data tracing method for power data services. Figure 2 , including the following steps: S1, automatic parsing: used to obtain power data traceability information; the power data traceability information includes source, type, mode, processing process, dependency, etc. The automatic parsing includes parsing SQL statement files, stored procedure files and ETL process files to obtain data traceability information in various fields of power production services.

[0046] The present invention can fully and accurately obtain the traceability information of various types of data in various fields involved in power services, thereby improving the accuracy and reliability of data traceability.

[0047] The present invention can trace the power data more accurately and efficiently, thereby providing a more reliable basis for investigating subsequent problems in various fields and processes, and further promoting the generation of more effective optimization decisions, thereby improving the level of power production, sales and services.

[0048] In the automatic analysis process, it is necessary to first process the multi-source data in order to efficiently obtain the power data traceability information, specifically, to process heterogeneous data, dynamic data and uncertain data.

[0049] The present invention can improve the efficiency of power data tracing, reduce data deviations, errors or duplications in the tracing process, and thus improve the effectiveness and reliability of the tracing work.

[0050] Processing the heterogeneous data requires integrating, merging and integrating the heterogeneous data.

[0051] The present invention can achieve the purposes of data unification, standardization, cleaning, conversion, etc., thereby improving the consistency and availability of data.

[0052] Processing the dynamic data requires the use of technologies such as streaming computing, real-time analysis, and incremental updates. The present invention can achieve rapid processing, response, and feedback of data, thereby effectively improving the efficiency of multi-source data tracing.

[0053] Processing the uncertain data requires the use of data quality assessment, data repair, data completion, data fusion and other technologies. The present invention can achieve error correction, optimization and improvement of data, thereby ensuring the accuracy, authenticity and reliability of the data source, thus laying a good foundation for subsequent data tracing work, ensuring that reliable data tracing results can be obtained in the future to support relevant personnel to draw more accurate and reliable conclusions, thereby effectively improving the correctness of relevant decisions.

[0054] S2, system tracking: used to mark and track various types of electricity data; specifically, various types of basic electricity data include electricity production, electricity sales price, electricity production and sales cost, and electricity utilization rate, etc.

[0055] The system tracking is to add an identifier corresponding to the traceability information of the already parsed basic power information by adding corresponding logs or annotations in the data processing program. The present invention can add identifiers to the relevant basic power data, so as to facilitate the clear identification of the information in the subsequent machine learning process, and then calculate the similarity or dependency between the various data through these identifiers, thereby forming a virtuous circle, accelerating the speed of machine learning, and ultimately more and more effectively improving the speed and accuracy of data traceability, and can provide a basis for subsequent data traceability and further improve the speed of data traceability by tracking the corresponding data status in the entire power production service process through the identifier.

[0056] S3, machine learning: used to improve the ability to analyze and process power data; The specific method of machine learning is to infer data traceability information by calculating the similarity or dependency between data. The present invention can realize data traceability from point to surface through the similarity or dependency between data, thereby significantly improving the efficiency of data traceability work.

[0057] S4, manual collection: used to supplement and verify data traceability information.

[0058] The present invention can accurately obtain relevant power traceability information by parsing various types of basic power data, and then add corresponding logs or annotations in the data processing program to add identifiers corresponding to the traceability information of the data, thereby realizing the tracking of data flow, and can enable the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method.

[0059] Embodiment 4: This embodiment optimizes S4 on the basis of embodiment 3 and proposes a multi-source data tracing method for power data services. Figure 2 , including the following steps: S1, automatic parsing: used to obtain power data traceability information; the power data traceability information includes source, type, mode, processing process, dependency, etc. The automatic parsing includes parsing SQL statement files, stored procedure files and ETL process files to obtain data traceability information in various fields of power production services.

[0060] The present invention can fully and accurately obtain the traceability information of various types of data in various fields involved in power services, thereby improving the accuracy and reliability of data traceability.

[0061] The present invention can trace the power data more accurately and efficiently, thereby providing a more reliable basis for investigating subsequent problems in various fields and processes, and further promoting the generation of more effective optimization decisions, thereby improving the level of power production, sales and services.

[0062] In the automatic analysis process, it is necessary to first process the multi-source data in order to efficiently obtain the power data traceability information, specifically, to process heterogeneous data, dynamic data and uncertain data.

[0063] The present invention can improve the efficiency of power data tracing, reduce data deviations, errors or duplications in the tracing process, and thus improve the effectiveness and reliability of the tracing work.

[0064] Processing the heterogeneous data requires integrating, merging and integrating the heterogeneous data.

[0065] The present invention can achieve the purposes of data unification, standardization, cleaning, conversion, etc., thereby improving the consistency and availability of data.

[0066] Processing the dynamic data requires the use of technologies such as streaming computing, real-time analysis, and incremental updates. The present invention can achieve rapid processing, response, and feedback of data, thereby effectively improving the efficiency of multi-source data tracing.

[0067] Processing the uncertain data requires the use of data quality assessment, data repair, data completion, data fusion and other technologies. The present invention can achieve error correction, optimization and improvement of data, thereby ensuring the accuracy, authenticity and reliability of the data source, thus laying a good foundation for subsequent data tracing work, ensuring that reliable data tracing results can be obtained in the future to support relevant personnel to draw more accurate and reliable conclusions, thereby effectively improving the correctness of relevant decisions.

[0068] S2, system tracking: used to mark and track various types of electricity data; specifically, various types of basic electricity data include electricity production, electricity sales price, electricity production and sales cost, and electricity utilization rate, etc.

[0069] The system tracking is to add an identifier corresponding to the traceability information of the already parsed basic power information by adding corresponding logs or annotations in the data processing program. The present invention can add identifiers to the relevant basic power data, so as to facilitate the clear identification of the information in the subsequent machine learning process, and then calculate the similarity or dependency between the various data through these identifiers, thereby forming a virtuous circle, accelerating the speed of machine learning, and ultimately more and more effectively improving the speed and accuracy of data traceability, and can provide a basis for subsequent data traceability and further improve the speed of data traceability by tracking the corresponding data status in the entire power production service process through the identifier.

[0070] S3, machine learning: used to improve the ability to analyze and process power data; The specific method of machine learning is to infer data traceability information by calculating the similarity or dependency between data. The present invention can realize data traceability from point to surface through the similarity or dependency between data, thereby significantly improving the efficiency of data traceability work.

[0071] S4, manual collection: used to supplement and verify data traceability information.

[0072] The manual collection specifically refers to manually collecting and collating data and manually labeling the data. The present invention can check for omissions and fill in gaps in basic power data, further improving the accuracy and reliability of the power data tracing method.

[0073] The present invention can accurately obtain relevant power traceability information by parsing various types of basic power data, and then add corresponding logs or annotations in the data processing program to add identifiers corresponding to the traceability information of the data, thereby realizing the tracking of data flow, and can enable the machine to quickly identify the data and its traceability information in subsequent machine learning, and calculate the similarity and dependency between the data based on the identifiers of these data, so as to continuously improve the traceability speed and accuracy of various types of power data, and then cooperate with manual collection to check for omissions in the basic power data, further improving the accuracy and reliability of the power data traceability method.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source data tracing method for power data services, characterized in that: The steps include: S1, automatic analysis, obtaining power data traceability information; S2, system tracking, marking and tracking various types of power data; S3, machine learning, improves the ability to analyze and process power data; S4, manually collect, supplement and verify data traceability information.

2. A multi-source data tracing method for power data services according to claim 1, characterized in that: The automatic parsing includes parsing SQL statement files, parsing stored procedure files and ETL process files, and obtaining data traceability information in various fields of power production services.

3. A multi-source data tracing method for power data services according to claim 1 or 2, characterized in that: The traceability information of the power data includes one or more of source, type, mode, processing procedure or dependency.

4. The multi-source data tracing method for power data service according to claim 3 is characterized by: In the automatic parsing process, multi-source data is first processed, especially dynamic data, heterogeneous data and uncertain data.

5. The multi-source data tracing method for power data service according to claim 4 is characterized in that: Processing the heterogeneous data requires integrating the heterogeneous data.

6. A multi-source data tracing method for power data services according to claim 4, characterized in that: One or more of stream computing processing, real-time analysis processing, or incremental update processing is performed on the dynamic data.

7. The multi-source data tracing method for power data service according to claim 4 is characterized in that: One or more of data quality assessment processing, data repair processing, data completion processing, and data fusion processing are performed on the uncertain data.

8. A multi-source data tracing method for power data services according to any one of claims 4 to 7, characterized in that: The system tracking includes adding an identification corresponding to the traceability information of the parsed basic power information in the data processing program, and the identification includes a log or an annotation.

9. A multi-source data tracing method for power data services according to any one of claim 1, characterized in that: The machine learning includes inferring data provenance information based on the dependencies or similarities between calculated data.

10. A multi-source data tracing method for power data services according to any one of claim 1, characterized in that: The manual collection includes manually labeling the data and manually collecting and arranging the data.

Images