Full-life-cycle management method and system for metering business and assets
By automatically cleaning up abnormal data in the database, the problem of inefficiency in metering business and asset management is solved, and efficient and accurate full life cycle management is achieved.
Patent Information
- Application Number
- CN202510923102.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-04
- Publication Date
- 2025-10-17
AI Technical Summary
In existing technologies, the full life cycle management of measurement services and assets is inefficient due to missed inspections and repeated inspections caused by manual removal of abnormal data.
By generating data cleansing scripts, abnormal data in the database can be automatically cleaned, including obtaining cleansing plans, generating cleansing rules and scripts, performing data cleansing and confirmation, and finally synchronizing with the target database without abnormal data.
It improves the efficiency and accuracy of measurement business and asset life cycle management, and reduces missed inspections and duplicate inspections.
Smart Images
Figure CN120806859A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, in particular to a full life cycle management method and system for metering business and assets. BACKGROUND
[0002] With the advent of the digital era, power grid enterprises have accumulated a large amount of data, and reasonably playing the value of these data assets can help power grid enterprises achieve new breakthroughs, but the basis of these activities is dependent on high-quality data. Therefore, it is necessary for power grid enterprises to consolidate the data quality foundation and strengthen data quality management, so as to better release the value of data, improve decision-making, reduce costs, and reduce risks, help power grid enterprises digital transformation, and achieve high-quality development.
[0003] In the existing scheme, the full life cycle management of metering business and assets in some important scenarios still uses the method of manually removing abnormal data in the database, which leads to the situation of missing inspection and repeated inspection due to the negligence of relevant personnel when performing the full life cycle management of metering business and assets, resulting in low efficiency when performing the full life cycle management of metering business and assets. SUMMARY
[0004] The embodiments of the present application provide a full life cycle management method and system for metering business and assets, which can improve the efficiency of full life cycle management of metering business and assets.
[0005] The first aspect of the embodiments of the present application provides a full life cycle management method for metering business and assets, the method comprising:
[0006] obtaining a data cleaning scheme for cleaning abnormal data in a database to obtain a first cleaning scheme;
[0007] cleaning the data in the database according to the first cleaning scheme to obtain a first data set;
[0008] obtaining a data cleaning rule for cleaning abnormal data in a database to obtain a target cleaning rule;
[0009] generating a corresponding data cleaning script according to the target cleaning rule to obtain a first script;
[0010] cleaning the first data in the first data set using the first script to obtain a second data set;
[0011] confirming the second data set to obtain a target data set;
[0012] Synchronize target data in the target data set to the database to obtain a target database, wherein no abnormal data exists in the target database.
[0013] In this example, by obtaining a data cleaning scheme for cleaning abnormal data in the database, a first cleaning scheme is obtained, the data in the database is cleaned according to the first cleaning scheme, a first data set is obtained, a data cleaning rule for cleaning abnormal data in the database is obtained, a target cleaning rule is obtained, a corresponding data cleaning script is generated according to the target cleaning rule, a first script is obtained, the first data in the first data set is cleaned using the first script, a second data set is obtained, the second data set is confirmed, a target data set is obtained, and the target data in the target data set is synchronized to the database to obtain a target database, thereby improving the efficiency of the whole life cycle management of the metering business and assets.
[0014] In one possible implementation, a method for obtaining a data cleaning rule for cleaning abnormal data in a database to obtain a target cleaning rule includes:
[0015] Obtaining a data cleaning rule corresponding to an execution unit to obtain a first cleaning rule;
[0016] Obtaining a data cleaning rule corresponding to a supervision unit to obtain a second cleaning rule;
[0017] Integrating the first cleaning rule and the second cleaning rule to obtain a target cleaning rule.
[0018] In one possible implementation, a method for confirming the second data set to obtain a target data set includes:
[0019] Sending the second data set to an execution unit and a supervision unit respectively;
[0020] The execution unit and the supervision unit determine whether the second data in the second data set meets a preset cleaning effect, if the second data in the second data set meets the preset cleaning effect, the second data set is determined as a target data set, and if the second data in the second data set does not meet the preset cleaning effect, the reference second data in the second data set is cleaned again to obtain a target data set.
[0021] In a possible implementation, a method for performing unit and supervision unit to determine whether second data in the second data set meets a preset cleaning effect, if the second data in the second data set meets the preset cleaning effect, the second data set is determined as a target data set, if the second data in the second data set does not meet the preset cleaning effect, the reference second data set in the second data set is cleaned again to obtain the target data set, comprising:
[0022] If the second data in the second data set does not meet the preset cleaning effect, the first script is adjusted to obtain a second script;
[0023] The second data in the second data set is cleaned again using the second script to obtain a third data set;
[0024] Determine whether the third data in the third data set meets the preset cleaning effect, if the third data in the third data set meets the preset cleaning effect, the third data set is determined as a target data set, if the third data in the third data set does not meet the preset cleaning effect, the step of adjusting the first script to obtain the second script, cleaning the second data in the second data set using the second script to obtain the third data set is repeated until the target data set is obtained.
[0025] In a possible implementation, a method for obtaining a data cleaning rule for cleaning abnormal data in a database to obtain a target cleaning rule, the method further comprises:
[0026] Obtaining a data specification corresponding to the first data set to obtain a target data specification;
[0027] Determine whether the first data in the first data set meets the target data specification, if yes, the first data set is determined as a target data set, if no, a data cleaning rule for cleaning abnormal data in the database is obtained to obtain a target cleaning rule.
[0028] The second aspect of the embodiment of the application provides a full life cycle management system of metering business and assets, the system comprises:
[0029] A first obtaining unit is configured to obtain a data cleaning scheme for cleaning abnormal data in a database to obtain a first cleaning scheme;
[0030] A first cleaning unit is configured to clean data in the database according to the first cleaning scheme to obtain a first data set;
[0031] The second obtaining unit is configured to obtain a data cleaning rule for cleaning abnormal data in the database, and obtain a target cleaning rule.
[0032] The generating unit is configured to generate a corresponding data cleaning script according to the target cleaning rule, and obtain a first script.
[0033] The second cleaning unit is configured to clean first data in the first data set by using the first script, and obtain a second data set.
[0034] The confirming unit is configured to confirm the second data set, and obtain a target data set.
[0035] The synchronizing unit is configured to synchronize target data in the target data set to the database, and obtain a target database, wherein the target database does not contain abnormal data.
[0036] In one possible implementation, the second obtaining unit is specifically configured to:
[0037] obtain a first cleaning rule corresponding to an execution unit;
[0038] obtain a second cleaning rule corresponding to a supervision unit;
[0039] integrate the first cleaning rule and the second cleaning rule, and obtain the target cleaning rule.
[0040] In one possible implementation, the confirming unit is specifically configured to:
[0041] send the second data set to the execution unit and the supervision unit respectively;
[0042] The execution unit and the supervision unit determine whether second data in the second data set meets a preset cleaning effect, if the second data meets the preset cleaning effect, the second data set is determined as the target data set, and if the second data does not meet the preset cleaning effect, reference second data in the second data set is cleaned again, and a target data set is obtained.
[0043] In one possible implementation, in the process of obtaining a data cleaning rule for cleaning abnormal data in the database, and obtaining a target cleaning rule, the second obtaining unit is specifically configured to:
[0044] obtain a target data specification corresponding to the first data set;
[0045] Determine whether the first data in the first data set meets a target data specification. If yes, determine the first data set as a target data set. If no, obtain a data cleaning rule for cleaning abnormal data in the database, and obtain a target cleaning rule.
[0046] A third aspect of the embodiment of the present application provides a terminal, comprising a processor, an input device, an output device and a memory, which are connected with each other, wherein the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to invoke the program instructions to execute the step instructions as in the first aspect of the embodiment of the present application.
[0047] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application.
[0048] A fifth aspect of the embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product can be a software installation package. BRIEF DESCRIPTION OF DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.
[0050] Figure 1 A flowchart of a full life cycle management method of metering business and assets is provided for the embodiment of the present application;
[0051] Figure 2 A structural diagram of a terminal is provided for the embodiment of the present application;
[0052] Figure 3 A structural diagram of a full life cycle management system of metering business and assets is provided for the embodiment of the present application. DETAILED DESCRIPTION
[0053] With reference to the drawings and briefly describing the technical schemes in the embodiments of the present application, obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0054] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product, or device.
[0055] In the present application, "embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily refer to the same embodiment, nor is it mutually exclusive or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.
[0056] In order to better understand the method for managing the full life cycle of metering business and assets provided by the embodiments of the present application, first, the method for managing the full life cycle of metering business and assets in the prior art will be briefly introduced. In the prior art, the full life cycle management of metering business and assets in some important scenarios still uses the method of manually clearing the abnormal data in the database, which leads to the situation of missing inspection and repeated inspection due to the negligence of relevant personnel when the full life cycle management of metering business and assets is performed, and thus the efficiency of the full life cycle management of metering business and assets is low.
[0057] To solve the above technical problems, the embodiments of the present application provide a method for managing the full life cycle of metering business and assets, which can automatically clean the abnormal data in the database by generating a data cleaning script, and thus the efficiency of the full life cycle management of metering business and assets is improved.
[0058] Please refer to Figure 1 , Figure 1 A flowchart of the method for managing the full life cycle of metering business and assets provided by the embodiments of the present application is provided. As shown in Figure 1 , the method comprises:
[0059] 101. Obtain a data cleaning scheme for cleaning abnormal data in the database to obtain a first cleaning scheme.
[0060] The data cleaning scheme for cleaning abnormal data in the database can be obtained by a relevant business department, and the first cleaning scheme can be obtained by the relevant business department. Specifically, the relevant business department can obtain the data cleaning scheme for cleaning abnormal data in the database by understanding and sorting the business logic of the relevant business, and then upload the data cleaning scheme for cleaning abnormal data in the database to the server, so as to obtain the first cleaning scheme. The first cleaning scheme includes the priority of cleaning data, the fields corresponding to the data to be cleaned, and the reference values corresponding to the data of each field.
[0061] 102. Clean the data in the database according to the first cleaning scheme to obtain a first data set.
[0062] Specifically, the abnormal data in the database can be cleaned according to the priority of cleaning data, the fields corresponding to the data to be cleaned, and the reference values corresponding to the data of each field in the first cleaning scheme by using the record number checking method, the key indicator total amount verification method, the historical data comparison method, the value range judgment method, the experience audit method, and the matching judgment method, etc. to obtain the first data set.
[0063] 103. Obtain a data cleaning rule for cleaning abnormal data in the database to obtain a target cleaning rule.
[0064] The target cleaning rule can be obtained by integrating the data cleaning rules published by the execution unit and the supervision unit respectively.
[0065] 104. Generate a corresponding data cleaning script according to the target cleaning rule to obtain a first script.
[0066] The data cleaning script that meets the target cleaning rule can be generated by using a general script generation method according to the target cleaning rule to obtain the first script.
[0067] 105. Use the first script to clean the first data in the first data set to obtain a second data set.
[0068] The first script can be run by the server to automatically clean the abnormal data in the first data set by the first script to obtain the second data set.
[0069] 106. Confirm the second data set to obtain a target data set.
[0070] The first data in the first data set can be cleaned by using the first script, and the second data set is obtained. The target data set is obtained by confirming the second data set. The target data in the target data set is synchronized to the database to obtain a target database. The efficiency of the full life cycle management of the metering business and assets is improved.
[0071] 107. The target data in the target data set is synchronized to the database to obtain a target database, wherein the target database does not have abnormal data.
[0072] The target data in the target data set can be synchronized to the database to obtain a target database by using the target data in the target data set to replace the data in the database having the same field corresponding to the field of the target data in the target data set.
[0073] In this example, a data cleaning scheme for cleaning abnormal data in the database is obtained to obtain a first cleaning scheme. The data in the database is cleaned according to the first cleaning scheme to obtain a first data set. A data cleaning rule for cleaning abnormal data in the database is obtained to obtain a target cleaning rule. A corresponding data cleaning script is generated according to the target cleaning rule to obtain a first script. The first data in the first data set is cleaned by using the first script to obtain a second data set. The target data set is obtained by confirming the second data set. The target data in the target data set is synchronized to the database to obtain a target database. The efficiency of the full life cycle management of the metering business and assets is improved.
[0074] In one possible implementation, a method for obtaining a data cleaning rule for cleaning abnormal data in a database to obtain a target cleaning rule includes:
[0075] A1. Obtain a data cleaning rule corresponding to an execution unit to obtain a first cleaning rule;
[0076] A2. Obtain a data cleaning rule corresponding to a supervision unit to obtain a second cleaning rule;
[0077] A3. Integrate the first cleaning rule and the second cleaning rule to obtain a target cleaning rule.
[0078] The first cleaning rule can be obtained by obtaining the data cleaning rule for data cleaning published by the executing unit on the official website of the executing unit.
[0079] After obtaining the first cleaning rule, the second cleaning rule can be obtained by obtaining the data cleaning rule for data cleaning published by the supervising unit on the official website of the supervising unit.
[0080] After obtaining the first cleaning rule and the second cleaning rule, the target cleaning rule can be obtained by processing the first cleaning rule and the second cleaning rule by using a general processing method.
[0081] In this example, the first cleaning rule is obtained by obtaining the corresponding data cleaning rule for data cleaning on the official website of the executing unit, and the second cleaning rule is obtained by obtaining the corresponding data cleaning rule for data cleaning on the official website of the supervising unit. The first cleaning rule and the second cleaning rule are processed to obtain the target cleaning rule, which improves the accuracy of obtaining the target cleaning rule and thus improves the accuracy of the whole life cycle management of the measurement business and assets.
[0082] In one possible implementation, a method for confirming the second data set to obtain a target data set includes:
[0083] B1, the second data set is sent to the executing unit and the supervising unit respectively;
[0084] B2, the executing unit and the supervising unit judge whether the second data in the second data set meets the preset cleaning effect. If the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set. If the second data in the second data set does not meet the preset cleaning effect, the reference second data in the second data set is cleaned again to obtain the target data set.
[0085] The second data in the second data set can be obtained by obtaining the timeliness, accuracy and completeness of the second data in the second data set, and then sending the timeliness, accuracy and completeness of the second data in the second data set to the executing unit and the supervising unit respectively by using a general data transmission method.
[0086] After the execution unit and the supervision unit receive the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set, the server determines the second data set as the target data set if the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set meet the preset cleaning effect, and the server continues to clean the second data in the second data set after adjusting the first script if the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set do not meet the preset cleaning effect, until the second data in the second data set meets the preset cleaning effect, and the target data set is obtained; wherein the preset cleaning effect can be understood as the minimum value of the timeliness rate corresponding to the second data in the second data set, the minimum value of the accuracy rate corresponding to the second data in the second data set and the minimum value of the completeness rate corresponding to the second data in the second data set formulated by the execution unit and the supervision unit. If the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set are greater than or equal to the preset cleaning effect, it indicates that the second data in the second data set meets the cleaning effect, and if the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set are less than the preset cleaning effect, it indicates that the second data in the second data set does not meet the cleaning effect, and it is indicated that there is still abnormal data in the second data set.
[0087] In the example, by sending the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set to the execution unit and the supervision unit, the execution unit and the supervision unit determine the target data set by judging whether the timeliness rate, the accuracy rate and the completeness rate corresponding to the second data in the second data set meet the preset cleaning effect, thereby improving the accuracy when the metering business and assets are managed in the whole life cycle.
[0088] In one possible implementation, a method for an execution unit and a supervision unit to determine whether the second data in the second data set meets the preset cleaning effect, and if the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set, and if the second data in the second data set does not meet the preset cleaning effect, the reference second data in the second data set is cleaned again to obtain the target data set, comprising:
[0089] C1, if the second data in the second data set does not meet the preset cleaning effect, the first script is adjusted to obtain a second script;
[0090] C2, the second data in the second data set is cleaned again using the second script to obtain a third data set;
[0091] C3, determine whether the third data in the third data set meets the preset cleaning effect, if the third data in the third data set meets the preset cleaning effect, determine the third data set as the target data set, if the third data in the third data set does not meet the preset cleaning effect, repeatedly execute the step of adjusting the first script to obtain the second script, and using the second script to clean the second data in the second data set to obtain the third data set until the target data set is obtained.
[0092] If the second data in the second data set does not meet the preset cleaning effect, the first script can be adjusted by adjusting the priority of cleaning data in the first script and expanding the field range corresponding to the data to be cleaned, thereby adjusting the first script to obtain the second script.
[0093] The server runs the second script to clean the abnormal data in the second data set to obtain the third data set.
[0094] After obtaining the third data set, the server determines whether the third data in the third data set meets the preset cleaning effect, if the third data in the third data set meets the preset cleaning effect, the third data set is determined as the target data set, if the third data in the third data set does not meet the preset cleaning effect, the step of repeatedly executing the first script to obtain the second script is executed, and using the second script to clean the second data in the second data set to obtain the third data set until the target data set is obtained.
[0095] In this example, the second data in the second data set is cleaned by generating a data cleaning script, which realizes automatic cleaning of the second data in the second data set, thereby improving the efficiency of the whole life cycle management of the metering business and assets.
[0096] In one possible implementation, before the target cleaning rule is obtained by obtaining the data cleaning rule for cleaning the abnormal data in the database, the method further includes:
[0097] D1, obtaining the data specification corresponding to the first data set to obtain the target data specification;
[0098] D2, determine whether the first data in the first data set meets the target data specification, if it meets, determine the first data set as the target data set, if it does not meet, obtain the data cleaning rule for cleaning the abnormal data in the database to obtain the target cleaning rule.
[0099] The target data specification can be obtained by obtaining the data specification corresponding to the historical data that has been confirmed as normal data after cleaning in the database as the first data set.
[0100] After obtaining the target data specification, the server determines whether the first data in the first data set conforms to the target data specification. If yes, the first data set is determined as the target data set. If no, the data cleaning rule for cleaning the abnormal data in the database is obtained to obtain the target cleaning rule.
[0101] In this example, the target data specification is obtained by obtaining the data specification corresponding to the first data in the first data set, and the first data in the first data set is preliminarily judged according to the target data specification, thereby improving the efficiency in the full life cycle management of the metering business and assets.
[0102] In one specific implementation, the embodiment of the application further provides another full life cycle management method of metering business and assets, specifically:
[0103] The data quality evaluation method is specifically:
[0104] The quality inspection mode of specific data adopts a record number inspection method, a key indicator total amount verification method, a historical data comparison method, a value range judgment method, an experience audit method, and a matching judgment method. Through these methods, the data accuracy of a single data point can be checked, and data quality problems can be found in time.
[0105] (1) Record number inspection method
[0106] The data situation is verified by comparing the record number. Mainly, whether the record number of the data table is a certain value or within a certain range is checked.
[0107] Applicable scope: For the data in the data table loaded incrementally by date, the number of records increased in each loading period is a constant value or can be determined within a certain range, and the record number inspection must be performed.
[0108] (2) Key indicator total amount verification method
[0109] For the key indicators, whether the total amount of data is consistent is compared. Mainly, the check of the summary logic of the same business meaning from different dimensions.
[0110] Applicable scope: When the same field in the same table is statistically summarized from different dimensions, the total amount inspection must be performed.
[0111] The fields in this table have the same business meaning as the fields in other tables, are statistically different from different dimensions, have a summary relationship, and the data in the two tables is not processed by the same data source. When this condition is met, the total amount test must be performed.
[0112] (3) Historical data comparison method
[0113] Observe the change rule of the data through historical data to verify the data quality. Usually, the same development speed is used for judgment. When evaluating, according to the development characteristics of various indicators, the data with larger same development speed increase (or decrease) should be audited. Historical data comparison method includes same period and period comparison.
[0114] Applicable scope: cannot perform record number check method, key indicator total amount verification method, and the record number of the fact table is less than 1000 million.
[0115] (4) Value range judgment method
[0116] Determine the reasonable variation interval of the indicator data within a certain period, and focus on auditing the data outside the interval. The reasonable variation interval of the data is determined directly according to business experience.
[0117] Applicable scope: the fields in the fact table can determine the value range, and the data outside the range must be wrong. If this condition is met, the value range judgment method must be performed.
[0118] (5) Experience auditing method
[0119] For the logical relationship between indicators in the report, which cannot be confirmed by computer program auditing, quantization, or some audits have set quantity limits, but the limits are difficult to determine, manual experience auditing is needed.
[0120] Applicable scope: when the above methods are not applicable, the experience auditing method can be used.
[0121] (6) Matching judgment method
[0122] Compare and verify with the relevant data provided or released by relevant departments.
[0123] Applicable scope: when the data provided or released by relevant departments is consistent in scope, the matching judgment method can be used.
[0124] Data quality control process
[0125] (1) Data quality verification process
[0126] Data warehouse has many ETL tasks executed daily to load data, and ensuring the completeness and accuracy of ETL loaded data is the basic requirement of data quality management.
[0127] 1) Daily data check
[0128] Data quality manager checks the ETL loading task execution every day. Data check method selection ETL task data quality check requirements must use at least one of the following three methods to determine: record number check method; Key indicators total volume verification method; Value domain judgment method.
[0129] Data check cycle
[0130] There are many ETL loading tasks every day, and if all data checks are performed, it will take too long, so the check frequency is determined according to the trust level of each theme data.
[0131] The correspondence between the trust level and the check frequency is as follows:
[0132] Level 1: Data check must be performed every time loading
[0133] Level 2: Data check is performed every three times loading
[0134] Level 3: Data check is performed every six times loading
[0135] For theme data that needs special protection, the check frequency can be adjusted and additional experience audit method can be added.
[0136] 2) Timely data sampling
[0137] Data check ensures the integrity and accuracy of incremental data loaded every day, and on this basis, the data quality management team must organize a regular sampling of the data warehouse every quarter.
[0138] The scope of regular sampling must include all theme data with a trust level of one, two themes with a trust level of two, and one theme with a trust level of three. Regular sampling must use all methods defined in the data quality evaluation method.
[0139] 3) Comprehensive data check
[0140] The data quality management team must organize a comprehensive check of the data warehouse every year.
[0141] The scope of comprehensive check includes all theme data of enterprise data center platform.
[0142] Comprehensive check must use all methods defined in the data quality evaluation method.
[0143] (II) Data exception handling process
[0144] Data quality management personnel should check and verify data errors in time, fill in the data problem processing form according to the checking and verifying results, describe the current situation, causes and correction, prevention measures of data quality problems.
[0145] (2) After the data quality management team leader approves, report to the data center management department for approval to execute the data correction task.
[0146] (Three) Data quality evaluation report
[0147] According to the data quality inspection, the data quality management team will generate relevant data quality evaluation reports regularly or irregularly. Data quality reports are divided into two categories:
[0148] Monthly data quality report, i.e. monthly data quality problem statement.
[0149] Data quality report submitted after data quality sampling or comprehensive inspection.
[0150] 1. Monthly data quality evaluation report
[0151] The monthly data quality report is a monthly execution report prepared at the end of each month or at the beginning of the next month. The data quality management team will summarize and count the data quality of the month, and adjust the inspection strategy according to the changes in the "data quality problem frequency" of each theme. The report format is as follows:
[0152] A, non-system problem
[0153] (1) Data quality problem phenomenon
[0154] (2) Business data range
[0155] (3) Problem statement
[0156] B, system problem
[0157] (1) Data quality problem phenomenon
[0158] (2) Business data range
[0159] (3) Problem statement
[0160] 2. Data quality report submitted after data quality sampling or comprehensive inspection
[0161] The data quality sampling or comprehensive inspection report is a quality report prepared after each enterprise data center sampling or comprehensive inspection. Compared with the monthly data quality report, in addition to counting the data quality and reclassifying the theme level, it also needs to evaluate and improve the operation of the entire quality system. The report format is as follows:
[0162] A. Basic Overview
[0163] It includes: relevant organizations and personnel involved in data quality inspection; time of data quality inspection; location of data quality inspection; form of data quality inspection; scope of data quality inspection and other aspects.
[0164] B. Data quality check
[0165] (1) Evaluation process and steps
[0166] (2) Data checking method
[0167] It is mainly divided into full inspection and spot inspection. Full inspection must state the scope, content and method of inspection. Spot inspection must state the sampling plan, process and scope, content and method of data inspection.
[0168] (3) Data quality evaluation method
[0169] C. Data Quality Review and Conclusion
[0170] (1) Commentary
[0171] A comprehensive description of the data quality (including any problems).
[0172] (2) Conclusions and Recommendations
[0173] Including inspection results, trust level adjustment suggestions, etc.
[0174] For the same example as above, please refer to Figure 2 , Figure 2 A schematic diagram of the structure of a terminal provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the system comprises a processor, an input device, an output device and a memory, which are connected to each other, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions, and the program includes instructions for executing the following steps;
[0175] Obtaining a data cleaning solution for cleaning abnormal data in a database to obtain a first cleaning solution;
[0176] Cleaning the data in the database according to the first cleaning solution to obtain a first data set;
[0177] Obtaining data cleaning rules for cleaning abnormal data in the database to obtain target cleaning rules;
[0178] Generate a corresponding data cleaning script according to the target cleaning rule to obtain a first script;
[0179] cleaning first data in the first data set by using the first script to obtain a second data set;
[0180] confirming the second data set to obtain a target data set;
[0181] synchronizing target data in the target data set to a database to obtain a target database, wherein no abnormal data exists in the target database.
[0182] The above mainly introduces the scheme of the embodiments of the present application from the perspective of the execution process of the method. It can be understood that the terminal includes a hardware structure and / or a software module corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the unit and algorithm steps of each example described in the embodiments provided in the present application, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0183] The embodiments of the present application can divide the functional units of the terminal according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division. When actually implemented, there can be another division manner.
[0184] Consistent with the above, please refer to Figure 3 , Figure 3 The embodiments of the present application provide a structural schematic diagram of a full life cycle management system of a metering service and asset. As shown in Figure 3 , the system includes:
[0185] A first acquisition unit 301 is configured to acquire a data cleaning scheme for cleaning abnormal data in a database to obtain a first cleaning scheme.
[0186] A first cleaning unit 302 is configured to clean data in the database according to the first cleaning scheme to obtain a first data set.
[0187] A second acquisition unit 303 is configured to acquire a data cleaning rule for cleaning abnormal data in the database to obtain a target cleaning rule.
[0188] The generating unit 304 is configured to generate a corresponding data cleaning script according to the target cleaning rule, to obtain a first script.
[0189] The second cleaning unit 305 is configured to clean first data in the first data set by using the first script, to obtain a second data set.
[0190] The confirming unit 306 is configured to confirm the second data set, to obtain a target data set.
[0191] The synchronizing unit 307 is configured to synchronize target data in the target data set to a database, to obtain a target database, wherein no abnormal data exists in the target database.
[0192] In one possible implementation, the second obtaining unit 303 is specifically configured to:
[0193] obtain a first cleaning rule corresponding to the execution unit;
[0194] obtain a second cleaning rule corresponding to the supervision unit;
[0195] integrate the first cleaning rule and the second cleaning rule, to obtain a target cleaning rule.
[0196] In one possible implementation, the confirming unit 306 is specifically configured to:
[0197] send the second data set to the execution unit and the supervision unit respectively;
[0198] The execution unit and the supervision unit judge whether second data in the second data set meets a preset cleaning effect, if the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set, if the second data in the second data set does not meet the preset cleaning effect, reference second data in the second data set is cleaned again, to obtain the target data set.
[0199] In one possible implementation, in the aspect that the execution unit and the supervision unit judge whether second data in the second data set meets a preset cleaning effect, if the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set, if the second data in the second data set does not meet the preset cleaning effect, reference second data in the second data set is cleaned again, to obtain the target data set, the confirming unit 306 is specifically configured to:
[0200] If the second data in the second data set does not conform to the preset cleaning effect, the first script is adjusted to obtain a second script;
[0201] The second data in the second data set is cleaned again using the second script to obtain a third data set;
[0202] It is judged whether the third data in the third data set conforms to the preset cleaning effect. If the third data in the third data set conforms to the preset cleaning effect, the third data set is determined as a target data set. If the third data in the third data set does not conform to the preset cleaning effect, the steps of adjusting the first script to obtain a second script, cleaning the second data in the second data set again using the second script to obtain a third data set, and so on, are repeated until the target data set is obtained.
[0203] In one possible implementation, before the target cleaning rule is obtained by acquiring the data cleaning rule for cleaning the abnormal data in the database, the second acquisition unit 303 is specifically configured to:
[0204] The data specification corresponding to the first data set is acquired to obtain a target data specification;
[0205] It is judged whether the first data in the first data set conforms to the target data specification. If yes, the first data set is determined as a target data set. If no, the data cleaning rule for cleaning the abnormal data in the database is acquired to obtain a target cleaning rule.
[0206] The embodiment of the application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute part or all steps of any one of the full life cycle management methods of measurement business and assets described in the above method embodiments.
[0207] The embodiment of the application further provides a computer program product, which includes a non-transitory computer readable storage medium storing a computer program. The computer program causes a computer to execute part or all steps of any one of the full life cycle management methods of measurement business and assets described in the above method embodiments.
[0208] It should be noted that, for the foregoing method embodiments, the sequences of the described actions are not necessarily required to achieve the objects of the application, and certain steps can be performed in other sequences or even concurrently. Additionally, the described embodiments are merely provided as examples, and not all of the actions described are necessarily required to achieve desired results.
[0209] In the above embodiments, the description of each embodiment is focused on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.
[0210] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, another division manner can be adopted. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical or other forms.
[0211] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
[0212] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software program module.
[0213] If the integrated unit is realized in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.
[0214] A person of ordinary skill in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable memory, and the memory can include a flash disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.
[0215] The embodiments of the present application are described in detail above, and specific examples are applied in this paper to describe the principles and implementation modes of the present application. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description of the embodiments should not be understood as a limitation of the present application.
Claims
1. A method for managing the entire life cycle of measurement services and assets, characterized in that: The method comprises: Obtaining a data cleaning solution for cleaning abnormal data in a database to obtain a first cleaning solution; Cleaning the data in the database according to the first cleaning solution to obtain a first data set; Obtaining data cleaning rules for cleaning abnormal data in the database to obtain target cleaning rules; Generate a corresponding data cleaning script according to the target cleaning rule to obtain a first script; Cleaning the first data in the first data set using the first script to obtain a second data set; confirming the second data set to obtain a target data set; The target data in the target data set is synchronized into a database to obtain a target database, wherein no abnormal data exists in the target database.
2. The method for managing the entire life cycle of measurement services and assets according to claim 1, characterized in that: The step of obtaining a data cleaning rule for cleaning abnormal data in a database to obtain a target cleaning rule includes: Obtain the data cleansing rule corresponding to the execution unit to obtain a first cleansing rule; Obtain the data cleaning rule corresponding to the supervisory unit to obtain the second cleaning rule; The first cleaning rule and the second cleaning rule are integrated to obtain a target cleaning rule.
3. The method for managing the entire life cycle of measurement services and assets according to claim 2, characterized in that: The confirming the second data set to obtain a target data set includes: sending the second data set to the execution unit and the supervision unit respectively; The execution unit and the supervision unit determine whether the second data set in the second data set meets the preset cleaning effect. If the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set. If the second data in the second data set does not meet the preset cleaning effect, the reference second data set in the second data set is cleaned again to obtain the target data set.
4. The method for managing the entire life cycle of measurement services and assets according to claim 3, characterized in that: The execution unit and the supervision unit determine whether the second data set in the second data set meets the preset cleaning effect, and if the second data in the second data set meets the preset cleaning effect, determine the second data set as the target data set; if the second data in the second data set does not meet the preset cleaning effect, clean the reference second data set in the second data set again to obtain the target data set, including: If the second data in the second data set does not meet the preset cleaning effect, adjusting the first script to obtain a second script; Performing secondary cleaning on the second data in the second data set using a second script to obtain a third data set; Determine whether the third data in the third data set meets the preset cleaning effect. If the third data in the third data set meets the preset cleaning effect, determine the third data set as the target data set. If the third data in the third data set does not meet the preset cleaning effect, repeat the steps of adjusting the first script to obtain the second script, and using the second script to perform secondary cleaning on the second data in the second data set to obtain the third data set, until the target data set is obtained.
5. The method for managing the entire life cycle of measurement services and assets according to claim 4, characterized in that: Before obtaining the data cleaning rule for cleaning abnormal data in the database and obtaining the target cleaning rule, the method further includes: Obtaining a data specification corresponding to the first data set to obtain a target data specification; Determine whether the first data in the first data set meets the target data specification. If so, determine the first data set as the target data set. If not, obtain a data cleaning rule for cleaning abnormal data in the database to obtain a target cleaning rule.
6. A full life cycle management system for measurement services and assets, characterized by: The system comprises: A first acquiring unit is configured to acquire a data cleaning solution for cleaning abnormal data in a database, thereby obtaining a first cleaning solution; A first cleaning unit, configured to clean the data in the database according to the first cleaning scheme to obtain a first data set; The second acquisition unit is used to acquire a data cleaning rule for cleaning abnormal data in the database to obtain a target cleaning rule; A generating unit, configured to generate a corresponding data cleaning script according to the target cleaning rule to obtain a first script; a second cleaning unit, configured to clean the first data in the first data set using the first script to obtain a second data set; a confirmation unit, configured to confirm the second data set to obtain a target data set; The synchronization unit is used to synchronize the target data in the target data set into a database to obtain a target database, wherein there is no abnormal data in the target database.
7. The full life cycle management system for measurement services and assets according to claim 6 is characterized in that: The second acquiring unit is specifically configured to: Obtain the data cleansing rule corresponding to the execution unit to obtain a first cleansing rule; Obtain the data cleaning rule corresponding to the supervisory unit to obtain the second cleaning rule; The first cleaning rule and the second cleaning rule are integrated to obtain a target cleaning rule.
8. The full life cycle management system for measurement services and assets according to claim 7 is characterized in that: The confirmation unit is specifically used for: sending the second data set to the execution unit and the supervision unit respectively; The execution unit and the supervision unit determine whether the second data set in the second data set meets the preset cleaning effect. If the second data in the second data set meets the preset cleaning effect, the second data set is determined as the target data set. If the second data in the second data set does not meet the preset cleaning effect, the reference second data set in the second data set is cleaned again to obtain the target data set.
9. A terminal, characterized in that: The system comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the full life cycle management method for metering services and assets according to any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the full life cycle management method for metering services and assets according to any one of claims 1 to 5.