Remote Meter Reading Data Quality Evaluation Method, System and Medium Based on Evaluation Model

Abstract

The present invention relates to the technical field of meter reading data analysis, and specifically relates to a method, system and medium for remotely reading meter data quality assessment based on an evaluation model. The method includes: performing format standardization processing on historical remote meter reading data; extracting principal component features from the standardized historical remote meter reading data by using the principal component analysis method; constructing an evaluation model, and training the evaluation model based on the principal component features of the historical remote meter reading data by using an ensemble learning method, where the input of the evaluation model is the principal component features and the output is a number of evaluation indicators; assigning weights to each evaluation indicator; obtaining remote meter reading data in real time; performing format standardization processing on the remotely read meter data obtained in real time, extracting the required principal component features, inputting them into the trained evaluation model to obtain evaluation indicators, and calculating the quality score of the remotely read meter data based on each evaluation indicator and the corresponding weights, realizing data-driven evaluation indicator correlation analysis, and improving the accuracy and comprehensiveness of the evaluation results.

Description

Technical Field

[0001] The present invention relates to the technical field of meter reading data analysis, and specifically relates to a method, system and medium for remotely reading meter data quality assessment based on an evaluation model. Background Art

[0002] With the rapid development of technologies such as the Internet of Things, big data, and cloud computing, the remote meter reading technology has emerged. This technology uses devices such as smart meters and sensors to achieve real-time monitoring and accurate measurement of energy usage, and transmits the data to the remote meter reading system for processing and analysis. The emergence of the remote meter reading technology has greatly improved the meter reading efficiency and data accuracy, providing strong support for energy management and optimization.

[0003] In the remote meter reading technology, data quality is the key to ensuring the accuracy of energy measurement and the scientific nature of management decisions. High-quality data can truly reflect the energy usage situation and provide a reliable basis for energy management and optimization. If the data quality is low, with problems such as errors, missing values or anomalies, it will lead to inaccurate energy measurement, and further affect the effect of energy management and optimization. Therefore, it is of great significance to analyze and evaluate the quality of remote meter reading data.

[0004] Currently, certain development achievements have been made in the existing methods for remotely reading meter data quality assessment. For example, in Chinese Patent CN112286899A, a method for meter reading data quality assessment, a meter reading center, a system, a device and a medium, the meter reading center sends a meter reading request to the meter device to obtain meter reading data, parses the meter reading data to obtain each field value, and then judges whether each field value is compliant according to the preset meter reading data rules to obtain the compliance rate of each field value, and calculates the quality score of the meter reading data through the compliance rate of each field value, thus completing the quality scoring of the meter reading data.

[0005] However, the existing methods for remotely reading meter data quality assessment often rely on manually preset meter reading data rules for judgment in actual applications. Different energy enterprises, different regions or even different evaluators may adopt different evaluation indicators and weights, resulting in the lack of unity and comparability of evaluation results, and it is difficult to comprehensively cover all possible data quality problems. For example, some evaluation methods may focus on evaluating the integrity and compliance of data while ignoring the accuracy of data, etc. Summary of the Invention

[0006] In view of the above technical problems, the present invention proposes a method, system and medium for remotely reading meter data quality assessment based on an evaluation model, aiming to achieve data-driven evaluation index correlation analysis, provide scientific decision support for the quality assessment of remotely read meter data, and improve the accuracy and comprehensiveness of evaluation results.

[0007] In a first aspect, the present application provides a method for evaluating the quality of remote meter reading data based on an evaluation model, including the following steps:

[0008] Obtain historical remote meter reading data;

[0009] Perform format standardization processing on the historical remote meter reading data to obtain the standardized historical remote meter reading data;

[0010] Use the principal component analysis method to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data;

[0011] Construct an evaluation model, and use the ensemble learning method to train the evaluation model based on the principal component features of the historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is several evaluation indicators;

[0012] Assign weights to each evaluation indicator to obtain the weights of each evaluation indicator;

[0013] Obtain remote meter reading data in real time;

[0014] Perform format standardization processing on the remotely obtained meter reading data in real time and extract the required principal component features;

[0015] Input the extracted principal component features into the trained evaluation model to obtain several evaluation indicators, and calculate the quality score of the remote meter reading data based on each evaluation indicator and the weight of the corresponding evaluation indicator.

[0016] In some embodiments, using the principal component analysis method to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data includes:

[0017] Calculate the covariance matrix between each feature in the historical remote meter reading data;

[0018] Solve the eigenvalues and eigenvectors of the covariance matrix;

[0019] According to the magnitudes of the eigenvalues, arrange the eigenvalues in descending order and calculate the ratio of the explained variance corresponding to each eigenvalue;

[0020] Calculate the cumulative sum of the ratios of the explained variance corresponding to each eigenvalue in the order of the arrangement of the eigenvalues to form a cumulative explained variance sequence;

[0021] Search the cumulative explained variance sequence to determine the minimum number of principal components that reach the preset threshold;

[0022] Select the eigenvectors corresponding to the first N eigenvalues as the principal component features of the historical remote meter reading data in the order of the arrangement of the eigenvalues, where N is the minimum number of principal components.

[0023] In some embodiments, an evaluation model is trained using an ensemble learning method based on the principal component features of historical remote meter reading data, including:

[0024] Based on the principal component features of historical remote meter reading data, list all evaluation metrics and the calculation rules for the evaluation metrics;

[0025] Use the random forest algorithm to determine the preferred evaluation metric calculation rules from all the evaluation metric calculation rules, and use the preferred evaluation metric calculation rules as the labels of the evaluation model;

[0026] Use the dataset containing the principal component features and labels as the training set to train the evaluation model. The input of the trained evaluation model is the principal component features, and the output is several evaluation metrics calculated according to the preferred evaluation metric calculation rules.

[0027] In some embodiments, using the random forest algorithm to determine the preferred evaluation metric calculation rules from all the evaluation metric calculation rules includes:

[0028] Use the historical remote meter reading data as samples, each evaluation metric calculation rule as a feature, and randomly select several features to form a feature subset;

[0029] Construct a random forest model containing multiple decision trees, and for each decision tree, train it with the randomly selected samples and feature subsets;

[0030] Calculate the average decrease in impurity of each feature over all the decision trees in the random forest as a quantitative measure of feature importance;

[0031] Arrange the features in descending order according to the magnitude of feature importance, and select the evaluation metric calculation rules corresponding to the top several features as the preferred evaluation metric calculation rules.

[0032] In some embodiments, weights are assigned to each evaluation metric to obtain the weights of each evaluation metric, including:

[0033] Use the entropy weight method to assign weights to each evaluation metric to obtain the objective weights of each evaluation metric.

[0034] In some embodiments, using the entropy weight method to assign weights to each evaluation metric to obtain the objective weights of each evaluation metric includes:

[0035] Standardize the evaluation metrics to obtain the standardized values of each metric;

[0036] Use the historical remote meter reading data as samples, and calculate the proportion of the standardized value of each metric in each sample to the total standardized value of that metric in all samples;

[0037] Calculate the entropy value of each evaluation index based on the specific gravity;

[0038] Determine the difference coefficient of each evaluation index based on the entropy value of each evaluation index;

[0039] Calculate the objective weight of each evaluation index based on the difference coefficient of each evaluation index.

[0040] In some embodiments, weighting each evaluation index to obtain the weights of each evaluation index further includes:

[0041] Cluster each evaluation index using a clustering algorithm;

[0042] Analyze the feature distribution in different clusters, identify the evaluation indexes with similar features, and converge them into several categories of evaluation dimensions according to the evaluation indexes;

[0043] Weight each category of evaluation dimensions to obtain the weights of each category of evaluation dimensions;

[0044] Based on the weights of each category of evaluation dimensions, weight the evaluation indexes in each category of evaluation dimensions to obtain the weights of each evaluation index in the evaluation dimension.

[0045] In some embodiments, calculating the quality score of the remote meter reading data based on each evaluation index and the weight of the corresponding evaluation index includes:

[0046] Perform weighted calculation on the values of each evaluation index and the weights of the corresponding evaluation indexes to obtain the quality score of the remote meter reading data.

[0047] In a second aspect, the present application provides a remote meter reading data quality evaluation system based on an evaluation model, including:

[0048] A data acquisition module for acquiring historical remote meter reading data and real-time acquiring remote meter reading data;

[0049] A format standardization module for performing format standardization processing on the historical remote meter reading data to obtain the standardized historical remote meter reading data, and performing format standardization processing on the real-time acquired remote meter reading data;

[0050] A feature extraction module for extracting the principal component features of the historical remote meter reading data by using the principal component analysis method, and extracting the required principal component features from the standardized real-time remote meter reading data;

[0051] An evaluation model training module for constructing an evaluation model and training the evaluation model based on the principal component features of the historical remote meter reading data by using an ensemble learning method, where the input of the evaluation model is the principal component features and the output is several evaluation indexes;

[0052] An empowerment module for assigning weights to each evaluation index to obtain the weights of each evaluation index;

[0053] A quality score output module for inputting the extracted principal component features into the trained evaluation model to obtain several evaluation indexes, and calculating the quality score of the remote meter reading data based on each evaluation index and the corresponding weight of the evaluation index.

[0054] Thirdly, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for evaluating the quality of remote meter reading data based on an evaluation model as described above is implemented.

[0055] The beneficial technical effects of the present invention at least include:

[0056] 1. By adopting the method, system and medium for evaluating the quality of remote meter reading data based on an evaluation model, firstly, historical remote meter reading data is obtained and format standardized processing is carried out to ensure the consistency and comparability of the data. Secondly, the principal component analysis method is applied to extract features from the standardized data, reducing the data dimension while retaining the principal component features most important for quality evaluation. Then, an evaluation model is constructed and trained based on the principal component features using an ensemble learning method (i.e., the random forest algorithm), so that the evaluation model can output multiple evaluation indexes. Then, an objective weight assignment is carried out for each evaluation index output by the evaluation model to reflect the importance of different evaluation indexes in data quality evaluation. Finally, the evaluation model is applied in practice, that is, the latest remote meter reading data is obtained in real time, subjected to the same format standardization and feature extraction processing, and input into the trained evaluation model. According to the evaluation indexes output by the model and the corresponding weights, the quality score of the remote meter reading data is calculated, thereby realizing data-driven evaluation index correlation analysis, providing scientific decision-making support for the quality evaluation of remote meter reading data, and greatly improving the accuracy and comprehensiveness of the quality evaluation results of remote meter reading data;

[0057] 2. Since the remote meter reading system usually involves multiple data sources, different communication methods, and transmission protocols, this diversity may cause the remote meter reading data to be affected by different factors during the processes of collection, transmission, and processing, thereby increasing the difficulty of evaluating the quality of remote meter reading data. Therefore, this application extracts features from the standardized historical remote meter reading data by using the principal component analysis method, which helps to integrate the data from different meters into a common framework, enabling the data from different sources to be compared and analyzed in the same dimension. During the process, the cumulative explained variance is further calculated to select a sufficient number of principal component features to retain the information of the original data to the greatest extent, which helps to more comprehensively reflect the quality of remote meter reading data. At the same time, those components with smaller variances are ignored to achieve data dimensionality reduction, focusing on the main variability of the data, and reducing the interference caused by different communication methods and transmission protocols to a certain extent, thereby reducing the impact of data source diversity on the quality assessment results;

[0058] 3. Compared with the prior art that directly uses the manually preset meter data rules for data quality assessment, this application lists all potential evaluation indicators and their calculation rules based on the principal component features of the historical remote meter reading data, and uses the random forest algorithm to screen the listed evaluation indicator calculation rules to determine the preferred evaluation indicators that have a greater impact on data quality assessment, greatly reducing manual intervention and improving the automation and intelligence levels of the assessment process. The preferred evaluation indicators are used as the labels of the evaluation model, and the random forest model is trained with the dataset containing the principal component features and the labels of the preferred evaluation indicators, enabling the evaluation model to predict multiple evaluation indicators simultaneously, thereby providing a more comprehensive data quality assessment. Moreover, the trained evaluation model can be used for subsequent real-time monitoring and early warning of remote meter reading data to timely discover data quality problems, which is difficult to achieve by the prior art method based on preset rules. Therefore, this application provides a more advanced and effective solution for the quality assessment of remote meter reading data;

[0059] 4. By using the entropy weight method to assign weights to each evaluation indicator, the objective weights of each evaluation indicator are calculated based on the dispersion degree of the evaluation indicator itself, ensuring the rationality of weight allocation and not being limited to a specific data distribution. Therefore, it can be applied to the meter reading data of different types of meters, further reducing the impact of data source diversity on the quality assessment results.

[0060] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The following further describes the present invention with reference to the drawings:

[0062] Figure 1This is the flowchart of the remote meter reading data quality assessment method based on the evaluation model in the embodiments of the present invention.

[0063] Figure 2 This is the schematic structural diagram of the remote meter reading data quality assessment system based on the evaluation model in the embodiments of the present invention. Detailed implementation manners

[0064] The technical solutions in the embodiments of the present invention will be explained and illustrated below with reference to the accompanying drawings of the embodiments of the present invention. However, the following embodiments are only the preferred embodiments of the present invention, not all of them. Based on the embodiments in the implementation manners, other embodiments obtained by those skilled in the art without creative efforts all fall within the protection scope of the present invention.

[0065] In the following description, terms such as "inner", "outer", "upper", "lower", "left", "right", etc. indicating orientation or position relationships are only for the convenience of describing the embodiments and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention.

[0066] Please refer to the atta Figure 1 , Figure 1 which shows the schematic flowchart of the remote meter reading data quality assessment method provided by an embodiment of this specification.

[0067] As Figure 1 shown, the remote meter reading data quality assessment method based on the evaluation model may at least include the following steps:

[0068] Step 101, obtain historical remote meter reading data.

[0069] It can be understood that remote meter reading data usually involves multiple data sources, including different types of meters such as smart electricity meters, water meters, gas meters, etc. Specifically, the historical remote meter reading data of different data sources in this embodiment includes but is not limited to reading timestamps, meter readings, meter types, meter IDs, working states of meters, communication signal strengths between meters and remote reading systems, and so on. Among them, since the meter ID is the unique identity code for distinguishing each meter device, to a certain extent, the data quality assessment of different meter devices can also reflect the stability and reliability levels of the corresponding meter devices.

[0070] Step 102, perform format standardization processing on the historical remote meter reading data to obtain the standardized historical remote meter reading data.

[0071] Specifically, the implementation manner of performing format standardization processing on the historical remote meter reading data is as follows:

[0072] 1. Unify feature columns:

[0073] First, list the feature columns of all data sources and identify the unique columns of each data source;

[0074] Secondly, according to business requirements, define a standard feature set that includes all feature columns;

[0075] Next, for each data source, establish a mapping relationship between the original data feature columns and the standard data feature columns, rename the original feature columns with the names in the standard feature set, and ensure that each original data feature column can be correctly mapped to the standard data feature column to avoid data loss or errors;

[0076] Then, for the standard features missing in some data sources, default values, calculated values or interpolation methods can be filled.

[0077] 2. Data format unification:

[0078] First, check the data types (such as integers, floating-point numbers, strings, etc.) of each feature column in the standard feature set, and then convert the data types of the same feature column in all data sources to the data types in the standard format. For example, convert the data types of all feature columns representing quantities to floating-point numbers.

[0079] Step 103: Use the principal component analysis method to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data.

[0080] It can be understood that the principal component features of the historical remote meter reading data are the features related to the quality assessment of the meter reading data. Taking the water meter as an example, the principal component features of the historical remote meter reading data in this embodiment may include meter ID, reading timestamp, collection frequency, water meter reading, over-range data, data fluctuation, water meter stop code data, etc.

[0081] Since the remote meter reading system usually involves multiple data sources, including different types of meters such as smart electricity meters, water meters, and gas meters, as well as different communication methods and transmission protocols, this diversity may cause the remote meter reading data to be affected by different factors during the collection, transmission, and processing processes, thus increasing the difficulty of quality assessment of the remote meter reading data. Therefore, further in this embodiment, the principal component analysis method is used to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data, including:

[0082] Step 201: Calculate the covariance matrix between each feature in the historical remote meter reading data.

[0083] It can be understood that the covariance matrix describes the correlation between each feature. For a vector X = [x1, x2,... containing n features , the calculation formula of the covariance matrix Cov(X) can be:

[0084]

[0085] Among them, n represents the total number of features, represents the i-th eigenvalue, represents the mean value of the eigenvector X currently being calculated.

[0086] Step 202, solve the eigenvalues and eigenvectors of the covariance matrix.

[0087] Specifically, the eigenvalues and eigenvectors are obtained by solving the characteristic equation of the covariance matrix. The characteristic equation can be expressed as:

[0088]

[0089] Among them, represents the eigenvalue, and I represents the identity matrix.

[0090] Step 203, according to the magnitudes of the eigenvalues, sort the eigenvalues in descending order and calculate the ratio of the explained variance corresponding to each eigenvalue.

[0091] Specifically, after sorting the eigenvalues in descending order, solve the ratio of each eigenvalue divided by the sum of all eigenvalues to obtain the ratio of the explained variance corresponding to each eigenvalue.

[0092] Step 204, calculate the cumulative sum of the ratios of the explained variance corresponding to each eigenvalue in the order of the eigenvalue arrangement to form a cumulative explained variance sequence.

[0093] It can be understood that in the cumulative explained variance sequence, each cumulative explained variance is the cumulative sum of the ratios of the explained variance corresponding to the eigenvalues from top to bottom in the order of the eigenvalue arrangement.

[0094] Step 205, search the cumulative explained variance sequence to determine the minimum number of principal components that reach the preset threshold.

[0095] It can be understood that among the cumulative explained variances greater than the preset threshold, the number of the ratios of the explained variance corresponding to the eigenvalues included in the cumulative explained variance closest to the preset threshold is the minimum number of principal components.

[0096] Step 206, select the eigenvectors corresponding to the top N eigenvalues as the principal component features of the historical remote meter reading data in the order of the eigenvalue arrangement, where N is the minimum number of principal components.

[0097] In this embodiment, the principal component analysis method is used to extract features from the standardized historical remote meter reading data, which helps to integrate data from different meters into a common framework, enabling data from different sources to be compared and analyzed in the same dimension. During the process, a sufficient number of principal component features are further selected by calculating the cumulative explained variance to retain the information of the original data to the greatest extent, which helps to more comprehensively reflect the quality of the remote meter reading data. At the same time, those components with smaller variances are ignored to achieve data dimensionality reduction, focusing on the main variability of the data, and reducing the interference caused by different communication methods and transmission protocols to a certain extent, thereby reducing the impact of the diversity of data sources on the quality assessment results.

[0098] Step 104: Build an evaluation model. Use the ensemble learning method to train the evaluation model based on the principal component features of the historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is several evaluation indicators.

[0099] Specifically, in this embodiment, using the ensemble learning method to train the evaluation model based on the principal component features of the historical remote meter reading data includes:

[0100] Step 301: Based on the principal component features of the historical remote meter reading data, list all the evaluation indicators and the calculation rules for the evaluation indicators.

[0101] Among them, the calculation rules for the evaluation indicators define how to calculate evaluation indicators such as the data completeness rate and the data validity rate from the original remote meter reading data.

[0102] Step 302: Use the random forest algorithm to determine the preferred calculation rules for the evaluation indicators from all the calculation rules for the evaluation indicators, and use the preferred calculation rules for the evaluation indicators as the labels of the evaluation model.

[0103] Among them, the random forest algorithm is a type of ensemble learning method. The ensemble learning method improves the performance of the overall evaluation model by combining the prediction results of multiple base models (such as decision trees in a random forest).

[0104] Furthermore, in this embodiment, using the random forest algorithm to determine the preferred calculation rules for the evaluation indicators from all the calculation rules for the evaluation indicators includes:

[0105] Step 401: Use the historical remote meter reading data as samples, and each calculation rule for the evaluation indicator as a feature. Randomly select several features to form a feature subset to form a diverse tree structure to enhance the generalization ability of the model.

[0106] Among them, a sample refers to a row of a data set, where each row represents a complete historical remote meter reading data. A feature refers to a column in the data set. A feature subset refers to a part of features randomly selected from all features when constructing each decision tree, and the feature subset is used to make a splitting decision at each node.

[0107] Step 402: Construct a random forest model including multiple decision trees. For each decision tree, train it with randomly selected samples and feature subsets. Each tree recursively splits the feature space to minimize impurity.

[0108] Among them, in a random forest, decision trees usually use impurity measures (such as Gini impurity or information gain) to evaluate the purity of nodes. The lower the impurity, the more the samples in the node belong to the same class.

[0109] Step 403: Calculate the average impurity reduction of each feature over all decision trees in the random forest as a quantitative indicator of feature importance.

[0110] Among them, the impurity reduction represents, for each feature, the difference in impurity before and after splitting the data using this feature. If a certain feature can significantly reduce the impurity, then its importance for constructing the decision tree is relatively high. Since there are multiple trees in the random forest and each tree may use different features for splitting, for each feature, it is necessary to calculate the average of the impurity reduction over all decision trees, that is, the average impurity reduction, which reflects the average importance of the feature in the entire random forest.

[0111] Specifically, for each feature, divide the sum of the impurity reduction over all trees by the number of trees to calculate the average impurity reduction.

[0112] Step 404: Sort the features in descending order according to the magnitude of feature importance, and select the calculation rules of the evaluation indicators corresponding to the top several features as the preferred evaluation indicator calculation rules.

[0113] Among them, the selection of the number of the top several features depends on the variance ratio of the indicators we hope to evaluate. These features are the evaluation indicators and their calculation rules that have a greater impact on the analysis of the quality of meter reading data. In data quality assessment, we can pay more attention to the factors that have a direct impact on quality changes, or adopt the "80 / 20 rule" to select features that can evaluate at least 80% of the indicators. This embodiment does not make a limitation on this.

[0114] Exemplarily, taking a water meter as an example, the preferred evaluation indicators and their calculation rules can at least include the following items:

[0115] 1. Data Completeness Rate: Calculate the ratio of the actual number of acquisitions to the number of acquisitions that should be made, and divide the value of the data completeness rate according to the size of the ratio. For example, the values of the evaluation indicators can be divided into several levels: when the ratio ≥ 98%, it is 100; when the ratio is between 95% (inclusive) - 98%, it is 80; when the ratio is between 90% (inclusive) - 95%, it is 60; when the ratio is between 85% (inclusive) - 90%, it is 40; when the ratio < 80%, it is 20.

[0116] 2. Data Efficiency Rate: Calculate the ratio of the number of effective acquisitions (i.e., the actual number of acquisitions minus the number of occurrences of out-of-range data) to the actual number of acquisitions, and divide the value of the data efficiency rate according to the size of the ratio. For example, the values of the evaluation indicators can be divided into several levels: when the ratio ≥ 98%, it is 100; when the ratio is between 95% (inclusive) - 98%, it is 80; when the ratio is between 90% (inclusive) - 95%, it is 60; when the ratio is between 85% (inclusive) - 90%, it is 40; when the ratio < 80%, it is 20.

[0117] 3. Water Meter Base Value Zero: Create a binary feature indicating whether the water meter stop code data is zero, and count the cases where the water meter stop code data shows a zero value. If there is no case where the water meter stop code data shows a zero value, the value of the evaluation indicator is 100; if there is a case where the water meter stop code data shows a zero value, the value of the evaluation indicator is 0.

[0118] 4. Consecutive Decrease Times of Meter Reading Data: Count the number of times that the meter reading data satisfies that the last acquisition value of the current day is less than the last acquisition value of the previous day for seven consecutive days, reflecting the continuous change of the meter reading data. Each time it occurs, the value of the evaluation indicator is -10, that is, 10 points will be deducted from the evaluation score each time it occurs.

[0119] 5. Consecutive 7-day Sudden Increase Times of Meter Reading Data: Count the number of times that the average daily water volume in the previous 7 days has an increase rate of 10% or more compared with the same period of the previous month, reflecting the year-on-year change of the meter reading data.

[0120] 6. Consecutive 7-day Sudden Decrease Times of Meter Reading Data: Count the number of times that the average daily water volume in the previous 7 days has a decrease rate of 10% or more compared with the same period of the previous month, reflecting the year-on-year change of the meter reading data.

[0121] Step 303: Use the dataset containing the principal component features and labels as the training set to train the evaluation model. The input of the trained evaluation model is the principal component features, and the output is several evaluation indicators calculated according to the preferred evaluation index calculation rules.

[0122] For example, taking the selection of the random forest model as the evaluation model, the implementation method of using the dataset containing the principal component features and labels as the training set to train the evaluation model is as follows:

[0123] First, perform random sampling with replacement from the dataset to form multiple different training subsets. Then, for each training subset, construct a decision tree. At each node of the tree, randomly select a training subset, and in the selected training subset, find the optimal splitting point to maximize the information gain or other splitting criteria (such as Gini impurity). Since each tree is trained independently of other trees, parallel processing can be performed, predicting multiple labels simultaneously, and integrating their prediction results, thereby comprehensively considering multiple evaluation metrics and reducing the limitations of a single model.

[0124] Compared with the prior art that directly uses manually preset meter data rules for data quality assessment, in this embodiment, based on the principal component features of historical remote meter reading data, all potential evaluation metrics and their calculation rules are listed, and the random forest algorithm is used to screen the calculation rules of the listed evaluation metrics to determine the preferred evaluation metrics that have a greater impact on data quality assessment, greatly reducing manual intervention, improving the automation and intelligence level of the evaluation process, and using the preferred evaluation metrics as the labels of the evaluation model. The random forest model is trained using the dataset containing the principal component features and the labels of the preferred evaluation metrics, enabling the evaluation model to predict multiple evaluation metrics simultaneously, thereby providing a more comprehensive data quality assessment. Moreover, the trained evaluation model can be used for subsequent real-time monitoring and early warning of remote meter reading data to timely detect data quality problems, which is difficult to achieve by the prior art method based on preset rules. Therefore, this embodiment provides a more advanced and effective solution for remote meter reading data quality assessment.

[0125] Step 105: Assign weights to each evaluation metric to obtain the weights of each evaluation metric.

[0126] Specifically, in this embodiment, assigning weights to each evaluation metric to obtain the weights of each evaluation metric includes:

[0127] Use the entropy weight method to assign weights to each evaluation metric to obtain the objective weights of each evaluation metric.

[0128] Furthermore, in this embodiment, using the entropy weight method to assign weights to each evaluation metric to obtain the objective weights of each evaluation metric includes:

[0129] Step 501: Standardize the evaluation metrics to obtain the standardized values of each metric.

[0130] For example, for the evaluation metrics of data completeness rate and data validity rate, the min-max normalization formula can be selected for normalization processing; for the evaluation metric of the number of consecutive sudden increases / decreases in meter reading data, the Z-score normalization formula can be selected for normalization processing; for the evaluation metric of the zero value of the initial meter reading, whether to perform normalization processing can be selected according to the situation, etc. This embodiment does not limit this.

[0131] Step 502: Use the historical remote meter reading data as samples, and calculate the proportion of the normalized value of each indicator in each sample to the sum of the normalized values of this indicator in all samples. The proportion can be expressed as:

[0132]

[0133] where represents the normalized value of the i-th sample on the j-th indicator, and n represents the total number of samples.

[0134] Step 503: Based on the proportion, calculate the entropy value of each evaluation indicator to measure the information uncertainty of the indicator. The entropy value of each evaluation indicator can be expressed as:

[0135]

[0136] where is a constant used to ensure that the entropy value is between 0 and 1.

[0137] Step 504: Based on the entropy value of each evaluation indicator, determine the coefficient of variation of each evaluation indicator.

[0138] Specifically, the coefficient of variation of each evaluation indicator is 1 minus the entropy value of the corresponding evaluation indicator. The larger the coefficient of variation, the higher the importance of the evaluation indicator in the data quality evaluation system.

[0139] Step 505: Based on the coefficient of variation of each evaluation indicator, calculate the objective weight of each evaluation indicator, which can be expressed as:

[0140]

[0141] where represents the relative importance of the j-th evaluation indicator in the overall evaluation, that is, the objective weight of the j-th evaluation indicator, represents the coefficient of variation of each evaluation indicator, and m represents the total number of evaluation indicators.

[0142] Allocate to the corresponding evaluation indicators, and these weights represent the relative importance of each evaluation indicator in the analysis of the quality of meter reading data.

[0143] In this embodiment, the entropy weight method is used to assign weights to each evaluation index, and the objective weights of each evaluation index are calculated based on the dispersion degree of the evaluation index itself, ensuring the rationality of weight distribution and not being limited to a specific data distribution. Therefore, it can be applied to the meter reading data of different types of instruments, thereby further reducing the impact of data source diversity on the quality evaluation results.

[0144] On the other hand, in this embodiment, assigning weights to each evaluation index to obtain the weights of each evaluation index further includes:

[0145] Using a clustering algorithm to cluster each evaluation index;

[0146] Analyzing the feature distributions in different clusters, identifying evaluation indexes with similar features, and aggregating them into several types of evaluation dimensions according to the evaluation indexes;

[0147] Assigning weights to each type of evaluation dimension to obtain the weights of each type of evaluation dimension;

[0148] Based on the weights of each type of evaluation dimension, assigning weights to the evaluation indexes in each type of evaluation dimension to obtain the weights of each evaluation index in the evaluation dimension.

[0149] It can be understood that since the remote meter reading data may contain noise and outliers, and there are large differences in each evaluation index, this embodiment preferably uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to cluster each evaluation index, which can identify clusters of any shape and can handle noise points. When using the DBSCAN clustering algorithm to cluster each evaluation index, it is necessary to determine the neighborhood radius and the minimum number of points, and then determine the formation of clusters by calculating the density between data points.

[0150] Among them, the implementation method of analyzing the feature distributions in different clusters, identifying evaluation indexes with similar features, and aggregating them into several types of evaluation dimensions according to the evaluation indexes is as follows:

[0151] First, observe the evaluation index feature distributions of each cluster, and use the silhouette coefficient to evaluate the cohesion and separation of each cluster to judge the quality of the cluster;

[0152] Second, compare the similarities within and between clusters through the internal distance and the external distance, and identify evaluation indexes with similar features in different clusters;

[0153] Finally, combine the clustering results with business background knowledge, interpret the features of the clustering results, and then converge them into several evaluation dimensions according to the evaluation indicators, and name each evaluation dimension. For example, the data completeness rate, the data validity rate, and whether the water meter stop code data has a zero value can be converged into the evaluation dimension of data reliability, and the number of consecutive decreases in meter reading data and the number of sudden increases / decreases in meter reading data for 7 consecutive days can be converged into the evaluation dimension of data stability, etc.

[0154] In this embodiment, the internal structure and characteristics of the evaluation indicators are understood through the clustering algorithm, the evaluation indicators with similar features are identified, and they are converged into several evaluation dimensions according to the evaluation indicators. First, weights are assigned to the evaluation dimensions, and then based on the weights of each type of evaluation dimension, differential weights are further assigned to the evaluation indicators in different evaluation dimensions, which can optimize resource allocation, put more attention into the dimensions and indicators that have the greatest impact on data quality, and at the same time improve the standardization of the remote meter reading data quality evaluation method to a certain extent.

[0155] Step 106, obtain remote meter reading data in real time.

[0156] Step 107, perform format standardization processing on the remotely obtained meter reading data in real time, and extract the required principal component features.

[0157] Among them, the implementation method of performing format standardization processing on the remotely obtained meter reading data in real time in step 107 is similar to the implementation method of performing format standardization processing on the historical remote meter reading data in the foregoing step 102, and this embodiment will not elaborate here.

[0158] Among them, the "required principal component features" are the principal component features selected from the standardized historical remote meter reading data in step 103.

[0159] Step 108, input the extracted principal component features into the trained evaluation model to obtain several evaluation indicators, and calculate the quality score of the remote meter reading data based on each evaluation indicator and the weight of the corresponding evaluation indicator.

[0160] Furthermore, in this embodiment, calculating the quality score of the remote meter reading data based on each evaluation indicator and the weight of the corresponding evaluation indicator includes:

[0161] Perform weighted calculation on the values of each evaluation indicator and the weight of the corresponding evaluation indicator to obtain the quality score of the remote meter reading data.

[0162] In summary, in this embodiment, first, historical remote meter reading data is obtained and format standardized to ensure data consistency and comparability. Second, the principal component analysis method is applied to extract features from the standardized data, reducing the data dimension while retaining the principal component features most important for quality assessment. Then, an evaluation model is constructed and trained using an ensemble learning method (i.e., the random forest algorithm) based on the principal component features, enabling the evaluation model to output multiple evaluation indicators. Next, objective weight assignments are made to each evaluation indicator output by the evaluation model to reflect the importance of different evaluation indicators in data quality assessment. Finally, the evaluation model is applied in practice, that is, the latest remote meter reading data is obtained in real time, subjected to the same format standardization and feature extraction processes, and input into the trained evaluation model. According to the evaluation indicators and corresponding weights output by the model, the quality score of the remote meter reading data is calculated, thus realizing data-driven correlation analysis of evaluation indicators and providing scientific decision support for the quality assessment of remote meter reading data, significantly improving the accuracy and comprehensiveness of the quality assessment results of remote meter reading data.

[0163] Please refer to the attached Figure 2 , Figure 2 which is a schematic structural diagram of a remote meter reading data quality assessment system based on an evaluation model provided by an embodiment of this specification.

[0164] As Figure 2 shown, the remote meter reading data quality assessment system based on the evaluation model can at least include a data acquisition module 1, a format standardization module 2, a feature extraction module 3, an evaluation model training module 4, a weight assignment module 5, and a quality score output module 6, where:

[0165] The data acquisition module 1 is used to acquire historical remote meter reading data and real-time remote meter reading data;

[0166] The format standardization module 2 is used to perform format standardization processing on the historical remote meter reading data to obtain the standardized historical remote meter reading data, and perform format standardization processing on the real-time acquired remote meter reading data;

[0167] The feature extraction module 3 is used to extract features from the standardized historical remote meter reading data using the principal component analysis method to obtain the principal component features of the historical remote meter reading data, and extract the required principal component features from the standardized real-time remote meter reading data;

[0168] The evaluation model training module 4 is used to construct an evaluation model and train the evaluation model using an ensemble learning method based on the principal component features of the historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is several evaluation indicators;

[0169] An empowerment module 5 for assigning weights to each evaluation index to obtain the weights of each evaluation index;

[0170] A quality score output module 6 for inputting the extracted principal component features into the trained evaluation model to obtain several evaluation indexes, and calculating the quality score of the remote meter reading data based on each evaluation index and the corresponding weight of the evaluation index.

[0171] It can be understood that the technical concept of the remote meter reading data quality evaluation system based on the evaluation model provided in this embodiment is similar to that of the remote meter reading data quality evaluation method based on the evaluation model described above, and will not be elaborated herein.

[0172] Another embodiment of this specification provides a computer-readable storage medium, in which instructions are stored. When they run on a computer or a processor, the computer or the processor is enabled to execute one or more steps in the embodiments of the remote meter reading data quality evaluation method based on the evaluation model described above. If each component module of the above electronic device is implemented in the form of a software functional unit and used as an independent downstream task prediction or used, it can be stored in a computer-readable storage medium.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of this specification are generated in whole or in part. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0174] As described above, this is only a preferred embodiment disclosed in the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the protection scope involved in the present disclosure is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solution formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present disclosure.

[0175] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments may also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

Claims

1. A remote meter reading data quality assessment method based on an assessment model, characterized in that: The following steps are involved: Obtain historical remote meter reading data; Perform format standardization processing on historical remote meter reading data to obtain standardized historical remote meter reading data; The principal component analysis method is used to extract the features of the standardized historical remote meter reading data, and the principal component features of the historical remote meter reading data are obtained; Construct an evaluation model based on the principal component characteristics of historical remote meter reading data, list all evaluation indicators and evaluation indicator calculation rules; A random forest algorithm is used to determine the optimal evaluation indicator calculation rule from all evaluation indicator calculation rules, and the optimal evaluation indicator calculation rule is used as the label of the evaluation model; The evaluation model is trained using a data set containing principal component features and labels as a training set, wherein the input of the trained evaluation model is the principal component features, and the output is a number of evaluation indicators calculated according to the preferred evaluation indicator calculation rules; Clustering algorithms are used to cluster various evaluation indicators; Analyze the feature distribution in different clusters, identify evaluation indicators with similar features, and aggregate them into several categories of evaluation dimensions based on the evaluation indicators; Assign weights to each evaluation dimension to obtain the weights of each evaluation dimension; Based on the weights of various evaluation dimensions, the entropy weight method is used to weight the evaluation indicators in each evaluation dimension to obtain the objective weights of each evaluation indicator in the evaluation dimension; Obtain remote meter reading data in real time; Standardize the format of remote meter reading data acquired in real time and extract the required principal component features; The extracted principal component features are input into the trained evaluation model to obtain several evaluation indicators, and the quality score of the remote meter reading data is calculated based on each evaluation indicator and the weight of the corresponding evaluation indicator; The format of the historical remote meter reading data is standardized to obtain the standardized historical remote meter reading data, including: List the feature columns of all data sources in the historical remote meter reading data, and identify the unique columns of each data source, which are recorded as the original data feature columns; Define a standard feature set containing all feature columns according to business requirements. The feature columns in the standard feature set are recorded as standard data feature columns. For each data source, a mapping relationship between the original data feature columns and the standard data feature columns is established, and the original feature columns are renamed to the names in the standard feature set; Fill in missing standard features in the data source; The data type of each feature column in the standard feature set is checked, and the data types corresponding to the same feature column in all data sources are converted into data types in a standard format to obtain standardized historical remote meter reading data.

2. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The principal component analysis method is used to extract the features of the standardized historical remote meter reading data, and the principal component features of the historical remote meter reading data are obtained, including: Calculate the covariance matrix between each feature in the historical remote meter reading data; Solve for the eigenvalues ​​and eigenvectors of the covariance matrix; According to the size of the eigenvalue, the eigenvalues ​​are arranged in descending order, and the explained variance ratio corresponding to each eigenvalue is calculated; According to the order of eigenvalues, the cumulative sum of the explained variance ratios corresponding to each eigenvalue is calculated to form a cumulative explained variance sequence; Find the cumulative explained variance sequence and determine the minimum number of principal components that reaches the preset threshold; According to the arrangement order of the eigenvalues, the eigenvectors corresponding to the first N eigenvalues ​​are selected as the principal component features of the historical remote meter reading data, where N is the minimum number of principal components.

3. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The random forest algorithm is used to determine the optimal evaluation index calculation rules from all evaluation index calculation rules, including: The historical remote meter reading data is used as a sample, each evaluation index calculation rule is used as a feature, and several features are randomly selected to form a feature subset; Build a random forest model containing multiple decision trees, and train each decision tree with randomly selected samples and feature subsets; Calculate the average impurity reduction of each feature over all decision trees in the random forest as a quantitative indicator of feature importance; According to the importance of the features, the features are arranged in descending order, and the evaluation index calculation rules corresponding to the first several features are selected as the preferred evaluation index calculation rules.

4. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator, including: Standardize the evaluation indicators to obtain the standardized values ​​of each indicator; Taking the historical remote meter reading data as samples, calculate the proportion of the standardized value of each indicator in each sample to the sum of the standardized values ​​of the indicator in all samples; Based on the proportion, the entropy value of each evaluation index is calculated; Based on the entropy value of each evaluation indicator, determine the difference coefficient of each evaluation indicator; Based on the coefficient of difference of each evaluation indicator, the objective weight of each evaluation indicator is calculated.

5. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The quality score of remote meter reading data is calculated based on each evaluation indicator and the weight of the corresponding evaluation indicator, including: The value of each evaluation index is weighted and calculated with the weight of the corresponding evaluation index to obtain the quality score of the remote meter reading data.

6. The remote meter reading data quality assessment system based on the assessment model is characterized by: include: Data acquisition module, used to acquire historical remote meter reading data and real-time remote meter reading data; The format standardization module is used to perform format standardization processing on the historical remote meter reading data to obtain standardized historical remote meter reading data, and to perform format standardization processing on the remote meter reading data obtained in real time; A feature extraction module is used to extract features from the standardized historical remote meter reading data using a principal component analysis method to obtain the principal component features of the historical remote meter reading data, and to extract the required principal component features from the standardized real-time remote meter reading data; An evaluation model training module is used to construct an evaluation model. Based on the principal component features of historical remote meter reading data, all evaluation indicators and evaluation indicator calculation rules are listed. A random forest algorithm is used to determine the preferred evaluation indicator calculation rule from all evaluation indicator calculation rules. The preferred evaluation indicator calculation rule is used as the label of the evaluation model. The data set containing the principal component features and the label is used as the training set to train the evaluation model. The input of the trained evaluation model is the principal component features, and the output is a number of evaluation indicators calculated according to the preferred evaluation indicator calculation rule. A weighting module is used to cluster various evaluation indicators using a clustering algorithm, analyze the characteristic distribution in different clusters, identify evaluation indicators with similar characteristics, and aggregate them into several types of evaluation dimensions according to the evaluation indicators, weight each type of evaluation dimension, and obtain the weight of each type of evaluation dimension. Based on the weight of each type of evaluation dimension, the entropy weight method is used to weight the evaluation indicators in each type of evaluation dimension to obtain the objective weight of each evaluation indicator in the evaluation dimension; The quality score output module is used to input the extracted principal component features into the trained evaluation model to obtain a number of evaluation indicators, and calculate the quality score of the remote meter reading data based on each evaluation indicator and the weight of the corresponding evaluation indicator; The format standardization module is used to perform the following steps: List the feature columns of all data sources in the historical remote meter reading data, and identify the unique columns of each data source, which are recorded as the original data feature columns; Define a standard feature set containing all feature columns according to business requirements. The feature columns in the standard feature set are recorded as standard data feature columns. For each data source, a mapping relationship between the original data feature columns and the standard data feature columns is established, and the original feature columns are renamed to the names in the standard feature set; Fill in missing standard features in the data source; The data type of each feature column in the standard feature set is checked, and the data types corresponding to the same feature column in all data sources are converted into data types in a standard format to obtain standardized historical remote meter reading data.

7. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the remote meter reading data quality assessment method based on the assessment model according to any one of claims 1 to 5 is implemented.

Images