Remote meter reading data quality evaluation method and system based on evaluation model and medium

Abstract

The invention relates to the technical field of meter reading data analysis, in particular to a remote meter reading data quality evaluation method and system based on an evaluation model and a medium. The method comprises the following steps: performing format standardization processing on historical remote meter reading data; extracting principal component characteristics from the standardized historical remote meter reading data by adopting a principal component analysis method; an evaluation model is constructed, an integrated learning method is adopted to train the evaluation model based on the principal component characteristics of the historical remote meter reading data, the input of the evaluation model is the principal component characteristics, and the output is a plurality of evaluation indexes; weighting each evaluation index; acquiring remote meter reading data in real time; and performing format standardization processing on remote meter reading data acquired in real time, extracting required principal component features, inputting the extracted principal component features into the trained evaluation model to obtain evaluation indexes, and performing calculation based on the evaluation indexes and corresponding weights to obtain quality scores of the remote meter reading data so as to realize data-driven evaluation index association analysis. And the accuracy and comprehensiveness of the evaluation result are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of meter reading data analysis, and in particular to a remote meter reading data quality assessment method, system and medium based on an assessment model. Background Art

[0002] With the rapid development of technologies such as the Internet of Things, big data, and cloud computing, remote meter reading technology has emerged. This technology uses smart meters, sensors and other devices to achieve real-time monitoring and accurate measurement of energy usage, and transmits data to the remote meter reading system for processing and analysis. The emergence of remote meter reading technology has greatly improved meter reading efficiency and data accuracy, providing strong support for energy management and optimization.

[0003] In remote meter reading technology, data quality is the key to ensuring the accuracy of energy measurement and the scientific nature of management decisions. High-quality data can truly reflect energy usage and provide a reliable basis for energy management and optimization. If the data quality is poor, errors, omissions or anomalies will lead to inaccurate energy measurement, which will in turn affect the effectiveness of energy management and optimization. Therefore, it is of great significance to analyze and evaluate the quality of remote meter reading data.

[0004] At present, the existing remote meter reading data quality assessment method has achieved certain development results. For example, Chinese patent CN112286899A, meter data quality assessment method, meter reading center, system, equipment and medium, the meter reading center sends a meter reading request to the meter device to obtain meter data, parses the meter data to obtain the value of each field, and then judges whether each field value is compliant according to the preset meter data rules, obtains the compliance rate of each field value, calculates the quality score of the meter data by the compliance rate of each field value, and completes the quality score of the meter data.

[0005] However, in actual applications, existing remote meter reading data quality assessment methods often rely on manually preset meter data rules for judgment, and different energy companies, different regions, and even different assessors may use different assessment indicators and weights, resulting in a lack of uniformity and comparability in the assessment results, and it is difficult to fully cover all possible data quality issues. For example, some assessment methods may focus on the integrity and compliance of the assessment data, while ignoring the accuracy of the data, etc. Summary of the invention

[0006] In response to the above technical problems, the present invention proposes a remote meter reading data quality assessment method, system and medium based on an assessment model, aiming to realize data-driven assessment indicator correlation analysis, provide scientific decision support for the quality assessment of remote meter reading data, and improve the accuracy and comprehensiveness of the assessment results.

[0007] In a first aspect, the present application provides a remote meter reading data quality assessment method based on an assessment model, comprising the following steps: Obtain historical remote meter reading data; Perform format standardization processing on historical remote meter reading data to obtain standardized historical remote meter reading data; The principal component analysis method is used to extract the features of the standardized historical remote meter reading data, and the principal component features of the historical remote meter reading data are obtained; Construct an evaluation model, and use an integrated learning method to train the evaluation model based on the principal component features of historical remote meter reading data, wherein the input of the evaluation model is the principal component features, and the output is a number of evaluation indicators; Assign weights to each evaluation indicator to obtain the weights of each evaluation indicator; Obtain remote meter reading data in real time; Standardize the format of remote meter reading data acquired in real time and extract the required principal component features; The extracted principal component features are input into the trained evaluation model to obtain several evaluation indicators, and the quality score of the remote meter reading data is calculated based on each evaluation indicator and the weight of the corresponding evaluation indicator.

[0008] In some embodiments, the principal component analysis method is used to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data, including: Calculate the covariance matrix between each feature in the historical remote meter reading data; Solve for the eigenvalues ​​and eigenvectors of the covariance matrix; According to the size of the eigenvalue, the eigenvalues ​​are arranged in descending order, and the explained variance ratio corresponding to each eigenvalue is calculated; According to the order of eigenvalues, the cumulative sum of the explained variance ratios corresponding to each eigenvalue is calculated to form a cumulative explained variance sequence; Find the cumulative explained variance sequence and determine the minimum number of principal components that reaches the preset threshold; According to the arrangement order of the eigenvalues, the eigenvectors corresponding to the first N eigenvalues ​​are selected as the principal component features of the historical remote meter reading data, where N is the minimum number of principal components.

[0009] In some embodiments, an ensemble learning method is used to train an evaluation model based on the principal component features of historical remote meter reading data, including: Based on the principal component characteristics of historical remote meter reading data, all evaluation indicators and evaluation indicator calculation rules are listed; A random forest algorithm is used to determine the optimal evaluation indicator calculation rule from all evaluation indicator calculation rules, and the optimal evaluation indicator calculation rule is used as the label of the evaluation model; The evaluation model is trained using a data set containing principal component features and labels as a training set. The input of the trained evaluation model is the principal component features, and the output is a number of evaluation indicators calculated according to the preferred evaluation indicator calculation rules.

[0010] In some embodiments, a random forest algorithm is used to determine a preferred evaluation index calculation rule from all evaluation index calculation rules, including: The historical remote meter reading data is used as a sample, each evaluation index calculation rule is used as a feature, and several features are randomly selected to form a feature subset; Build a random forest model containing multiple decision trees, and train each decision tree with randomly selected samples and feature subsets; Calculate the average impurity reduction of each feature over all decision trees in the random forest as a quantitative indicator of feature importance; According to the importance of the features, the features are arranged in descending order, and the evaluation index calculation rules corresponding to the first several features are selected as the preferred evaluation index calculation rules.

[0011] In some embodiments, each evaluation indicator is weighted to obtain the weight of each evaluation indicator, including: The entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator.

[0012] In some embodiments, the entropy weight method is used to assign weights to each evaluation indicator to obtain an objective weight of each evaluation indicator, including: Standardize the evaluation indicators to obtain the standardized values ​​of each indicator; Taking the historical remote meter reading data as samples, calculate the proportion of the standardized value of each indicator in each sample to the sum of the standardized values ​​of the indicator in all samples; Based on the proportion, the entropy value of each evaluation index is calculated; Based on the entropy value of each evaluation indicator, determine the difference coefficient of each evaluation indicator; Based on the coefficient of difference of each evaluation indicator, the objective weight of each evaluation indicator is calculated.

[0013] In some embodiments, each evaluation indicator is weighted to obtain the weight of each evaluation indicator, and further includes: Clustering algorithms are used to cluster various evaluation indicators; Analyze the feature distribution in different clusters, identify evaluation indicators with similar features, and aggregate them into several categories of evaluation dimensions based on the evaluation indicators; Assign weights to each evaluation dimension to obtain the weights of each evaluation dimension; Based on the weights of various evaluation dimensions, the evaluation indicators in each evaluation dimension are weighted to obtain the weights of each evaluation indicator in the evaluation dimension.

[0014] In some embodiments, the quality score of the remote meter reading data is obtained based on the evaluation indicators and the weights of the corresponding evaluation indicators, including: The value of each evaluation index is weighted and calculated with the weight of the corresponding evaluation index to obtain the quality score of the remote meter reading data.

[0015] In a second aspect, the present application provides a remote meter reading data quality assessment system based on an assessment model, comprising: Data acquisition module, used to acquire historical remote meter reading data and real-time remote meter reading data; The format standardization module is used to perform format standardization processing on the historical remote meter reading data to obtain standardized historical remote meter reading data, and to perform format standardization processing on the remote meter reading data obtained in real time; A feature extraction module is used to extract features from the standardized historical remote meter reading data using a principal component analysis method to obtain the principal component features of the historical remote meter reading data, and to extract the required principal component features from the standardized real-time remote meter reading data; An evaluation model training module is used to construct an evaluation model, and an ensemble learning method is used to train the evaluation model based on the principal component features of historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is a number of evaluation indicators. The weighting module is used to weight each evaluation indicator and obtain the weight of each evaluation indicator; The quality score output module is used to input the extracted principal component features into the trained evaluation model to obtain a number of evaluation indicators, and obtain the quality score of the remote meter reading data based on the evaluation indicators and the weight calculation of the corresponding evaluation indicators.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the remote meter reading data quality assessment method based on the assessment model as described above is implemented.

[0017] The beneficial technical effects of the present invention include at least: 1. A remote meter reading data quality assessment method, system and medium based on an assessment model are adopted. First, historical remote meter reading data is obtained and formatted in a standardized manner to ensure data consistency and comparability. Then, the principal component analysis method is used to extract features from the standardized data to reduce data dimensions while retaining the most important principal component features for quality assessment. Then, an assessment model is constructed and trained based on principal component features using an ensemble learning method (i.e., random forest algorithm) so that the assessment model can output multiple assessment indicators. Then, objective weights are assigned to each assessment indicator output by the assessment model to reflect the importance of different assessment indicators in data quality assessment. Finally, the assessment model is applied in practice, i.e., the latest remote meter reading data is obtained in real time, and the same format standardization and feature extraction processing are performed, and the data is input into the trained assessment model. According to the assessment indicators output by the model and the corresponding weights, the quality score of the remote meter reading data is calculated, thereby realizing data-driven assessment indicator correlation analysis, providing scientific decision support for the quality assessment of remote meter reading data, and greatly improving the accuracy and comprehensiveness of the remote meter reading data quality assessment results. 2. Since the remote meter reading system usually involves multiple data sources and different communication methods and transmission protocols, these diversities may cause the remote meter reading data to be affected by different factors during the collection, transmission and processing, thereby increasing the difficulty of remote meter reading data quality assessment. To this end, this application uses the principal component analysis method to extract features from the standardized historical remote meter reading data, which helps to integrate data from different meters into a common framework, so that data from different sources can be compared and analyzed on the same dimension, and further selects a sufficient number of principal component features by calculating the cumulative explained variance in the process to retain the information of the original data to the greatest extent, which helps to more comprehensively reflect the quality of the remote meter reading data, while ignoring those components with smaller variances, achieving data dimensionality reduction, focusing on the main variability of the data, and reducing the interference caused by different communication methods and transmission protocols to a certain extent, thereby reducing the impact of the diversity of data sources on the quality assessment results; 3. Compared with the prior art that directly uses manually preset meter data rules to evaluate data quality, this application lists all potential evaluation indicators and their calculation rules based on the principal component characteristics of historical remote meter reading data, and uses the random forest algorithm to screen the listed evaluation indicator calculation rules to determine the preferred evaluation indicators that have a greater impact on data quality evaluation, which greatly reduces manual intervention and improves the automation and intelligence level of the evaluation process. The preferred evaluation indicators are used as labels for the evaluation model, and the random forest model is trained using a data set containing principal component characteristics and preferred evaluation indicator labels, so that the evaluation model can simultaneously predict multiple evaluation indicators, thereby providing a more comprehensive data quality evaluation. The trained evaluation model can be used for subsequent real-time monitoring and early warning of remote meter reading data, so as to timely discover data quality problems, which is difficult to achieve with the prior art method based on preset rules. Therefore, this application provides a more advanced and effective solution for remote meter reading data quality evaluation; 4. By adopting the entropy weight method to weight each evaluation indicator, the objective weight of each evaluation indicator is calculated based on the discrete degree of the evaluation indicator itself to ensure the rationality of the weight distribution. At the same time, it is not limited to a specific data distribution. Therefore, it can be applied to meter reading data of different types of instruments, thereby further reducing the impact of the diversity of data sources on the quality assessment results.

[0018] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with the accompanying drawings: Figure 1 The present invention is a flowchart of a remote meter reading data quality assessment method based on an assessment model according to an embodiment of the present invention.

[0020] Figure 2 The figure is a schematic diagram of the structure of a remote meter reading data quality assessment system based on an assessment model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0022] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0023] Please see attached Figure 1 , Figure 1 A flow chart of a remote meter reading data quality assessment method based on an assessment model provided in one embodiment of the present specification is shown.

[0024] like Figure 1 As shown, the remote meter reading data quality assessment method based on the assessment model may at least include the following steps: Step 101, obtaining historical remote meter reading data.

[0025] It is understandable that remote meter reading data usually involves multiple data sources, including different types of meters such as smart electricity meters, water meters, and gas meters. Specifically, the historical remote meter reading data from different data sources in this embodiment include but are not limited to reading timestamps, meter readings, meter types, meter IDs, meter working status, communication signal strength between meters and remote reading systems, etc. Among them, since the meter ID is a unique identity code that distinguishes each meter device, the data quality evaluation of different meter devices can also reflect the stability and reliability level of the corresponding meter devices to a certain extent.

[0026] Step 102: Perform format standardization processing on the historical remote meter reading data to obtain standardized historical remote meter reading data.

[0027] Specifically, the implementation method of format standardization of historical remote meter reading data is as follows: 1. Unification of feature columns: First, list the feature columns of all data sources and identify the unique columns of each data source; Secondly, according to business requirements, define a standard feature set that includes all feature columns; Next, for each data source, establish a mapping relationship between the original data feature column and the standard data feature column, rename the original feature column to the name in the standard feature set, and ensure that each original data feature column can be correctly mapped to the standard data feature column to avoid data loss or error; Then, for standard features that are missing in certain data sources, default values ​​can be filled in, values ​​calculated, or interpolation methods used.

[0028] 2. Unified data format: First check the data type of each feature column in the standard feature set (such as integer, floating point number, string, etc.), and then convert the data type of the same feature column in all data sources to the data type in the standard format. For example, convert the data type of all feature columns representing quantities to floating point numbers.

[0029] Step 103: extracting features from the standardized historical remote meter reading data using principal component analysis to obtain principal component features of the historical remote meter reading data.

[0030] It can be understood that the principal component features of the historical remote meter reading data are features related to the quality assessment of the meter reading data. Taking the water meter as an example, the principal component features of the historical remote meter reading data in this embodiment may include meter ID, reading timestamp, collection frequency, meter reading water volume, over-range data, data fluctuation, water meter stop code data, etc.

[0031] Since the remote meter reading system usually involves multiple data sources, including different types of meters such as smart electricity meters, water meters, and gas meters, as well as different communication methods and transmission protocols, these diversities may cause the remote meter reading data to be affected by different factors during the collection, transmission and processing, thereby increasing the difficulty of remote meter reading data quality assessment. To this end, further in this embodiment, the principal component analysis method is used to extract features from the standardized historical remote meter reading data to obtain the principal component features of the historical remote meter reading data, including: Step 201, calculating the covariance matrix between various features in the historical remote meter reading data.

[0032] It can be understood that the covariance matrix describes the correlation between each feature. For a vector X = [x1,x2,…, ], the calculation formula of the covariance matrix Cov(X) can be: Where n represents the total number of features. represents the i-th eigenvalue, Indicates the mean of the currently calculated feature vector X.

[0033] Step 202, solving the eigenvalues ​​and eigenvectors of the covariance matrix.

[0034] Specifically, the eigenvalues ​​and eigenvectors are obtained by solving the characteristic equation of the covariance matrix. The characteristic equation can be expressed as: in, represents the eigenvalue and I represents the identity matrix.

[0035] Step 203: Arrange the eigenvalues ​​in descending order according to their magnitudes, and calculate the explained variance ratio corresponding to each eigenvalue.

[0036] Specifically, after the eigenvalues ​​are arranged in descending order, each eigenvalue is divided by the sum of all eigenvalues ​​to obtain the explained variance ratio corresponding to each eigenvalue.

[0037] Step 204, according to the arrangement order of the eigenvalues, the cumulative sum of the explained variance ratios corresponding to the eigenvalues ​​is calculated to form a cumulative explained variance sequence.

[0038] It can be understood that in the cumulative explained variance sequence, each cumulative explained variance is the cumulative sum of the explained variance ratios corresponding to the eigenvalues ​​from top to bottom in the order of arrangement of the eigenvalues.

[0039] Step 205, searching for the cumulative explained variance sequence, and determining the minimum number of principal components that reaches a preset threshold.

[0040] It can be understood that, among the cumulative explained variances greater than the preset threshold, the number of explained variance ratios corresponding to the eigenvalues ​​contained in the cumulative explained variance closest to the preset threshold is the minimum number of principal components.

[0041] Step 206 , according to the arrangement order of the eigenvalues, select the eigenvectors corresponding to the first N eigenvalues ​​as the principal component features of the historical remote meter reading data, where N is the minimum number of principal components.

[0042] This embodiment uses principal component analysis to extract features from standardized historical remote meter reading data, which helps to integrate data from different meters into a common framework, so that data from different sources can be compared and analyzed on the same dimension, and further selects a sufficient number of principal component features by calculating the cumulative explained variance in the process to retain the information of the original data to the greatest extent, which helps to more comprehensively reflect the quality of the remote meter reading data, while ignoring those components with smaller variances, achieving data dimensionality reduction, focusing on the main variability of the data, and reducing the interference caused by different communication methods and transmission protocols to a certain extent, thereby reducing the impact of the diversity of data sources on the quality evaluation results.

[0043] Step 104, constructing an evaluation model, using an integrated learning method to train the evaluation model based on the principal component features of historical remote meter reading data, the input of the evaluation model is the principal component features, and the output is a number of evaluation indicators.

[0044] Specifically, in this embodiment, an ensemble learning method is used to train an evaluation model based on the principal component features of historical remote meter reading data, including: Step 301, based on the principal component characteristics of the historical remote meter reading data, all evaluation indicators and evaluation indicator calculation rules are listed.

[0045] Among them, the evaluation index calculation rules define how to calculate evaluation indicators such as data completeness rate and data efficiency from the original remote meter reading data.

[0046] Step 302: A random forest algorithm is used to determine a preferred evaluation index calculation rule from all evaluation index calculation rules, and the preferred evaluation index calculation rule is used as a label of the evaluation model.

[0047] Among them, the random forest algorithm is one of the ensemble learning methods. The ensemble learning method improves the performance of the overall evaluation model by combining the prediction results of multiple basic models (such as decision trees in random forests).

[0048] Furthermore, in this embodiment, a random forest algorithm is used to determine a preferred evaluation index calculation rule from all evaluation index calculation rules, including: Step 401, taking the historical remote meter reading data as a sample, taking each evaluation index calculation rule as a feature, randomly selecting several features to form a feature subset, and forming a diversified tree structure to enhance the generalization ability of the model.

[0049] Here, a sample refers to a row in a data set, where each row represents a complete historical remote meter reading data. A feature refers to a column in a data set, and a feature subset refers to a part of the features randomly selected from all the features when building each decision tree. The feature subset is used to make a split decision at each node.

[0050] Step 402, construct a random forest model including multiple decision trees, train each decision tree with randomly selected samples and feature subsets, and minimize impurity by recursively partitioning the feature space for each tree.

[0051] Among them, in random forests, decision trees usually use impurity metrics (such as Gini impurity or information gain) to evaluate the purity of nodes. The lower the impurity, the more samples in the node belong to the same class.

[0052] Step 403, calculate the average impurity reduction of each feature on all decision trees in the random forest as a quantitative indicator of feature importance.

[0053] The impurity reduction means that for each feature, the difference between the impurity before and after the feature is used to segment the data is calculated. If a feature can significantly reduce the impurity, then its importance for building a decision tree is higher. Since there are multiple trees in a random forest, each tree may use different features for segmentation. Therefore, for each feature, it is necessary to calculate the average of its impurity reduction on all decision trees, that is, the average impurity reduction. The average impurity reduction reflects the average importance of the feature in the entire random forest.

[0054] Specifically, for each feature, the average impurity reduction is calculated by summing its impurity reduction over all trees and dividing it by the number of trees.

[0055] Step 404 , according to the importance of the features, the features are arranged in descending order, and the evaluation index calculation rules corresponding to the first several features are selected as the preferred evaluation index calculation rules.

[0056] Among them, the selection of the number of the first several features depends on the variance ratio of the indicators we want to evaluate. These features are the evaluation indicators and evaluation indicator calculation rules that have a greater impact on the meter reading data quality analysis. In data quality assessment, we can pay more attention to those factors that have a direct impact on quality changes, or we can use the "80 / 20 rule" to select features that can evaluate at least 80% of the indicators. This embodiment does not limit this.

[0057] For example, taking a water meter as an example, the preferred evaluation index and evaluation index calculation rules may include at least the following items: 1. Data completeness rate: Calculate the ratio of the actual number of collections to the number of collections that should be collected, and divide the data completeness rate values ​​according to the ratio. For example, the evaluation index value can be divided into 100 for a ratio ≥ 98%, 80 for a ratio between 95% (inclusive) and 98%, 60 for a ratio between 90% (inclusive) and 95%, 40 for a ratio between 85% (inclusive) and 90%, and 20 for a ratio less than 80%.

[0058] 2. Data efficiency: Calculate the ratio of the effective acquisition times (i.e., the actual acquisition times minus the occurrence times of over-range data) to the actual acquisition times, and divide the data efficiency values ​​according to the ratio. For example, the evaluation index values ​​can be divided into 100 for a ratio ≥ 98%, 80 for a ratio between 95% (inclusive) and 98%, 60 for a ratio between 90% (inclusive) and 95%, 40 for a ratio between 85% (inclusive) and 90%, and 20 for a ratio less than 80%.

[0059] 3. Water meter bottom zero value: Create a binary feature to indicate whether the water meter stop code data is zero, and count the cases where the water meter stop code data is zero. If the water meter stop code data does not have a zero value, the value of the evaluation index is 100, and if the water meter stop code data has a zero value, the value of the evaluation index is 0.

[0060] 4. Number of consecutive decreases in meter reading data: Count the number of times the last collected value on the day is less than the last collected value on the previous day for seven consecutive days, reflecting the continuous changes in meter reading data. Each time the evaluation indicator appears, the value is -10, that is, 10 points are deducted from the evaluation score each time it appears.

[0061] 5. Number of sudden increases in meter reading data for seven consecutive days: The number of times the average water volume in the previous seven days increased by 10% or more compared with the average daily increase in the same period last month is counted, reflecting the year-on-year changes in meter reading data.

[0062] 6. Number of times meter reading data suddenly decreases for seven consecutive days: The number of times the average water volume in the previous seven days decreases by more than or equal to 10% compared with the average daily volume in the same period last month is counted, reflecting the year-on-year changes in meter reading data.

[0063] Step 303, using the data set containing the principal component features and labels as a training set to train the evaluation model, the input of the trained evaluation model is the principal component features, and the output is a number of evaluation indicators calculated according to the preferred evaluation indicator calculation rules.

[0064] For example, taking the random forest model as the evaluation model, the implementation method of using the data set containing the principal component features and labels as the training set to train the evaluation model is as follows: First, random sampling with replacement is performed from the data set to form multiple different training subsets; then for each training subset, a decision tree is constructed, a training subset is randomly selected at each node of the tree, and the optimal split point is found in the selected training subset to maximize the information gain or other split criteria (such as Gini impurity). Since each tree is trained independently of other trees, it can be processed in parallel, predicting multiple labels at the same time, and integrating their prediction results, thereby comprehensively considering multiple evaluation indicators and reducing the limitations of a single model.

[0065] Compared with the prior art that directly uses manually preset meter data rules to evaluate data quality, this embodiment lists all potential evaluation indicators and their calculation rules based on the principal component characteristics of historical remote meter reading data, and uses the random forest algorithm to screen the listed evaluation indicator calculation rules to determine the preferred evaluation indicators that have a greater impact on data quality evaluation, thereby greatly reducing manual intervention and improving the automation and intelligence level of the evaluation process. The preferred evaluation indicators are used as labels for the evaluation model, and the random forest model is trained using a data set containing principal component characteristics and preferred evaluation indicator labels, so that the evaluation model can simultaneously predict multiple evaluation indicators, thereby providing a more comprehensive data quality evaluation, and the trained evaluation model can be used for subsequent real-time monitoring and early warning of remote meter reading data, so as to timely discover data quality problems, which is difficult to achieve with the prior art method based on preset rules. Therefore, this embodiment provides a more advanced and effective solution for remote meter reading data quality evaluation.

[0066] Step 105, assign weights to each evaluation indicator to obtain the weights of each evaluation indicator.

[0067] Specifically, in this embodiment, each evaluation indicator is weighted to obtain the weight of each evaluation indicator, including: The entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator.

[0068] Furthermore, in this embodiment, the entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator, including: Step 501, standardize the evaluation indicators to obtain the standardized values ​​of each indicator.

[0069] For example, for the evaluation indicators of data completeness and data efficiency, the minimum-maximum normalization formula can be used for normalization processing; for the evaluation indicator of the number of consecutive sudden increases / decreases in meter reading data, the Z-score normalization formula can be used for normalization processing; for the evaluation indicator of the zero value of the water meter bottom, it can be selected whether to perform normalization processing according to the situation, etc., and this embodiment does not limit this.

[0070] Step 502: Take the historical remote meter reading data as samples and calculate the proportion of the standardized value of each indicator in each sample to the sum of the standardized values ​​of the indicator in all samples. The expression can be: in, It represents the standardized value of the i-th sample on the j-th indicator, and n represents the total number of samples.

[0071] Step 503, based on the proportion, calculate the entropy value of each evaluation index to measure the information uncertainty of the index. The expression can be: in, is a constant that ensures that the entropy value is between 0 and 1.

[0072] Step 504: Determine the coefficient of difference of each evaluation indicator based on the entropy value of each evaluation indicator.

[0073] Specifically, the coefficient of difference of each evaluation indicator is 1 minus the entropy value of the corresponding evaluation indicator. The larger the coefficient of difference, the higher the importance of the evaluation indicator in the data quality evaluation system.

[0074] Step 505, based on the difference coefficient of each evaluation index, the objective weight of each evaluation index is calculated, which can be expressed as: in, It represents the relative importance of the j-th evaluation indicator in the overall evaluation, that is, the objective weight of the j-th evaluation indicator. represents the coefficient of difference of each evaluation index, and m represents the total number of evaluation indicators.

[0075] Assigned to the corresponding evaluation indicators, these weights represent the relative importance of each evaluation indicator in the meter reading data quality analysis.

[0076] This embodiment uses the entropy weight method to assign weights to each evaluation indicator, and calculates the objective weight of each evaluation indicator based on the discrete degree of the evaluation indicator itself, so as to ensure the rationality of the weight distribution. At the same time, it is not limited to a specific data distribution, so it can be applied to meter reading data of different types of instruments, thereby further reducing the impact of the diversity of data sources on the quality evaluation results.

[0077] On the other hand, in this embodiment, each evaluation indicator is weighted to obtain the weight of each evaluation indicator, which also includes: Clustering algorithms are used to cluster various evaluation indicators; Analyze the feature distribution in different clusters, identify evaluation indicators with similar features, and aggregate them into several categories of evaluation dimensions based on the evaluation indicators; Assign weights to each evaluation dimension to obtain the weights of each evaluation dimension; Based on the weights of various evaluation dimensions, the evaluation indicators in each evaluation dimension are weighted to obtain the weights of each evaluation indicator in the evaluation dimension.

[0078] It is understandable that, since the remote meter reading data may contain noise and outliers, and the evaluation indicators vary greatly, this embodiment preferably uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to cluster the evaluation indicators, which can identify clusters of any shape and can handle noise points. When using the DBSCAN clustering algorithm to cluster the evaluation indicators, it is necessary to determine the neighborhood radius and the minimum number of points, and then determine the formation of the cluster by calculating the density between the data points.

[0079] The method of analyzing the feature distribution in different clusters, identifying evaluation indicators with similar features, and aggregating them into several types of evaluation dimensions according to the evaluation indicators is as follows: First, observe the characteristic distribution of evaluation indicators of each cluster, and use the silhouette coefficient to evaluate the cohesion and separation of each cluster to judge the quality of the cluster; Secondly, the similarities within and between clusters are compared by internal and external distances to identify evaluation indicators with similar characteristics in different clusters; Finally, the clustering results are combined with business background knowledge, and the clustering results are interpreted. Then, they are aggregated into several evaluation dimensions according to the evaluation indicators, and each evaluation dimension is named. For example, data completeness, data efficiency, and whether the water meter stop code data has zero values ​​can be aggregated into the evaluation dimension of data reliability, and the number of consecutive decreases in meter reading data and the number of sudden increases / decreases in meter reading data for 7 consecutive days can be aggregated into the evaluation dimension of data stability.

[0080] This embodiment uses a clustering algorithm to understand the internal structure and characteristics of the evaluation indicators, identifies evaluation indicators with similar characteristics, and aggregates them into several types of evaluation dimensions according to the evaluation indicators. The evaluation dimensions are first weighted, and then based on the weights of each type of evaluation dimensions, the evaluation indicators in different evaluation dimensions are further differentiatedly weighted. This can optimize resource allocation and pay more attention to the dimensions and indicators that have the greatest impact on data quality. At the same time, it improves the standardization of remote meter reading data quality evaluation methods to a certain extent.

[0081] Step 106: Acquire remote meter reading data in real time.

[0082] Step 107, standardize the format of the remote meter reading data acquired in real time, and extract the required principal component features.

[0083] The implementation manner of format standardization processing of the remote meter reading data acquired in real time in step 107 is similar to the implementation manner of format standardization processing of the historical remote meter reading data in the aforementioned step 102, and will not be described in detail in this embodiment.

[0084] The “required principal component features” are the principal component features selected from the standardized historical remote meter reading data in step 103 .

[0085] Step 108, input the extracted principal component features into the trained evaluation model to obtain a number of evaluation indicators, and calculate the quality score of the remote meter reading data based on the evaluation indicators and the weights of the corresponding evaluation indicators.

[0086] Furthermore, in this embodiment, the quality score of the remote meter reading data is obtained based on the evaluation indicators and the weights of the corresponding evaluation indicators, including: The value of each evaluation index is weighted and calculated with the weight of the corresponding evaluation index to obtain the quality score of the remote meter reading data.

[0087] In summary, this embodiment first obtains historical remote meter reading data and performs format standardization processing to ensure data consistency and comparability. Secondly, the principal component analysis method is used to extract features from the standardized data to reduce data dimensions while retaining the most important principal component features for quality assessment. Then, an evaluation model is constructed and trained based on the principal component features using an ensemble learning method (i.e., a random forest algorithm) so that the evaluation model can output multiple evaluation indicators. Then, an objective weight is assigned to each evaluation indicator output by the evaluation model to reflect the importance of different evaluation indicators in data quality assessment. Finally, the evaluation model is applied in practice, that is, the latest remote meter reading data is obtained in real time, the same format standardization and feature extraction processing are performed, and the data is input into the trained evaluation model. According to the evaluation indicators output by the model and the corresponding weights, the quality score of the remote meter reading data is calculated, thereby realizing data-driven evaluation indicator correlation analysis, providing scientific decision support for the quality assessment of remote meter reading data, and greatly improving the accuracy and comprehensiveness of the remote meter reading data quality assessment results.

[0088] Please see attached Figure 2 , Figure 2 This is a schematic diagram of the structure of a remote meter reading data quality assessment system based on an assessment model provided in one embodiment of this specification.

[0089] like Figure 2 As shown, the remote meter reading data quality assessment system based on the assessment model may at least include a data acquisition module 1, a format standardization module 2, a feature extraction module 3, an assessment model training module 4, a weighting module 5 and a quality score output module 6, wherein: Data acquisition module 1, used to acquire historical remote meter reading data and real-time remote meter reading data; The format standardization module 2 is used to perform format standardization processing on the historical remote meter reading data to obtain standardized historical remote meter reading data, and to perform format standardization processing on the remote meter reading data obtained in real time; The feature extraction module 3 is used to extract features from the standardized historical remote meter reading data using the principal component analysis method to obtain the principal component features of the historical remote meter reading data, and to extract the required principal component features from the standardized real-time remote meter reading data; Evaluation model training module 4 is used to construct an evaluation model, and uses an integrated learning method to train the evaluation model based on the principal component features of historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is a number of evaluation indicators; The weighting module 5 is used to weight each evaluation indicator to obtain the weight of each evaluation indicator; The quality score output module 6 is used to input the extracted principal component features into the trained evaluation model to obtain a number of evaluation indicators, and obtain the quality score of the remote meter reading data based on the evaluation indicators and the weight calculation of the corresponding evaluation indicators.

[0090] It can be understood that the technical concept of the remote meter reading data quality assessment system based on the assessment model provided in this embodiment is similar to the technical concept of the remote meter reading data quality assessment method based on the assessment model described above, and this embodiment will not be repeated here.

[0091] Another embodiment of the present specification provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above-mentioned embodiment of the remote meter reading data quality assessment method based on the assessment model. If the component modules of the above-mentioned electronic device are implemented in the form of software functional units and predicted or used as independent downstream tasks, they can be stored in a computer-readable storage medium.

[0092] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, tapes), optical media (eg, digital versatile discs (DVD)), or semiconductor media (eg, solid state drives (SSD)).

[0093] The above description is only the preferred embodiment disclosed in this application and the description of the technical principle used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.

[0094] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A remote meter reading data quality assessment method based on an assessment model, characterized in that: The following steps are involved: Obtain historical remote meter reading data; Perform format standardization processing on historical remote meter reading data to obtain standardized historical remote meter reading data; The principal component analysis method is used to extract the features of the standardized historical remote meter reading data, and the principal component features of the historical remote meter reading data are obtained; Construct an evaluation model, and use an integrated learning method to train the evaluation model based on the principal component features of historical remote meter reading data, wherein the input of the evaluation model is the principal component features, and the output is a number of evaluation indicators; Assign weights to each evaluation indicator to obtain the weights of each evaluation indicator; Obtain remote meter reading data in real time; Standardize the format of remote meter reading data acquired in real time and extract the required principal component features; The extracted principal component features are input into the trained evaluation model to obtain several evaluation indicators, and the quality score of the remote meter reading data is calculated based on each evaluation indicator and the weight of the corresponding evaluation indicator.

2. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The principal component analysis method is used to extract the features of the standardized historical remote meter reading data, and the principal component features of the historical remote meter reading data are obtained, including: Calculate the covariance matrix between each feature in the historical remote meter reading data; Solve for the eigenvalues ​​and eigenvectors of the covariance matrix; According to the size of the eigenvalue, the eigenvalues ​​are arranged in descending order, and the explained variance ratio corresponding to each eigenvalue is calculated; According to the order of eigenvalues, the cumulative sum of the explained variance ratios corresponding to each eigenvalue is calculated to form a cumulative explained variance sequence; Find the cumulative explained variance sequence and determine the minimum number of principal components that reaches the preset threshold; According to the arrangement order of the eigenvalues, the eigenvectors corresponding to the first N eigenvalues ​​are selected as the principal component features of the historical remote meter reading data, where N is the minimum number of principal components.

3. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The ensemble learning method is used to train the evaluation model based on the principal component features of historical remote meter reading data, including: Based on the principal component characteristics of historical remote meter reading data, all evaluation indicators and evaluation indicator calculation rules are listed; A random forest algorithm is used to determine the optimal evaluation indicator calculation rule from all evaluation indicator calculation rules, and the optimal evaluation indicator calculation rule is used as the label of the evaluation model; The evaluation model is trained using a data set containing principal component features and labels as a training set. The input of the trained evaluation model is the principal component features, and the output is a number of evaluation indicators calculated according to the preferred evaluation indicator calculation rules.

4. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 3, characterized in that: The random forest algorithm is used to determine the optimal evaluation index calculation rules from all evaluation index calculation rules, including: The historical remote meter reading data is used as a sample, each evaluation index calculation rule is used as a feature, and several features are randomly selected to form a feature subset; Build a random forest model containing multiple decision trees, and train each decision tree with randomly selected samples and feature subsets; Calculate the average impurity reduction of each feature over all decision trees in the random forest as a quantitative indicator of feature importance; According to the importance of the features, the features are arranged in descending order, and the evaluation index calculation rules corresponding to the first several features are selected as the preferred evaluation index calculation rules.

5. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: Each evaluation indicator is weighted to obtain the weight of each evaluation indicator, including: The entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator.

6. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 5, characterized in that: The entropy weight method is used to assign weights to each evaluation indicator to obtain the objective weight of each evaluation indicator, including: Standardize the evaluation indicators to obtain the standardized values ​​of each indicator; Taking the historical remote meter reading data as samples, calculate the proportion of the standardized value of each indicator in each sample to the sum of the standardized values ​​of the indicator in all samples; Based on the proportion, the entropy value of each evaluation index is calculated; Based on the entropy value of each evaluation indicator, determine the difference coefficient of each evaluation indicator; Based on the coefficient of difference of each evaluation indicator, the objective weight of each evaluation indicator is calculated.

7. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: Each evaluation indicator is weighted to obtain the weight of each evaluation indicator, including: Clustering algorithms are used to cluster various evaluation indicators; Analyze the feature distribution in different clusters, identify evaluation indicators with similar features, and aggregate them into several categories of evaluation dimensions based on the evaluation indicators; Assign weights to each evaluation dimension to obtain the weights of each evaluation dimension; Based on the weights of various evaluation dimensions, the evaluation indicators in each evaluation dimension are weighted to obtain the weights of each evaluation indicator in the evaluation dimension.

8. The remote meter reading data quality assessment method based on the assessment model as claimed in claim 1, characterized in that: The quality score of remote meter reading data is calculated based on each evaluation indicator and the weight of the corresponding evaluation indicator, including: The value of each evaluation index is weighted and calculated with the weight of the corresponding evaluation index to obtain the quality score of the remote meter reading data.

9. The remote meter reading data quality assessment system based on the assessment model is characterized by: include: Data acquisition module, used to acquire historical remote meter reading data and real-time remote meter reading data; The format standardization module is used to perform format standardization processing on the historical remote meter reading data to obtain standardized historical remote meter reading data, and to perform format standardization processing on the remote meter reading data obtained in real time; A feature extraction module is used to extract features from the standardized historical remote meter reading data using a principal component analysis method to obtain the principal component features of the historical remote meter reading data, and to extract the required principal component features from the standardized real-time remote meter reading data; An evaluation model training module is used to construct an evaluation model, and an ensemble learning method is used to train the evaluation model based on the principal component features of historical remote meter reading data. The input of the evaluation model is the principal component features, and the output is a number of evaluation indicators. The weighting module is used to weight each evaluation indicator and obtain the weight of each evaluation indicator; The quality score output module is used to input the extracted principal component features into the trained evaluation model to obtain a number of evaluation indicators, and obtain the quality score of the remote meter reading data based on the evaluation indicators and the weight calculation of the corresponding evaluation indicators.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the remote meter reading data quality assessment method based on the assessment model according to any one of claims 1 to 8 is implemented.

Images