Abnormal gas usage judgment method based on gas user classification model

Abstract

The present invention relates to a method for judging abnormal gas usage based on a gas user classification model, which collects group gas user data, extracts gas usage data to construct a sample data set, removes dirty data; extracts gas usage features and normalizes the features; uses KMO test and Bartlett test to test the correlation between data variables; principal component analysis; cluster analysis; introduces user attribute data to construct user tags, and after labeling each user, uses a random forest algorithm to establish a user classification model. The random forest result is to generate multiple user classification models, which are used to analyze user gas usage anomalies after being tested. The present invention establishes a user group portrait model to solve the problem that new users without historical gas usage information cannot perform abnormal gas usage behavior analysis.

Description

Technical Field

[0001] The present invention belongs to the technical field of gas safety management, and specifically is a method for determining abnormal gas usage based on a gas user classification model. Background Art

[0002] Profiling gas users can help relevant personnel accurately analyze the behavioral characteristics of users, analyze users' abnormal gas usage behaviors in real time, realize intelligent diagnosis and monitoring of gas usage, and timely warn and remind gas companies to stop losses in time. The prior art gas user portrait is an individual user portrait, that is, a gas usage behavior portrait of the user is established based on each user's own historical gas usage information, and then the user's future gas usage behavior is analyzed. For example, my country's invention patent, patent number 202110818902.6, application date: 2021.07.20, publication date: 2021.10.19, discloses a gas anomaly detection method based on user portrait, including: preprocessing the user data of the gas user to obtain time period data; using the features extracted from the time period data to construct a user portrait of the gas user; using the user portrait of the gas user and the time period data to detect the abnormal behavior of the gas user; wherein the user data includes instantaneous flow data and gas usage data.

[0003] Disadvantages of existing technologies: 1) It has limitations: for the users to be analyzed, there must be a certain amount of historical data, and the data must be instantaneous; 2) It is unstable: if the historical data of individual users contains a certain proportion of abnormal data, then the established portrait model is inaccurate, and the resulting analysis model is even more inaccurate. Summary of the invention

[0004] The technical problem to be solved by the present invention is: for new network users without historical gas usage data, it is impossible to use user profiles to perform anomaly detection.

[0005] The present invention solves the above-mentioned problem by adopting the following technical solution:

[0006] A method for determining abnormal gas usage based on a gas user classification model is performed in the following steps:

[0007] The first step is to collect sample data: collect user data of group gas users for at least one year, extract gas usage data from the user data to construct a sample data set;

[0008] The second step is data exploration and preprocessing to obtain the purified sample data set: 1) Traverse the sample data set, make a descriptive statistics of the variables in the data, and remove some unreasonable abnormal data and obvious erroneous data by consulting; 2) Use the 3σ principle to remove some outliers; 3) Use box plot analysis to remove some outliers; 4) For the processing of time series data, the steps are: a) unify the time series data format; b) delete the duplicate time series data; c) rebuild the index; d) fill in the missing time series data; e) perform differential processing on the accumulated data to obtain the actual occurrence value of the current time series;

[0009] The third step is to extract gas usage features: extract gas usage features from the purified sample data set and normalize all the extracted features;

[0010] Step 4: Use KMO test and Bartlett test to test the correlation between data variables, and plan dimensionality reduction according to the correlation analysis results;

[0011] Step 5: Principal component analysis: S1. Calculate the correlation coefficient matrix of the normalized data; S2. Calculate the eigenvalues ​​and eigenvectors of the correlation coefficient matrix; S3. Calculate the cumulative contribution, stop when it reaches more than 80%, and select the corresponding eigenvalues ​​and corresponding eigenvectors; S4. Modify the positive and negative signs of the eigenvectors so that the sum of the components of each eigenvector is positive, which is the final eigenvector; S5. Use the final eigenvector to linearly combine the original features to obtain new features, thus completing the dimensionality reduction process;

[0012] Step 6: Use K-Means to perform cluster analysis;

[0013] Step 7: Use the random forest algorithm to establish a user classification model: extract user attribute data from the user data, use the user attribute data as the independent variable, use the clustering result of step 6 as the dependent variable, and use the random forest algorithm to establish multiple classification models;

[0014] Step 8: Use cross-validation to test the accuracy of the model;

[0015] Step 9: Analyze abnormal user data:

[0016] Each classification model corresponds to a box plot; for old users, their daily gas consumption is extracted and input into the model to obtain the abnormal judgment structure of the day; for new users, they are classified according to the obtained classification model. If the daily gas consumption of the new user exceeds the upper and lower limits of the box plot of the model, it is judged as abnormal gas consumption; if the daily gas consumption of the new user exceeds the box plot of the classification model, it is judged as abnormal gas consumption. Figure 4 If the readings are below one-third and one-quarter, it is considered abnormal; otherwise, the gas usage is normal.

[0017] Compared with the prior art, the present invention using the above scheme has the following beneficial effects:

[0018] The present invention is different from individual user portraits. It establishes a user group portrait model based on a group of original data. It can not only solve the problem that new users without historical gas usage information and old users without instantaneous data cannot perform abnormal gas usage behavior analysis, but also eliminate the phenomenon of unstable prediction caused by inaccurate historical information of individual users.

[0019] Preferably, a further technical solution of the present invention is:

[0020] Optionally, the gas consumption data includes gas consumption indicating the gas consumption on the day, date indicating the gas consumption date and user indicating the user; the user attribute data includes location indicating the user location, type indicating the user type, area indicating the user area and number indicating the number of meters.

[0021] Optionally, the extracted features include:

[0022] 1) Maximum value (max): .

[0023] 2) Mean (avg): ,

[0024] 3) Standard deviation (std): ,

[0025] 4) Median: Arrange the daily gas consumption from small to large, and the daily gas consumption in the middle position ,

[0026] 5) Total gas consumption (total): ,

[0027] 6) Skew: 1 n ∑ i = 1 n ( g i − g ¯ ) 3 [ 1 n − 1 ∑ i = 1 n ( g i − g ¯ ) 2 ] 3 2 ,

[0028] 7) Hurt: 1 n ∑ i = 1 n ( g i − g ¯ ) 4 [ 1 n ∑ i = 1 n ( g i − g ¯ ) 2 ] 2 − 3 ,

[0029] 8) Work rate (work_rate): ,

[0030] 9) Gas consumption shock (jerk): First use five-point smoothing to get a set of estimated gas consumption data , and then calculate the gas impact degree as ,

[0031] in , , Indicates daily gas consumption, Indicates the number of days with non-zero gas usage.

[0032] Optionally, multiple classification models are established in step seven as follows: user attribute data is used to establish user labels, each gas user is labeled, the area label is selected as a quantitative variable, and the user attribute data is processed using unique hot encoding; four labels in the user attribute data are used as independent variables, and the clustering results are used as dependent variables to establish multiple classification models.

[0033] Optionally, step eight of testing the model is as follows: 70% of the sample data set is selected as training data and 30% as test data, which are input into the model to verify the model, and the verification process is repeated multiple times to evaluate the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the contour coefficient diagram of the present invention, which is analyzed based on gas usage data;

[0035] Figure 2 This is a box plot of Class A users of the present invention, where the vertical axis is gas consumption, the thick solid line represents the median gas consumption of this type of users, and the thick dotted line represents the mean gas consumption of this type of users;

[0036] Figure 3 This is a box plot of Class B users of the present invention, where the vertical axis is gas consumption, the thick solid line represents the median gas consumption of this type of users, and the thick dotted line represents the mean gas consumption of this type of users;

[0037] Figure 4 This is a box plot of Class C users of the present invention, where the vertical axis is gas consumption, the thick solid line represents the median gas consumption of this type of users, and the thick dotted line represents the mean gas consumption of this type of users. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with embodiments, the purpose of which is only to provide a better understanding of the content of the present invention. Therefore, the examples given do not limit the protection scope of the present invention.

[0039] In view of the fact that the current abnormality detection methods in the gas field are relatively single, the mainstream still uses sensor monitoring, and the technology based on gas user portrait detection is still relatively rudimentary, the present invention proposes an abnormality detection method using gas user portraits. This method uses group data to construct a portrait of gas user data, which is different from the individual user portrait, and establishes a portrait model for the user group. It can not only solve the problem of abnormal gas usage behavior analysis of new users without historical gas usage information or old users without instantaneous data, but also eliminate the phenomenon of unstable prediction caused by inaccurate historical information of individual users.

[0040] See also Figures 1 to 4The method for determining abnormal gas usage based on a gas user classification model provided by the present invention specifically comprises the following steps:

[0041] The first step is sample data collection:

[0042] Collect user data of group gas users for at least one year. In order to conduct targeted gas usage monitoring, the civil and non-civilian users can be analyzed separately according to the user type, or analyzed separately according to the type of enterprise. For example, in the embodiment of the present invention, the gas usage data of a catering company in a northern city is selected, and the gas usage data of this type of users is extracted to construct a sample data set. The gas usage data includes:

[0043] Gas consumption--indicates the gas consumption on the day;

[0044] Date--indicates the date of gas use;

[0045] User--indicates user.

[0046] The second step is data exploration and preprocessing to obtain the purified sample data set:

[0047] First, perform data quality analysis to check whether there is dirty data in the original sample data set. Dirty data generally refers to data that does not meet the requirements and cannot be directly analyzed. Common types of dirty data include: 1) missing data; 2) abnormal data; 3) inconsistent data; 4) duplicate data; 5) data containing special symbols (such as #, ￥, *).

[0048] 1) Traverse the sample data set, make a descriptive statistics of the variables in the data set, and remove some unreasonable abnormal data and obviously erroneous data by consulting; 2) Use the 3σ principle to remove some outliers; 3) Use box plot analysis to remove some outlier data; 4) For the processing of time series data, the steps are: a) unify the time series data format; b) delete the duplicate time series data; c) rebuild the index; d) fill in the missing time series data; e) perform differential processing on the accumulated data to obtain the actual occurrence value of the current time series.

[0049] By testing the data quality of the above sample data sets, drawing charts, calculating certain characteristic quantities, etc., the structure and rules of the sample data sets can be analyzed, which will help to select appropriate data preprocessing and modeling methods, and even solve some problems that are usually solved by data mining.

[0050] The abnormal value analysis method of the present invention is:

[0051] 1) Simple statistical analysis: Perform a descriptive statistical analysis on the variables to see which data is unreasonable. The most commonly used statistics are the maximum and minimum values, which are used to determine whether the value of the variable exceeds a reasonable range, such as the user's gas consumption is -100.

[0052] 2) 3σ principle: If the data follows a normal distribution, under the 3σ principle, an outlier is defined as a value in a set of measured values ​​that deviates from the mean by more than 3 times the standard deviation. If the data does not follow a normal distribution, it can also be described by n times the standard deviation away from the mean.

[0053] 3) Box plot analysis: The box plot is drawn based on actual data and does not impose any restrictive requirements on the data (such as obeying a certain distribution form). It only truly and intuitively shows the true appearance of the data distribution. On the other hand, the standard for judging outliers in box plots is based on quartiles and interquartile ranges. Quartiles have a certain degree of robustness. Up to 25% of the data can become arbitrarily far away without greatly disturbing the quartiles, so outliers cannot affect this standard. It can be seen that the results of box plots in identifying outliers are relatively objective and have certain advantages in identifying outliers.

[0054] The third step is to extract gas usage characteristics:

[0055] Gas usage feature extraction is performed on the purified sample data set, including:

[0056] 1) Maximum value (max):

[0057] 2) Mean (avg):

[0058] 3) Standard deviation (std):

[0059] 4) Median: Arrange the daily gas consumption from small to large, and the daily gas consumption in the middle position

[0060] 5) Total gas consumption (total):

[0061] 6) Skew: 1 n ∑ i = 1 n ( g i − g ¯ ) 3 [ 1 n − 1 ∑ i = 1 n ( g i − g ¯ ) 2 ] 3 2

[0062] 7) Hurt: 1 n ∑ i = 1 n ( g i − g ¯ ) 4 [ 1 n ∑ i = 1 n ( g i − g ¯ ) 2 ] 2 − 3

[0063] 8) Work rate (work_rate):

[0064] 9) Gas consumption shock (jerk): First use five-point smoothing to get a set of estimated gas consumption data , and then calculate the gas impact degree as

[0065] in , , Indicates daily gas consumption, Indicates the number of days with non-zero gas consumption, and all the features extracted above are normalized.

[0066] The fourth step is to use the KMO test and Bartlett test to test the correlation between data variables, and plan the dimensionality reduction processing based on the correlation analysis results.

[0067] KMO test: KMO statistic takes values ​​between 0 and 1. When the sum of squares of simple correlation coefficients among all variables is much larger than the sum of squares of partial correlation coefficients, the KMO value is close to 1. The closer the KMO value is to 1, the stronger the correlation among variables is, and the more suitable the original variables are for factor analysis. When the sum of squares of simple correlation coefficients among all variables is close to 0, the KMO value is close to 0. The closer the KMO value is to 0, the weaker the correlation among variables is, and the less suitable the original variables are for factor analysis.

[0068] Bartlett test: The statistic of Bartlett's sphericity test is obtained based on the determinant of the correlation coefficient matrix. If the value is large and its corresponding associated probability value is less than the significance level in the user's mind, then the null hypothesis should be rejected, and it is believed that the correlation coefficient matrix cannot be a unit matrix, that is, there is a correlation between the original variables, which is suitable for principal component analysis; on the contrary, if the statistic is relatively small and its corresponding correlation probability is greater than the significance level, the null hypothesis cannot be rejected, and it is believed that the correlation coefficient matrix may be a unit matrix, which is not suitable for factor analysis.

[0069] In the present invention, the purified sample data set was analyzed, and the KMO value was 0.7020, and the p value obtained by the Bartlett test was less than 0.05, both indicating that there was a strong correlation between the variables, so factor analysis could be performed.

[0070] The fifth step is principal component analysis (PCA). The purpose of this step is not only to reduce the dimensionality of high-dimensional data, but more importantly, to remove the noise in the data through dimensionality reduction.

[0071] The specific steps are as follows:

[0072] S1. Calculate the correlation coefficient matrix of the normalized data;

[0073] S2, calculate the eigenvalues ​​and eigenvectors of the correlation coefficient matrix;

[0074] S3, calculate the cumulative contribution, stop when it reaches more than 80%, and select the corresponding eigenvalue and the corresponding eigenvector;

[0075] S4, modify the sign of the eigenvector so that the sum of the components of each eigenvector is positive, which is the final eigenvector;

[0076] S5. Use the final feature vector to linearly combine the original features to obtain new features, thus completing the dimensionality reduction process.

[0077] Step 6: Use K-Means for cluster analysis:

[0078] First, profile analysis is performed based on gas consumption data, combined with Figure 1 The contour curve graph shown is used to determine the number of clusters. In the embodiment shown in this figure, the best effect is achieved when the number of clusters is 3, so the K value is 3, and then K-Means clustering is performed based on K=3.

[0079] Step 7: Use the random forest algorithm to build a user classification model.

[0080] Extract user attribute data from user data and create user tags. Tags include

[0081] location indicates the user's location.

[0082] type indicates the user type,

[0083] Area represents the user area.

[0084] number indicates the number of meters.

[0085] Label each user. Since the label area is a quantitative variable, use One-Hot Encoding to process the user attribute data.

[0086] The four labels in the user attributes are used as independent variables, and the clustering results in the sixth step are used as dependent variables to build a classification model using the random forest algorithm. This step obtains three classification models based on the sample data set, and subdivides the catering companies in the sample data into three types of users.

[0087] Step 8: Use cross validation to test the accuracy of the model:

[0088] In order to test the accuracy of the classification model, the present invention selects a cross-validation algorithm, that is, 70% of the purified sample data set is selected as training data and 30% as test data, and each classification model obtained in step seven is input for verification. The process is repeated many times to evaluate the performance of the model. The final result is: the accuracy reaches 86.67%, indicating that the model has good performance.

[0089] Step 9: Analyze abnormal user data:

[0090] See also Figures 2 to 4 The present invention obtains the box-whisker plots corresponding to the classification models obtained by the gas consumption data of all catering companies in a northern city, and each classification model corresponds to a box-whisker plot; for old users, their daily gas consumption is extracted and input into the model to obtain the abnormal judgment structure of the day; for new users, they are classified according to the obtained classification model. If the daily gas consumption of the new user exceeds the upper and lower limits of the box-whisker plot of this type of model, it is judged as gas consumption abnormality; if the daily gas consumption of the new user exceeds the box-whisker plot of the classification model, it is judged as gas consumption abnormality. Figure 4 If the readings are below one-third and one-quarter, it is considered abnormal; otherwise, the gas usage is normal.

[0091] The above description is only a preferred feasible embodiment of the present invention, and does not limit the scope of rights of the present invention. All equivalent changes made using the contents of the present specification and its drawings are included in the scope of rights of the present invention.

Claims

1. A method for determining abnormal gas usage based on a gas user classification model, characterized in that: Follow these steps: The first step is to collect sample data: collect user data of group gas users for at least one year, extract gas usage data from the user data to construct a sample data set; The second step is data exploration and preprocessing to obtain the purified sample data set: 1) Traverse the sample data set, make a descriptive statistics of the variables in the data, and remove some unreasonable abnormal data and obvious erroneous data by consulting; 2) Use the 3σ principle to remove some outliers; 3) Use box plot analysis to remove some outlier data; 4) For the processing of time series data, the steps are: a) unify the time series data format; b) delete the duplicate time series data; c) rebuild the index; d) fill in the missing time series data; e) Perform differential processing on the accumulated data to obtain the actual value of a single day; The third step is to extract gas usage features: extract gas usage features from the purified sample data set and normalize all the extracted features; Step 4: Use KMO test and Bartlett test to test the correlation between data variables, and plan dimensionality reduction according to the correlation analysis results; Step 5: Principal component analysis: S1. Calculate the correlation coefficient matrix of the normalized data; S2. Calculate the eigenvalues ​​and eigenvectors of the correlation coefficient matrix; S3. Calculate the cumulative contribution, stop when it reaches more than 80%, and select the corresponding eigenvalues ​​and corresponding eigenvectors; S4. Modify the positive and negative signs of the eigenvectors so that the sum of the components of each eigenvector is positive, which is the final eigenvector; S5. Use the final eigenvector to linearly combine the original features to obtain new features, thus completing the dimensionality reduction process; Step 6: Use K-Means to perform cluster analysis; Step 7: Use the random forest algorithm to build a user classification model: extract user attribute data from user data, use the user attribute data as the independent variable, and use the clustering results in step 6 as the dependent variable to build multiple classification models; Step 8: Use cross-validation to test the accuracy of the model; Step 9: Analyze abnormal user data: Each classification model corresponds to a box plot; for old users, extract their daily gas consumption and input them into the model to obtain the abnormal judgment structure of the day; for new users, classify them according to the obtained classification model. If the daily gas consumption of the new user exceeds the upper and lower limits of the box plot of the classification model, it is judged as abnormal gas consumption; if the daily gas consumption of the new user exceeds the one-quarter and three-quarters digits of the box plot of the classification model, it is judged as abnormal; the rest are normal gas consumption.

2. The method for determining abnormal gas usage based on a gas user classification model according to claim 1, characterized in that: The gas consumption data includes the gas consumption of the day; the gas consumption date and the user; the user attribute data includes location indicating the user location, type indicating the user type, area indicating the user area and number indicating the number of meters.

3. The method for determining abnormal gas usage based on a gas user classification model according to claim 1, characterized in that: The extracted features include: 1) Maximum value (max): , 2) Mean (avg): , 3) Standard deviation (std): , 4) Median: Arrange the daily gas consumption from small to large, and the daily gas consumption in the middle position , 5) Total gas consumption (total): , 6) Skew: , 7) Hurt: , 8) Work rate (work_rate): , 9) Gas consumption shock (jerk): First use five-point smoothing to get a set of estimated gas consumption data , and then calculate the gas impact degree as , in , , Indicates daily gas consumption, Indicates the number of days with non-zero gas usage.

4. The method for determining abnormal gas usage based on a gas user classification model according to claim 2 is characterized in that: Use user attribute data to establish user labels, label each gas user, select the area label as a quantitative variable, and use one-hot encoding to process user attribute data; Taking the four labels in the user attribute data as independent variables and the clustering results as dependent variables, multiple classification models are constructed.

5. The method for determining abnormal gas usage based on a gas user classification model according to claim 1 is characterized in that: In step eight, the accuracy of the model is verified by selecting 70% of the purified sample data set as training data and 30% as test data, inputting them into the model to verify the model, and repeating the verification process multiple times to evaluate the performance of the model.

Images