A method and system for screening data set samples
By performing multi-dimensional extraction and outlier screening on the data set, combining principal component analysis method and entropy weight method to calculate the abnormal frequency, the problem of inability to effectively handle complex abnormal data in the power system in the existing technology is solved, and high-precision Sample filtering and operation optimization.
Patent Information
- Application Number
- CN202210122374.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-09
AI Technical Summary
The prior art is too simple in data set sample screening, and cannot effectively process complex abnormal data in power systems, especially hidden abnormal data and associated outliers, and cannot achieve accurate screening of data samples.
By performing multi-dimensional extraction and outlier screening on the data set, outliers are screened using box graph outliers or 28 law methods, sorting them with principal component analysis method and entropy weight method, and the abnormal frequency is calculated based on preset standards Final sample screening.
It improves the accuracy of sample screening, optimizes the calculation amount, reveals the correlation between outliers, and meets the needs of complex data processing of power systems.
Smart Images

Figure CN114462537B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and more specifically, to a method and system for screening data set samples. Background Art
[0002] In the field of data processing, the screening of data set samples is the initial step in data modeling and subsequent processing. It is a necessary process to effectively implement data mining, simplify data processing algorithms, and greatly reduce the amount of data processing operations. It can extract key data with potential value from massive data. Therefore, it plays a vital role in data processing.
[0003] In the prior art, the screening method for data set samples is relatively simple. Generally speaking, it only includes three processes: data extraction, data cleaning, and data loading. The actual process of processing the original collected data also only includes four parts commonly used in data cleaning: missing data processing, duplicate data processing, abnormal data processing, and inconsistent data sorting. Specifically, missing data processing and inconsistent data sorting can usually only delete the original data with obvious collection anomalies, and cannot effectively process and screen the more hidden abnormal data in the data collection process.
[0004] Furthermore, the abnormal data processing method used in the prior art is also relatively simple. In the prior art, usually only the abnormal data itself is analyzed and the abnormal values are eliminated. In the power system data processing, since a data sample itself includes not only an abnormal value data, but also other collected data values closely related to the abnormal value, the simple method of eliminating or screening abnormal values in the prior art cannot meet the requirements of complex data processing in the power system.
[0005] In addition, the existing technology is not able to screen the frequency of outliers for a single data sample. In other words, although the existing technology can screen out multiple unrelated single outliers, it is impossible to obtain multiple interrelated outliers for a data sample and achieve accurate screening of samples based on the severity of the abnormality of the data sample.
[0006] In view of the above problems, the present invention provides a method and system for screening data set samples. Summary of the invention
[0007] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for screening data set samples, by extracting dimensions and screening outliers from the data set, and implementing sample screening based on the abnormal frequency of the samples.
[0008] The present invention adopts the following technical solution.
[0009] The first aspect of the present invention relates to a method for screening samples of a data set, wherein the method comprises the following steps: step 1, performing multi-dimensional extraction on the data set, and screening all samples in the data set for outliers in each dimension; step 2, sorting all samples in the data set, and obtaining a preferred data set based on the sorting results; step 3, calculating the abnormal frequency of each sample in the preferred data set, and screening the data set samples based on preset standards.
[0010] Preferably, in step 1, after the data set is extracted in multiple dimensions, the dimension N is ≥ 3; the dimension N is extracted based on the same column name of all samples in the data set.
[0011] Preferably, the outlier screening method is a box plot outlier screening method or an 80 / 20 rule method.
[0012] Preferably, in step 2, the sorting methods are principal component analysis and entropy weight method.
[0013] Preferably, the preferred data set is obtained by: selecting the first 20% of samples from the sorting results; or, selecting the last 20% of samples from the sorting results; or, selecting samples based on expert opinions.
[0014] Preferably, in step 3, the method for calculating the abnormal frequency of each sample in the preferred data set is as follows: Step 3.1, based on the abnormal values [Z 1 , Z 2 , …, Z n , …, Z N ] and the preferred data set P obtained in step 2 calculate the outlier value of the preferred data set under each dimension n [E 1 , E 2 , …, E n , …, E N ], where n = 1, 2, ..., N; Step 3.2, for all outliers E in different dimensions n Take the union and obtain the common abnormal sample set; Step 3.3, for all the abnormal values E in different dimensions n Take the intersection and obtain the key abnormal sample set.
[0015] Preferably, in step 3.3, based on the outlier frequency M specified in the preset standard, a number of dimensions equal to the specified outlier frequency M is selected, and the intersection of multiple outliers under the number of dimensions is taken to obtain a key outlier sample set.
[0016] Preferably, the method for taking the intersection of multiple outliers under the number of dimensions is to select any M dimensions from N dimensions and based on Select the outliers E for all different dimensions respectively. nTake the intersection; the samples included in the intersection are the samples in the key abnormal sample set.
[0017] Preferably, the preset standard generates a specified frequency of outliers based on the screening requirements of the data set samples.
[0018] A second aspect of the present invention relates to a device for screening data set samples, wherein the device comprises a processor, and the processor is used to implement a method for screening data set samples as described in the first aspect of the present invention.
[0019] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a method and system for screening data set samples, which can extract dimensions and screen outliers from the data set, and achieve final sample screening based on the abnormal frequency of the samples. The method of the present invention is simple, and while optimizing the amount of calculation, it provides a correlation between outliers, so that the screened samples are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A schematic diagram of the steps of a method for screening data set samples in the present invention. DETAILED DESCRIPTION
[0021] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.
[0022] Figure 1 FIG. 1 is a schematic diagram of the steps of a method for screening a data set sample in the present invention. Figure 1 As shown, the first aspect of the present invention relates to a method for screening data set samples, wherein the method includes steps 1 to 3.
[0023] Step 1: Perform multi-dimensional extraction on the data set and filter outliers for all samples in the data set in each dimension.
[0024] It is understandable that before generating the data set, the data set may be preprocessed to filter out unnecessary data content. For example, if the frequency of data collection by the relevant equipment is too high, when the data to be filtered in the method of the present invention does not require such a high frequency, the irrelevant data may be pre-filtered. This filtering method may adopt the method commonly used in the prior art.
[0025] In addition, the data set in the present invention may come from multiple different sources. These multiple different sources may include multiple different acquisition devices, or multiple different data set processing devices. Therefore, multiple sub-data tables may be obtained in advance, and the multiple sub-data tables may be merged in advance in a manner commonly used in the prior art to generate the data set in the present invention.
[0026] It should be noted that due to the influence of multiple different acquisition devices or different data set processing devices, such as processors, etc., the contents of multiple sub-data tables may not be exactly the same. For example, the data in some data tables include relatively more column names, while some include relatively fewer column names. In the process of merging data tables, the column names included in all sub-data tables can be merged, but it should be noted that for column names not included in other sub-data tables, the corresponding data samples should annotate the completed content to prevent them from being identified as abnormal items in the subsequent processing process.
[0027] In addition, the method of the present invention can be applied to the processing of various different data. In other words, the method of the present invention does not impose any restrictions on the content of the data, nor does it impose too many restrictions on the association between multiple different data samples. However, multiple data samples should include at least multiple data contents, also referred to as data dimensions in the following text, which are exactly the same.
[0028] In one embodiment of the present invention, the data set used, the content of its data sample is about the relevant basic data collected by different substations in the power system. Specifically, the data content includes the unit where the substation is located, the name of the substation, the date of commissioning of the substation, the average household distribution transformer capacity, the number of overload anomalies, the number of over-re-changes, the number of low voltage changes, critical defects, the number of power outages, the average load rate, the collection success rate and other related content. Of course, the sub-data tables generated by multiple different collection devices or processing devices may also include other content of the sample data, and these other contents may be associated with the data set in the present invention, or may not be associated with the data set in the present invention. However, in the present invention, all samples in the data set should include the multiple dimensions mentioned in the above content, such as the unit where the substation is located, the name of the substation, the date of commissioning of the substation, the average household distribution transformer capacity, the number of overload anomalies, the number of over-re-changes, the number of low voltage changes, critical defects, the number of power outages, the average load rate, and the collection success rate.
[0029] Preferably, in step 1, after the data set is extracted in multiple dimensions, the dimension N is ≥ 3; the dimension N is extracted based on the same column name of all samples in the data set.
[0030] In other words, in the present invention, a single sample in the data set may include more than N data contents, and in the data table, it may also be displayed as including more than N column names, but all samples in the data set should include the corresponding data of the N column names represented by the extracted dimension N. Moreover, these corresponding data are all valid data contents that are actually collected.
[0031] It can be seen that in the process of extracting multiple dimensions, this principle should also be followed to extract from all samples in the data set. In order to effectively achieve the purpose of sample screening in the present invention and the subsequent calculations of steps 2 and 3, the number of dimensions obtained in this step should be at least 3.
[0032] Preferably, the outlier screening method is a box plot outlier screening method or an 80 / 20 rule method.
[0033] It should be noted that, after screening out multiple dimensions, the present invention will screen outliers from all samples for each dimension. The method for screening outliers can be a box plot outlier screening method or an 80 / 20 rule method, or it can also be a commonly used outlier screening method in the prior art.
[0034] In the present invention, for example, a box plot outlier screening method is used, and this method will be used N times, so that, for each dimension in turn, it is screened whether the data value corresponding to the dimension in each data sample under the dimension is abnormal.
[0035] It should be noted that, generally speaking, the box plot outliers and the 80 / 20 rule will analyze the general trend of all samples according to the corresponding values of all samples under the current dimension, and screen out a small number of outlier data from the general trend. Of course, in the present invention, the setting of the abnormal range of data for a certain dimension can also be realized according to the nature of the extracted data itself. For example, assuming that the acquisition success rate data is used as the current dimension, the data should be as high as possible. For example, when the acquisition success rate is 100, it means that the current data must not be abnormal data. However, in a certain actual situation, for example, the total number of samples in the data set is 50, and there are 48 samples with a collection success rate of 98.5, while the collection success rate of one sample is 100, and the collection success rate of another sample is 97. In this case, when only the box plot outliers and the 80 / 20 rule are considered, the abnormal samples will be two samples with collection success rates of 100 and 97. Therefore, in this case, the upper limit, lower limit, median and other parameters of the box plot outliers should be effectively preset according to the properties of the selected dimension itself, or the 80 / 20 rule algorithm should be preset based on the properties of the dimension itself.
[0036] Through the method in step 1, the present invention can first obtain the sample outlier value [Z 1 , Z 2 , …, Z n , …, Z N ]. It should be noted that N here is the total number of selected dimensions, and Z n contains the relevant information of all samples corresponding to all outliers in the current dimension n, such as the sample number, etc. For the convenience of subsequent calculations, here Zn It is not necessary to include the outliers themselves.
[0037] Step 2: sort all samples in the data set and obtain the preferred data set based on the sorting results.
[0038] Preferably, in step 2, the sorting methods are principal component analysis and entropy weight method.
[0039] It should be noted that the sorting method in the present invention can be implemented with reference to the principal component analysis method or entropy weight method used in the prior art. Specifically, each of the dimensions selected in the present invention, or the dimensions not selected in the present invention, can be used as an evaluation index to generate a comprehensive evaluation matrix. At the same time, samples in multiple different data sets are used as principal components, so that the principal component analysis method in the prior art is used to calculate the principal component eigenvalues of multiple samples. According to the size of the principal component eigenvalue, sorting is performed to obtain the order of multiple samples.
[0040] It should be noted that, in the present invention, the entropy weight method can be used to calculate the size of the principal component eigenvalue. For example, the proportion Pik occupied by the score of the i-th sample under the k-th evaluation index can be solved first, and then the entropy Ek and weight Ak of the k-th evaluation index can be calculated to generate the eigenvalue ∑Ek·Ak of the i-th sample.
[0041] It is worth noting that the evaluation index of the principal component analysis method can overlap with multiple dimensions extracted in advance, and in step 1, when screening outliers, the proportion of the data value of each sample in the current dimension to the average value of the data values of all samples in the current dimension can be calculated simultaneously.
[0042] Compared with the prior art, in step 1 of the present invention, the proportion of data values under a certain dimension has been pre-calculated by the box plot outlier method of each dimension or the 80 / 20 rule method. Therefore, when the dimension N extracted in step 1 coincides with the evaluation index k in this step, the proportion Pik of the score of the i-th sample under the k-th evaluation index can be quickly calculated, thereby simplifying the calculation process and improving the calculation speed.
[0043] Preferably, the preferred data set is obtained by: selecting the first 20% of samples from the sorting results; or, selecting the last 20% of samples from the sorting results; or, selecting samples based on expert opinions.
[0044] In the present invention, in step 2, after sorting by the method described above, sorting and extraction can also be achieved based on the size of the principal component eigenvalue. The amount of data extracted can be the 20% with the highest eigenvalue or the 20% with the lowest eigenvalue. Of course, the method in the prior art can also be used to select samples corresponding to eigenvalues of different numbers and sizes based on expert experience. The specific selection method will not be described in detail here.
[0045] Step 3: Calculate the abnormal frequency of each sample in the preferred data set and screen the data set samples based on preset standards.
[0046] After completing steps 1 and 2, step 3 can be performed by integrating the abnormal samples of each dimension obtained in step 1 and the preferred data set obtained in step 2 for processing.
[0047] Preferably, in step 3, the method for calculating the abnormal frequency of each sample in the preferred data set is:
[0048] Step 3.1: Based on the outliers of all samples in each dimension obtained in step 1 [Z 1 , Z 2 , …, Z n , …, Z N ] and the preferred data set P obtained in step 2 calculate the outlier value of the preferred data set under each dimension n [E 1 , E 2 , …, E n , …, E n ], where n = 1, 2, ..., N; Step 3.2, for all outliers E in different dimensions n Take the union and obtain the common abnormal sample set; Step 3.3, for all the abnormal values E in different dimensions n Take the intersection and obtain the key abnormal sample set.
[0049] Specifically, the present invention can firstly perform intersection calculation on the preferred data set P according to the outliers in multiple dimensions, and obtain [E 1 , E 2 , …, E n , …, E N Then, all the outliers E n Take the union set to obtain all samples with outlier frequencies. At this time, the outlier frequency can be only 1 or greater than 1, and meet the requirements of the preset standard. The sample set obtained here can be called a common outlier sample set. After obtaining this set, the priority of processing the outlier can actually be lower.
[0050] In addition, in the present invention, multiple dimensions can be arbitrarily selected from the N dimensions to perform intersection operations, thereby obtaining a key abnormal sample set, and the processing priority of the key abnormal sample set is relatively high.
[0051] Preferably, in step 3.3, based on the outlier frequency M specified in the preset standard, a number of dimensions equal to the specified outlier frequency M is selected, and the intersection of multiple outliers under the number of dimensions is taken to obtain a key outlier sample set.
[0052] Specifically, different frequencies of outliers can be set in the preset standards. In other words, when there are anomalies in data items under multiple dimensions in a sample, it means that the sample has key anomalies.
[0053] Preferably, the method for taking the intersection of multiple outliers under the number of dimensions is to select any M dimensions from N dimensions and based on Select the outliers E for all different dimensions respectively. n Take the intersection; the samples included in the intersection are the samples in the key abnormal sample set.
[0054] Adopt this method in the present invention, need to realize The dimensions are selected in multiple different combinations, so the value of M is not easy to be too large. In one embodiment of the present invention, M=2.
[0055] Preferably, the preset standard generates a specified frequency of outliers based on the screening requirements of the data set samples.
[0056] In addition, the value of M also needs to be set according to actual screening needs. In the present invention, the value of M can be set manually or generated according to an algorithm.
[0057] In the following, a method for screening poor substation data collection is described by taking substation data collection in a power system as an example.
[0058] The original data set is shown in Table 1. The column names of the table header include serial number, unit, substation name, commissioning date, average distribution transformer capacity per household, number of overload anomalies, number of overload anomalies, number of low voltage anomalies, critical defects, number of power outages, average load rate, and collection success rate. The data set includes a total of 90 data samples.
[0059]
[0060] Table 1 Original dataset
[0061] In step 1 of the present invention, 9 dimensions are first extracted from the original data set, including commissioning date, average distribution transformer capacity per household, number of overload anomalies, number of overload anomalies, number of low voltage anomalies, critical defects, number of power outages, average load rate, and collection success rate. Then, abnormal conditions are screened out from each dimension.
[0062] Table 2 shows the outlier value Z obtained by using the box plot outlier algorithm with the commissioning date as the current dimension. 1 The corresponding sample set.
[0063]
[0064] Table 2 Outlier values Z 1 The corresponding sample set
[0065] A similar method can also be used to calculate the sample outlier value Z in other dimensions. 2 , Z 3 and Z N wait.
[0066] In step 2, principal component analysis is performed on 90 samples to obtain a comprehensive ranking, and then the 20% of the last-ranked stations are obtained to obtain the preferred data set in Table 3.
[0067]
[0068]
[0069]
[0070] Table 3 Preferred datasets
[0071] By using the above method, the worst 18 relevant samples of the substations are selected from the 90 samples to obtain the optimal data set P. The optimal data set is compared with the sample outliers [Z 1 , Z 2 , …, Z n , …, Z N ] Take the intersection and get the outlier value of the optimal data set under each dimension n [E 1 , E 2 , …, E n , …, E N ].
[0072] Table 4 shows the outlier E 1 The corresponding sample set. As shown in Table 4, in Z 1 After intersecting with P, we get E 1 .
[0073]
[0074]
[0075] Table 4 Outlier E 1 The corresponding sample set
[0076] In one embodiment of the present invention, Z 2The intersection with P is empty, Z 4 The intersection with P is empty. In addition, Z 3 , Z 5 , Z 6 , Z 7 and Z 8 and Z 9 The intersections with P are shown in Tables 5 to 10 respectively.
[0077]
[0078]
[0079] Table 5 Z 3 The intersection with P
[0080]
[0081] Table 6 Z 5 The intersection with P
[0082]
[0083]
[0084]
[0085] Table 7 Z 6 The intersection with P
[0086]
[0087]
[0088] Table 8 Z 7 The intersection with P
[0089]
[0090] Table 9 Z 8 The intersection with P
[0091]
[0092] Table 10 Z 9 The intersection with P
[0093] If the method in step 3.2 is used, for all outliers E in different dimensions n Taking the union, we can get
[0094] The common abnormal sample set in Table 11.
[0095]
[0096]
[0097]
[0098]
[0099] Table 11 Common abnormal sample set
[0100] In this embodiment, samples corresponding to the stations with a frequency of 2 or more times are selected as individuals in the key abnormal sample set, thereby obtaining the key abnormal sample set described in Table 12.
[0101]
[0102]
[0103]
[0104] Table 12 Key abnormal sample sets
[0105] When the preset frequency is 2, it can be seen that the key abnormal sample sets selected from the 18 preferred data sets P by the method of the present invention include the 1st to 3rd, 6th, 8th to 10th, and 15th to 18th samples, and the remaining 7 samples are screened out as common abnormal samples.
[0106] According to pre-set rules, the areas corresponding to samples 1-3, 6, 8-10, and 15-18 can be listed as key areas to be invested in, while the areas corresponding to the other 7 samples are ordinary investment areas.
[0107] A second aspect of the present invention relates to a device for screening data set samples, wherein the device comprises a processor, and the processor is used to implement a method for screening data set samples as described in the first aspect of the present invention.
[0108] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a method and system for screening data set samples, which can extract dimensions and screen outliers from the data set, and achieve final sample screening based on the abnormal frequency of the samples. The method of the present invention is simple, and while optimizing the amount of calculation, it provides a correlation between outliers, so that the screened samples are more accurate.
[0109] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.
Claims
1. A method for screening a data set sample, characterized in that: The method comprises the following steps: Step 1, collecting equipment data from different collection devices or different data set processing devices in different areas of the power system, wherein the equipment data includes at least the following dimensions: the unit where the area is located, the name of the area, the date of commissioning of the area, the average distribution transformer capacity per household, the number of overload abnormalities, the number of overload abnormalities, the number of low voltage abnormalities, the number of critical defects, the number of power outages, the average load rate, and the collection success rate; constructing a data set for the equipment data based on the dimensions, and performing multi-dimensional extraction on the data set based on the same column name, after extraction, the dimension n≥3; and performing outlier screening on all samples in the data set for each dimension, and obtaining the sample outlier value [Z1, Z2, ..., Z n , …, Z N ], where Zn contains the relevant information of all samples corresponding to all outliers in the current dimension n; Step 2, sorting all samples in the data set, and obtaining a preferred data set P based on the sorting result; Step 3, calculating the abnormal frequency of each sample in the preferred data set according to the sample abnormal value in step 1 and the preferred data set in step 2, and screening the data set samples based on preset standards; specifically including: Step 3.1, based on the outliers [Z1, Z2, ..., Z n , …, Z N ] and the preferred data set P obtained in step 2 calculate the outliers [E1, E2, ..., E n ,…,E N ], where n = 1, 2, ..., N; Step 3.2: For all outliers E in different dimensions n Take the union and obtain the common abnormal sample set; Step 3.3, for all outliers E in different dimensions n Take the intersection, or based on the outlier frequency M specified in the preset standard, select a number of dimensions equal to the specified outlier frequency M, and take the intersection of multiple outliers under the number of dimensions to obtain a key outlier sample set.
2. A method for screening a data set sample according to claim 1, characterized in that: The outlier screening method is a box plot outlier screening method or an 80 / 20 rule method.
3. A method for screening a data set sample according to claim 2, characterized in that: In the step 2, The sorting methods are principal component analysis method and entropy weight method.
4. A method for screening a data set sample according to claim 3, characterized in that: The preferred data set is obtained in the following manner: Select the first 20% of samples from the ranking results; Alternatively, selecting the last 20% of samples from the ranking results; Alternatively, samples may be selected based on expert opinion.
5. A method for screening a data set sample according to claim 4, characterized in that: The method for taking the intersection of multiple outliers under the number of dimensions is to select any M dimensions from N dimensions, and based on Select the outliers E for all different dimensions respectively. n Take the intersection; The samples included in the intersection are the samples in the key abnormal sample set.
6. A method for screening data set samples according to claim 5, characterized in that: The preset standard generates the specified outlier frequency based on the screening requirements of the data set samples.
7. A device for screening data set samples, characterized in that: The device comprises a processor, The processor is used to implement a data set sample screening method as described in any one of claims 1-6.
Citation Information
Patent Citations
Spacecraft system anomaly detection method based on high-dimensional space mapping
CN111274543A
