Abnormal value analysis and missing value supplement system for station data

Abstract

The invention relates to the technical field of station data processing, in particular to a station data abnormal value analysis and missing value supplementation system, which comprises an acquisition unit, an analysis unit, a supplementation unit, a storage unit, a verification unit and a problem unit, and is characterized in that the acquisition unit is used for recording station data and carrying out primary processing on the data; the analysis unit is used for carrying out analysis and identification processing on the data processed by the acquisition unit and carrying out deletion processing on the analyzed and identified data, the analysis unit is established based on an improved fruit fly algorithm, and the analysis unit comprises the following specific steps: S1-1, data preprocessing: preprocessing the data through data cleaning, data standardization and data normalization technologies; the analysis unit can easily analyze data abnormal values by improving the fruit fly algorithm, and meanwhile, the analysis unit is high in search capability and high in convergence speed, and can quickly process a large amount of data, so that the data processing cost is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of station data processing, and in particular to a station data abnormal value analysis and missing value supplementation system. Background Art

[0002] Station data refers to various real-time data about station operations collected by sensors, meters, cameras and other equipment. These data cover all aspects of the station, including the operating status of equipment, personnel scheduling, work processes, etc., and are an important basis for station management and optimization.

[0003] For example, a method for identifying and correcting bad data in a new energy station based on deep learning, with application number CN202210230736.2 and publication date 20220701, includes: obtaining historical operating data of the identification object in the new energy station, and marking historical normal data and historical bad data in the historical operating data; establishing an identification model, and performing deep learning training on the identification model based on the historical normal data; establishing a correction model, inputting historical bad data into the correction model and the trained identification model, and performing deep learning training on the correction model in combination with the output of the identification model; obtaining real-time operating data of the identification object, and distinguishing real-time normal data and real-time bad data in the real-time operating data by inputting the real-time operating data into the trained identification model; inputting the real-time bad data into the trained correction model to obtain a correction value of the real-time bad data. The present invention can significantly improve the efficiency of bad data identification and correction, and ensure the real-time, safe and stable operation of new energy power stations.

[0004] The amount of various types of data at the station is huge, and the relationships between the data are numerous, so data processing is rather troublesome. Although there are some ways on the market to process data outliers and missing values, the processing is difficult. At the same time, because there is a lot of data to be processed, the processing process is time-consuming, which increases the cost of data processing. Therefore, it is urgent to design a station data outlier analysis and missing value supplement system to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a site data outlier analysis and missing value supplementation system to solve the above-mentioned deficiencies in the prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The station data outlier analysis and missing value supplement system includes an acquisition unit, an analysis unit, a supplement unit, a storage unit, a verification unit and a problem unit.

[0008] The acquisition unit is used to collect station data and perform preliminary processing on the data. The acquisition unit includes an interface module and a noise reduction conversion module. The interface module is a sensor interface. The interface module facilitates the connection between the compensation system and the sensor set in the station. The noise reduction conversion module performs noise reduction processing on the data received by the interface module through an adaptive filtering method, and the noise reduction conversion module also transforms the noise-reduced data through data transformation technology.

[0009] The analysis unit is used to analyze and identify the data processed by the acquisition unit, and delete the analyzed and identified data. The analysis unit is established based on the improved fruit fly algorithm. The specific steps of the analysis unit are as follows:

[0010] S1-1. Data preprocessing: Data is preprocessed through data cleaning, data standardization and data normalization techniques.

[0011] S1-2. Data clustering analysis: The data is clustered using the fruit fly algorithm, and similar data points are grouped together based on the analysis results.

[0012] S1-3. Data anomaly detection: set a threshold, then calculate the distance from each data point to its nearest cluster center, and the data points exceeding the threshold are considered outliers;

[0013] The improved fruit fly algorithm formula used by the analysis unit is as follows:

[0014]

[0015] Among them, D i The distance between the ith data point and the cluster center, Y i is the XY axis coordinate of the ith data point, and X i , Y i The calculation formula is as follows:

[0016]

[0017] In the formula, randomvariables is a random variable, X _axis , Y _axis is the initial position of the data;

[0018] The analysis steps of the analysis unit fruit fly algorithm are as follows:

[0019] S2-1. Initialization: Randomly select the initial cluster center;

[0020] S2-2. Density peak clustering: Calculate the neighborhood density of each data point within a certain radius, and then select the cluster center based on the density peak principle;

[0021] S2-3. Construct a distance matrix: For each data point, calculate the distance between it and all cluster centers, and use this to construct a distance matrix for induction.

[0022] The compensation unit is used to supplement the data analyzed by the analysis unit. The compensation unit includes a comparison module and a supplementation module. The comparison module is used to compare the data analyzed and processed by the analysis unit with the previous related data stored in the storage unit. The comparison formula is as follows:

[0023]

[0024] Among them, S is the data analyzed and processed by the analysis unit, and G is the previous related data stored by the storage unit;

[0025] The supplementation module supplements the data according to the comparison result of the comparison module, and the supplementation process is as follows:

[0026] S3-1. The comparison module comparison formula is established, the supplement module searches the data in G through the search engine, and calls out the data different from S to supplement S;

[0027] S3-2. If the comparison formula is not valid, re-acquire data through the acquisition unit for calculation.

[0028] The verification unit constructs a simulation model through digital twin technology, and the specific steps of constructing the simulation model are as follows:

[0029] S4-1. Data collection: collect various data of the physical system through sensors, monitoring equipment, etc., including parameters such as temperature, humidity, pressure, speed, etc.;

[0030] S4-1. Data preprocessing: Clean, integrate and process the collected data to ensure the accuracy and completeness of the data;

[0031] S4-2. Establish a digital twin model: Based on the collected and processed data, use modeling tools to establish a digital twin model to ensure that the model matches the actual object well;

[0032] S4-3. Model verification and optimization: Evaluate the accuracy and reliability of the model by comparing and verifying it with the actual object, and adjust parameters and improve the algorithm based on the verification results to improve the predictive ability of the model;

[0033] S4-4. Real-time update and application: Ensure that the digital twin model can obtain and process the latest data in a timely manner, reflect the latest status of the actual object, and apply it to the actual scenario.

[0034] The verification unit is constructed based on the digital twin technology. The verification unit is used to verify the data compensated by the compensation unit. The verification steps of the verification unit are as follows:

[0035] S5-1. Data input: The data processed by the compensation unit is input into the corresponding model of the simulation model;

[0036] S5-2. Run: Run the model and determine whether the data is normal based on the model's actions.

[0037] The problem unit automatically generates a solution based on the verification result of the verification unit. The problem unit includes a search module and a generation module. The search module can search the Internet and data in the storage unit through keyword search technology. The generation module summarizes the data retrieved by the search module through a text summary algorithm and generates a solution text.

[0038] The storage unit is used to store log data, processing results and previous station operation data of the acquisition unit, analysis unit, supplementation unit, verification unit and problem unit during operation.

[0039] In the above technical solution, the station data outlier analysis and missing value supplementation system provided by the present invention has the following beneficial effects:

[0040] (1) The analysis unit in the present invention can easily analyze data outliers by improving the fruit fly algorithm. At the same time, the analysis unit has strong search capability and fast convergence speed, and can quickly process a large amount of data, greatly reducing the cost of data processing.

[0041] (2) The compensation unit designed in the present invention supplements the missing values ​​after the data outlier processing by comparing with the previous data. Compared with the original mean filling method and other methods, the missing values ​​compensated by the compensation unit are more in line with reality and can ensure the correctness of the station operation. At the same time, this method has a faster speed for supplementing missing values ​​and is simpler to operate. It will not occupy much computing power of the system and can reduce the probability of system operation jams. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0043] Figure 1 A schematic diagram of the system structure provided for an embodiment of the site data outlier analysis and missing value supplementation system of the present invention.

[0044] Figure 2 A schematic diagram of the compensation unit structure provided for the station data outlier analysis and missing value supplementation system embodiment of the present invention.

[0045] Figure 3 A schematic diagram of the problem unit structure provided for an embodiment of the site data outlier analysis and missing value supplementation system of the present invention.

[0046] Figure 4 A schematic diagram of the acquisition unit structure provided for an embodiment of the site data outlier analysis and missing value supplementation system of the present invention. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0048] like Figure 1-4 As shown, the station data abnormal value analysis and missing value supplementation system provided by the embodiment of the present invention includes an acquisition unit, an analysis unit, a supplementation unit, a storage unit, a verification unit and a question unit.

[0049] The acquisition unit is used to collect station data and perform preliminary processing on the data. The acquisition unit includes an interface module and a noise reduction conversion module. The interface module is a sensor interface. The interface module facilitates the connection between the compensation system and the sensor set in the station. The noise reduction conversion module performs noise reduction processing on the data received by the interface module through an adaptive filtering method, and the noise reduction conversion module also transforms the noise-reduced data through a data transformation technology;

[0050] It should be noted that adaptive filtering: automatically adjusts filter parameters according to the statistical characteristics of the data to adapt to different data noise conditions;

[0051] Data transformation: By performing power, logarithmic, exponential and other transformations on the data, the data distribution is made closer to the normal distribution and the impact of noise is reduced.

[0052] The analysis unit is used to analyze and identify the data processed by the acquisition unit, and delete the analyzed and identified data. The analysis unit is established based on the improved fruit fly algorithm. The specific steps of the analysis unit are as follows:

[0053] S1-1. Data preprocessing: Data is preprocessed through data cleaning, data standardization and data normalization techniques.

[0054] S1-2. Data clustering analysis: The data is clustered using the fruit fly algorithm, and similar data points are grouped together based on the analysis results.

[0055] S1-3. Data anomaly detection: set a threshold, then calculate the distance from each data point to its nearest cluster center, and the data points exceeding the threshold are considered outliers;

[0056] The improved fruit fly algorithm formula used by the analysis unit is as follows:

[0057]

[0058] Among them, D i The distance between the ith data point and the cluster center, Y i is the XY axis coordinate of the ith data point, and X i , Y i The calculation formula is as follows:

[0059]

[0060] In the formula, randomvariables is a random variable, X _axis , Y _axis is the initial position of the data;

[0061] The analysis steps of the analysis unit fruit fly algorithm are as follows:

[0062] S2-1. Initialization: Randomly select the initial cluster center;

[0063] S2-2. Density peak clustering: Calculate the neighborhood density of each data point within a certain radius, and then select the cluster center based on the density peak principle;

[0064] S2-3. Construct a distance matrix: For each data point, calculate the distance between it and all cluster centers, and use this to construct a distance matrix for induction.

[0065] The compensation unit is used to supplement the data analyzed by the analysis unit. The compensation unit includes a comparison module and a supplementation module. The comparison module is used to compare the data analyzed and processed by the analysis unit with the previous related data stored in the storage unit. The comparison formula is as follows:

[0066]

[0067] Among them, S is the data analyzed and processed by the analysis unit, and G is the previous related data stored by the storage unit;

[0068] The supplementation module supplements the data according to the comparison result of the comparison module, and the supplementation process is as follows:

[0069] S3-1. The comparison module comparison formula is established, the supplement module searches the data in G through the search engine, and calls out the data different from S to supplement S;

[0070] S3-2. If the comparison formula is not valid, re-acquire data through the acquisition unit for calculation.

[0071] The verification unit constructs a simulation model through digital twin technology, and the specific steps of constructing the simulation model are as follows:

[0072] S4-1. Data collection: collect various data of the physical system through sensors, monitoring equipment, etc., including parameters such as temperature, humidity, pressure, speed, etc.;

[0073] S4-1. Data preprocessing: Clean, integrate and process the collected data to ensure the accuracy and completeness of the data;

[0074] S4-2. Establish a digital twin model: Based on the collected and processed data, use modeling tools to establish a digital twin model to ensure that the model matches the actual object well;

[0075] S4-3. Model verification and optimization: Evaluate the accuracy and reliability of the model by comparing and verifying it with the actual object, and adjust parameters and improve the algorithm based on the verification results to improve the predictive ability of the model;

[0076] S4-4. Real-time update and application: Ensure that the digital twin model can obtain and process the latest data in a timely manner, reflect the latest status of the actual object, and apply it to the actual scenario.

[0077] The verification unit is constructed based on the digital twin technology. The verification unit is used to verify the data compensated by the compensation unit. The verification steps of the verification unit are as follows:

[0078] S5-1. Data input: The data processed by the compensation unit is input into the corresponding model of the simulation model;

[0079] S5-2. Run: Run the model and determine whether the data is normal based on the model's actions.

[0080] The problem unit automatically generates a solution according to the result verified by the verification unit. The problem unit includes a search module and a generation module. The search module can search the Internet and data in the storage unit through keyword search technology. The generation module summarizes the data retrieved by the search module through a copywriting summary algorithm and generates a solution copywriting.

[0081] It should be noted that the copywriting summary algorithm is a technology that analyzes, modifies and optimizes copywriting content to improve its attractiveness, expressiveness and effectiveness. It includes five steps: vocabulary replacement and modification, sentence structure adjustment, punctuation and paragraph optimization, keyword emphasis and repetition, A / B testing and data-driven optimization.

[0082] The storage unit is used to store log data, processing results and previous station operation data of the acquisition unit, analysis unit, supplementation unit, verification unit and problem unit during operation;

[0083] It should be noted that the storage unit may select a cloud database.

[0084] Working principle: The acquisition unit is used to collect station data and perform preliminary processing on the data. In this process, the acquisition unit first receives the data and then converts the data. Then the analysis unit analyzes and identifies the data processed by the acquisition unit and deletes the abnormal values ​​of the data identified in the analysis. After that, the compensation unit supplements the data analyzed by the analysis unit. The verification unit built based on digital twin technology is used to verify the data compensated by the compensation unit. According to the verification results, the problem unit automatically generates a solution, which is then transmitted to the station workers so that the workers can repair the station in time.

[0085] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. The station data outlier analysis and missing value supplement system is characterized by: It includes acquisition unit, analysis unit, supplement unit, storage unit, verification unit and question unit. The acquisition unit is used to collect the station data and perform preliminary processing on the data; The analysis unit is used to analyze and identify the data processed by the acquisition unit, and delete the analyzed and identified data. The analysis unit is established based on the improved fruit fly algorithm. The specific steps of the analysis unit are as follows: S1-1. Data preprocessing: Data is preprocessed through data cleaning, data standardization and data normalization techniques. S1-2. Data clustering analysis: The data is clustered using the fruit fly algorithm, and similar data points are grouped together based on the analysis results. S1-3. Data anomaly detection: set a threshold, then calculate the distance from each data point to its nearest cluster center, and the data points exceeding the threshold are considered outliers; The compensation unit is used to perform supplementary processing on the data analyzed by the analysis unit; The verification unit is constructed based on the digital twin technology, and is used to verify the data compensated by the compensation unit; The problem unit automatically generates a solution according to the verification result of the verification unit; The storage unit is used to store log data, processing results and previous station operation data of the acquisition unit, analysis unit, supplementation unit, verification unit and problem unit during operation.

2. The station data outlier analysis and missing value supplementation system according to claim 1 is characterized in that: The acquisition unit includes an interface module and a noise reduction conversion module. The interface module is a sensor interface, and the interface module facilitates the connection between the compensation system and the sensor arranged in the station.

3. The station data outlier analysis and missing value supplementation system according to claim 2 is characterized in that: The noise reduction conversion module performs noise reduction processing on the data received by the interface module through an adaptive filtering method, and the noise reduction conversion module also transforms the noise-reduced data through a data transformation technology.

4. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The improved fruit fly algorithm formula used by the analysis unit is as follows: Among them, D i The distance between the ith data point and the cluster center, Y i is the XY axis coordinate of the ith data point, and X i , Y i The calculation formula is as follows: In the formula, randomvariables is a random variable, X _axis , Y _axis The initial location of the data.

5. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The analysis steps of the analysis unit fruit fly algorithm are as follows: S2-1. Initialization: Randomly select the initial cluster center; S2-2. Density peak clustering: Calculate the neighborhood density of each data point within a certain radius, and then select the cluster center based on the density peak principle; S2-3. Construct a distance matrix: For each data point, calculate the distance between it and all cluster centers, and use this to construct a distance matrix for induction.

6. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The compensation unit includes a comparison module and a supplementation module. The comparison module is used to compare the data analyzed and processed by the analysis unit with the previous related data stored in the storage unit, and the comparison formula is as follows: Among them, S is the data analyzed and processed by the analysis unit, and G is the previous related data stored by the storage unit.

7. The site data outlier analysis and missing value supplementation system according to claim 6, characterized in that: The supplementation module supplements the data according to the comparison result of the comparison module, and the supplementation process is as follows: S3-1. The comparison module comparison formula is established, and the supplement module searches for data in G through the search engine, and calls out the data that is different from S and supplements it into S; S3-2. If the comparison formula is not valid, re-acquire data through the acquisition unit for calculation.

8. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The verification unit constructs a simulation model through digital twin technology, and the specific steps of constructing the simulation model are as follows: S4-1. Data collection: collect various data of the physical system through sensors, monitoring equipment, etc., including parameters such as temperature, humidity, pressure, speed, etc.; S4-1. Data preprocessing: Clean, integrate and process the collected data to ensure the accuracy and completeness of the data; S4-2. Establish a digital twin model: Based on the collected and processed data, use modeling tools to establish a digital twin model to ensure that the model matches the actual object well; S4-3. Model verification and optimization: Evaluate the accuracy and reliability of the model by comparing and verifying it with the actual object, and adjust parameters and improve the algorithm based on the verification results to improve the predictive ability of the model; S4-4. Real-time update and application: Ensure that the digital twin model can obtain and process the latest data in a timely manner, reflect the latest status of the actual object, and apply it to the actual scenario.

9. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The verification steps of the verification unit are as follows: S5-1. Data input: The data processed by the compensation unit is input into the corresponding model of the simulation model; S5-2. Run: Run the model and determine whether the data is normal based on the model's actions.

10. The site data outlier analysis and missing value supplementation system according to claim 1, characterized in that: The problem unit includes a search module and a generation module. The search module can search the Internet and data in the storage unit through keyword search technology. The generation module summarizes the data retrieved by the search module through a copy summary algorithm and generates a solution copy.

Images