Data cleaning and filling method and system in photovoltaic power generation scene

Abstract

The invention relates to a data cleaning and filling method and system in a photovoltaic power generation scene, and the method comprises the steps: obtaining a real-time or historical data set generated by a photovoltaic panel, carrying out the data cleaning, recognizing and processing a missing value and an abnormal value in the data, and marking the missing value and the abnormal value; generating an overview of corresponding missing and abnormal conditions for a user to compare and analyze, and quickly positioning an area needing to be filled and repaired; according to missing values and abnormal values existing in the cleaned data, missing data filling and abnormal data correction results adopting different interpolation algorithms are visually output, and the filling similarity between the missing data filling and abnormal data correction results and historical similar data is calculated for a user to judge and select the most suitable interpolation algorithm; missing data filling and abnormal data correction results are visually displayed, and the whole data condition and the processed data state are visually displayed to a user, so that the user can manually adjust part of abnormal values through an interactive interface. According to the invention, the accuracy and flexibility of photovoltaic panel current data processing can be significantly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of solar power generation, and in particular to a data cleaning and filling method and system in a photovoltaic power generation scenario. Background Art

[0002] Photovoltaic panels are key components in solar power generation systems, and their performance and power generation efficiency have a crucial impact on the output of the entire system. With the large-scale application of photovoltaic power generation systems, real-time monitoring of the current data generated by photovoltaic panels has become particularly important. However, in actual operation, the current data of photovoltaic panels is often interfered by factors such as external environmental conditions, sensor failures, and communication noise, resulting in outliers, missing values, and unreasonable extreme data in the data.

[0003] Traditional data processing methods usually use simple statistical methods or empirical rules to clean and complete data when dealing with outliers in photovoltaic current data. Although these methods can improve data quality to a certain extent, due to the complexity and time series characteristics of photovoltaic current data, simple methods are difficult to effectively capture abnormal fluctuations in the data, and for different types of outliers, a single interpolation or completion method often cannot meet the accuracy and adaptability requirements.

[0004] In addition, most current data processing tools lack user interactivity, and it is difficult for users to intuitively understand the impact of different interpolation algorithms on the outlier filling effect through visualization. Most existing methods are difficult to provide users with the function of freely selecting interpolation algorithms, and users lack the ability to interact and adjust when faced with different filling effects. This leads to low efficiency in PV panel data cleaning and repair, and the repair results are difficult to flexibly adjust, which further affects the fault diagnosis, performance analysis and prediction accuracy of the PV system.

[0005] Therefore, there is an urgent need for a data cleaning and filling system that integrates multiple algorithms, can perform efficient data cleaning according to various anomaly types, and provide users with a variety of interpolation algorithm options through a visual interface, display the repair effect in real time, and support manual adjustment of abnormal data to ensure that the cleaned and repaired data is more accurate and controllable, providing a higher quality data foundation for subsequent analysis, prediction and decision-making of photovoltaic panel current data. Summary of the invention

[0006] The purpose of the present invention is to provide a data cleaning and filling method and system in a photovoltaic power generation scenario, so as to overcome the limitations of existing photovoltaic current data cleaning and interpolation algorithms in terms of insufficient accuracy in abnormal data identification, single interpolation method selection, poor flexibility, etc., as well as the problem that it is difficult for users to intuitively understand the data repair effect and manually adjust it, so as to meet the requirements of photovoltaic systems in terms of data processing accuracy, flexibility and user interactivity.

[0007] The present invention provides a data cleaning and filling method in a photovoltaic power generation scenario, comprising the following steps:

[0008] Step 1: obtain the real-time or historical data set generated by the photovoltaic panel to perform data cleaning, identify and process missing values ​​and outliers in the data, and mark them;

[0009] Step 2: Generate an overview of the corresponding missing and abnormal situations for users to compare and analyze, and quickly locate the areas that need to be filled and repaired;

[0010] Step 3: For missing values ​​and outliers in the cleaned data, visualize the missing data filling and outlier correction results using different interpolation algorithms, and calculate the filling similarity with historical similar data, so that users can judge and select the most appropriate interpolation algorithm;

[0011] Step 4: Visually display the missing data filling and abnormal data correction results, intuitively show the user the entire data status and the processed data status, and allow the user to manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results.

[0012] Furthermore, in step 1, missing types include zero values ​​and non-numeric values, and abnormal types include negative values ​​and mutation glitches.

[0013] Furthermore, negative values ​​and non-numeric values ​​are directly detected and marked; for sudden glitches, the differential method is used to compare whether the difference between the current value and the previous and next values ​​exceeds the set threshold, thereby identifying abnormal fluctuations in a short period of time.

[0014] Furthermore, for zero values, a non-zero value mask is created, the data is grouped, and the forward and backward cumulative sums are calculated to determine whether there is a zero value between the start and end points of the time series. If a zero value exists between the start and end points, it is marked as a missing value.

[0015] Furthermore, in step 2, an overview of current missing and abnormal situations is presented in the form of a line graph and a table.

[0016] Furthermore, the different interpolation algorithms in step 3 include five deep learning prediction algorithms, namely: iTransformer, ETSformer, FreTS, saits, and PatchTST.

[0017] Furthermore, in step 3, after the deep learning prediction algorithm trains the corresponding model through historical data, it fills in the missing data or corrects the abnormal data. Each algorithm is suitable for different abnormal types and data distributions according to its different characteristics.

[0018] The present invention also provides a data cleaning and filling system in a photovoltaic power generation scenario, comprising:

[0019] The data import and cleaning module is used to obtain the real-time or historical data sets generated by photovoltaic panels for data cleaning, identify and process missing values ​​and abnormal values ​​in the data, and mark them;

[0020] The data overview module is used to generate an overview of the corresponding missing and abnormal situations for users to compare and analyze, and quickly locate the areas that need to be filled and repaired;

[0021] The missing data filling and abnormal data repair module is used to visually output the missing data filling and abnormal data correction results using different interpolation algorithms for the missing values ​​and abnormal values ​​in the cleaned data, and calculate the filling similarity with the historical similar data, so that users can judge and choose the most appropriate interpolation algorithm;

[0022] The visualization and user interaction module is used to visualize the missing data filling and abnormal data correction results, intuitively show the user the entire data status and the processed data status, and allow users to manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results.

[0023] The present invention also provides a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the data cleaning and filling method in the photovoltaic power generation scenario is implemented.

[0024] The present invention also provides an electronic device, comprising:

[0025] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data cleaning and filling method in a photovoltaic power generation scenario by executing the computer instructions.

[0026] Through the above scheme, through the data cleaning and filling method and system in the photovoltaic power generation scenario, various abnormal data can be effectively identified and cleared, and data repair can be performed in combination with various interpolation algorithms. Through an intuitive visual interface, users are allowed to freely select different interpolation algorithms and view the interpolation effect in real time. At the same time, it supports manual adjustment of the repaired data value, which can improve the accuracy and flexibility of photovoltaic panel data, thereby improving the operating efficiency of the photovoltaic system and the reliability of data analysis.

[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the data cleaning and filling method in the photovoltaic power generation scenario of the present invention;

[0029] Figure 2 Schematic diagram of the data import and cleaning module of the present invention

[0030] Figure 3 It is a schematic diagram of a line graph of the data overview module of the present invention;

[0031] Figure 4 A schematic diagram of a table of a data overview module of the present invention;

[0032] Figure 5 A schematic diagram of a date overview of the data overview module of the present invention;

[0033] Figure 6 It is a schematic diagram of the missing data filling and abnormal data repair module of the present invention;

[0034] Figure 7 A result visualization schematic diagram of the visualization and user interaction module of the present invention;

[0035] Figure 8 A user adjustment schematic diagram of the visualization and user interaction module of the present invention;

[0036] Fig. 9 It is a schematic diagram of the system interface of the present invention;

[0037] Fig.10 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0039] Ginseng Figure 1 As shown, this embodiment provides a data cleaning and filling method in a photovoltaic power generation scenario, including the following steps:

[0040] Step S1, obtain the real-time or historical data set generated by the photovoltaic panel for data cleaning, identify and process missing values ​​and abnormal values ​​in the data, and mark them. Import the data set, and use multiple rules to identify missing or abnormal conditions such as non-numeric data, negative values, zero values, mutation glitches and missing values ​​in the data set for subsequent processing.

[0041] Step S2: Generate an overview of the corresponding missing and abnormal conditions for the user to compare and analyze, and quickly locate the area that needs to be filled and repaired. Generate an overview of the corresponding missing and abnormal conditions for the user to compare and analyze, and quickly locate the problem.

[0042] Step S3: For missing values ​​and outliers in the cleaned data, the missing data filling and outlier correction results using different interpolation algorithms are visually output, and the filling similarity with historical similar data is calculated for users to judge and select the most appropriate interpolation algorithm. Users can freely select multiple interpolation algorithms in the visual interface, view the repair effects of different algorithms in real time, and enhance the accuracy and applicability of data filling and repair.

[0043] Step S4, visualize the missing data filling and abnormal data correction results, and intuitively show the user the entire data status and the processed data status, so that the user can manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results. By providing an intuitive visualization interface, users can not only intuitively observe the effect of data repair, but also view the differences between different algorithms by switching different algorithms. For some abnormal values, the algorithm cannot perfectly repair or fill, and the user can flexibly adjust the data through the operation interface.

[0044] Through the data cleaning and filling method in the photovoltaic power generation scenario, the accuracy and flexibility of photovoltaic panel current data processing are significantly improved. Users can efficiently process photovoltaic panel current data in an intuitive and interactive way, significantly improving the accuracy of data cleaning and repair, meeting the high-quality requirements of photovoltaic system operation data analysis, while simplifying the data processing process and improving the user's operating experience.

[0045] This embodiment also provides a data cleaning and filling system in a photovoltaic power generation scenario, including:

[0046] The data import and cleaning module is used to obtain the real-time or historical data sets generated by photovoltaic panels for data cleaning, identify and process missing values ​​and outliers in the data, and mark them. This module is responsible for importing the real-time or historical data sets generated by photovoltaic panels. Users can use this module to identify and process missing values ​​and outliers in the data. This module supports multiple cleaning rules. The system will automatically identify non-numeric values, negative values, zero values, and other data that do not conform to physical meanings in the data, and mark them for subsequent modules to fill or correct.

[0047] The data overview module is used to generate an overview of the corresponding missing and abnormal situations for users to compare and analyze, and quickly locate the areas that need to be filled and repaired. The data overview module is used to present the results of data cleaning. This module generates an overview of the corresponding missing and abnormal situations according to the selected time period for users to compare and analyze, and quickly locate the areas that need to be filled and repaired.

[0048] The missing data filling and abnormal data repair module is used to visually output the missing data filling and abnormal data correction results of different interpolation algorithms for the missing values ​​and abnormal values ​​in the cleaned data, and calculate the filling similarity with historical similar data, so that users can judge and select the most appropriate interpolation algorithm. For the missing values ​​and abnormal values ​​in the cleaned data, this module uses a variety of deep learning prediction models and learns historical rules to provide a variety of filling or correction results for users to choose. By visually comparing the results of different interpolation algorithms and calculating the filling similarity with historical similar data, users can quickly judge and select the most appropriate algorithm.

[0049] The visualization and user interaction module is used to visualize the missing data filling and abnormal data correction results, and to intuitively show the user the entire data status and the processed data status, so that the user can manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results. This module is used to display the visualization effect of data filling and correction results, helping users to intuitively understand the entire data status and the processed data status. Users can use this module to view the cleaning and interpolation results in real time, and compare and analyze different interpolation algorithms to select the optimal solution. This module also allows users to combine their own domain knowledge and relevant experience to manually adjust individual data points through the interactive interface to further optimize the data filling results.

[0050] Ginseng Figure 2 As shown in the figure, users import the current data of photovoltaic panels through the data import and cleaning module. The supported input formats include common file formats such as CSV and Excel, and the time can be customized. The data cleaning process uses a variety of detection algorithms to automatically detect missing and abnormal points in the data. Missing types include zero values ​​and non-numeric values, and abnormal types include negative values ​​and mutation glitches. For negative values ​​and non-numeric values, the system can directly detect and mark them. For mutation glitches, the system uses the difference method to compare whether the difference between the current value and the previous and next values ​​exceeds the set threshold to identify abnormal fluctuations in a short period of time. For zero values, the system first creates a non-zero value mask, then groups the data, and calculates the forward and backward cumulative sums to determine whether there is a zero value between the start and end of the time series. If a zero value exists between the start and end points, it is marked as a missing value.

[0051] Ginseng Figure 3 , Figure 4 , Figure 5 As shown in the figure, the data overview module displays the overview of the current data after cleaning in the form of a line chart and a table. The two curves in the line chart reflect the missing data changes over time ( Figure 3 The upper middle curve shows the abnormal situation ( Figure 3The lower middle curve shows Figure 4 The missing details are shown in Figure 1, with the severity of the missing and abnormal rates coded by color. Figure 3 and Figure 4 The information presented in the table allows users to have an overall understanding of the data and quickly locate the string currents and their corresponding time periods that are seriously missing or abnormal. After selecting a specific string current for analysis, a block view containing the corresponding selected time span will appear. The color depth of each block encodes the missing or abnormal situation of the string current at the corresponding time (such as Figure 5 shown).

[0052] Ginseng Figure 6 As shown in the figure, the missing data filling and abnormal data repair module integrates five deep learning prediction algorithms (iTransformer, ETSformer, FreTS, saits, PatchTST). After training the corresponding model with historical data, the missing data is filled or the abnormal data is corrected. Each algorithm can be applied to different abnormal types and data distributions according to its different characteristics. Users can freely select the recommended interpolation algorithm on the visual interface provided by the system and view the repair effect in real time. The system will also give the historical filling accuracy corresponding to different interpolation algorithms. This indicator is calculated based on the similarity of historical data highly correlated with non-missing values, which is convenient for comparing multiple interpolation algorithms, so as to select the optimal repair method.

[0053] Ginseng Figure 7 , Figure 8 , Fig. 9 As shown in the figure, after the model processing is completed, the visualization and user interaction module displays the processed data in a visual way. Users can switch different models to observe the change trend of the data before and after repair and the difference between different algorithms. After clicking on the processing result of a certain model, the line chart above will simultaneously present the result of the original data cleaning and the filling or correction result after the corresponding model processing (such as Figure 7 After the user automatically repairs the data, the system allows the user to manually adjust the data points. After clicking the button to draw the ideal filling curve, the user can modify the current value at a certain moment by clicking on the line graph to improve the accuracy of the repair (as shown in Figure 2). Figure 8 Once the adjustment is complete, the user can choose to export the repaired data for subsequent analysis or integration into other systems.

[0054] The data cleaning and filling system in the photovoltaic power generation scenario realizes an efficient, flexible and intelligent data processing process, greatly improves the accuracy of cleaning and repairing photovoltaic panel current data, and meets the high standards of photovoltaic systems in data processing and analysis.

[0055] This embodiment also provides a non-transitory computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the data cleaning and filling method in the photovoltaic power generation scenario is implemented.

[0056] Ginseng Fig.10 As shown, this embodiment also provides an electronic device, including:

[0057] A memory 201 and a processor 202, wherein the memory 201 and the processor 202 are communicatively connected to each other, the memory 201 stores computer instructions, and the processor 202 executes the computer instructions to execute the data cleaning and filling method in the photovoltaic power generation scenario.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data cleaning and filling method in a photovoltaic power generation scenario, characterized in that: The steps include: Step 1: obtain the real-time or historical data set generated by the photovoltaic panel to perform data cleaning, identify and process missing values ​​and outliers in the data, and mark them; Step 2: Generate an overview of the corresponding missing and abnormal situations for users to compare and analyze, and quickly locate the areas that need to be filled and repaired; Step 3: For missing values ​​and outliers in the cleaned data, visualize the missing data filling and outlier correction results using different interpolation algorithms, and calculate the filling similarity with historical similar data, so that users can judge and select the most appropriate interpolation algorithm; Step 4: Visually display the missing data filling and abnormal data correction results, intuitively show the user the entire data status and the processed data status, and allow the user to manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results.

2. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 1 is characterized in that: In step 1, missing types include zero values ​​and non-numeric values, and abnormal types include negative values ​​and mutation glitches.

3. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 2 is characterized in that: Negative values ​​and non-numeric values ​​are directly detected and marked; for sudden glitches, the differential method is used to compare whether the difference between the current value and the previous and next values ​​exceeds the set threshold, and abnormal fluctuations in a short period of time are identified.

4. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 3 is characterized in that: For zero values, create a non-zero mask, group the data, calculate the forward and backward cumulative sums to determine if there is a zero value between the start and end of the time series, and if a zero value exists between the start and end, mark it as a missing value.

5. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 4 is characterized in that: In step 2, an overview of the current missing and abnormal situations is presented in the form of a line graph and a table.

6. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 5 is characterized in that: The different interpolation algorithms in step 3 include five deep learning prediction algorithms, namely: iTransformer, ETSformer, FreTS, saits, and PatchTST.

7. The data cleaning and filling method in the photovoltaic power generation scenario according to claim 6 is characterized in that: In step 3, after the deep learning prediction algorithm trains the corresponding model through historical data, it fills in the missing data or corrects the abnormal data. Each algorithm is suitable for different abnormal types and data distributions according to its different characteristics.

8. A data cleaning and filling system in a photovoltaic power generation scenario, characterized in that: include: The data import and cleaning module is used to obtain the real-time or historical data sets generated by photovoltaic panels for data cleaning, identify and process missing values ​​and abnormal values ​​in the data, and mark them; The data overview module is used to generate an overview of the corresponding missing and abnormal situations for users to compare and analyze, and quickly locate the areas that need to be filled and repaired; The missing data filling and abnormal data repair module is used to visually output the missing data filling and abnormal data correction results using different interpolation algorithms for the missing values ​​and abnormal values ​​in the cleaned data, and calculate the filling similarity with the historical similar data, so that users can judge and choose the most appropriate interpolation algorithm; The visualization and user interaction module is used to visualize the missing data filling and abnormal data correction results, intuitively show the user the entire data status and the processed data status, and allow users to manually adjust some abnormal values ​​through the interactive interface to further optimize the data filling results.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a processor, implement a data cleaning and filling method in a photovoltaic power generation scenario as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data cleaning and filling method in a photovoltaic power generation scenario as described in any one of claims 1 to 7 by executing the computer instructions.

Images