New energy stand-alone data restoration method based on clustering algorithm and computing device

Abstract

The invention provides a new energy stand-alone data restoration method based on a clustering algorithm and computing equipment. The method comprises the following steps: processing correct data in historical data of each station; clustering and grouping are carried out according to the processed correct data, and a cluster to which each station belongs is determined; missing data and abnormal values in the real-time data of each station are identified and marked; and repairing the missing data and the abnormal value according to the cluster to which the station in which the missing data and the abnormal value are identified belongs. According to the technical scheme of the invention, the data processing efficiency can be improved, the data restoration time can be shortened, real-time online data restoration can be realized, and the monitoring strength and the maintenance capability of the equipment operation state can be improved.

Description

Technical Field

[0001] The present invention relates to the field of new energy, and in particular to a method and computing device for repairing new energy stand-alone data based on a clustering algorithm. Background Art

[0002] Due to the dual influence of the physical environment and the transmission environment, the current new energy sites have prominent problems in data quality. Specifically, there are a lot of missing and abnormal data in the data collected by the site units. With the rapid development of the new energy industry, the number of site units continues to increase, resulting in a significant increase in the amount of data uploaded to the power grid dispatching department, and the phenomenon of missing and abnormal data is becoming increasingly serious. This not only affects the accuracy and reliability of the data, but also brings challenges to the dispatching and operation of the power grid. Therefore, it is urgent to effectively manage the data quality issues of new energy sites to ensure the quality of the data and the stable operation of the power grid.

[0003] To this end, a technical solution is needed that can improve data processing efficiency, shorten data repair time, realize real-time online repair of data, and enhance the monitoring and maintenance capabilities of equipment operating status. Summary of the invention

[0004] The present invention aims to provide a method and computing device for data repair of a new energy stand-alone machine based on a clustering algorithm, which can improve data processing efficiency, shorten data repair time, realize real-time online repair of data, and enhance the monitoring intensity and maintenance capability of the equipment operation status.

[0005] According to one aspect of the present invention, a method for repairing data of a new energy stand-alone machine based on a clustering algorithm is provided, the method comprising:

[0006] Process the correct data in the historical data of each station;

[0007] Perform clustering and grouping according to the processed correct data to determine the cluster to which each station belongs;

[0008] Identify and mark missing data and outliers in real-time data from each station;

[0009] Missing data and outliers are repaired based on the clusters to which the stations with the missing data and outliers are identified belong.

[0010] According to some embodiments, missing data and outliers are identified in the historical data of each station;

[0011] Missing data and outliers in the historical data are marked to facilitate their removal before clustering and grouping.

[0012] According to some embodiments, the correct data in the historical data of each station is processed, including:

[0013] The correct data in the historical data is processed in a first manner so that the data maintains the same scale, and the first manner is normalization processing.

[0014] According to some embodiments, the simultaneous rate is active power per unit capacity;

[0015] The simultaneity rate is calculated by dividing the active power required by each station by the installed capacity.

[0016] According to some embodiments, clustering and grouping are performed according to the processed correct data to determine the cluster to which each station belongs, including:

[0017] determining the number of clusters by a second method, wherein the second method is an elbow rule;

[0018] Allocate each processed correct data to the corresponding cluster center;

[0019] After allocation, the average of the correct data in each cluster was calculated;

[0020] According to some embodiments, each cluster center is updated using the average value;

[0021] The values ​​of the cluster centers are calculated repeatedly until the change of the cluster centers is less than a preset threshold.

[0022] According to some embodiments, repairing missing data and outliers according to the cluster to which the station to which the missing data and outliers are identified belongs includes:

[0023] According to the clusters to which the stations with missing data and outliers belong, the mean and / or median of the clusters are obtained to fill the missing data and outliers;

[0024] After repairing the missing data and outliers, the repaired data is processed in a third manner, which is a median smoothing algorithm.

[0025] According to some embodiments, the repaired data is processed in a third manner, including:

[0026] Check the labeled real-time data for missing data and outliers;

[0027] Using a fixed-size window to calculate the median of missing data and outliers of the real-time data;

[0028] Missing data and outliers of the real-time data are replaced with the median.

[0029] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0030] According to another aspect of the present invention, there is provided a computing device, comprising:

[0031] Processor; and

[0032] A memory stores a computer program, and when the computer program is executed by the processor, the method described in any one of the above items is implemented.

[0033] According to an embodiment of the present invention, historical data is collected from each station, and missing and abnormal values ​​are identified and marked. The correct data of the station is processed, and the station is clustered and grouped according to the processed historical data of the station, and the station data is divided into a number of clusters. The real-time data of the station is processed, missing and abnormal values ​​are identified, and for stations with abnormal missing data, the average value or median of the cluster is used to fill the missing value according to the cluster to which the station belongs. Through the present invention, data processing efficiency can be improved, data repair time can be shortened, real-time online repair of data can be achieved, and the monitoring intensity and maintenance capability of equipment operation status can be improved.

[0034] According to some embodiments, by filling missing values, the integrity of the data is ensured, making subsequent data analysis more reliable. Identifying and marking outliers helps understand potential problems in the data, avoids outliers from adversely affecting the results, and facilitates better data analysis and decision support.

[0035] According to some embodiments, by clustering the stations, groups of stations with similar characteristics can be found, which helps to formulate unified management strategies or optimization measures. After the stations are divided into different clusters, a specific management plan can be formulated according to the characteristics of each cluster to reduce workload and improve efficiency.

[0036] According to some embodiments, missing values ​​in real-time data are filled using the mean or median of the cluster to which they belong. This method is flexible and practical and can quickly solve the problem without affecting overall accuracy. Filling missing values ​​with statistical data within the cluster reduces the system's reliance on the integrity of individual station data and enhances the overall robustness of the system.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0039] Figure 1 A flow chart of a data repair method according to an example embodiment is shown.

[0040] Figure 2 A flow chart of a real-time data repair method according to an example embodiment is shown.

[0041] Figure 3 A block diagram of a computing device is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0043] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0045] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0046] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the present inventive concept. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0047] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0048] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present invention, and therefore cannot be used to limit the protection scope of the present invention.

[0049] When dealing with the data quality problem of new energy stations, the current technical means is to use interpolation methods such as KNN to fill the gaps in missing data. This method estimates the possible values ​​of missing data by mathematically processing known data, thereby restoring the integrity of the data to a certain extent. However, this technical means has some limitations in practical applications: First, the timeliness of the interpolation method is not high. Since it relies on existing data to infer missing values, the interpolation method cannot meet the needs of rapid response in scenarios with high data real-time requirements. Especially in the operation of new energy stations, the accuracy and timeliness of real-time data are crucial for power grid dispatching and stable operation. Secondly, when faced with a large amount of missing data, the effectiveness of the interpolation method will be greatly reduced. If the amount of missing data is too large, the interpolation result will deviate far from the actual value, resulting in the accuracy and reliability of the data being affected.

[0050] To this end, the present invention proposes a method for repairing data of a new energy stand-alone machine based on a clustering algorithm, which can improve data processing efficiency, shorten data repair time, realize real-time online repair data, and enhance the monitoring intensity and maintenance capability of equipment operation status.

[0051] Exemplary embodiments of the present invention are described below with reference to the accompanying drawings.

[0052] Figure 1 A flow chart of a data repair method according to an example embodiment is shown.

[0053] See also Figure 1 , in S101, the correct data in the historical data of each station is processed.

[0054] According to some embodiments, missing data and outliers in the historical data of each station are identified; the missing data and outliers in the historical data are marked to facilitate their removal before clustering and grouping. In the historical data of each station, negative numbers of active power data, values ​​greater than installed capacity, and null values ​​all need to be marked as outliers. One purpose of marking outliers is to remove the marked data when clustering, and to only process the marked data when correcting and smoothing the data later to prevent affecting normal data.

[0055] According to some embodiments, the correct data in the historical data is processed in a first manner so that the data maintains the same scale. For example, the active power is processed in a first manner to obtain the simultaneity rate, and the first manner is normalization processing. The simultaneity rate is the active power per unit capacity; the simultaneity rate is calculated by dividing the active power of each station by the installed capacity.

[0056] Normalization is a common preprocessing technique in data processing, which is used to adjust the range of values ​​of different variables so that they are on the same scale. This is usually to eliminate the impact of different dimensions and magnitudes between attributes to facilitate further data analysis or model training. Normalization can be applied to a variety of scenarios, such as database management, image processing, statistics, machine learning and other fields.

[0057] Common normalization methods include: Min-Max Normalization, which linearly transforms the original data into a specific interval, usually [0,1] or [-1,1]; Z-Score Normalization, which standardizes the data by calculating the standard deviation and mean, so that the data is converted into a standard normal distribution with zero mean and unit variance; Decimal Scaling Normalization, which normalizes the data by moving the decimal point position until all values ​​are in the interval [-1,1].

[0058] The choice of normalization method depends on the characteristics of the dataset and the requirements of the subsequent task. When implementing normalization, it is important to ensure that the distribution characteristics of the data are not changed and to take into account the impact of outliers. In machine learning, normalization can increase the speed of model training and help the algorithm converge better.

[0059] In S103, clustering and grouping are performed according to the processed correct data to determine the cluster to which each station belongs.

[0060] According to some embodiments, the number of clusters is determined by a second method, which is the elbow rule; each processed correct data is assigned to a corresponding cluster center; after assignment, the average value of the correct data in each cluster is calculated; the average value is used to update each cluster center; and the value of the cluster center is repeatedly calculated until the change in the cluster center is less than a preset threshold.

[0061] Based on the processed historical data of the stations, the K-means clustering algorithm is used to cluster the stations. First, the K value is selected, and the elbow rule is used to determine how many clusters the station data should be divided into. Each data point is assigned to the nearest cluster center, and the new center of each cluster is recalculated, and it is repeated until the cluster center no longer changes significantly.

[0062] The K-means clustering algorithm is an iterative clustering analysis algorithm that aims to divide n objects into k clusters according to their attributes so that the similarity between objects in a cluster is maximized and the similarity between clusters is minimized.

[0063] For different values ​​of k, calculate the total squared error (SSE) of the clusters corresponding to each k. As the value of k increases, the SSE usually decreases because the number of samples in each cluster decreases. Plot the curve of SSE as a function of k and look for the "elbow" of the curve, the point where the rate of reduction of SSE slows down significantly, which is usually considered the optimal k value.

[0064] In the K-means algorithm, the steps of calculating the new center of each cluster and assigning the data points to the nearest cluster center are as follows.

[0065] The first step is to assign data points to the nearest cluster center: initialize the cluster center and select k data points as the initial cluster center by the elbow rule. For each data point in the data set, use the Euclidean distance to calculate its distance to each cluster center. The calculation formula of the Euclidean distance is:

[0066]

[0067] Here, p and q are two data points, and n is the dimension of the data points.

[0068] According to the calculated distance, each data point is assigned to the nearest cluster center. That is, for each data point x, find the cluster center uj so that d(x,uj) is minimized, and assign x to cluster j.

[0069] The second step is to recalculate the new center of each cluster: calculate the mean of the data points in the cluster, and for each cluster j, calculate the mean of all the data points in the cluster, which will be used as the new cluster center. Then use the calculated mean to update the center of each cluster.

[0070] Assuming that cluster j contains data points x1, x2, ..., xm, the calculation formula for the new cluster center uj is:

[0071]

[0072] Here, m is the number of data points in cluster j.

[0073] The third step is iteration: repeat the above two steps until the change in cluster center is less than a preset threshold, such as 0.01.

[0074] At S105 , missing data and outliers in the real-time data of each station are identified and marked.

[0075] In the real-time data of each station, negative active power data, values ​​greater than the installed capacity, and null values ​​need to be marked as abnormal values. Marking helps to distinguish which data are corrected, and comparison can be made before and after correction.

[0076] In S107, the missing data and the outliers are repaired according to the clusters to which the stations with the identified missing data and the outliers belong.

[0077] According to the cluster to which the stations with missing data and outliers belong, the average value and / or median of the cluster is obtained to fill the missing data and outliers; after repairing the missing data and outliers, the repaired data is processed in a third way, which is a median smoothing algorithm.

[0078] According to some embodiments, missing data and outliers of the marked real-time data are checked, a fixed-size window is used to calculate the median of the missing data and outliers of the real-time data; and the missing data and outliers of the real-time data are replaced with the median.

[0079] The real-time data of the station is processed to identify missing and abnormal values. For stations with abnormal missing data, the missing values ​​are filled with the average or median of the cluster according to the cluster to which the station belongs. The repaired data is smoothed using the median smoothing algorithm.

[0080] Median smoothing is a time series smoothing technique that smooths data jumps by calculating the median of the data of a certain window size. For each data point, check whether it is marked as an outlier. If it is marked, use a fixed-size window (usually an odd number) to calculate the median of the point and its neighbors, and finally replace the value of the jump point with the calculated median.

[0081] This method uses the K-means clustering algorithm to cluster and group new energy stations, repairs the missing and abnormal data of the stations according to the clustering results, and smoothes the repaired data using the median smoothing algorithm to ensure that there are no mutations or jumps between normal data and repaired data.

[0082] This method has good adaptability and can adjust the division of clusters according to the changes in data, so as to better adapt to the changes in the operating status of new energy station equipment. The clustering algorithm has high computational efficiency and is suitable for processing large-scale data sets. For single-machine data of new energy stations, this can increase the speed of data processing and shorten the repair time. It has a better effect on the repair of long-term data loss of single-machine data of new energy stations, and the K-means clustering algorithm has a fast iterative process, which can realize real-time online repair. For single-machine data of new energy stations, real-time repair is of great significance for the monitoring and maintenance of equipment operating status.

[0083] Figure 2 A flow chart of a real-time data repair method according to an example embodiment is shown.

[0084] This method pre-groups the stations through the K-means clustering algorithm, establishes an intelligent feedback mechanism, captures and analyzes the dynamic data of the stations in real time, and automatically adjusts the classification of the stations. Data uploaded by the stations are cleaned, missing data, abnormal data, etc. are identified and marked, and finally the missing and abnormal data are repaired in real time by calculating the data average or median of the cluster to which the station belongs. This solution is highly timely and can handle the situation where the station has long-term data missing and abnormal.

[0085] In S201, the real-time active power of each station is collected; in S203, the station active power that is negative, greater than the installed capacity, and null value is marked as anomaly; in S205, the normal value is normalized, and the normalization is to calculate the simultaneity rate of each point of all stations equal to the ratio of the active power of the point to the installed capacity of the point; in S207, the K-means clustering algorithm is used to cluster and group the stations point by point based on the simultaneity rate of the processed normal values.

[0086] For the marked outliers, in S209, according to the cluster to which the station belongs after clustering, the average simultaneity rate of the stations in the same cluster is used as the simultaneity rate of the outlier, and the active power is calculated by multiplying the simultaneity rate by the installed capacity; in S211, after the processing of the outliers is completed, the marked data is smoothed using the median smoothing algorithm to prevent jumps between the processed data and the normal data.

[0087] Figure 3 A block diagram of a computing device is shown according to an exemplary embodiment.

[0088] like Figure 3 As shown, computing device 30 includes processor 12 and memory 14. Computing device 30 may also include bus 22, network interface 16, and I / O interface 18. Processor 12, memory 14, network interface 16, and I / O interface 18 may communicate with each other via bus 22.

[0089] The processor 12 may include one or more general-purpose CPUs (Central Processing Units, processors), microprocessors, or application-specific integrated circuits, etc., for executing relevant program instructions. According to some embodiments, the computing device 30 may also include a high-performance graphics card (GPU) 20 for accelerating the processor 12.

[0090] The memory 14 may include a machine system readable medium in the form of a volatile memory, such as a random access memory (RAM), a read-only memory (ROM) and / or a cache memory. The memory 14 is used to store one or more programs including instructions and data. The processor 12 can read the instructions stored in the memory 14 to execute the above-mentioned method according to the embodiment of the present invention.

[0091] The computing device 30 may also communicate with one or more networks via the network interface 16. The network interface 16 may be a wireless network interface.

[0092] The bus 22 may include an address bus, a data bus, a control bus, etc. The bus 22 provides a path for exchanging information between components.

[0093] It should be noted that, in the specific implementation process, the computing device 30 may also include other components necessary for normal operation. In addition, those skilled in the art may understand that the above device may only include components necessary for implementing the embodiments of this specification, and need not include all components shown in the figure.

[0094] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a microdrive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), a network storage device, a cloud storage device, or any type of medium or device suitable for storing instructions and / or data.

[0095] An embodiment of the present invention further provides a computer program product, which includes a computer program. The computer program is operable to enable a computer to execute part or all of the steps of any one of the methods described in the above method embodiments.

[0096] Those skilled in the art can clearly understand that the technical solution of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array, an integrated circuit, etc.

[0097] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided by the present invention, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention.

[0103] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended clauses.

Claims

1. A method for repairing data of a new energy stand-alone machine based on a clustering algorithm, the method comprising: Process the correct data in the historical data of each station; Perform clustering and grouping according to the processed correct data to determine the cluster to which each station belongs; Identify and mark missing data and outliers in real-time data from each station; Missing data and outliers are repaired based on the clusters to which the stations with the missing data and outliers are identified belong.

2. The method according to claim 1, characterized in that include: Identify missing data and outliers in the historical data of each station; Missing data and outliers in the historical data are marked to facilitate their removal before clustering and grouping.

3. The method according to claim 1, characterized in that Process the correct data in the historical data of each station, including: The correct data in the historical data is processed in a first manner so that the data maintains the same scale, and the first manner is normalization processing.

4. The method according to claim 3, characterized in that The simultaneous rate is the active power per unit capacity; The simultaneity rate is calculated by dividing the active power required by each station by the installed capacity.

5. The method according to claim 1, characterized in that Clustering and grouping are performed based on the processed correct data to determine the cluster to which each station belongs, including: determining the number of clusters by a second method, wherein the second method is an elbow rule; Allocate each processed correct data to the corresponding cluster center; After allocation, the average of the correct data in each cluster was calculated.

6. The method according to claim 5, characterized in that include: Using the average value to update each cluster center; The values ​​of the cluster centers are calculated repeatedly until the change of the cluster centers is less than a preset threshold.

7. The method according to claim 1, characterized in that Repair missing data and outliers based on the clusters to which the stations with missing data and outliers belong, including: According to the clusters to which the stations with missing data and outliers belong, the mean and / or median of the clusters are obtained to fill the missing data and outliers; After repairing the missing data and outliers, the repaired data is processed in a third manner, which is a median smoothing algorithm.

8. The method according to claim 7, characterized in that The repaired data is processed in a third way, including: Check the labeled real-time data for missing data and outliers; Using a fixed-size window to calculate the median of missing data and outliers of the real-time data; Missing data and outliers of the real-time data are replaced with the median.

9. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

10. A computing device, characterized in that: include: processor; as well as A memory storing a computer program, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images