New energy equipment operation index data cleaning method and device and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a new energy equipment operation index data cleaning method and device, equipment and a computer storage medium. According to the new energy equipment operation index data cleaning method, through the four main steps of data preprocessing, anomaly detection and data restoration and reconstruction, it is ensured that the cleaning process can adapt to different types of new energy equipment and operation data characteristics of the new energy equipment, and the data quality and the reliability of an analysis result are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, equipment, and computer storage medium for cleaning data of operating indicators of new energy equipment. Background Technology

[0002] The quality and accuracy of operational data from new energy equipment directly impact system performance analysis and optimization. However, several pressing issues exist in current new energy systems. First, the individual capacity of new energy equipment (such as wind turbines and photovoltaic panels) is relatively small, typically at the megawatt level or below. Compared to the megawatt-level individual capacity of traditional coal-fired, hydropower, and nuclear power plants, new energy equipment is numerous, with diverse operating parameters and extremely rapid data generation. This massive data volume places enormous pressure on data management, collection, and analysis. Furthermore, the technical parameters, operating indicators, and data collection methods for new energy equipment lack unified standards. Significant differences exist in the calculation methods and data aggregation methods across different smart parks and power stations, leading to frequent errors and inconsistencies during data collection. These data quality issues not only affect the accuracy of analysis results but can also lead to decision-making biases and even mislead operational planning.

[0003] Secondly, with the development of integrated smart parks, the operating scenarios of new energy systems are becoming increasingly complex. Existing data processing methods often only clean and process data for a single scenario, lacking comprehensive solutions. Current methods cannot effectively cope with the complexity of multi-source heterogeneous data, nor can they handle common problems such as frequent outliers and missing data. This limitation greatly restricts the application and in-depth analysis of new energy equipment operating data in smart parks.

[0004] Therefore, the operation of new energy equipment in integrated smart parks urgently requires a standardized and scalable data cleaning method that can effectively handle different energy forms and adapt to complex and diverse data sources to improve data quality and consistency. This is not only key to improving the accuracy of data analysis results, but also an important foundation for ensuring the operating efficiency of new energy equipment, optimizing maintenance strategies, and supporting future new energy planning. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology may destroy the original data distribution during the data cleaning and transformation process, and does not clearly handle multi-source heterogeneous data and data scenario weight allocation, resulting in poor data quality.

[0006] To address the aforementioned technical problems, this invention provides a method for cleaning operational indicator data of new energy equipment, comprising:

[0007] Based on the enterprise affiliation model, equipment operation records from different data sources are uniformly encoded and mapped.

[0008] The device operation records after unified coding mapping are used for outlier detection and missing value filling, and are then marked.

[0009] Based on the scoring rule base, each device operation record is scored according to its weight according to the tags;

[0010] Based on the scoring results, the equipment operation records were sampled proportionally according to the key dimensions of the data to obtain the cleaned data.

[0011] Preferably, the step of uniformly encoding and mapping equipment operation records from different data sources based on the enterprise affiliation model includes:

[0012] Obtain information on all equipment, sites, subsidiaries, and parent companies of the enterprise, and determine the hierarchy and relationship of each entity based on the enterprise's organizational structure to construct an enterprise affiliation model;

[0013] Based on the enterprise affiliation model, a unique code is assigned to each device, site, subsidiary, and parent company, and all enterprise entity names in the original dataset are replaced with the corresponding codes.

[0014] Retrieve the manufacturer names appearing in all data sources, and perform deduplication and normalization on the manufacturer names;

[0015] Assign a unique code to each unique manufacturer name;

[0016] In the original dataset, all manufacturer names were replaced with their corresponding codes;

[0017] Create and maintain an encoding mapping table so that the original name can be queried when needed.

[0018] Preferably, outlier detection and marking of the device operation records after the unified encoding mapping includes:

[0019] Based on industry mechanisms, a normal range is defined for each operational indicator, and operational indicator data that exceeds the corresponding predefined normal range is marked as abnormal.

[0020] Determine the expected mathematical relationship between operational indicators, and verify the relevant operational indicator data according to the predefined mathematical relationship. If the relationship does not hold, mark it as an anomaly.

[0021] Based on statistics, the confidence range of the operating indicators is determined, and operating indicator data that exceed the corresponding predefined confidence range are marked as anomalies;

[0022] The normal range of operating indicators is determined based on a machine learning model, and operating indicator data that exceeds the normal range is marked as abnormal.

[0023] Preferably, the missing value imputation and marking of the device operation records after the unified encoding mapping includes:

[0024] Scan and retrieve missing values;

[0025] Missing values ​​are imputed using statistical methods or predictive models, and the original values, imputed data, and unimputed data are classified and labeled.

[0026] Preferably, the imputation of missing values ​​using statistical methods or predictive models includes:

[0027] The K nearest neighbors of a missing value are calculated using Euclidean distance, and the missing value is filled in based on the average of the K nearest neighbors.

[0028] A regression prediction model is established based on historical data, and the missing values ​​are filled in using the predicted values ​​calculated by the regression prediction model.

[0029] Preferably, the step of scoring each device operation record according to the weight based on the tag, based on the scoring rule base, includes:

[0030] Assign weights to the data based on the label type and build a scoring rule library;

[0031] Each device operation record is scored based on the weights and the scoring rule library.

[0032] Preferably, the step of sampling the equipment operation records proportionally based on the scoring results and stratifying them according to key data dimensions to obtain the cleaned data includes:

[0033] Identify the key dimensions of the data;

[0034] The dataset is stratified according to the key dimensions of the data.

[0035] Within each layer, a certain proportion of the equipment operation records with high scores are randomly selected.

[0036] This invention also provides a data cleaning device for operating indicators of new energy equipment, comprising:

[0037] The data preprocessing module is used to perform unified encoding and mapping on equipment operation records from different data sources based on the enterprise affiliation model;

[0038] The data detection and repair module is used to detect outliers and fill in missing values ​​in the device operation records after unified coding mapping, and to mark them.

[0039] The data scoring module is used to score each device operation record according to weight based on the tags, using a scoring rule library;

[0040] The data sampling module is used to sample equipment operation records proportionally based on the scoring results and according to the key dimensions of the data, so as to obtain cleaned data.

[0041] This invention also provides a data cleaning device for operating indicators of new energy equipment, comprising:

[0042] Memory, used to store computer programs;

[0043] A processor is used to implement the steps of the above-mentioned method for cleaning data of operating indicators of new energy equipment when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for cleaning data of operating indicators of new energy equipment.

[0045] The technical solution of the present invention has the following advantages compared with the prior art:

[0046] The new energy equipment operation index data cleaning method described in this invention, through four main steps—data preprocessing, anomaly detection, data repair and reconstruction—ensures that the cleaning process can adapt to different types of new energy equipment and their operation data characteristics, thereby improving data quality and the reliability of analysis results. Attached Figure Description

[0047] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0048] Figure 1 This is a flowchart illustrating the implementation of a data cleaning method for operating indicators of new energy equipment provided by the present invention. Detailed Implementation

[0049] The core of this invention is to provide a method, apparatus, equipment, and computer storage medium for cleaning data of operating indicators of new energy equipment, which effectively improves data quality and the reliability of analysis results.

[0050] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Please refer to Figure 1 , Figure 1 The flowchart illustrates the implementation of a data cleaning method for operating indicators of new energy equipment provided by this invention; the specific operation steps are as follows:

[0052] S101: Based on the enterprise affiliation model, uniformly encode and map equipment operation records from different data sources;

[0053] S102: Perform outlier detection and missing value imputation on the device operation records after unified coding mapping, and mark them accordingly;

[0054] S103: Based on the scoring rule base, score each device operation record according to the weight based on the tag;

[0055] S104: Based on the scoring results, the equipment operation records are sampled proportionally according to the key dimensions of the data to obtain the cleaned data.

[0056] Based on the above embodiments, this embodiment will provide a detailed description of step S101:

[0057] • A unified coding mapping is implemented based on the ownership relationship model of equipment, sites, subsidiaries, and parent companies according to the actual situation of the enterprise:

[0058] Objective: To ensure consistency and comparability in data analysis and processing for enterprise entities at different levels and of different types. Steps:

[0059] Data: Information on all equipment, sites, subsidiaries, and parent companies of the enterprise.

[0060] Create an attribution model: Based on the company's organizational structure, create an attribution model to clarify the hierarchy and relationships of each entity.

[0061] Assign a unique code: Assign a unique code to each device, site, subsidiary, and parent company.

[0062] Update the dataset: In the original dataset, replace all enterprise entity names with the corresponding codes.

[0063] Maintain an encoding mapping table: Create and maintain an encoding mapping table so that the original name can be queried when needed.

[0064] • Implement unified variable coding for unordered categorical data such as manufacturer names.

[0065] Objective: To ensure consistency and comparability of manufacturer names across different data sources during data analysis and processing.

[0066] step:

[0067] Data collection: Collect manufacturer names appearing in all data sources.

[0068] Deduplication and standardization: Deduplicate and standardize manufacturer names, for example, by using English names uniformly and removing redundant spaces and symbols.

[0069] Create an encoding mapping table: Assign a unique code to each unique manufacturer name. For example, "Manufacturer A" can be encoded as "M001", "Manufacturer B" as "M002", and so on.

[0070] Replace the original data: In the original dataset, replace all manufacturer names with the corresponding codes.

[0071] Update the dataset: Update the dataset with the replaced encoding to ensure data consistency.

[0072] Table 1: Example 1

[0073] Device ID Station Name Subsidiary Name Parent company name Installed capacity (kW) Grid connection date 001 Station 1 Subsidiary 1 Parent Company A 1000 2023-01-15 002 Station 2 Subsidiary 2 Parent Company B 1500 2023-02-20

[0074] Table 2: Example 1 - Encoding Mapping Table

[0075]

[0076]

[0077] Table 3: Example 1 - Updated Dataset

[0078] Device ID Station Name Subsidiary Name Parent company name Installed capacity (kW) Grid connection date 001 S001 CS01 P001 1000 2023-01-15 002 S002 CS02 P002 1500 2023-02-20

[0079] Data standardization processing: This invention standardizes data from different sources, devices, and unordered categories (such as manufacturer names) through a unified coding mapping, solving the inconsistency problem caused by data diversity and effectively improving the efficiency of data integration and analysis.

[0080] Based on the above embodiments, this embodiment will provide a detailed description of step S102:

[0081] • Define the normal range for each operational indicator based on industry mechanisms, and mark operational indicator data that exceed the corresponding predefined normal range as abnormal;

[0082] Objective: To identify and label data points that are impossible or logically illogical according to industry mechanisms, such as wind speeds less than 0 m / s.

[0083] step:

[0084] Define normal ranges: Based on industry knowledge and historical data, define the normal range for each indicator. For example, the normal range for wind speed might be 0 to 30 m / s.

[0085] Data collection: Collect all operational indicator data of new energy equipment.

[0086] Data Scan: Scan each metric in the dataset to check if any data points are outside the predefined normal range.

[0087] Mark outlier data: Mark data points that are outside the normal range. This can be done by adding a new column, such as "Outlier Markers", and assigning specific labels, such as "Outlier", to outlier data points.

[0088] Record anomaly information: Record detailed information about abnormal data, including the original value of the data point, the type of anomaly detected, and possible causes.

[0089] Preserve the original data: Even if the data is marked as anomalous, the original data should be preserved for further analysis or for training machine learning models.

[0090] Suppose we have a dataset of wind farms, including wind speed and power generation.

[0091] Table 4: Example 2

[0092] Device ID Wind speed (m / s) Electricity generation (kWh) date 001 -2 500 2023-01-15 002 25 750 2023-01-15 003 30 800 2023-01-15

[0093] Define the normal range:

[0094] Wind speed: 0 to 30 m / s

[0095] Processing steps:

[0096] Scan data: Check if the wind speed data is within the range of 0 to 30 m / s.

[0097] Mark anomalous data:

[0098] The wind speed of device ID 001 is -2m / s, which is below the normal range and is marked as "abnormal".

[0099] The wind speed of device ID 003 is 30m / s, which is within the range but close to the upper limit, so it deserves special attention.

[0100] Table 5: Example 2 - Updated Dataset

[0101] Device ID Wind speed (m / s) Electricity generation (kWh) date Exception marking 001 -2 500 2023-01-15 abnormal 002 25 750 2023-01-15 normal 003 30 800 2023-01-15 normal

[0102] • Determine the expected mathematical relationship between operational indicators, and verify the relevant operational indicator data according to the predefined mathematical relationship. If the relationship does not hold, mark it as an anomaly.

[0103] Objective: To identify data points that violate known indicative relationships, such as the product of hours and equivalent capacity not equaling the generated electricity.

[0104] step:

[0105] Define the relationship between indicators: Determine the expected mathematical relationship between indicators. For example, electricity generated = utilization hours ×

[0106] Equivalent capacity.

[0107] Data collection: Collect data for all relevant metrics.

[0108] Validate the relationship: For each data point, validate whether the relationship between the metrics holds. If the relationship does not hold, mark it as an anomaly.

[0109] Record anomaly information: Record detailed information about abnormal data, including the original value of the data point, the type of anomaly detected, and possible causes.

[0110] Suppose we have the following data:

[0111] Table 6: Example 3

[0112] Device ID Hours of use Equivalent capacity (kW) Electricity generated (kWh) 001 2000 1000 1800000 002 2000 1000 2500000

[0113] Verify relationship:

[0114] For device ID 001: Generated electricity = 2000 × 1000 = 2000000 kWh (normal)

[0115] For device ID 002: Generated electricity = 2000 × 1000 = 2000000 kWh (abnormal, actual is 2500000 kWh)

[0116] Table 7: Example 3 - Updated Dataset

[0117] Device ID Hours of use Equivalent capacity (kW) Electricity generated (kWh) Exception marking 001 2000 1000 1800000 normal 002 2000 1000 2500000 abnormal

[0118] • Determine the confidence range of operational indicators based on statistics, and mark operational indicator data that exceed the corresponding predefined confidence range as anomalies;

[0119] Objective: To use statistical methods to determine the confidence range of data and to mark data points that are outside the range.

[0120] step:

[0121] Choose a statistical method: Select an appropriate statistical method, such as a t-test, to determine the confidence range.

[0122] Calculate confidence ranges: Calculate the confidence ranges for each dimension using the selected method.

[0123] Mark outlier data: Mark data points that are outside the confidence range.

[0124] Example:

[0125] Suppose we have data on the grid curtailment rate, with an average of 10% and a standard deviation of 5%.

[0126] Calculate the confidence range:

[0127] 95% confidence interval: Mean ± 1.96 × Standard Deviation = 10% ± 1.96 × 5% = 10% ± 9.8%

[0128] Mark anomalous data:

[0129] Any grid curtailment rate exceeding 19.8% or falling below 0.2% will be flagged as abnormal.

[0130] • Determine the normal range of operating indicators based on machine learning models, and mark operating indicator data that exceeds the normal range as abnormal.

[0131] Objective: To use machine learning models to identify normal regions of data and mark outlier data points.

[0132] step:

[0133] Choosing a machine learning model: Select an appropriate model, such as DBSCAN or SVM.

[0134] Training the model: Train the model using normal data.

[0135] Identify the normal range: Use the model to identify the normal range of the data.

[0136] Mark abnormal data: Mark data points that fall outside the normal area.

[0137] Example:

[0138] Suppose we use the DBSCAN model to determine the normal range of data related to the operation of the facility.

[0139] Identify the normal area:

[0140] The DBSCAN model is used to determine the normal zone based on the density and distance of the station's operational data.

[0141] Mark anomalous data:

[0142] If the data for a power station's operation in a specific dimension is lower than the average for that dimension, but the power generation is much higher than the average, it is marked as an anomaly.

[0143] Abnormal data retention and marking: This invention retains and marks abnormal data while performing anomaly identification, ensuring that the abnormal data can still be traced and used in subsequent analysis, thereby avoiding the possible loss of effective information in traditional anomaly handling methods.

[0144] Anomaly Identification Based on Industry Mechanisms: Combining industry mechanisms and statistical methods, this invention can effectively identify various data anomalies, such as anomalies in key indicators like wind speed and power generation, ensuring the accuracy of data analysis. It is especially suitable for multi-source data in complex energy systems.

[0145] Based on the above embodiments, this embodiment further explains step S102:

[0146] Objective: To ensure the integrity and consistency of the data, and to provide an accurate data foundation for subsequent analysis.

[0147] step:

[0148] Identify missing values: Scan the dataset to identify which records contain missing values.

[0149] Categorical missing values: Based on the type of missing value and the context, determine which imputation strategy to use.

[0150] Imputing missing values: Use appropriate methods to impute missing values.

[0151] Tagging imputed data: The imputed records are tagged so that the original data and the imputed data can be distinguished during analysis.

[0152] Imputing missing values ​​using statistical methods or predictive models includes:

[0153] • Calculate the K nearest neighbors of the missing value using Euclidean distance, and fill in the missing value based on the average of the K nearest neighbors;

[0154] Suppose we have the following dataset, with missing temperature values:

[0155] Table 8: Example 4

[0156]

[0157]

[0158] Table 9: Example 4 - Updated Dataset

[0159] Device ID Wind speed (m / s) Temperature (°C) Electricity generation (kWh) Temperature fill mark 001 5 13.5 100 Fill 002 7 15 150 original 003 6 12 120 original

[0160] • Establish a regression prediction model based on historical data, and use the predicted values ​​calculated by the regression prediction model to fill in the missing values.

[0161] Suppose we have the following dataset, which is missing "uptime":

[0162] Table 10: Example 5

[0163] Device ID Normal operating hours Maintenance times Electricity generation (kWh) 001 Missing 1 500 002 2000 2 1000 003 1800 1 800

[0164] Filling method:

[0165] Build a regression model to predict "uptime" using "maintenance count".

[0166] Use the model to predict the missing "normal operating hours".

[0167] Table 11: Example 5 - Updated Dataset

[0168] Device ID Normal operating hours Maintenance times Electricity generation (kWh) Normal operating hours fill mark 001 Predicted value 1 500 Fill 002 2000 2 1000 original 003 1800 1 800 original

[0169] Intelligent missing value imputation: Utilizing statistical methods and model predictions to intelligently imput missing values, ensuring data integrity and consistency, while appropriately marking missing values ​​that cannot be imputed, thereby enhancing the data's usability.

[0170] Based on the above embodiments, this embodiment will provide a detailed description of step S103:

[0171] Objective: To assign a score to each record to reflect its data quality.

[0172] step:

[0173] Establish a scoring rule library: Based on the importance of the data and the nature of the imputation / labeling, establish scoring rules for different types of labels.

[0174] Assign weights: Assign a weight to each tag type; for example, "raw data" may have a higher score than "filled data".

[0175] Scoring records: Assign a score to each record based on the record's tag type and the scoring rule library.

[0176] Example:

[0177] Table 12: Scoring Rule Base

[0178] Tag type Weight Raw data 1 Imputation (statistical method) 0.8 Fill in (model predictions) 0.6 Missing value marker 0.4

[0179] Table 13: Scoring Records

[0180] Device ID Temperature fill mark Normal operating hours fill mark score 001 Fill original 0.8 002 original original 1 003 original Fill 0.6

[0181] Based on the above embodiments, this embodiment will provide a detailed description of step S104:

[0182] Objective: To extract representative samples from a dataset based on different data analysis needs.

[0183] • Determine whether sampling is necessary.

[0184] step:

[0185] Define the analysis objectives: Clarify the goals and needs of the data analysis.

[0186] Assess dataset size: Determine whether sampling is necessary based on the size of the dataset and the analysis objectives.

[0187] Determine the sampling method: If sampling is required, decide which sampling method to use.

[0188] Example:

[0189] If the analysis objective is to predict power generation and the dataset is too large, then sampling may be necessary.

[0190] • Ensure sampling of key dimension distribution characteristics

[0191] Objective: To ensure that the sampled data can represent the key features of the overall data.

[0192] step:

[0193] Identify key dimensions: Determine which dimensions are key dimensions in the analysis, such as "month" and "electric field".

[0194] Stratified sampling: stratifying the dataset according to key dimensions.

[0195] Proportional sampling: In each stratum, records with better data quality are extracted according to a certain proportion.

[0196] Suppose we have the following dataset and need to perform stratified sampling by "month" and "electric field":

[0197] Table 14 Example 6

[0198] Device ID month electric field Electricity generation (kWh) score 001 1 Electric field A 500 0.8 002 1 Electric field A 1000 1 003 2 electric field B 800 0.6 004 2 electric field B 700 1

[0199] Stratified sampling:

[0200] The data is stratified by "month" and "electric field".

[0201] Within each layer, samples are drawn from high to low scores.

[0202] Table 15 Example 6 - Sampling Results

[0203] Device ID month electric field Electricity generation (kWh) score 002 1 Electric field A 1000 1 004 2 electric field B 700 1

[0204] Data quality scoring and sampling optimization: Through scoring mechanisms and sampling methods, the quality of key dimension data is ensured. Especially when the data is used in different scenarios, efficient training sample selection is achieved, which optimizes the performance of subsequent data models.

[0205] This invention provides a method for cleaning operational index data of new energy equipment in integrated smart parks. It aims to effectively solve common data quality problems of wind power, photovoltaic, and other new energy equipment in integrated smart parks, such as outliers, missing values, and noise. It is particularly optimized for the diverse and complex data sources of new energy equipment. This invention employs four main steps: data preprocessing, anomaly detection, data repair, and reconstruction. This ensures that the cleaning process can adapt to different types of new energy equipment and their operational data characteristics, improving data quality and the reliability of analysis results.

[0206] In the data preprocessing stage, this method preserves the time series and environmental characteristics of the original data, ensuring that no key information is lost during the cleaning process. In the anomaly detection process, combining traditional statistical analysis methods (such as standard deviation analysis and box plots) with advanced machine learning algorithms (such as isolated forests and support vector machines) effectively handles nonlinear anomalies caused by weather changes, equipment aging, etc., adapting to the complex operating scenarios of new energy equipment.

[0207] The data repair component not only employs interpolation and regression analysis but also incorporates machine learning models based on real-world application scenarios. This ensures that the repaired data conforms to the physical laws of the energy system while preserving the distribution characteristics of the original data. Particularly considering the characteristics of distributed new energy systems in integrated smart parks, this method fully considers both local features and global trends when repairing missing data, thereby guaranteeing data integrity and consistency.

[0208] Furthermore, this invention specifically considers the operational characteristics of different renewable energy sources, such as wind speed and light intensity, and employs a multi-level, multi-dimensional cleaning strategy to ensure the efficiency, accuracy, and applicability of data processing. Compared with existing technologies, this invention not only improves the precision of data cleaning but also preserves the distribution characteristics of the original data to the greatest extent possible. It is suitable for large-scale, multi-source, heterogeneous renewable energy equipment operation data processing scenarios, and particularly meets the operational needs of renewable energy equipment in integrated smart parks.

[0209] This invention also provides a data cleaning device for operating indicators of new energy equipment; the specific device may include:

[0210] The data preprocessing module is used to perform unified encoding and mapping on equipment operation records from different data sources based on the enterprise affiliation model;

[0211] The data detection and repair module is used to detect outliers and fill in missing values ​​in the device operation records after unified coding mapping, and to mark them.

[0212] The data scoring module is used to score each device operation record according to weight based on the tags, using a scoring rule library;

[0213] The data sampling module is used to sample equipment operation records proportionally based on the scoring results and according to the key dimensions of the data, so as to obtain cleaned data.

[0214] The new energy equipment operation index data cleaning device in this embodiment is used to implement the aforementioned new energy equipment operation index data cleaning method. Therefore, the specific implementation of the new energy equipment operation index data cleaning device can be found in the previous embodiment section of the new energy equipment operation index data cleaning method. For example, the data preprocessing module, data detection and repair module, data scoring module, and data sampling module are used to implement steps S101, S102, S103, S104, and S105 in the above-mentioned new energy equipment operation index data cleaning method. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0215] A specific embodiment of the present invention also provides a data cleaning device for operating indicators of new energy equipment, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-mentioned data cleaning method for operating indicators of new energy equipment.

[0216] A specific embodiment of the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for cleaning data of operating indicators of new energy equipment.

[0217] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0218] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0219] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0221] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for cleaning new energy equipment operation index data, characterized in that, include: Based on the enterprise affiliation model, equipment operation records from different data sources are uniformly encoded and mapped. The device operation records after unified coding mapping are used for outlier detection and missing value filling, and are then marked. Based on the scoring rule base, each device operation record is scored according to its weight according to the tags; Based on the scoring results, the equipment operation records were sampled proportionally according to the key dimensions of the data to obtain the cleaned data.

2. The new energy equipment operation index data cleaning method according to claim 1, characterized in that, The unified encoding and mapping of equipment operation records from different data sources based on the enterprise affiliation model includes: Obtain information on all equipment, sites, subsidiaries, and parent companies of the enterprise, and determine the hierarchy and relationship of each entity based on the enterprise's organizational structure to construct an enterprise affiliation model; Based on the enterprise affiliation model, a unique code is assigned to each device, site, subsidiary, and parent company, and all enterprise entity names in the original dataset are replaced with the corresponding codes. Retrieve the manufacturer names appearing in all data sources, and perform deduplication and normalization on the manufacturer names; Assign a unique code to each unique manufacturer name; In the original dataset, all manufacturer names were replaced with their corresponding codes; Create and maintain an encoding mapping table so that the original name can be queried when needed.

3. The new energy equipment operation index data cleaning method according to claim 1, characterized in that, Anomaly detection and marking of the device operation records after the unified encoding mapping includes: Based on industry mechanisms, a normal range is defined for each operational indicator, and operational indicator data that exceeds the corresponding predefined normal range is marked as abnormal. Determine the expected mathematical relationship between operational indicators, and verify the relevant operational indicator data according to the predefined mathematical relationship. If the relationship does not hold, mark it as an anomaly. Based on statistics, the confidence range of the operating indicators is determined, and operating indicator data that exceed the corresponding predefined confidence range are marked as anomalies; The normal range of operating indicators is determined based on a machine learning model, and operating indicator data that exceeds the normal range is marked as abnormal.

4. The new energy equipment operation index data cleaning method according to claim 1, characterized in that, The missing value imputation and marking of the device operation records after the unified encoding mapping includes: Scan and retrieve missing values; Missing values ​​are imputed using statistical methods or predictive models, and the original values, imputed data, and unimputed data are classified and labeled.

5. The new energy equipment operation index data cleaning method according to claim 4, characterized in that, The use of statistical methods or predictive models for missing value imputation includes: The K nearest neighbors of a missing value are calculated using Euclidean distance, and the missing value is filled in based on the average of the K nearest neighbors. A regression prediction model is established based on historical data, and the missing values ​​are filled in using the predicted values ​​calculated by the regression prediction model.

6. The new energy equipment operation index data cleaning method according to claim 1, characterized in that, The step of scoring each device operation record according to the weight based on the scoring rule base and the tag includes: Assign weights to the data based on the label type and build a scoring rule library; Each device operation record is scored based on the weights and the scoring rule library.

7. The new energy equipment operation index data cleaning method according to claim 1, characterized in that, Based on the scoring results, the data is stratified according to key dimensions, and the equipment operation records are sampled proportionally to obtain the cleaned data, which includes: Identify the key dimensions of the data; The dataset is stratified according to the key dimensions of the data. Within each layer, a certain proportion of the equipment operation records with high scores are randomly selected.

8. A new energy equipment operation index data cleaning device, characterized in that, include: The data preprocessing module is used to perform unified encoding and mapping on equipment operation records from different data sources based on the enterprise affiliation model; The data detection and repair module is used to detect outliers and fill in missing values ​​in the device operation records after unified coding mapping, and to mark them. The data scoring module is used to score each device operation record according to weight based on the tags, using a scoring rule library; The data sampling module is used to sample equipment operation records proportionally based on the scoring results and according to the key dimensions of the data, so as to obtain cleaned data.

9. A new energy equipment operation index data cleaning device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the new energy equipment operation index data cleaning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data cleaning method for operating indicators of new energy equipment as described in any one of claims 1 to 7.

Images