Adaptive cleaning method and system for full-link cross-regional energy big data

Abstract

The application provides a kind of self-adaptive cleaning method and system for full-link cross-energy big data, the method comprises: collecting cross-energy big data and classifying;Adopt isolated forest anomaly detection to identify and eliminate abnormal samples in each category data;For different data categories, construct adaptive hyperparameter search space, use Bayesian optimization algorithm to optimize the hyperparameters of SAITS missing value completion model, obtain the optimal hyperparameter combination matched with each data category;Call the SAITS model under the corresponding parameter configuration, through the double-branch encoder architecture, diagonal line mask multi-head self-attention mechanism and joint optimization target, iteratively complete the missing values in each data category, and output the cleaned cross-energy big data set.The application can adaptively process the complex quality problems of multiple energy varieties and full-service link data, effectively guarantee the integrity, accuracy and availability of cross-energy data.

Description

Technical Field

[0001] This invention relates to the field of data cleaning technology, and in particular to an adaptive cleaning method and system for cross-domain energy big data across the entire data chain. Background Technology

[0002] With the comprehensive advancement of my country's energy digital transformation and the in-depth implementation of the "dual carbon" target, the energy system is accelerating its evolution towards a new energy system characterized by multi-energy complementarity and coordinated development of energy sources, grids, loads, and storage. The deep integration of multiple energy sources, including electricity, oil, and natural gas, and the explosive growth of data across all business processes, including production, transmission, and consumption, have made energy big data a core production factor driving the high-quality development of the energy industry. The fusion, governance, and quality improvement of multi-source heterogeneous energy data are the core foundation for breaking down cross-domain and cross-process data barriers in the energy industry, achieving unified management and efficient utilization of data resources, and directly determining the accuracy and reliability of upper-level applications such as intelligent scheduling, market transaction optimization, demand-side response, and security risk early warning in the energy system.

[0003] However, current energy data exhibits significant characteristics of multi-source heterogeneity and end-to-end distribution: data collection standards for different energy types such as electricity, oil, and natural gas vary considerably, formats differ significantly, and storage systems are independent; during data collection across different business stages such as production, transmission, and consumption, factors such as equipment failures, network fluctuations, and human error result in widespread and complex quality issues including data noise, missing data, inconsistencies, and redundancy. These problems hinder the effective integration and sharing of cross-domain data, preventing the full realization of data value and severely restricting the in-depth advancement of energy digital transformation. Therefore, conducting research on multi-source heterogeneous energy data fusion governance and quality improvement technologies, and constructing an efficient, intelligent, and integrated end-to-end data processing system, has become a critical technological issue urgently needing to be addressed in the current energy sector.

[0004] While some progress has been made in research on technologies related to energy data governance and quality improvement, several limitations remain. Traditional data cleaning and preprocessing methods are largely based on manually defined rules and statistical thresholds, capable of handling only known, single-type data quality issues and failing to adapt to the diverse data characteristics of different energy sources and business processes. Some data processing methods incorporating machine learning techniques primarily focus on localized data quality repair within a single energy system or business process, without in-depth analysis of the common characteristics and patterns of differences in multi-source, heterogeneous energy data. Furthermore, existing technologies generally lack a collaborative cleaning architecture covering the entire energy value chain; anomaly identification, data cleaning, and data completion are fragmented, resulting in low processing efficiency and a lack of self-learning and self-optimization capabilities, making it difficult to address the new data quality challenges brought about by dynamic changes in energy systems.

[0005] In summary, existing technologies cannot achieve integrated governance and intelligent quality improvement of data across multiple energy sources, including electricity, oil, and natural gas, as well as across all business processes from production and transmission to consumption. They struggle to guarantee data integrity, accuracy, and availability, failing to provide a solid and reliable data foundation for subsequent energy data value analysis and higher-level intelligent applications. Summary of the Invention

[0006] The purpose of this invention is to provide an adaptive cleaning method and system for cross-domain energy big data across the entire energy chain, aiming to solve at least one of the problems in the background art.

[0007] In a first aspect, the present invention provides an adaptive cleaning method for cross-domain energy big data across the entire energy supply chain, the method comprising: Collect cross-domain energy big data covering the entire energy chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results; An anomaly detection model is used to identify and remove abnormal samples in the data to be processed, thereby obtaining a normal sample set; Data feature descriptors are extracted from the normal sample set, and a hyperparameter search space is adaptively constructed for each data category based on the data feature descriptors. A Bayesian optimization algorithm is then used for adaptive optimization to obtain the optimal hyperparameter combination that matches each data category. Based on the optimal hyperparameter combination, missing values ​​of data in each data category of the normal sample set are filled in to obtain the cleaned cross-domain energy big data set.

[0008] Secondly, this invention provides an adaptive cleaning system for cross-domain energy big data across the entire energy supply chain, the system comprising: The data acquisition module is used to collect cross-domain energy big data across the entire chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results. The abnormal sample removal module is used to identify and remove abnormal samples in the data to be processed using an anomaly detection model, so as to obtain a normal sample set. The optimization module is used to extract data feature descriptors from the normal sample set, adaptively construct a hyperparameter search space for each data category based on the data feature descriptors, and use a Bayesian optimization algorithm to perform adaptive optimization in order to obtain the optimal hyperparameter combination that matches each data category. The cleaning execution module is used to fill in the missing values ​​of data in each data category of the normal sample set according to the optimal hyperparameter combination, so as to obtain the cleaned cross-domain energy big data set.

[0009] Thirdly, the present invention provides a storage medium that stores one or more programs, which, when executed by a processor, implement the above-described adaptive cleaning method for cross-domain energy big data across the entire value chain.

[0010] Fourthly, the present invention provides an electronic device, the electronic device comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the above-mentioned adaptive cleaning method for cross-domain energy big data across the entire chain.

[0011] Compared with the prior art, the present invention has the following advantages: This invention achieves integrated intelligent cleaning of data from multiple energy types and all business processes through isolated forest anomaly detection, adaptive hyperparameter search and Bayesian optimization, as well as SAITS iterative completion that integrates diagonal mask self-attention and bi-branch joint optimization. This effectively ensures the integrity, accuracy and availability of cross-domain energy data and provides a reliable data foundation for upper-level intelligent applications. Attached Figure Description

[0012] Figure 1 This is a flowchart of an adaptive cleaning method for cross-domain energy big data across the entire value chain, as proposed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an adaptive cleaning system for cross-domain energy big data across the entire value chain, as proposed in an embodiment of the present invention.

[0013] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects.

[0015] like Figure 1As shown, an embodiment of the present invention proposes an adaptive cleaning method for cross-domain energy big data across the entire value chain. The method includes steps S101 to S104, wherein: Step S101: Collect cross-domain energy big data for the entire chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results; It should be noted that in this step, daily data from the power system, oil system, and natural gas system are collected separately. The daily power system data includes three dimensions: generation, grid, and consumption. Generation data includes daily power generation of various types of generators, daily utilization hours of generator equipment, daily curtailment of wind and solar power from renewable energy units, and daily available capacity of generators. Grid data includes daily maximum load, minimum load, and daily average load factor of the entire grid, daily transmission volume, daily line loss rate and regional line loss data, daily voltage qualification rate, frequency qualification rate, and daily availability of transformers and transmission lines, as well as the number and duration of outages. Consumption data includes daily electricity consumption by industry, daily peak-to-valley difference and load curve characteristic parameters, daily demand response execution volume and response rate.

[0016] The daily data of the petroleum system includes three dimensions: production, storage and transportation, and sales. Production data includes daily crude oil production, imports, and inventory; daily refining volume and refined product yield; daily number of oil wells in operation; average daily production; and overall decline rate. Storage and transportation data includes daily crude oil and refined product pipeline transportation volume; daily oil depot inventory and turnover days; and daily transportation loss rate. Sales data includes daily sales volume by product type, daily wholesale and retail prices and price differences, and daily gas station operating efficiency indicators.

[0017] The daily data for the natural gas system includes three dimensions: supply side, pipeline network side, and consumption side. Supply side data includes daily natural gas production, import volume, and total supply, daily gas injection and extraction volume and working gas volume in gas storage facilities, and daily gas source structure ratio. Pipeline network data includes daily pipeline gas transmission volume, pipeline gas storage volume, daily pipeline network pressure monitoring average and extreme values, daily gas transmission efficiency and transmission deviation rate of distribution stations. Consumption side data includes daily consumption by user type, deviation between daily designated volume and actual extraction volume, and daily seasonal consumption fluctuation coefficient.

[0018] The collected data is systematically categorized according to the hierarchical structure of energy type, business process, and data dimension to construct a cross-domain energy dataset to be processed. Where k1 is the total number of data categories after partitioning. This refers to the data in the k1th data category.

[0019] Step S102: Use an anomaly detection model to identify and remove abnormal samples in the data to be processed to obtain a normal sample set; It should be noted that this applies to the data of the kth data category. The sample set in Construct T isolated trees for each sample. Calculate its path length in a single isolated tree. The global anomaly score for a sample is defined as the number of edges traversed from the root node to an external node. Calculate using the following formula: ; ; in, For the nth sample, For the sample The expected value of the path length in all isolated trees; For a given sample size n, the standardization factor for the path length. For harmonic numbers, is Euler's constant.

[0020] The value range is (0,1), and the closer its value is to 1, the stronger the sample. The higher the global anomaly level, the higher the initial global anomaly screening threshold should be. , will satisfy Samples that were classified as abnormal were removed from the abnormal set; the remaining samples were marked as normal samples. Then proceed to the next processing step.

[0021] Step S103: Extract data feature descriptors from the normal sample set, and adaptively construct a hyperparameter search space for each data category based on the data feature descriptors. Then, use a Bayesian optimization algorithm to perform adaptive optimization to obtain the optimal hyperparameter combination that matches each data category. It should be noted that in this step, the first step is to extract... Data feature descriptors ,in, For sample size, For feature dimension, The missing rate, denoted as the proportion of outlier samples, and T as the transpose.

[0022] Based on this feature descriptor, a hyperparameter search space is adaptively constructed for each data category. : ; Where d is the hyperparameter dimension. The search space boundary is dynamically adjusted based on data characteristics.

[0023] Then, the interpolation error of the SAITS interpolation model on the validation set is used as the objective function for Bayesian optimization. For data categories... Construct a missing mask M for verification. Given true and complete data, provide hyperparameter configurations. The interpolation results were obtained after training. The objective function uses the normalized mean absolute error: ; The smaller the objective function value, the better the hyperparameter configuration. The higher the interpolation accuracy, the better. The goal of Bayesian optimization is to find the method that maximizes interpolation accuracy. Minimize the optimal combination of hyperparameters.

[0024] Furthermore, Bayesian optimization constructs an objective function. The probabilistic proxy model utilizes observed hyperparameter-performance data. This predicts the performance distribution in unexplored regions, thereby efficiently locating the optimal solution. In each iteration, the expected improvement criterion is used to select the next set of hyperparameters to be evaluated. ; In the formula, For the function to be improved, let represent the configuration of candidate hyperparameters. Below, the model performance is expected to exceed the expected improvement of the current best observations. This is the current optimal observation value. and Configuration for proxy models The performance prediction mean and standard deviation, and Let Z be the cumulative distribution function and probability density function of the standard normal distribution, respectively, and Z be the standardized variable of the standard normal distribution. To explore and utilize trade-off parameters, adaptively adjust based on the missing data category rate: ; In the formula, Basic exploration - utilizing trade-off parameters, The missing rate, This is the reference missing rate. The higher the missing rate, The smaller the value, the more likely it is to perform a fine-grained search in the current optimal neighborhood to quickly obtain reliable performance; the lower the missing value, the better. The larger the value, the more it encourages exploration of uncharted territories to discover optimal configurations. Each time, by maximizing... Determine the next set of hyperparameters, substitute them into SAITS for training, update the observation dataset, and iterate to optimize.

[0025] Convergence determination and optimal hyperparameter output. Repeat the above optimization process until any of the following conditions are met: (1) the number of iterations reaches the upper limit; (2) the improvement of the optimal value in multiple consecutive iterations is less than the threshold; (3) the EI value is less than the exploration threshold. After convergence, output the optimal hyperparameter combination: ; The above process is executed independently for each data category to obtain the optimal hyperparameter combination that matches its data characteristics. .

[0026] Step S104: Based on the optimal hyperparameter combination, fill in the missing values ​​of the data in each data category of the normal sample set to obtain the cleaned cross-domain energy big data set.

[0027] It should be noted that in this step, the data... Reconstruct it into a multivariate time series tensor ,in, For time step, For feature dimensions. Construction and Binary mask matrix of the same dimension Its elements are defined as: ; To avoid introducing initial bias due to missing values ​​during model forward propagation, an adaptive initial imputation strategy is employed. Based on the detected data distribution characteristics of this category, initial values ​​are estimated for the missing locations: ; In the formula, For the first k The input tensor after adaptive initial padding for each data category; The initial filling tensor is used. For stationary features in the energy data, the median value along the feature dimension is used for filling; for features with obvious temporal trends, linear interpolation is used for filling; and for highly volatile features, zero values ​​are used for filling. This adaptive strategy ensures that the initial filling values ​​are consistent with the physical characteristics of the energy data, accelerating subsequent model convergence.

[0028] Next, the tensor after initial filling Inputting the SAITS model: First, the feature vector at each time step t is mapped to the input embedding layer. Hidden space, then, hidden state Passing in sequence Layer Transformer encoder, first The output of the layer encoder It consists of stacked multi-head self-attention sub-layers and feedforward network sub-layers; Introducing a diagonal mask matrix in multi-head self-attention computation The diagonal mask matrix is ​​used to mask the main diagonal elements of the attention weight matrix, so that the hidden state of each time step is obtained only by aggregating the context information of the other time steps. The SAITS model employs a two-branch output structure with an interpolation branch and an observation branch. The two branches are based on the interpolation branch and the observation branch, respectively. The final hidden state output by the stacked Transformer encoder layers is used to obtain the interpolation values ​​for all positions generated by the interpolation branch based on the global temporal context through independent linear projection heads. And the reconstructed values ​​of the observation locations generated by the observation branch based on the non-missing location context. ; The joint optimization objective of the SAITS model is a weighted sum of the interpolation loss and the observation loss, plus a regularization term. ; in, To compensate for the loss, To observe the loss, Optimize the branch weight hyperparameters for the Bayesian output; The regularization coefficient is used. The L2 norm of all trainable parameters of the model. The total loss function for the bi-branch joint optimization of the SAITS model; By minimizing This generates imputation results that are physically consistent with the missing values; In the s-th iteration, the interpolation result from the previous round is used to fill the missing positions and merged with the original data at the observation positions as the input for the current round. The tensor filling, encoding, diagonal mask attention calculation, and joint optimization processes are repeated until the convergence condition is met or the maximum number of iterations is reached. The missing values ​​are then filled with the final interpolation result after convergence, resulting in the cleaned and completed data category. ; Thus, the above process is performed on all data categories to output a cleaned cross-domain energy big data dataset.

[0029] In summary, this invention achieves integrated intelligent cleaning of data from multiple energy types and all business processes through isolated forest anomaly detection, adaptive hyperparameter search and Bayesian optimization, as well as SAITS iterative completion that integrates diagonal mask self-attention and bi-branch joint optimization. This effectively ensures the integrity, accuracy and availability of cross-domain energy data and provides a reliable data foundation for upper-level intelligent applications.

[0030] like Figure 2 As shown, one embodiment of the present invention proposes an adaptive cleaning system for cross-domain energy big data across the entire value chain. The system includes: The data acquisition module 10 is used to collect cross-domain energy big data for the entire chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results. The abnormal sample removal module 20 is used to identify and remove abnormal samples in the data to be processed using an anomaly detection model to obtain a normal sample set. The optimization module 30 is used to extract data feature descriptors from the normal sample set, adaptively construct a hyperparameter search space for each data category based on the data feature descriptors, and use a Bayesian optimization algorithm to perform adaptive optimization in order to obtain the optimal hyperparameter combination that matches each data category. The cleaning execution module 40 is used to fill in the missing values ​​of the data in each data category of the normal sample set according to the optimal hyperparameter combination, so as to obtain the cleaned cross-domain energy big data set.

[0031] In another aspect, the present invention also proposes a storage medium on which one or more programs are stored, which, when executed by a processor, implement the above-described adaptive cleaning method for cross-domain energy big data across the entire chain.

[0032] In another aspect, the present invention also proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program and the processor is used to execute the computer program stored in the memory to realize the above-mentioned adaptive cleaning method for cross-domain energy big data across the entire chain.

[0033] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0034] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0035] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0036] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. An adaptive cleaning method for cross-domain energy big data across the entire value chain, characterized in that, The method includes: Collect cross-domain energy big data covering the entire energy chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results; An anomaly detection model is used to identify and remove abnormal samples in the data to be processed, thereby obtaining a normal sample set; Data feature descriptors are extracted from the normal sample set, and a hyperparameter search space is adaptively constructed for each data category based on the data feature descriptors. A Bayesian optimization algorithm is then used for adaptive optimization to obtain the optimal hyperparameter combination that matches each data category. Based on the optimal hyperparameter combination, missing values ​​of data in each data category of the normal sample set are filled in to obtain the cleaned cross-domain energy big data set.

2. The adaptive cleaning method for cross-domain energy big data across the entire value chain according to claim 1, characterized in that, The steps of collecting cross-domain energy big data across the entire energy supply chain, classifying the cross-domain energy big data into data categories, and constructing a dataset to be processed based on the classification results include: The collected data is categorized according to a hierarchical structure based on energy type, business process, and data dimension to construct a cross-domain energy dataset to be processed. Where k1 is the total number of data categories after partitioning. This refers to the data in the k1th data category.

3. The adaptive cleaning method for cross-domain energy big data across the entire value chain according to claim 2, characterized in that, The step of using an anomaly detection model to identify and remove abnormal samples in the data to be processed, and obtaining a normal sample set, includes: For the data of the kth data category The sample set in Construct T isolated trees for each sample. Calculate its path length in a single isolated tree. The global anomaly score for a sample is defined as the number of edges traversed from the root node to an external node. Calculate using the following formula: ； ； in, For the nth sample, For the sample The expected value of the path length in all isolated trees; For a given sample size n, the standardization factor for the path length. For harmonic numbers, Here is Euler's constant; Set global anomaly initial screening threshold , will satisfy The abnormal samples were classified into the outlier set and removed, while the remaining samples were marked as normal samples. .

4. The adaptive cleaning method for cross-domain energy big data across the entire value chain according to claim 3, characterized in that, The steps of extracting data feature descriptors from the normal sample set, adaptively constructing a hyperparameter search space for each data category based on the data feature descriptors, and using a Bayesian optimization algorithm for adaptive optimization to obtain the optimal hyperparameter combination matching each data category include: extract Data feature descriptors ,in, For sample size, For feature dimension, The missing rate, The percentage of outlier samples is represented by T, which is the transpose of the sample. Adaptively construct the hyperparameter search space for the data of the k-th data category. : ； Where d is the hyperparameter dimension. This indicates that the data is for the k-th data category. The lower bound of the search space for its j-th dimension hyperparameter, This indicates that the data is for the k-th data category. The upper bound of the search space for its j-th dimension hyperparameter; The interpolation error of the SAITS interpolation model on the validation set is used as the objective function for Bayesian optimization. ； in, For the Bayesian optimization objective function corresponding to the k-th data category, The result is the interpolation result, and M is the verification missing mask. For true and complete data, Configure hyperparameters; In each iteration, the expected improvement criterion is used to select the next set of hyperparameters to be evaluated: ； in, For the function to be improved, it means the hyperparameter configuration. Below, the model performance is expected to exceed the expected improvement of the current best observations. This is the current optimal observation value. and Configuration for proxy models The performance prediction mean and standard deviation, and Let Z be the cumulative distribution function and probability density function of the standard normal distribution, respectively, and Z be the standardized variable of the standard normal distribution. To explore and utilize trade-off parameters, adaptively adjust based on the missing data rate for each data category. Basic exploration - utilizing trade-off parameters, The missing rate, Used as a reference missing rate; Repeat the above optimization process until any of the following conditions are met: (1) The number of iterations reaches the upper limit; (2) The improvement of the optimal value in multiple consecutive iterations is less than the threshold; (3) The EI value is lower than the exploration threshold. After convergence, the optimal hyperparameter combination is output: ； in, For the optimal combination of hyperparameters, For the first One hyperparameter configuration.

5. The adaptive cleaning method for cross-domain energy big data across the entire value chain according to claim 4, characterized in that, The step of completing the missing values ​​of data in each data category of the normal sample set according to the optimal hyperparameter combination to obtain the cleaned cross-domain energy big data set includes: Will Reconstructed into multivariate time series tensors ,in, For time step, As the feature dimension, and construct with Binary mask matrix of the same dimension Its elements are defined as: ； in, This represents the missing state marker value at the j-th feature dimension position at the t-th time step in the binary mask matrix corresponding to the k-th data category. This represents the original data element at the t-th time step and the j-th feature dimension position in the multivariate time series tensor corresponding to the k-th data category; Initial value estimation for missing locations: ； in, For the first k The input tensor after adaptive initial padding for each data category; This is the initial filling tensor; The tensor after initial filling Inputting the SAITS model: First, the feature vector at each time step t is mapped to the input embedding layer. Hidden space, then, hidden state Passing in sequence Layer Transformer encoder, first The output of the layer encoder It consists of stacked multi-head self-attention sub-layers and feedforward network sub-layers; Introducing a diagonal mask matrix in multi-head self-attention computation The diagonal mask matrix is ​​used to mask the main diagonal elements of the attention weight matrix, so that the hidden state of each time step is obtained only by aggregating the context information of the other time steps. The SAITS model employs a two-branch output structure with an interpolation branch and an observation branch. The two branches are based on the interpolation branch and the observation branch, respectively. The final hidden state output by the stacked Transformer encoder layers is used to obtain the interpolation values ​​for all positions generated by the interpolation branch based on the global temporal context through independent linear projection heads. And the reconstructed values ​​of the observation locations generated by the observation branch based on the non-missing location context. ; The joint optimization objective of the SAITS model is a weighted sum of the interpolation loss and the observation loss, plus a regularization term. ； in, To compensate for the loss, To observe the loss, Optimize the branch weight hyperparameters for the Bayesian output; The regularization coefficient is used. The L2 norm of all trainable parameters of the model. The total loss function for the bi-branch joint optimization of the SAITS model; By minimizing This generates imputation results that are physically consistent with the missing values; In the s-th iteration, the interpolation result from the previous round is used to fill the missing positions and merged with the original data at the observation positions as the input for the current round. The tensor filling, encoding, diagonal mask attention calculation, and joint optimization processes are repeated until the convergence condition is met or the maximum number of iterations is reached. The missing values ​​are then filled with the final interpolation result after convergence, resulting in the cleaned and completed data category. .

6. An adaptive cleaning system for cross-domain energy big data across the entire value chain, characterized in that, The system includes: The data acquisition module is used to collect cross-domain energy big data across the entire chain, classify the cross-domain energy big data into data categories, and construct a dataset to be processed based on the classification results. The abnormal sample removal module is used to identify and remove abnormal samples in the data to be processed using an anomaly detection model, so as to obtain a normal sample set. The optimization module is used to extract data feature descriptors from the normal sample set, adaptively construct a hyperparameter search space for each data category based on the data feature descriptors, and use a Bayesian optimization algorithm to perform adaptive optimization in order to obtain the optimal hyperparameter combination that matches each data category. The cleaning execution module is used to fill in the missing values ​​of data in each data category of the normal sample set according to the optimal hyperparameter combination, so as to obtain the cleaned cross-domain energy big data set.

7. A storage medium, characterized in that, The storage medium stores one or more programs, which, when executed by a processor, implement the adaptive cleaning method for cross-domain energy big data as described in any one of claims 1-5.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the adaptive cleaning method for cross-domain energy big data according to any one of claims 1-5.

Images