Multi-format data management method and platform for smart comprehensive energy

Abstract

The application provides a multi-industry data management method and platform for smart comprehensive energy, and relates to the field of big data processing.The method comprises the following steps: collecting original industry data of different sources and formats to perform data standardization processing and obtain standard industry data; performing metadata management on the standard industry data to obtain multi-industry data, wherein the multi-industry data comprises observation variables, potential state data and hidden state parameters; constructing an initial generation model and an initial inference model according to the observation variables, the potential state data and the hidden state parameters, training the initial inference model based on an encoder of a variational autoencoder and a recurrent neural network, training the initial generation model based on a decoder of the variational autoencoder and the recurrent neural network, and obtaining an optimized inference model and an optimized generation model; constructing a robust deep state space model based on the optimized inference model, the optimized generation model, an initial deep state space model and a variational lower bound function; and inputting the multi-industry data into the robust deep state space model to perform quality detection and repair on the multi-industry data.The application is helpful for realizing collaborative processing of multi-industry data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data processing, and in particular to a multi-format data management method and platform for smart comprehensive energy. BACKGROUND

[0002] The performance of big data analysis depends on the quality of data, and the correct use of high-quality data can help make better predictions and decisions, and more reliable data analysis. The success of big data analysis depends largely on how data is cleaned, transformed and integrated. With the development of society, the amount of data in various industries also gradually increases exponentially, and the sources of data are also more and more complex and numerous. In practical applications, data often contains incomplete, incorrect or irrelevant data, and data inaccuracy is caused by many reasons. In particular, when large-scale data from different sources (such as multi-energy variety data) are integrated, the quality is particularly worrying; these data sources are usually derived from homogeneous or heterogeneous databases, file systems and service interfaces, thus reducing the reliability of data.

[0003] Chinese patent with publication number CN115729993A discloses a method and system for unified management of multi-source heterogeneous data based on metadata. The method includes: in response to the operation of uploading metadata of multi-source heterogeneous data by a data production user, converting the metadata to a running state; wherein the metadata is data obtained by the data production user based on the preset metadata definition rule to characterize the multi-source heterogeneous data; configuring corresponding permissions for users with different roles; in response to the compilation operation of the data directory by the platform management user and / or the data production user, establishing the association between the data directory and the data object according to the compilation operation, and forming a classified and layered data directory; returning the corresponding data attribute information and content information to the data consumption user. However, the above-mentioned scheme cannot realize complete data fusion for multi-element data, and the processing and processing capacity of data only stays within a single category of data, and cannot complete analysis and comparison between different categories of data. Therefore, it is necessary to provide a multi-format data management method and platform for smart comprehensive energy to realize collaborative processing of multi-format data. SUMMARY

[0004] Therefore, the present application provides a multi-format data management method and platform for smart comprehensive energy to realize collaborative processing of multi-format data.

[0005] The present application provides a multi-format data management method for smart comprehensive energy, which comprises:

[0006] Collecting original format data of different sources and formats to perform data standardization processing to obtain standard format data;

[0007] Performing metadata management on the standard format data to obtain multi-format data, wherein the multi-format data includes observation variables, latent state data and hidden state parameters;

[0008] According to the observation variables, the latent state data and the hidden state parameters, an initial generation model and an initial inference model are respectively constructed, and the initial inference model is trained based on the encoder of the variational autoencoder and the recurrent neural network, and the initial generation model is trained based on the decoder of the variational autoencoder and the recurrent neural network, to obtain an optimized inference model and an optimized generation model respectively;

[0009] Based on the optimized inference model, the optimized generation model, an initial deep state space model and a variational lower bound function, a robust deep state space model is constructed;

[0010] The multi-format data is input into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0011] On the basis of the above technical solutions, preferably, the collecting of original format data of different sources and formats to perform data standardization processing to obtain standard format data specifically includes:

[0012] A data collection network is established, and the original format data is collected according to the data collection network;

[0013] The original format data is preprocessed to obtain transition format data;

[0014] A standard data model is constructed, and the transition format data is standardized and converted to map the transition format data to the standard data model to obtain the standard format data.

[0015] On the basis of the above technical solutions, preferably, the construction process of the initial deep state space model includes:

[0016] Based on the time series obtained by the data collection network, a corresponding spatial distribution condition is generated for the observation variables in each time series and the latent state data corresponding to the observation variables;

[0017] According to the spatial distribution condition and the observation variables and latent state data at different time steps in the time series, an initial deep state space model is constructed.

[0018] More preferably, the expression of the spatial distribution condition is:

[0019] ;

[0020] wherein, x 1:t denotes the observation sequence from time step 1 to T, z 1:t denotes the latent state data from time step 1 to T, denotes the conditional probability distribution function of the observation sequence, Π denotes the continuous product, denotes the state transition model corresponding to the latent state data in the time series, denotes the first observation variable generation function corresponding to the latent state data.

[0021] More preferably, the expression of the variational lower bound function is:

[0022] ;

[0023] wherein, denotes the variational lower bound function, θ denotes the output parameter of the generation model in the variational autoencoder, denotes the output parameter of the inference model in the variational autoencoder, denotes the expectation of the approximate posterior distribution in the inference model, denotes the likelihood probability function of the generation model in the variational autoencoder, i.e. the probability of observing data z when the latent variable is given x , KL( ) denotes the KL divergence function, denotes the approximate posterior distribution of the inference model in the variational autoencoder, denotes the prior distribution of the generation model in the variational autoencoder.

[0024] More preferably, the expression of the robust deep state-space model is:

[0025] ;

[0026] ;

[0027] wherein, x t denotes the observation variable at time step t, Φ( ) denotes the probability distribution function, z t denotes the latent state data at time step t, μ θz ( ) denotes the mean function of the state transition Gaussian distribution, Σ θz ( ) denotes the covariance function, h tdenotes the recurrent neural network hidden state parameter at time step t, δ( ) denotes the delta distribution function, f θh ( ) denotes the transition function of the recurrent neural network, denotes the joint distribution of the observed variables, the latent state data and the hidden state parameter in the recurrent neural network in the robust deep state space model, p θx ( ) denotes the parameterized distribution function of the emission model, p θz ( ) denotes the parameterized distribution function of the state transition model, p θz ( ) denotes the transition function of the recurrent neural network, Π denotes the continuous product.

[0028] More preferably, the method further comprises:

[0029] representing the conditional probability distribution function of the observed sequence by a t-distribution, and generalizing the location parameter and the scale parameter of the conditional probability distribution function represented by the t-distribution to obtain a conditional probability distribution function based on the t-distribution;

[0030] When the degree of freedom parameter, the location parameter and the scale parameter in the conditional probability distribution function based on the t-distribution satisfy a preset constraint condition, constructing a second observed variable generation function according to the latent state data and the hidden state parameter.

[0031] More preferably, the expression of the conditional probability distribution function based on the t-distribution is:

[0032] ;

[0033] wherein, denotes the probability density function based on the t-distribution, υ denotes the degree of freedom parameter, μ denotes the location parameter of the t-distribution, σ denotes the scale parameter of the t-distribution, and Γ( ) denotes the gamma function.

[0034] More preferably, the expression of the second observed variable generation function is:

[0035] ;

[0036] wherein, denotes the second observed variable generation function, x t denotes the observed variable at time step t, z t denotes the latent state data at time step t, ht denotes a recurrent neural network hidden state parameter at time step t, ST() denotes a tail re-distribution function, NN 1() denotes a network function for computing a distribution mean, NN 2() denotes a network function for computing a distribution scale, NN 3() denotes a network function for computing a t-distribution degree of freedom parameter, SoftPlus () denotes an activation function.

[0037] In a second aspect of the present application, a multi-format data management platform for smart comprehensive energy is provided, the multi-format data management platform comprises a data acquisition management module, a model construction optimization module and a data monitoring repair module, wherein,

[0038] The data acquisition management module is used for acquiring original multi-format data of different sources and formats for data standardization processing, obtaining standard multi-format data, performing metadata management on the standard multi-format data, and obtaining multi-format data, wherein the multi-format data comprises observation variables, latent state data and hidden state parameters;

[0039] The model construction optimization module is used for constructing an initial generation model and an initial inference model according to the observation variables, the latent state data and the hidden state parameters respectively, training the initial inference model based on an encoder of a variational autoencoder and a recurrent neural network, training the initial generation model based on a decoder of the variational autoencoder and the recurrent neural network, to obtain an optimized inference model and an optimized generation model respectively, and constructing a robust deep state space model based on the optimized inference model, the optimized generation model, an initial deep state space model and a variational lower bound function;

[0040] The data monitoring repair module is used for inputting the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0041] The multi-format data management method and platform for smart comprehensive energy provided by the present application have the following beneficial effects compared with the prior art:

[0042] (1) By standardizing original multi-format data of different sources and formats, data standardization and unified management are realized, and by using a variational autoencoder and a recurrent neural network to optimize inference and generation models, the robust deep state space model constructed can better process noise and abnormalities in multi-format data, has the regularization effect of the variational autoencoder and the memory mechanism of the recurrent neural network, can utilize long-term dependence information and help prevent overfitting, and can realize collaborative processing of multi-format data and capture the mutual relationship between different energy forms;

[0043] (2) By combining time series data and spatial distribution conditions, the modeling of data time and spatial characteristics is realized at the same time, and the spatial distribution conditions are generated according to the actual observation data, which improves the adaptability of the robust deep state space model to different types of time series data, and the combination of time series and spatial distribution enables the robust deep state space model to analyze different time and spatial scales at the same time, significantly improves the performance, adaptability and interpretability of the robust deep state space model, and enhances the generalization ability and anomaly detection ability of the robust deep state space model, and improves the state estimation accuracy and prediction performance. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A step schematic diagram of a multi-format data management method for smart comprehensive energy provided by the present application;

[0046] Figure 2 A structure schematic diagram of an optimization generation model provided by the present application;

[0047] Figure 3 A structure schematic diagram of an optimization inference model provided by the present application;

[0048] Figure 4 A structure schematic diagram of a multi-format data management platform provided by the present application.

[0049] The figure mark explanation: 1, communication protocol dynamic adaptation device; 11, data acquisition management module; 12, model construction optimization module; 13, data monitoring repair module. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] Before introducing the embodiments of the present application, first, some terms and their abbreviations involved in the embodiments of the present application are defined and explained.

[0052] Variational Autoencoder (VAE) is a generative model that combines the advantages of deep learning and probabilistic graphical models. VAE mainly consists of two parts: an encoder and a decoder. The encoder is used to map the input data x to the distribution parameters (usually mean μ and variance σ) of the latent space, and the decoder is used to reconstruct the original data from the sampled vector z reconstruct the original data x .

[0053] The following is a detailed description of the present application in conjunction with the accompanying Figures 1-4 The present application is further described in detail.

[0054] Reference Figure 1 The present application provides a multi-format data management method for smart comprehensive energy, which comprises steps S1-S5.

[0055] Step S1, collect different sources and formats of original format data for data standardization processing, and obtain standard format data.

[0056] In this embodiment, by identifying all relevant data sources (such as different business systems, external data providers, etc.), and evaluating the data quality, format, update frequency and other characteristics of each data source, the priority and importance of the data source are determined, the data collection strategy (such as real-time collection, batch import, API docking, etc.) is designed and implemented, the connection mechanism with each data source is established, the data extraction process is implemented, and the original data is cleaned, that is, the repeated data is removed, the missing values are processed (such as filling, deleting or marking) and the obvious data errors (such as format error, range out of bounds, etc.) are corrected. Define the standard data format and structure, convert different formats of data to a unified format, and unify the representation of date, time, currency, etc.

[0057] In this step, steps S11-S13 are also included.

[0058] Step S11, establish a data collection network, and collect original format data according to the data collection network.

[0059] In this step, the data sources of the data collection network can be one or more of enterprise resource planning (ERP) systems, customer relationship management (CRM) systems, point-of-sale (POS) systems, production management systems, warehouse management systems, internal databases and data warehouses, and enterprise internal documents and reports, and can also be obtained from external data sources such as data provided by suppliers and partners, open data sets (such as government open data), third-party data providers (such as market research companies), social media platforms, reports of industry associations and organizations, and open web data collected by web crawlers, data quality assessment (accuracy, completeness, consistency). Data source evaluation of the data collection network can be evaluated from data update frequency, data format and structure, data volume and growth rate, data access method and limitation, data security and privacy compliance requirements, and data usage cost. Data source priority ranking can be sorted according to the importance of business needs, data quality and reliability, data acquisition difficulty, and the potential value of the evaluation data for analysis and decision-making.

[0060] The data collection network can collect raw format data through real-time data stream collection. Real-time data collection can be performed through a high-throughput distributed stream processing platform (Apache Kafka), a distributed computing engine that unifies stream processing and batch processing (Apache Flink), a real-time computing system (Apache Storm), or a provided real-time data stream processing service (Amazon Kinesis). When the data collection network needs to collect batch data, ETL tools (such as Talend, Informatica) or data integration platforms (such as Apache NiFi) can be used to periodically process large amounts of historical data.

[0061] In step S12, the raw format data is preprocessed to obtain transition format data.

[0062] In this step, first, the raw format data is cleaned to delete records containing missing values, and statistical methods are used to fill in the missing values. Finally, machine learning methods are used to predict missing values, identify and delete completely duplicated records, handle partial field duplication, correct inconsistent data (such as uniform date format, uniform unit of measurement, and correct obvious input errors), and use statistical methods to identify outliers (such as Z-score, IQR rule) in the corrected raw format data.

[0063] The raw format data with identified outliers is normalized to ensure that all dates use the same format, uniform variable name naming rules, ensure that numerical values are within a reasonable range, normalize the possible values of categorical variables, ensure that the data structure conforms to the predefined pattern, and handle nested or hierarchical data.

[0064] Step S13, a standard data model is constructed, and the transition industry data is standardized and converted to make the transition industry data map to the standard data model, and standard industry data is obtained.

[0065] In this step, the construction of the standard data model specifically includes collecting and analyzing existing business documents and systems, determining the main business processes and data requirements, and defining the scope and goals of the data model, while identifying the main business entities and defining the relationships between entities, refining the attributes of each entity to determine primary and foreign keys, applying database normalization rules to solve many-to-many relationships between business entities, and creating an entity relationship diagram, selecting an appropriate database management system, converting the logical model to a physical model for a specific database management system, and defining data types, lengths, and constraints. For metadata management, a data dictionary needs to be built to record detailed descriptions of each table and column, define business rules and data quality standards, and establish data lineage relationships, identify key data quality dimensions (such as accuracy, completeness, consistency), define specific rules for each dimension, design data verification and cleaning processes, and finally define data classification and sensitivity levels.

[0066] Further, the mapping relationship between the transition industry data and the standard data model is created, the fields that need to be converted and the conversion rules are identified, the table structure is adjusted to match the standard model, different fields in the transition industry data are split or merged, and changes in the hierarchical structure or relationships in the transition industry data are handled to ensure that the data types converted from the transition industry data to the standard data model are consistent with the standard data model. In the process of implementing the transition industry data to the standard data model, ETL tools or custom scripts can be used to perform the conversion, such as Talend, Informatica, and Apache NiFi, etc.

[0067] In one example, the original format data can also include real-time data acquisition through sensors, periodic manual data entry, and data acquisition through third-party system interface; establish a data acquisition network to connect scattered data sources; pre-process the collected original format data, including cleaning the data, removing error values and duplicate values, filling in missing values, converting the data, unifying the data format, normalizing the data, and unifying the naming rules and measurement units; establish a standard data model, including designing a unified data structure and field definition, developing standard data classification and coding rules, and establishing a mapping relationship between business objects and data; standardize the pre-processed data, including developing data conversion programs, mapping the original data to the standard data model, converting the data format, performing unit conversion, and using a unified measurement unit; quality control the standardized data, including setting data validity rules, filtering invalid data, checking data consistency to ensure consistent data from different sources, establishing a data review mechanism, and manually reviewing abnormal data; store the quality-controlled standard data, including establishing a unified data warehouse or data lake, organizing the storage structure according to the standard data model, establishing data indexes, and improving query efficiency; manage the stored standard data, including developing data governance systems, standardizing data collection and usage processes, establishing data dictionaries, explaining data meanings and usage methods, implementing data version management, and recording data change history.

[0068] Step S2, metadata management is performed on the standard format data to obtain multi-format data, wherein the multi-format data includes observation variables, latent state data, and hidden state parameters.

[0069] In this embodiment, in a smart comprehensive energy system, the standard format data usually includes energy production data (such as power generation, heat generation), energy consumption data (such as electricity consumption, heat consumption), energy transmission and distribution data (such as power grid load, pipe network pressure), environmental data (such as temperature, humidity, solar intensity), and equipment operation state data (such as generator set operation parameters, transformer state).

[0070] Metadata management is a process of structurally describing and organizing these standard format data. The tasks of metadata management mainly include determining the sources of various data, such as sensors, smart meters, SCADA systems, etc.; defining data properties, such as data types (such as numerical, character, timestamp), units (such as kWh, MW, °C), acquisition frequency (such as every second, every minute, every hour), and data range (such as voltage range 0-380V); setting data relationship mapping, i.e. establishing the relationship between different data, such as the relationship between power generation and fuel consumption; defining data accuracy, completeness, and consistency standards; specifying data storage period, archiving strategy, and deletion strategy.

[0071] Further, the observation variables obtained through metadata management refer to data that can be directly measured and observed, such as real-time power load, real-time power generation of photovoltaic panels, power consumption of buildings, and outdoor temperature and humidity, etc.; the latent state data are derived or estimated from the observation variables, such as stability indicators of power grids, overall efficiency of energy systems, short-term load prediction, and availability prediction of renewable energy, etc.; and the hidden state parameters refer to parameters that need to be estimated by complex models, such as health status of generator sets, user behavior patterns, elasticity coefficient of energy demand, impact factor of energy price on consumption, and integration and application of multi-format data.

[0072] Step S3, constructing an initial generation model and an initial inference model according to the observation variables, the latent state data, and the hidden state parameters respectively, and training the initial inference model based on the encoder of the variational autoencoder and the recurrent neural network, and training the initial generation model based on the decoder of the variational autoencoder and the recurrent neural network, to obtain an optimized inference model and an optimized generation model respectively.

[0073] Please refer to Figure 2 and Figure 3 , Figure 2 the structure diagram of the optimized generation model provided by the present application, Figure 3 the structure diagram of the optimized inference model.

[0074] In this step, steps S31-S32 are further included.

[0075] Step S31, generating a corresponding spatial distribution condition for the observation variables and the latent state data corresponding to the observation variables in each time series based on the time series obtained by the data acquisition network.

[0076] In this step, first, data preprocessing is performed, including time synchronization, standardization, and outlier processing; then, spatial information is integrated, including geographic coordinate mapping and spatial partitioning; then, spatio-temporal correlation analysis is performed, including time autocorrelation, spatial autocorrelation, and spatio-temporal cross-correlation analysis; subsequently, spatial interpolation is performed, including selecting an appropriate method and performing interpolation; then, a conditional probability distribution model is established, including selecting an appropriate distribution and estimating parameters; then, time-varying characteristic modeling is performed, including time series decomposition and dynamic model construction; then, uncertainty is quantified, including error analysis, sensitivity analysis, and Monte Carlo simulation; then, result verification and optimization are performed, including cross-validation, model integration, and iterative optimization.

[0077] In one example, the spatio-temporal correlation analysis needs to calculate the autocorrelation function (ACF) of each time series, and identify the periodicity and trend in the time series, while calculating the global spatial autocorrelation index, identifying the local spatial clustering pattern, such as using the LISA statistic, to analyze the correlation between different locations and different variables, and constructing the spatio-temporal correlation matrix. Conditional probability distribution modeling can select appropriate probability distribution according to the characteristics of the data, such as Gaussian distribution, t distribution, etc. For non-parametric distribution, kernel density estimation can be considered, and maximum likelihood estimation or Bayesian method is used to estimate the distribution parameters, and based on the observed variables, the conditional distribution of the latent state is constructed, which can use conditional random field (CRF) or Gaussian process (GP) method, etc.

[0078] The expression of the spatial distribution condition is:

[0079] ;

[0080] wherein, x 1:t denotes the observation sequence from time step 1 to T, z 1:t denotes the latent state data from time step 1 to T, denotes the conditional probability distribution function of the observation sequence, and Π denotes the continuous product, denotes the state transition model corresponding to the latent state data in the time series, denotes the first observation variable generation function corresponding to the latent state data.

[0081] Step S32, according to the spatial distribution condition and the observation variables and latent state data in different time steps in the time series, an initial deep state space model is constructed.

[0082] In this step, the observation variable data set X= { x 1, x 2,..., x T} is arranged, wherein, x t denotes the observation variable at time step t, and the corresponding latent state data set Z={ z 1, z 2,..., z T} is prepared, wherein, z tThe latent state parameter at time step t is denoted as θt, and a suitable deep neural network structure, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), is selected to parameterize these models. The spatial distribution condition generated in step S31 is incorporated into the model structure by adjusting the prior distribution of the observation model or the state transition model. For example, the spatial distribution condition is used to initialize or constrain the model parameters, and initial values are set for the weights and biases of the deep neural network.

[0083] The expression of the initial deep state space model is:

[0084] ;

[0085] ;

[0086] and are called transition model and emission model, respectively, and θ is the parameter of the transition model and the emission model. The initial deep state space model provides a unified framework for modeling and density estimation of time series data.

[0087] The expression of the variational lower bound function is:

[0088] ;

[0089] wherein, denotes the variational lower bound function, θ denotes the output parameter of the generation model in the variational autoencoder, denotes the output parameter of the inference model in the variational autoencoder, denotes the expectation of the approximate posterior distribution in the inference model, denotes the likelihood probability function of the generation model in the variational autoencoder, i.e., the probability of observing data z when the latent variable is given as x , and KL( ) denotes the KL divergence function, denotes the approximate posterior distribution of the inference model in the variational autoencoder, denotes the prior distribution of the generation model in the variational autoencoder.

[0090] In this step, steps S33-S34 are also included.

[0091] In step S33, the conditional probability distribution function of the observation sequence is represented by a t-distribution, and the location parameter and scale parameter of the t-distribution after representing the conditional probability distribution function are generalized to obtain a conditional probability distribution function based on the t-distribution.

[0092] The expression of the conditional probability distribution function based on the t-distribution is:

[0093] ;

[0094] wherein, denotes a probability density function based on a t-distribution, υ denotes a degree of freedom parameter, μ denotes a location parameter of the t-distribution, σ denotes a scale parameter of the t-distribution, and Γ( ) denotes a gamma function.

[0095] Step S34, constructing a second observation variable generating function according to the latent state data and the hidden state parameter when the degree of freedom parameter, the location parameter and the scale parameter in the conditional probability distribution function based on the t-distribution satisfy the preset constraint condition formula.

[0096] The expression of the second observation variable generating function is:

[0097] ;

[0098] wherein, denotes the second observation variable generating function, x t denotes an observation variable at time step t, z t denotes latent state data at time step t, h t denotes a recurrent neural network hidden state parameter at time step t, and ST( ) denotes a tail re-distribution function, NN 1( ) denotes a network function for calculating a distribution mean value, NN 2( ) denotes a network function for calculating a distribution scale, NN 3( ) denotes a network function for calculating a t-distribution degree of freedom parameter, SoftPlus ( ) denotes an activation function.

[0099] In one example, the network function 1( ) for calculating the distribution mean value can be expressed as: NN

[0100] NN 1( r )=W3×φ(W2×φ(W1× r +b1)+b2)+b3;

[0101] wherein, r denotes an input vector, the input vector including the latent state data and the recurrent neural network hidden state parameter at time step t, and φ( ) denotes an activation function, φ( u )=max(0, u ​), W1 represents a first weight matrix, W2 represents a second weight matrix, W3 represents a third weight matrix, b1 represents a first bias vector, b2 represents a second bias vector, and b3 represents a third bias vector.

[0102] Network function for computing distribution scale NN 2( ) can be represented as:

[0103] NN 2( r )= SoftPlus (W3x φ(W2x φ(W1x r +b1)+b2)+b3) ;

[0104] wherein φ( ) represents an activation function, SoftPlus ( ) represents an activation function, SoftPlus ( u )=ln(1+e u ) is an activation function used to ensure the output is positive.

[0105] Network function for computing t-distribution degrees of freedom parameter NN 3( ) can be represented as:

[0106] NN 3( r )= SoftPlus (W3x φ(W2x φ(W1x r +b1)+b2)+b3)+c ;

[0107] wherein c is a positive constant offset used to ensure the output of the network function NN 3( ) has a large initial value.

[0108] In one example, the goal of variational inference is to find a tractable distribution q ( z | x ) to approximate the true posterior distribution p ( z | x ). The ELBO is a lower bound on the log-likelihood p ( x ), and because the goal is to maximize the ELBO, this is equivalent to minimizing the KL divergence between the true posterior distribution and the approximate posterior distribution.

[0109] Inference model q ( z | x ) is typically parameterized by a neural network, denoted as q φ ( z | x ), φLet L represent the network parameters. The objective function for optimizing the inference model is L. inference ( φ )= E [log p θ ( x | z )]- KL ( q φ ( z | x )|| p ( z )),in p θ (x|z) represents the generative model, and θ represents the parameters corresponding to the generative model.

[0110] Similarly, the generative model p(x|z) is also parameterized using a neural network, denoted as... p θ ( x | z The objective function for optimizing the generative model is L. generative ( θ )= E [log p θ ( x | z )).

[0111] Since both the inference model and the generative model need to be optimized simultaneously, the objective function for joint optimization can be expressed as:

[0112] ;

[0113] By combining time series data and spatial distribution conditions, simultaneous modeling of the temporal and spatial characteristics of the data is achieved. Furthermore, spatial distribution conditions are generated based on actual observation data, which improves the robust deep state space model's adaptability to different types of time series data. Combining time series and spatial distribution enables the robust deep state space model to perform analysis at different time and spatial scales simultaneously, significantly improving the performance, adaptability, and interpretability of the robust deep state space model. It also enhances the generalization ability and anomaly detection capability of the robust deep state space model, and improves the accuracy of state estimation and prediction performance.

[0114] Step S4: Construct a robust deep state space model based on the optimized inference model, optimized generative model, initial deep state space model, and variational lower bound function.

[0115] The expression for the robust deep state-space model is:

[0116] ;

[0117] ;

[0118] wherein, x t represents the observed variable at time step t, Φ( ) represents the probability distribution function, z t represents the latent state data at time step t, μ θz ( ) represents the mean function of the state transition Gaussian distribution, Σ θz ( ) represents the covariance function, h t represents the recurrent neural network hidden state parameter at time step t, δ( ) represents the delta distribution function, f θh ( ) represents the transition function of the recurrent neural network, represents the joint distribution of the observed variable, the latent state data and the hidden state parameter in the recurrent neural network in the robust deep state space model, p θx ( ) represents the parameterized distribution function of the emission model, p θz ( ) represents the parameterized distribution function of the state transition model, p θz ( ) represents the transition function of the recurrent neural network, Π represents the continuous product.

[0119] Step S5, inputting the multi-format data into the robust deep state space model to detect and repair the quality of the multi-format data.

[0120] In this step, first, the multi-format data is integrated, including time series data of different energy forms such as electricity, heat and gas, and the data is standardized and normalized, missing values and outliers are processed, and interpolation or simple statistical methods can be used for preliminary filling. The multi-format data is organized into an input format suitable for the robust deep state space model, a sliding window is created to divide the time series data into fixed-length sequence segments, a batch data loader is constructed to efficiently input data into the model, and when the preprocessed multi-format data is input into the robust deep state space model, the input data is mapped to the latent space through the encoder network, the next time latent state is predicted using the state transition model, and the latent state is mapped back to the observation space through the decoder network.

[0121] Further, the reconstruction error can be calculated by comparing the difference between the model output and the original input, calculating the log-likelihood of each data point using the characteristics of the t-distribution, and setting a threshold to mark data points below the threshold as potential low-quality data or anomalies. For data points marked as low-quality or anomalies, the reconstruction data generated by the model is used for replacement, and for missing data, the prediction result of the model is used for filling. By considering the correlation between multi-format data, the repaired data is ensured to be consistent between different energy forms. By using the probabilistic characteristics of the robust deep state space model, an uncertainty estimate is provided for each repaired or filled data point. That is, the Monte Carlo sampling method can be used to run the model multiple times to obtain the prediction distribution. The repaired data is re-input into the robust deep state space model for multiple iterations, and the model parameters are updated in each iteration to improve the accuracy of detection and repair.

[0122] By standardizing the original multi-format data from different sources and formats, data standardization and unified management are achieved. At the same time, by using variational autoencoder and recurrent neural network to optimize inference and generation model, the constructed robust deep state space model can better handle noise and anomalies in multi-format data, and has the regularization effect of variational autoencoder and the memory mechanism of recurrent neural network, so that the robust deep state space model can utilize long-term dependency information and help prevent overfitting, while realizing the collaborative processing of multi-format data and capturing the mutual relationship between different energy forms.

[0123] Based on the above method, the embodiment of the present application discloses a multi-format data management platform for smart comprehensive energy, which refers to Figure 4 The multi-format data management platform 1 includes a data collection and management module 11, a model construction and optimization module 12, and a data monitoring and repair module 13, wherein

[0124] The data collection and management module 11 is used to collect original multi-format data from different sources and formats for data standardization processing, to obtain standard multi-format data, and to perform metadata management on the standard multi-format data to obtain multi-format data, wherein the multi-format data includes observed variables, latent state data, and hidden state parameters.

[0125] The model construction and optimization module 12 is used to construct an initial generation model and an initial inference model based on the observed variables, the latent state data, and the hidden state parameters, respectively, and to train the initial inference model based on the encoder of the variational autoencoder and the recurrent neural network, and to train the initial generation model based on the decoder of the variational autoencoder and the recurrent neural network, to obtain an optimized inference model and an optimized generation model, respectively, and to construct a robust deep state space model based on the optimized inference model, the optimized generation model, the initial deep state space model, and the variational lower bound function.

[0126] The data monitoring and repairing module 13 is configured to input the multi-format data into a robust deep state space model to detect and repair the quality of the multi-format data.

[0127] In one example, the data collection management module 11 is configured to establish a data collection network, collect original format data according to the data collection network, pre-process the original format data to obtain transition format data, construct a standard data model, and perform standardized conversion on the transition format data to map the transition format data to the standard data model to obtain standard format data.

[0128] In one example, the model construction optimization module 12 is configured to generate a corresponding spatial distribution condition for each observation variable and the corresponding latent state data in each time series based on the time series obtained by the data collection network, and construct an initial deep state space model according to the spatial distribution condition and the observation variables and the latent state data at different time steps in the time series.

[0129] In one example, the expression of the spatial distribution condition is as follows:

[0130] ;

[0131] wherein, x 1:t represents an observation sequence from time step 1 to T, z 1:t represents latent state data from time step 1 to T, represents a conditional probability distribution function of the observation sequence, and Π represents a continuous product, represents a state transition model corresponding to the latent state data in the time series, represents a first observation variable generation function corresponding to the latent state data.

[0132] In one example, the expression of the variational lower bound function is as follows:

[0133] ;

[0134] wherein, represents the variational lower bound function, θ represents an output parameter of a generation model in the variational autoencoder, represents an output parameter of an inference model in the variational autoencoder, represents an expectation of an approximate posterior distribution in the inference model, represents a likelihood probability function of the generation model in the variational autoencoder, i.e., a probability of observing data z when the latent variable is x , and KL( ) represents a KL divergence function. represents an approximate posterior distribution of an inference model in a variational autoencoder, represents a prior distribution of a generative model in a variational autoencoder.

[0135] In one example, the expression of the robust deep state-space model is:

[0136] ;

[0137] ;

[0138] wherein, x t represents an observation variable at time step t, Φ( ) represents a probability distribution function, z t represents latent state data at time step t, μ θz ( ) represents a mean function of a state transition Gaussian distribution, Σ θz ( ) represents a covariance function, h t represents a recurrent neural network hidden state parameter at time step t, δ( ) represents a delta distribution function, f θh ( ) represents a transition function of a recurrent neural network, represents a joint distribution of the observation variable, the latent state data, and the hidden state parameter in the recurrent neural network in the robust deep state-space model, p θx ( ) represents a parameterized distribution function of an emission model, p θz ( ) represents a parameterized distribution function of a state transition model, p θz ( ) represents a transition function of a recurrent neural network, Π represents a continuous product.

[0139] In one example, the model construction optimization module 12 is configured to represent the conditional probability distribution function of the observation sequence by using a t-distribution, generalize the location parameter and the scale parameter of the conditional probability distribution function represented by the t-distribution, and obtain a conditional probability distribution function based on the t-distribution; when the degree of freedom parameter, the location parameter, and the scale parameter in the conditional probability distribution function based on the t-distribution satisfy a preset constraint condition, construct a second observation variable generation function according to the latent state data and the hidden state parameter.

[0140] In one example, the expression of the conditional probability distribution function based on the t-distribution is:

[0141] ;

[0142] wherein, denotes a probability density function based on a t-distribution, υ denotes a degree of freedom parameter, μ denotes a location parameter of the t-distribution, σ denotes a scale parameter of the t-distribution, and

[0143] In one example, the expression of the second observation variable generating function is:

[0144] ;

[0145] wherein, denotes a second observation variable generating function, x t denotes an observation variable at time step t, z t denotes a latent state data at time step t, h t denotes a recurrent neural network hidden state parameter at time step t, and NN 1( ) denotes a network function for computing a distribution mean, NN 2( ) denotes a network function for computing a distribution scale, NN 3( ) denotes a network function for computing a t-distribution degree of freedom parameter, and SoftPlus ( ) denotes an activation function.

[0146] The above description is merely that of the preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacements, improvements, and the like made within the spirit and principle of the present application shall fall within the protection scope of the present application.

Claims

1. A multi-format data management method for smart integrated energy, characterized in that, The method comprises: Collecting original format data of different sources and formats for data standardization processing to obtain standard format data; Performing metadata management on the standard format data to obtain multi-format data, wherein the multi-format data comprises observation variables, latent state data and hidden state parameters; According to the observation variables, the latent state data and the hidden state parameters, an initial generation model and an initial inference model are respectively constructed, and the initial inference model is trained based on an encoder of a variational autoencoder and a recurrent neural network, and the initial generation model is trained based on a decoder of the variational autoencoder and the recurrent neural network to respectively obtain an optimized inference model and an optimized generation model; Based on the optimized inference model, the optimized generation model, an initial deep state space model and a variational lower bound function, a robust deep state space model is constructed; The multi-format data is input into the robust deep state space model for quality detection and repair of the multi-format data; The expression of the robust deep state space model is: ； ； where, x t denotes the observed variable at time step t, Φ( ) denotes the probability distribution function, z t denotes the latent state data at time step t, μ θz ( ) denotes the mean function of the state transition Gaussian distribution, Σ θz ( ) denotes the covariance function, h t denotes the recurrent neural network hidden state parameter at time step t, δ( ) denotes the delta distribution function, f θh ( ) denotes the transition function of the recurrent neural network, denotes the joint distribution of the observed variable, the latent state data, and the hidden state parameter in the recurrent neural network in the robust deep state space model, p θx ( ) denotes the parameterized distribution function of the emission model, p θz ( ) denotes the parameterized distribution function of the state transition model, p θh ( ) denotes the transition function of the recurrent neural network, Π denotes the continuous product.

2. The method of claim 1, wherein, The collecting of the original format data of different sources and formats for data standardization processing to obtain standard format data specifically comprises: A data collection network is established, and original format data is collected according to the data collection network; The original format data is preprocessed to obtain transition format data; A standard data model is constructed, and the transition format data is standardized and converted to map the transition format data to the standard data model to obtain the standard format data.

3. The method of claim 2, wherein, The construction process of the initial deep state space model comprises: Based on the time series obtained by the data collection network, a corresponding spatial distribution condition is generated for the observation variables and the latent state data corresponding to the observation variables in each time series; According to the spatial distribution condition and the observation variables and the latent state data at different time steps in the time series, an initial deep state space model is constructed.

4. The method of claim 3, wherein, The expression of the spatial distribution condition is: ； wherein, x 1:t denotes the observation sequence from time step 1 to T, z 1:t denotes the latent state data from time step 1 to T, denotes the conditional probability distribution function of the observation sequence, and Π denotes the continuous product, denotes the state transition model corresponding to the latent state data in the time series, denotes the first observation variable generation function corresponding to the latent state data.

5. The method of claim 1, wherein, The expression of the variational lower bound function is: ； wherein, represents a variational lower bound function, θ represents an output parameter of a generative model in a variational autoencoder, represents an output parameter of an inference model in a variational autoencoder, represents an expectation of an approximate posterior distribution in an inference model, represents a likelihood probability function of a generative model in a variational autoencoder, i.e. a probability of observing data z given that the latent variable is x , KL() represents a KL divergence function, represents an approximate posterior distribution of an inference model in a variational autoencoder, represents a prior distribution of a generative model in a variational autoencoder.

6. The method of claim 4, wherein, The method further comprises: The conditional probability distribution function of the observation sequence is represented by a t-distribution, and the location parameter and the scale parameter of the conditional probability distribution function represented by the t-distribution are generalized to obtain a conditional probability distribution function based on the t-distribution; When the degree of freedom parameter, the location parameter and the scale parameter in the conditional probability distribution function based on the t-distribution satisfy a preset constraint condition, a second observation variable generation function is constructed according to the latent state data and the hidden state parameters.

7. The method of claim 6, wherein, The expression of the conditional probability distribution function based on the t-distribution is: ； wherein, denotes a probability density function based on a t-distribution, υ denotes a degree of freedom parameter, μ denotes a location parameter of the t-distribution, σ denotes a scale parameter of the t-distribution, and Γ( ) denotes a gamma function.

8. The method of claim 7, wherein, The expression of the second observation variable generation function is: ； wherein, represents a second observation variable generating function, x t represents an observation variable at time step t, z t represents latent state data at time step t, h t represents recurrent neural network hidden state parameters at time step t, ST() represents a tail re-distribution function, NN 1() represents a network function for computing a distribution mean, NN 2() represents a network function for computing a distribution scale, NN 3() represents a network function for computing a t-distribution degrees of freedom parameter, SoftPlus () represents an activation function.

9. A multi-format data management platform for smart integrated energy, which adopts the method of any one of claims 1 to 8 for multi-format data management, characterized in that, The multi-format data management platform (1) comprises a data collection management module (11), a model construction optimization module (12) and a data monitoring and repair module (13), wherein, The data collection management module (11) is used for collecting original format data of different sources and formats, performing data standardization processing, obtaining standard format data, performing metadata management on the standard format data, and obtaining multi-format data, wherein the multi-format data includes observation variables, potential state data, and hidden state parameters; The model construction optimization module (12) is used for constructing an initial generation model and an initial inference model according to the observation variables, the potential state data, and the hidden state parameters, respectively, training the initial inference model based on an encoder of a variational autoencoder and a recurrent neural network, training the initial generation model based on a decoder of the variational autoencoder and the recurrent neural network, respectively obtaining an optimized inference model and an optimized generation model, and constructing a robust deep state space model based on the optimized inference model, the optimized generation model, an initial deep state space model, and a variational lower bound function; The data monitoring and repairing module (13) is used for inputting the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

Images