Multi-business data management method and platform for intelligent comprehensive energy

Abstract

The invention provides a multi-business data management method and platform for smart integrated energy, and relates to the field of big data processing.The method comprises the steps that original business data of different sources and formats are collected for data standardization processing, and standard business data are obtained; metadata management is carried out on the standard business state data to obtain multi-business state data, and the multi-business state data comprise observation variables, potential state data and hidden state parameters; respectively constructing an initial generation model and an initial reasoning model according to the observation variables, the potential state data and the hidden state parameters, training the initial reasoning model based on an encoder of the variational auto-encoder and a recurrent neural network, and training the initial generation model based on a decoder of the variational auto-encoder and the recurrent neural network; respectively obtaining an optimization reasoning model and an optimization generation model; constructing a robust depth state space model based on the optimization reasoning model, the optimization generation model, the initial depth state space model and the variational lower bound function; and inputting the multi-business data into the robust depth state space model so as to carry out quality detection and restoration on the multi-business data. According to the invention, cooperative processing of multi-business data can be realized.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a multi-format data management method and platform for smart integrated energy. Background Art

[0002] The performance of big data analysis depends on the quality of the data. The correct use of high-quality data can help make better predictions and decisions, as well as more reliable data analysis. The success of big data analysis depends largely on how the data is cleaned, transformed and integrated. With the development of society, the amount of data in all walks of life has gradually increased exponentially, and the sources of data have become more and more complex and numerous. In practical applications, data often contains incomplete, incorrect or irrelevant data. Data inaccuracy is caused by many reasons. In particular, when large-scale data from different sources (such as multi-energy variety data) are integrated, the quality is particularly worrying; these data sources are usually derived from homogeneous or heterogeneous databases, file systems and service interfaces, which reduces the reliability of the data.

[0003] The Chinese patent with publication number CN115729993A discloses a unified management method and system for multi-source heterogeneous data based on metadata, and the above method includes: the method includes: responding to the operation of data production users uploading metadata of multi-source heterogeneous data, converting metadata to running state; wherein metadata is data obtained by data production users characterizing multi-source heterogeneous data based on preset metadata definition rules; configuring corresponding permissions for users of different roles; responding to the compilation operation of data directory by platform management users and / or data production users, establishing the association between data directory and data objects according to the compilation operation, and forming a classified and layered data directory; returning corresponding data attribute information and content information to data consumption users. However, the above scheme cannot achieve complete data fusion for multivariate data, and the processing and processing capabilities of data only stay within a single category of data, and cannot complete the analysis and comparison between different categories of data. Therefore, it is very necessary to provide a multi-format data management method and platform for smart integrated energy to realize the collaborative processing of multi-format data. Summary of the invention

[0004] In view of this, the present invention proposes a multi-format data management method and platform for smart integrated energy. By standardizing the original format data of different sources and formats, data standardization and unified management are achieved. At the same time, by using variational autoencoders and recurrent neural networks to optimize the inference and generation models, the robust deep state space model can be used to achieve the collaborative processing of multi-format data.

[0005] The present invention provides a multi-format data management method for smart integrated energy, the method comprising: Collect original business data from different sources and formats for data standardization and obtain standard business data; Performing metadata management on the standard business data to obtain multi-business data, wherein the multi-business data includes observed variables, potential state data, and hidden state parameters; An initial generation model and an initial inference model are respectively constructed according to the observed variables, the potential state data, and the hidden state parameters, and the initial inference model is trained based on an encoder and a recurrent neural network of a variational autoencoder, and the initial generation model is trained based on a decoder and a recurrent neural network of a variational autoencoder, so as to obtain an optimized inference model and an optimized generation model respectively; Based on the optimization inference model, the optimization generation model, the initial deep state space model and the variational lower bound function, construct a robust deep state space model; The multi-format data is input into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0006] On the basis of the above technical solution, preferably, the collecting of original business data from different sources and formats for data standardization to obtain standard business data specifically includes: Establishing a data collection network, and collecting original business data according to the data collection network; Performing data preprocessing on the original business format data to obtain transitional business format data; A standard data model is constructed, and the transitional business format data is subjected to standardized transformation so as to map the transitional business format data into the standard data model, thereby obtaining the standard business format data.

[0007] On the basis of the above technical solution, preferably, the construction process of the initial deep state space model includes: Based on the time series acquired by the data acquisition network, generating corresponding spatial distribution conditions for the observed variables in each time series and the potential state data corresponding to the observed variables; An initial deep state space model is constructed according to the spatial distribution conditions and the observed variables and potential state data at different time steps in the time series.

[0008] More preferably, the expression of the spatial distribution condition is: ; in, x 1:t represents the observation sequence from time step 1 to T, z 1:t represents the potential state data from time step 1 to T, represents the conditional probability distribution function of the observation sequence, Π represents the continuous product, represents the state transition model corresponding to the potential state data in the time series, Represents the first observed variable generating function corresponding to the latent state data.

[0009] More preferably, the expression of the variational lower bound function is: ; in, represents the variational lower bound function, i represents the output parameters of the generative model in the variational autoencoder, represents the output parameters of the inference model in the variational autoencoder, represents the expectation of the approximate posterior distribution in the inference model, represents the likelihood probability function of the generative model in the variational self-encoder, that is, given the latent variable z Observed data x The probability of , KL () represents the KL divergence function, represents the approximate posterior distribution of the inference model in the variational autoencoder, Represents the prior distribution of the generative model in the variational autoencoder.

[0010] More preferably, the expression of the robust deep state space model is: ; ; in, x t represents the observed variable at time step t, Φ( ) represents the probability distribution function, z t represents the potential state data at time step t, m θz ( ) represents the mean function of the state transition Gaussian distribution, Σ θz ( ) represents the covariance function, h t represents the hidden state parameter of the recurrent neural network at time step t, δ( ) represents the delta distribution function, f θh ( ) represents the transition function of the recurrent neural network, represents the joint distribution of observed variables, latent state data in a robust deep state-space model, and hidden state parameters in a recurrent neural network, p θx ( ) represents the parameterized distribution function of the emission model, p θz( ) represents the parameterized distribution function of the state transition model, p θz ( ) represents the transfer function of the recurrent neural network, and Π represents the continuous product.

[0011] More preferably, the method further comprises: The conditional probability distribution function of the observation sequence is represented by t distribution, and the location parameter and scale parameter of the conditional probability distribution function represented by t distribution are generalized to obtain a conditional probability distribution function based on t distribution; When the degree of freedom parameter, location parameter and scale parameter in the conditional probability distribution function based on t-distribution meet the preset constraint condition formula, a second observation variable generation function is constructed according to the potential state data and the hidden state parameter.

[0012] More preferably, the expression of the conditional probability distribution function based on t distribution is: ; in, represents the probability density function based on t distribution, u represents the degree of freedom parameter, m represents the location parameter of the t-distribution, s represents the scale parameter of t-distribution, and Γ( ) represents the gamma function.

[0013] More preferably, the expression of the second observed variable generating function is: ; in, represents the second observed variable generating function, x t represents the observed variable at time step t, z t represents the potential state data at time step t, h t represents the hidden state parameter of the recurrent neural network at time step t, ST( ) represents the tail weight distribution function, NN 1( ) represents the network function used to calculate the mean of the distribution, NN 2( ) represents the network function used to calculate the distribution scale, NN 3( ) represents the network function used to calculate the degree of freedom parameter of the t distribution, SoftPlus ( ) represents the activation function.

[0014] In a second aspect of the present application, a multi-format data management platform for smart integrated energy is provided, wherein the multi-format data management platform includes a data acquisition management module, a model building optimization module, and a data monitoring and repair module, wherein: The data acquisition management module is used to collect original business data from different sources and formats for data standardization, obtain standard business data, and perform metadata management on the standard business data to obtain multi-business data, wherein the multi-business data includes observed variables, potential state data, and hidden state parameters; The model construction optimization module is used to respectively construct an initial generation model and an initial inference model according to the observed variables, the potential state data, and the hidden state parameters, and train the initial inference model based on the encoder and recurrent neural network of the variational autoencoder, and train the initial generation model based on the decoder and recurrent neural network of the variational autoencoder to obtain an optimized inference model and an optimized generation model, respectively, and construct a robust deep state space model based on the optimized inference model, the optimized generation model, the initial deep state space model, and the variational lower bound function; The data monitoring and repair module is used to input the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0015] The multi-format data management method and platform for smart integrated energy provided by the present invention has the following beneficial effects compared with the prior art: (1) By standardizing the raw business data from different sources and formats, data standardization and unified management are achieved. At the same time, by adopting variational autoencoders and recurrent neural networks to optimize the inference and generation models, the constructed robust deep state-space model can better handle the noise and anomalies in multi-business data. In addition, it has the regularization effect of variational autoencoders and the memory mechanism of recurrent neural networks, so that the robust deep state-space model can utilize long-term dependency information and help prevent overfitting. At the same time, it can realize the coordinated processing of multi-business data and capture the relationship between different energy forms. (2) By combining time series data and spatial distribution conditions, we can achieve simultaneous modeling of the temporal and spatial characteristics of the data, and generate spatial distribution conditions based on actual observation data, thereby improving the adaptability of the robust deep state-space model to different types of time series data. In addition, the combination of time series and spatial distribution enables the robust deep state-space model to simultaneously analyze different temporal and spatial scales, significantly improving the performance, adaptability, and interpretability of the robust deep state-space model, and enhancing the generalization and anomaly detection capabilities of the robust deep state-space model, thereby improving the state estimation accuracy and prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A schematic diagram of the steps of a multi-format data management method for smart integrated energy provided by the present invention; Figure 2 A schematic diagram of the structure of the optimization generation model provided by the present invention; Figure 3 A schematic diagram of the structure of the optimization reasoning model provided by the present invention; Figure 4 A schematic diagram of the structure of the multi-format data management platform provided by the present invention.

[0018] Explanation of the accompanying drawings: 1. Communication protocol dynamic adaptation device; 11. Data acquisition management module; 12. Model building and optimization module; 13. Data monitoring and repair module. DETAILED DESCRIPTION

[0019] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Before introducing the embodiments of the present invention, some terms and their abbreviations involved in the embodiments of the present invention are first defined and explained.

[0021] Variational Autoencoder (VAE) is a generative model that combines the advantages of deep learning and probabilistic graph models. VAE mainly consists of two parts: encoder and decoder. The encoder is used to map the input data x to the distribution parameters of the latent space (usually mean μ and variance σ), and the decoder is used to sample the vector from the latent space. z Reconstructing the original data x .

[0022] The following is combined with Figure 1-4 This application is described in further detail.

[0023] refer to Figure 1, the present application provides a multi-format data management method for smart integrated energy, and the above method steps include S1-S5.

[0024] Step S1, collect original business data from different sources and formats, perform data standardization processing, and obtain standard business data.

[0025] In this embodiment, by identifying all relevant data sources (such as different business systems, external data providers, etc.), and evaluating the data quality, format, update frequency and other characteristics of each data source, the priority and importance of the data source are determined, and data collection strategies (such as real-time collection, batch import, API docking, etc.) are designed and implemented, and a connection mechanism with each data source is established. The data extraction process is implemented, and the original data is cleaned, that is, duplicate data is removed, missing values ​​are processed (such as filling, deleting or marking), and obvious data errors (such as format errors, range crossing, etc.) are corrected. Define standard data formats and structures, convert data in different formats into a unified format, and unify the representation of date, time, currency, etc.

[0026] This step also includes steps S11 to S13.

[0027] Step S11, establish a data collection network, and collect original business data according to the data collection network.

[0028] In this step, the data source of the data collection network can be one or more of the enterprise resource planning (ERP) system, customer relationship management (CRM) system, point of sale (POS) system, production management system, warehouse management system, internal database and data warehouse, and internal documents and reports of the enterprise. It can also be obtained from external data sources, such as data public data sets provided by suppliers and partners (such as government open data), third-party data providers (such as market research companies), social media platforms, reports from industry associations and organizations, and public web page data collected by web crawlers, data quality assessment (accuracy, completeness, consistency). The data source evaluation of the data collection network can be judged from the aspects of data update frequency, data format and structure, data volume and growth rate, data access methods and restrictions, data security and privacy compliance requirements, and data usage costs. Data source priority sorting can be sorted according to the importance of business needs, data quality and reliability, the difficulty of data acquisition, and the potential value of data for analysis and decision-making.

[0029] The data collection network can collect raw business data through real-time data stream collection, and can collect real-time data through a high-throughput distributed stream processing platform (Apache Kafka) or a distributed computing engine that unifies stream processing and batch processing (Apache Flink) or a real-time computing system (Apache Storm) or a real-time data stream processing service (Amazon Kinesis). When the data collection network needs to collect batch data, ETL tools (such as Talend, Informatica) or data integration platforms (such as Apache NiFi) can be used to regularly process large amounts of historical data.

[0030] Step S12, preprocessing the original business format data to obtain transitional business format data.

[0031] In this step, the original business data is first cleaned to delete records containing missing values, and statistical methods are used to fill them. Finally, machine learning methods are used to predict missing values, identify and delete completely duplicate records, handle the duplication of some fields, and correct inconsistent data (such as unifying date formats, unifying measurement units, and correcting obvious input errors). Statistical methods are used to identify outliers (such as Z-score, IQR rule) for the corrected original business data.

[0032] The original business data that has completed outlier identification is normalized to ensure that all dates use the same format, unify the naming rules of variable names, ensure that the values ​​are within a reasonable range, normalize the possible values ​​of categorical variables, ensure that the data structure conforms to the predefined pattern, and handle nested or hierarchical data.

[0033] Step S13, constructing a standard data model, and performing a standardized transformation on the transitional business format data, so as to map the transitional business format data to the standard data model, and obtain the standard business format data.

[0034] In this step, building a standard data model specifically includes collecting and analyzing existing business documents and systems, determining the main business processes and data requirements, and defining the scope and objectives of the data model. At the same time, it identifies the main business entities and defines the relationship between entities, refines the attributes of each entity to determine the primary key and foreign key, applies database normalization rules to solve the many-to-many relationship between business entities, and creates an entity relationship diagram, selects an appropriate database management system, converts the logical model into a physical model for a specific database management system, and defines data types, lengths, and constraints. For metadata management, it is necessary to build a data dictionary, record a detailed description of each table and column, define business rules and data quality standards, establish data lineage, identify key data quality dimensions (such as accuracy, completeness, consistency), define specific rules for each dimension, design data verification and cleaning processes, and finally define data classification and sensitivity levels.

[0035] Furthermore, create a mapping relationship from the transitional business data to the standard data model, identify the fields and conversion rules that need to be converted, adjust the table structure to match the standard model, split or merge different fields in the transitional business data, and handle changes in the hierarchy or relationship in the transitional business data to ensure that the data type when the transitional business data is converted to the standard data model is consistent with the standard data model. In the process of implementing the transitional business data to the standard data model, you can use ETL tools or custom scripts to perform the conversion, such as Talend, Informatica, and Apache NiFi.

[0036] In one example, the original business data may also include real-time data collection through sensors, regular manual data entry, and data collection through third-party system interface docking; establish a data collection network to connect scattered data sources; pre-process the collected original business data, including cleaning the data, removing erroneous values ​​and duplicate values, filling missing values, converting the data, unifying the data format, normalizing the data, unifying the naming rules and measurement units; establish a standard data model, including designing a unified data structure and field definition, formulating standard data classification and coding rules, and establishing a mapping relationship between business objects and data; standardize the pre-processed data, including developing a data conversion program to convert the original data. The data is mapped to a standard data model, and data format conversion is performed, unit conversion is performed, and a unified unit of measurement is used; quality control is performed on the data after standardization conversion, including setting data validity rules, filtering invalid data, performing data consistency checks, ensuring that data from different sources are consistent, establishing a data review mechanism, and manually reviewing abnormal data; the standard data after quality control is stored, including establishing a unified data warehouse or data lake, organizing the storage structure according to the standard data model, establishing data indexes, and improving query efficiency; the stored standard data is managed, including formulating a data governance system, standardizing data collection and usage processes, establishing a data dictionary, explaining the meaning and usage of data, implementing data version management, and recording data change history.

[0037] Step S2, performing metadata management on the standard business format data to obtain multi-business format data, wherein the multi-business format data includes observed variables, potential state data, and hidden state parameters.

[0038] In this embodiment, in the smart integrated energy system, standard business data generally include energy production data (such as power generation, heat generation), energy consumption data (such as electricity consumption, heat consumption), energy transmission and distribution data (such as grid load, pipeline pressure), environmental data (such as temperature, humidity, sunshine intensity) and equipment operating status data (such as generator set operating parameters, transformer status).

[0039] Metadata management is the process of describing and organizing these standard business data in a structured manner. The tasks of metadata management mainly include determining the sources of various types of data, such as sensors, smart meters, SCADA systems, etc.; defining data attributes, such as data types (such as numeric, character, timestamp), units (such as kWh, MW, ℃), collection frequency (such as per second, per minute, per hour) and data range (such as voltage range 0-380V); setting data relationship mapping, that is, establishing the relationship between different data, such as the relationship between power generation and fuel consumption; defining data accuracy, completeness and consistency standards; and specifying data storage cycles, archiving strategies and deletion strategies.

[0040] Furthermore, observed variables obtained through metadata management refer to data that can be directly measured and observed, such as real-time power load, real-time power generation of photovoltaic panels, power consumption of buildings, outdoor temperature and humidity, etc.; potential state data are derived or estimated through observed variables, such as stability indicators of the power grid, the overall efficiency of the energy system, short-term load forecasts, and availability forecasts of renewable energy, etc.; hidden state parameters refer to those that require complex models to estimate, such as the health status of generator sets, user behavior patterns, the elasticity coefficient of energy demand, the impact of energy prices on consumption, and the integration and application of multi-format data.

[0041] Step S3, constructing an initial generation model and an initial inference model according to the observed variables, potential state data, and hidden state parameters, respectively, and training the initial inference model based on the encoder and recurrent neural network of the variational autoencoder, and training the initial generation model based on the decoder and recurrent neural network of the variational autoencoder to obtain an optimized inference model and an optimized generation model, respectively.

[0042] See also Figure 2 and Figure 3 , Figure 2 A schematic diagram of the structure of the optimization generation model provided by the present invention, Figure 3 Schematic diagram of the structure of the optimized inference model.

[0043] This step also includes steps S31~S32.

[0044] Step S31, based on the time series acquired by the data acquisition network, generating corresponding spatial distribution conditions for the observed variables in each time series and the potential state data corresponding to the observed variables.

[0045] In this step, data preprocessing is first performed, including time synchronization, standardization and outlier processing; then spatial information is integrated, geographic coordinate mapping and spatial partitioning are performed; then spatiotemporal correlation analysis is performed, which includes temporal autocorrelation, spatial autocorrelation and spatiotemporal cross-correlation analysis; then spatial interpolation is performed, and the appropriate method is selected and interpolation is performed; then a conditional probability distribution model is established, the appropriate distribution is selected and the parameters are estimated; then time-varying characteristic modeling is performed, including time series decomposition and dynamic model construction; then uncertainty is quantified, and error analysis, sensitivity analysis and Monte Carlo simulation are performed; then the results are verified and optimized, including cross-validation, model integration and iterative optimization.

[0046] In one example, spatiotemporal correlation analysis requires calculating the autocorrelation function (ACF) of each time series and identifying the periodicity and trend in the time series, while calculating the global spatial autocorrelation index and identifying the local spatial aggregation pattern, such as using the LISA statistic, to analyze the correlation between different locations and different variables and construct a spatiotemporal correlation matrix. Conditional probability distribution modeling can select a suitable probability distribution according to the data characteristics, such as Gaussian distribution, t distribution, etc. For non-parametric distributions, kernel density estimation can be considered, and distribution parameters can be estimated using maximum likelihood estimation or Bayesian methods. Based on the observed variables, the conditional distribution of the potential state can be constructed, and methods such as conditional random fields (CRF) or Gaussian processes (GP) can be used.

[0047] The expression of spatial distribution condition is: ; in, x 1:t represents the observation sequence from time step 1 to T, z 1:t represents the potential state data from time step 1 to T, represents the conditional probability distribution function of the observation sequence, Π represents the continuous product, represents the state transition model corresponding to the potential state data in the time series, Represents the first observed variable generating function corresponding to the latent state data.

[0048] Step S32, constructing an initial deep state space model according to the spatial distribution conditions and the observed variables and potential state data at different time steps in the time series.

[0049] In this step, the observed variable dataset is organized X= { x 1, x 2,..., x T},in, x t Represents the observed variable at time step t, and prepares the corresponding potential state data set Z={ z 1, z 2,..., z T},in, z tRepresenting the potential state parameters at time step t, selecting an appropriate deep neural network structure, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), to parameterize these models, integrating the spatial distribution conditions generated in step S31 into the model structure, and achieving this by adjusting the prior distribution of the observation model or the state transition model, for example, using the spatial distribution conditions to initialize or constrain model parameters, and setting initial values ​​for the weights and biases of the deep neural network.

[0050] The expression of the initial deep state space model is: ; ; and The transition model and emission model are called transition model and emission model respectively, and θ is the parameter of transition model and emission model. The initial deep state space model provides a unified framework for modeling and density estimation of time series data.

[0051] The expression of the variational lower bound function is: ; in, represents the variational lower bound function, i represents the output parameters of the generative model in the variational autoencoder, represents the output parameters of the inference model in the variational autoencoder, represents the expectation of the approximate posterior distribution in the inference model, represents the likelihood probability function of the generative model in the variational self-encoder, that is, given the latent variable z Observed data x The probability of , KL () represents the KL divergence function, represents the approximate posterior distribution of the inference model in the variational autoencoder, Represents the prior distribution of the generative model in the variational autoencoder.

[0052] This step also includes steps S33~S34.

[0053] Step S33, the conditional probability distribution function of the observation sequence is represented by t-distribution, and the location parameter and scale parameter of the conditional probability distribution function represented by t-distribution are generalized to obtain a conditional probability distribution function based on t-distribution.

[0054] The expression of the conditional probability distribution function based on t distribution is: ; in, represents the probability density function based on t distribution, urepresents the degree of freedom parameter, m represents the location parameter of the t-distribution, s represents the scale parameter of t-distribution, and Γ( ) represents the gamma function.

[0055] Step S34, when the degree of freedom parameter, location parameter and scale parameter in the conditional probability distribution function based on t distribution meet the preset constraint condition, construct a second observation variable generation function according to the potential state data and the hidden state parameter.

[0056] The expression of the second observed variable generating function is: ; in, represents the second observed variable generating function, x t represents the observed variable at time step t, z t represents the potential state data at time step t, h t represents the hidden state parameter of the recurrent neural network at time step t, ST( ) represents the tail weight distribution function, NN 1( ) represents the network function used to calculate the mean of the distribution, NN 2( ) represents the network function used to calculate the distribution scale, NN 3( ) represents the network function used to calculate the degree of freedom parameter of the t distribution, SoftPlus ( ) represents the activation function.

[0057] In one example, a network function for computing the mean of a distribution NN 1( ) can be expressed as: NN 1( r )=W3×φ(W2×φ(W1× r +b1)+b2)+b3; in, r represents the input vector, which includes the potential state data and the hidden state parameters of the recurrent neural network at time step t, φ( ) represents the activation function, φ( u )=max(0, u ), W1 represents the first weight matrix, W2 represents the second weight matrix, W3 represents the third weight matrix, b1 represents the first bias vector, b2 represents the second bias vector, and b3 represents the third bias vector.

[0058] Network function for calculating distribution size NN 2( ) can be expressed as: NN 2( r )= SoftPlus (W3×φ(W2×φ(W1× r +b1)+b2)+b3); in, SoftPlus ( ) represents the activation function, SoftPlus ( u )=ln(1+e u ) is the activation function used to ensure that the output is positive.

[0059] Web function for calculating the degrees of freedom parameter of the t-distribution NN 3( ) can be expressed as: NN 3( r )= SoftPlus (W3×φ(W2×φ(W1× r +b1)+b2)+b3)+c; Where c is a positive constant offset used to ensure that the network function NN The output of 3( ) has a larger initial value.

[0060] In one example, since the goal of variational inference is to find a distribution that is easy to compute q ( z | x ) to approximate the true posterior distribution p ( z | x ). ELBO is the log-likelihood p ( x ), since the goal is to maximize the ELBO, which is equivalent to minimizing the KL divergence between the true posterior distribution and the approximate posterior distribution.

[0061] Inference Model q ( z | x ) is usually parameterized by a neural network, denoted by q φ ( z | x ), f Represents the network parameters. The objective function of the optimization inference model is L inference ( f )= E [log p θ ( x | z )]- KL ( q φ ( z | x )|| p ( z )),inp θ (x|z) represents the generative model, and θ represents the parameters corresponding to the generative model.

[0062] Similarly, the generative model p(x|z) is also parameterized by a neural network and is denoted as p θ ( x | z ). The objective function of optimizing the generative model is L generative ( i )= E [log p θ ( x | z )].

[0063] Since both the inference model and the generation model need to be optimized simultaneously, the objective function of the joint optimization can be expressed as: ; By combining time series data and spatial distribution conditions, we can achieve simultaneous modeling of the temporal and spatial characteristics of the data, and generate spatial distribution conditions based on actual observation data, thereby improving the adaptability of the robust deep state-space model to different types of time series data. The combination of time series and spatial distribution enables the robust deep state-space model to simultaneously perform analysis at different time and spatial scales, significantly improving the performance, adaptability and interpretability of the robust deep state-space model, and enhancing the generalization and anomaly detection capabilities of the robust deep state-space model, thereby improving the state estimation accuracy and prediction performance.

[0064] Step S4, constructing a robust deep state space model based on the optimized inference model, the optimized generation model, the initial deep state space model and the variational lower bound function.

[0065] The expression of the robust deep state space model is: ; ; in, x t represents the observed variable at time step t, Φ( ) represents the probability distribution function, z t represents the potential state data at time step t, m θz ( ) represents the mean function of the state transition Gaussian distribution, Σ θz ( ) represents the covariance function, h t represents the hidden state parameter of the recurrent neural network at time step t, δ( ) represents the delta distribution function,f θh ( ) represents the transition function of the recurrent neural network, represents the joint distribution of observed variables, latent state data in a robust deep state-space model, and hidden state parameters in a recurrent neural network, p θx ( ) represents the parameterized distribution function of the emission model, p θz ( ) represents the parameterized distribution function of the state transition model, p θz ( ) represents the transfer function of the recurrent neural network, and Π represents the continuous product.

[0066] Step S5, inputting the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0067] In this step, we first integrate multi-format data, including time series data of different energy forms such as electricity, heat, and gas, and standardize and normalize the data. We handle missing values ​​and outliers, and can use interpolation or simple statistical methods for preliminary filling. We organize the multi-format data into an input format suitable for the robust deep state space model, create a sliding window to divide the time series data into sequence segments of fixed length, and build a batch data loader to efficiently input data into the model. When the preprocessed multi-format data is input into the robust deep state space model, the input data is mapped to the latent space through the encoder network, the state transition model is used to predict the potential state at the next moment, and the latent state is mapped back to the observation space through the decoder network.

[0068] Furthermore, the reconstruction error can be calculated by comparing the difference between the model output and the original input, using the characteristics of the t distribution to calculate the log-likelihood of each data point, and setting a threshold to mark data points below the threshold as potential low-quality data or anomalies. For data points marked as low-quality or anomalies, they are replaced with the reconstructed data generated by the model. For missing data, they are filled with the prediction results of the model. By considering the correlation between multi-format data, the repaired data is ensured to be consistent between different energy forms. The probabilistic characteristics of the robust deep state space model are used to provide uncertainty estimates for each repaired or filled data point. That is, the Monte Carlo sampling method can be used to run the model multiple times to obtain the predicted distribution, and the repaired data can be re-input into the robust deep state space model for multiple rounds of iterations. The model parameters are updated in each round of iteration to improve the accuracy of detection and repair.

[0069] By standardizing the original business data from different sources and formats, data standardization and unified management are achieved. At the same time, by adopting variational autoencoders and recurrent neural networks to optimize the inference and generation models, the constructed robust deep state-space model can better handle the noise and anomalies in multi-business data. It has the regularization effect of variational autoencoders and the memory mechanism of recurrent neural networks, so that the robust deep state-space model can utilize long-term dependency information and help prevent overfitting. At the same time, it can realize the coordinated processing of multi-business data and capture the relationship between different energy forms.

[0070] Based on the above method, the embodiment of the present application discloses a multi-format data management platform for smart integrated energy. Figure 4 The multi-format data management platform 1 includes a data collection management module 11, a model building optimization module 12 and a data monitoring and repair module 13, wherein: The data collection management module 11 is used to collect original business data from different sources and formats for data standardization, obtain standard business data, and perform metadata management on the standard business data to obtain multi-business data, wherein the multi-business data includes observed variables, potential state data, and hidden state parameters; The model building optimization module 12 is used to respectively build an initial generation model and an initial inference model according to the observed variables, the potential state data, and the hidden state parameters, and train the initial inference model based on the encoder and the recurrent neural network of the variational autoencoder, and train the initial generation model based on the decoder and the recurrent neural network of the variational autoencoder to obtain the optimized inference model and the optimized generation model, respectively, and build a robust deep state space model based on the optimized inference model, the optimized generation model, the initial deep state space model, and the variational lower bound function; The data monitoring and repairing module 13 is used to input the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

[0071] In one example, the data collection management module 11 is used to establish a data collection network, collect original business data according to the data collection network; perform data preprocessing on the original business data to obtain transitional business data; construct a standard data model, and perform standardized transformation on the transitional business data so that the transitional business data can be mapped to the standard data model to obtain standard business data.

[0072] In one example, the model building optimization module 12 is used to generate corresponding spatial distribution conditions for the observed variables and the potential state data corresponding to the observed variables in each time series based on the time series acquired by the data acquisition network; and to build an initial deep state space model based on the spatial distribution conditions and the observed variables and potential state data at different time steps in the time series.

[0073] In one example, the expression of the spatial distribution condition is: ; in, x 1:t represents the observation sequence from time step 1 to T, z 1:t represents the potential state data from time step 1 to T, represents the conditional probability distribution function of the observation sequence, Π represents the continuous product, represents the state transition model corresponding to the potential state data in the time series, Represents the first observed variable generating function corresponding to the latent state data.

[0074] In one example, the expression of the variational lower bound function is: ; in, represents the variational lower bound function, i represents the output parameters of the generative model in the variational autoencoder, represents the output parameters of the inference model in the variational autoencoder, represents the expectation of the approximate posterior distribution in the inference model, represents the likelihood probability function of the generative model in the variational self-encoder, that is, given the latent variable z Observed data x The probability of , KL () represents the KL divergence function, represents the approximate posterior distribution of the inference model in the variational autoencoder, Represents the prior distribution of the generative model in the variational autoencoder.

[0075] In one example, the robust deep state-space model is expressed as: ; ; in, x t represents the observed variable at time step t, Φ( ) represents the probability distribution function, z t represents the potential state data at time step t, m θz ( ) represents the mean function of the state transition Gaussian distribution, Σ θz ( ) represents the covariance function, h t represents the hidden state parameter of the recurrent neural network at time step t, δ( ) represents the delta distribution function, f θh( ) represents the transition function of the recurrent neural network, represents the joint distribution of observed variables, latent state data in a robust deep state-space model, and hidden state parameters in a recurrent neural network, p θx ( ) represents the parameterized distribution function of the emission model, p θz ( ) represents the parameterized distribution function of the state transition model, p θz ( ) represents the transfer function of the recurrent neural network, and Π represents the continuous product.

[0076] In one example, the model construction optimization module 12 is used to represent the conditional probability distribution function of the observation sequence using a t-distribution, and generalize the location parameters and scale parameters of the conditional probability distribution function after the t-distribution representation to obtain a conditional probability distribution function based on the t-distribution; when the degree of freedom parameter, location parameter and scale parameter in the conditional probability distribution function based on the t-distribution meet the preset constraint condition formula, a second observation variable generation function is constructed according to the potential state data and the hidden state parameters.

[0077] In one example, the expression of the conditional probability distribution function based on the t-distribution is: ; in, represents the probability density function based on t distribution, u represents the degree of freedom parameter, m represents the location parameter of the t-distribution, s represents the scale parameter of t-distribution, and Γ( ) represents the gamma function.

[0078] In one example, the expression of the second observation variable generating function is: ; in, represents the second observed variable generating function, x t represents the observed variable at time step t, z t represents the potential state data at time step t, h t represents the hidden state parameter of the recurrent neural network at time step t, ST( ) represents the tail weight distribution function, NN 1( ) represents the network function used to calculate the mean of the distribution, NN 2( ) represents the network function used to calculate the distribution scale, NN 3( ) represents the network function used to calculate the degree of freedom parameter of the t distribution, SoftPlus ( ) represents the activation function.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-format data management method for smart integrated energy, characterized in that: The method comprises: Collect original business data from different sources and formats for data standardization and obtain standard business data; Performing metadata management on the standard business data to obtain multi-business data, wherein the multi-business data includes observed variables, potential state data, and hidden state parameters; An initial generation model and an initial inference model are respectively constructed according to the observed variables, the potential state data, and the hidden state parameters, and the initial inference model is trained based on an encoder and a recurrent neural network of a variational autoencoder, and the initial generation model is trained based on a decoder and a recurrent neural network of a variational autoencoder, so as to obtain an optimized inference model and an optimized generation model respectively; Based on the optimization inference model, the optimization generation model, the initial deep state space model and the variational lower bound function, construct a robust deep state space model; The multi-format data is input into the robust deep state space model to perform quality detection and repair on the multi-format data.

2. The method according to claim 1, characterized in that The collecting of original business data from different sources and formats and performing data standardization processing to obtain standard business data specifically includes: Establishing a data collection network, and collecting original business data according to the data collection network; Performing data preprocessing on the original business format data to obtain transitional business format data; A standard data model is constructed, and the transitional business format data is subjected to standardized transformation so as to map the transitional business format data into the standard data model, thereby obtaining the standard business format data.

3. The method according to claim 2, characterized in that The construction process of the initial deep state space model includes: Based on the time series acquired by the data acquisition network, generating corresponding spatial distribution conditions for the observed variables in each time series and the potential state data corresponding to the observed variables; An initial deep state space model is constructed according to the spatial distribution conditions and the observed variables and potential state data at different time steps in the time series.

4. The method according to claim 3, characterized in that The expression of the spatial distribution condition is: ； in, x 1:t represents the observation sequence from time step 1 to T, z 1:t represents the potential state data from time step 1 to T, represents the conditional probability distribution function of the observation sequence, Π represents the continuous product, represents the state transition model corresponding to the potential state data in the time series, Represents the first observed variable generating function corresponding to the latent state data.

5. The method according to claim 1, characterized in that The expression of the variational lower bound function is: ； in, represents the variational lower bound function, θ represents the output parameters of the generative model in the variational autoencoder, represents the output parameters of the inference model in the variational autoencoder, represents the expectation of the approximate posterior distribution in the inference model, represents the likelihood probability function of the generative model in the variational self-encoder, that is, given the latent variable z Observed data x The probability of , KL () represents the KL divergence function, represents the approximate posterior distribution of the inference model in the variational autoencoder, Represents the prior distribution of the generative model in the variational autoencoder.

6. The method according to claim 1, characterized in that The expression of the robust deep state space model is: ； ； in, x t represents the observed variable at time step t, Φ( ) represents the probability distribution function, z t represents the potential state data at time step t, μ θz ( ) represents the mean function of the state transition Gaussian distribution, Σ θz ( ) represents the covariance function, h t represents the hidden state parameter of the recurrent neural network at time step t, δ( ) represents the delta distribution function, f θh ( ) represents the transition function of the recurrent neural network, represents the joint distribution of observed variables, latent state data in a robust deep state-space model, and hidden state parameters in a recurrent neural network, p θx ( ) represents the parameterized distribution function of the emission model, p θz ( ) represents the parameterized distribution function of the state transition model, p θz ( ) represents the transfer function of the recurrent neural network, and Π represents the continuous product.

7. The method according to claim 4, characterized in that The method further comprises: The conditional probability distribution function of the observation sequence is represented by t distribution, and the location parameter and scale parameter of the conditional probability distribution function represented by t distribution are generalized to obtain a conditional probability distribution function based on t distribution; When the degree of freedom parameter, location parameter and scale parameter in the conditional probability distribution function based on t-distribution meet the preset constraint condition formula, a second observation variable generation function is constructed according to the potential state data and the hidden state parameter.

8. The method according to claim 7, characterized in that The expression of the conditional probability distribution function based on t distribution is: ； in, represents the probability density function based on t distribution, υ represents the degree of freedom parameter, μ represents the location parameter of the t-distribution, σ represents the scale parameter of t-distribution, and Γ( ) represents the gamma function.

9. The method according to claim 8, characterized in that The expression of the second observed variable generating function is: ； in, represents the second observed variable generating function, x t represents the observed variable at time step t, z t represents the potential state data at time step t, h t represents the hidden state parameter of the recurrent neural network at time step t, ST( ) represents the tail weight distribution function, NN 1( ) represents the network function used to calculate the mean of the distribution, NN 2( ) represents the network function used to calculate the distribution scale, NN 3( ) represents the network function used to calculate the degree of freedom parameter of the t distribution, SoftPlus ( ) represents the activation function.

10. A multi-format data management platform for smart integrated energy, which uses the method of any one of claims 1 to 9 to manage multi-format data, characterized in that: The multi-format data management platform (1) comprises a data collection management module (11), a model building optimization module (12) and a data monitoring and repair module (13), wherein: The data collection management module (11) is used to collect original business data of different sources and formats, perform data standardization processing, obtain standard business data, and perform metadata management on the standard business data to obtain multi-business data, wherein the multi-business data includes observed variables, potential state data, and hidden state parameters; The model building optimization module (12) is used to respectively build an initial generation model and an initial inference model according to the observed variables, the potential state data and the hidden state parameters, and train the initial inference model based on the encoder and recurrent neural network of the variational autoencoder, and train the initial generation model based on the decoder and recurrent neural network of the variational autoencoder to respectively obtain an optimized inference model and an optimized generation model, and build a robust deep state space model based on the optimized inference model, the optimized generation model, the initial deep state space model and the variational lower bound function; The data monitoring and repairing module (13) is used to input the multi-format data into the robust deep state space model to perform quality detection and repair on the multi-format data.

Images