Missing data reconstruction method, system and device based on auto-encoder and storage medium

Abstract

The invention discloses a missing data reconstruction method, system and equipment based on an auto-encoder, and a storage medium, and relates to the technical field of power market data processing, and the method comprises the steps: collecting power market data indexes, carrying out the preprocessing of the collected data, employing a self-adaptive network architecture search algorithm to search and construct a network structure of the auto-encoder and LSTM, and carrying out the reconstruction of missing data based on the auto-encoder. Performing feature extraction on the preprocessed data by using an auto-encoder, extracting potential features of the data by using an encoder and a decoder, inputting feature data into an LSTM network model, generating missing power market data by using a generative adversarial network, training the LSTM network model by using the generated power market data, and obtaining the missing power market data. And reconstructing missing data by using the trained model, and evaluating a reconstruction effect through actual data. The problem of data missing can be effectively solved, and the integrity and reliability of power market data are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power market data processing, and in particular to a method, system, device and storage medium for reconstructing missing data based on an autoencoder. Background Art

[0002] In the power market, the integrity and accuracy of data are crucial. Power market data includes information on power load, price, and other aspects. These data not only provide support for the daily operation of the power market, but also are important bases for power system planning, operation optimization, and risk assessment. However, in actual operation, power market data is often missing due to equipment failure, communication interruption, human error, and other reasons. If these missing data are not processed, they will have a serious impact on the analysis and prediction of the power market, and then affect the operating efficiency and stability of the power system. In recent years, with the development of deep learning technology, many studies have begun to try to apply deep learning to missing data processing. By constructing an encoder-decoder structure, the autoencoder can effectively extract features from the data and capture the potential features and structural information of the data. In the missing data processing, the autoencoder can generate complementary data with a similar structure to the original data by learning the low-dimensional representation of the data. However, when processing time series data, it is difficult for the autoencoder to fully consider time dependency. The long short-term memory network is a special recursive neural network that can effectively capture long-term dependencies in time series data. In the prediction of power market data, LSTM has shown superior performance, but when LSTM is used alone to process missing data, it often faces the problem of insufficient training data, resulting in poor model generalization ability; although autoencoders and LSTMs each have their own advantages in processing missing data, the application of a single method often has limitations. When facing the problem of missing data, the prior art usually finds it difficult to simultaneously take into account data feature extraction and the capture of time dependencies, resulting in insufficient accuracy and reliability of reconstructed data. In view of the above-mentioned deficiencies in the prior art, the present invention proposes a method for reconstructing missing data in the power market based on a combination of autoencoders and LSTMs. The method extracts data features through an autoencoder and uses an LSTM network to capture the long-term dependencies of time series, thereby achieving high-precision reconstruction of missing data, solving the deficiencies in the prior art and significantly improving the accuracy and reliability of data reconstruction.

[0003] By combining the advantages of autoencoders and LSTMs, the present invention can effectively handle the missing data problem in the electricity market, improve the integrity and reliability of the data, and thus enhance the accuracy of electricity market analysis and prediction. The autoencoder can effectively extract the potential features of the data and improve the accuracy of data reconstruction; the LSTM can capture the long-term dependencies in time series data, solving the problem that traditional methods ignore time dependencies. By combining the advantages of autoencoders and LSTMs, the present invention can maintain a high reconstruction accuracy in the case of missing data and has a strong generalization ability. In summary, based on the prior art, the present invention proposes a new method for reconstructing missing data in the electricity market, which can effectively solve the problem of missing data, improve the integrity and reliability of electricity market data, and has important application value. Summary of the invention

[0004] In view of the above-mentioned existing problems, the present invention provides a method, system, device and storage medium for reconstructing missing data based on an autoencoder to solve the problem of missing data.

[0005] In order to solve the above technical problems, a missing data reconstruction method based on autoencoder is proposed, including: Collect electricity market data indicators and preprocess the collected data; use the adaptive network architecture search algorithm to search and construct the network structure of the autoencoder and LSTM, use the autoencoder to extract features from the preprocessed data, extract the potential features of the data through the encoder and decoder, and input the feature data into the LSTM network model; use the generative adversarial network to generate missing electricity market data, use the generated electricity market data to train the LSTM network model, use the trained model to reconstruct the missing data, and evaluate the reconstruction effect through actual data.

[0006] As a preferred solution of the method for reconstructing missing data based on an autoencoder described in the present invention, the collection of electricity market data indicators includes collecting electricity load data, electricity price data, supply and demand relationship data, weather-related data, user power consumption data, power grid operation data, socio-economic data, holidays and special events data, energy market data and electricity market policy data.

[0007] The weather-related data include temperature, humidity, wind speed and solar radiation intensity; the power grid operation data include power grid load rate, transmission line loss and transformer load condition.

[0008] The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods.

[0009] The anomaly detection formula based on the deep learning model is: , , in, is the comprehensive indicator of anomaly detection, N is the size of the data set, is the cth data point, is the variance of the data point distribution, is the mean of the data points, is the dimension of the data point, is the covariance matrix, is the loss function, is the output of the deep learning model after removing the cth data point, is the parameter of the deep learning model, P is the exponential function, is the normalization constant, is the transpose of the difference vector; when When it is 0, it means that the detected data point is an abnormal data point and is removed; when When it is 1, it indicates that the detected data point is a normal data point and the subsequent processing continues.

[0010] As a preferred solution of the method for reconstructing missing data based on an autoencoder described in the present invention, the adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search and construct a network structure of an autoencoder and LSTM.

[0011] The method for constructing the network structure of the autoencoder and LSTM includes defining the search space of the network architecture, selecting a neural architecture search algorithm, performing an iterative search in the search algorithm, and evaluating the network architecture in the current population according to an evaluation function in each iteration, and performing selection, crossover and mutation operations on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the search algorithm meets the performance requirements, the network architecture with the best evaluation results is selected as the final autoencoder and LSTM network structure.

[0012] As a preferred solution of the method for reconstructing missing data based on an autoencoder described in the present invention, the feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder.

[0013] The integrated periodic component analysis includes: performing Fourier transform on the preprocessed data, converting the time series data into frequency domain data, and capturing the periodic components in the data; the encoder maps the input data to a low-dimensional space and extracts the potential features of the data; the decoder reconstructs the low-dimensional features into the original data, and uses the periodic features extracted by the Fourier transform as the input of the autoencoder and captures the periodic components of the data; in the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data.

[0014] The Fourier transform formula is: , in, is the frequency domain data, is the time domain signal, b is the frequency, i is the imaginary unit, t is the time, is a complex exponential function.

[0015] As a preferred solution of the missing data reconstruction method based on the autoencoder described in the present invention, wherein: the generating of the missing electricity market data includes generating the missing electricity market data using a generative adversarial network.

[0016] The generative adversarial network formula is expressed as: , in, is the estimated missing data, G is the generator network, is the generator parameter, is a noise sample, is the characteristic function, is the observable data, A is the number of observable data points, is the mean of the noise distribution, is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient of the generator output, and l is the exponential parameter of the exponential function.

[0017] The exponential parameter is expressed as: , in, is a noise sample, is the mean of the noise distribution, is the standard deviation of the noise distribution, and l is the exponential parameter of the exponential function.

[0018] The gradient of the generator output is expressed as: , Among them, n is the gradient of the generator output, G is the generator network, is the generator parameter, is a noise sample.

[0019] As a preferred solution of the missing data reconstruction method based on the autoencoder described in the present invention, wherein: the training of the LSTM network model includes using a generative adversarial network to generate missing electricity market data to train the LSTM network model.

[0020] The LSTM network model includes LSTM units, and the calculation formula of LSTM units is: , , , , , , in, is the output of the forget gate, is the output of the input gate, is the output of the output gate, is the candidate cell state, is the cell state, is the hidden state, , , and is the weight, , , and is the bias, is the Sigmoid activation function, is the hyperbolic tangent activation function, is the element-wise product.

[0021] The training of the LSTM network model includes training the LSTM network using a loss function, and the loss function formula is expressed as: , Where L is the loss function, M is the number of samples in the batch, is the true value, is the predicted value, is the network parameter.

[0022] As a preferred solution of the method for reconstructing missing data based on an autoencoder described in the present invention, wherein: the evaluation of the reconstruction effect includes reconstructing the missing data using a trained model and evaluating the reconstruction effect through actual data.

[0023] The reconstruction of missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, the model outputting the predicted value of the missing data, denormalizing the predicted value to obtain the original data range, combining the predicted missing data with the known complete data to form a complete power market data set.

[0024] The anti-normalization formula is expressed as: , in, is the predicted data after denormalization, is the normalized prediction data, is the maximum value of the original data, is the minimum value of the original data.

[0025] The evaluation of the reconstruction effect includes dividing the original data set into a training set, a validation set and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the loss function value, accuracy and recall rate on the validation set meet the standards.

[0026] The accuracy is expressed as: , Among them, I is the accuracy, TP is the number of correctly predicted positive samples, is the number of correctly predicted negative samples, is the total number of samples in the validation set.

[0027] The recall rate is expressed as: , Among them, R is the recall rate, TP is the number of correctly predicted positive samples, and FN is the number of positive samples incorrectly predicted as negative samples.

[0028] Another object of the present invention is to provide a missing data reconstruction system based on an autoencoder, which improves the integrity and reliability of electricity market data; the system of the present invention combines the advantages of autoencoders and LSTMs to effectively handle missing data in electricity market data, improve data integrity and reliability, and thus improve the accuracy of electricity market analysis and prediction.

[0029] As a preferred solution of the autoencoder missing data reconstruction system of the present invention, it is characterized by comprising: Data acquisition module, used to collect power market data indicators.

[0030] The preprocessing module is used to preprocess the collected data.

[0031] The feature extraction module is used to extract features from the preprocessed data. Through the encoder and decoder, periodic component analysis is integrated and periodic components are captured through Fourier transform. In the encoder part of the autoencoder, the original data and periodic features are fused and feature extraction is performed.

[0032] Reconstruct the LSTM module to define the search space of the network architecture, select the neural architecture search algorithm, perform iterative search, iteratively evaluate the network architecture in the current population, and select, crossover and mutate the population based on the evaluation results, and dynamically adjust the search space until the search algorithm meets the performance requirements.

[0033] The model training module is used to define the calculation formula of the LSTM unit, use the loss function for training, and use the trained model to reconstruct missing data.

[0034] The model evaluation module is used to obtain the original data range using the denormalized prediction value, combine the missing data obtained by prediction with the complete data, and evaluate the reconstruction effect by calculating the loss function value, accuracy and recall rate.

[0035] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein when the processor executes the computer program, the steps of the method described in the method for reconstructing missing data based on an autoencoder are implemented.

[0036] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method described in the method for reconstructing missing data based on an autoencoder are implemented.

[0037] Beneficial effects of the present invention: The present invention combines the advantages of autoencoders and LSTM. The autoencoder can effectively capture the intrinsic structural information of the data by extracting the potential features of the data, and improve the accuracy of feature extraction; the LSTM network can capture the long-term dependencies in the time series data and effectively handle the dynamic changes of the sequence data. Therefore, through the joint modeling of the autoencoder and LSTM, the accuracy of the reconstruction of missing data can be greatly improved, ensuring that the reconstructed data is closer to the actual situation. When dealing with missing data in the power market, the existing technology often relies on a single statistical method or machine learning model, which is prone to overfitting or insufficient generalization when the data changes. Through the autoencoder for feature extraction, LSTM captures time dependency, and combines the advantages of both, the generalization ability of the model on different data sets is significantly improved, and it can better adapt to the complexity and diversity of data changes in the power market. Traditional methods such as mean filling and interpolation methods have limited effects when dealing with large-scale missing data, while the present invention can maintain a high reconstruction accuracy in the case of large-scale missing data through deep learning technology. The autoencoder can learn the potential features of the data, and can provide reliable feature representation even in the case of a large number of missing data; LSTM can use the historical information of the time series to generate reasonable prediction results, thereby effectively dealing with the problem of large-scale data missing. By combining the advantages of autoencoders and LSTM, a new method for reconstructing missing data in the power market is proposed, which significantly improves the accuracy and reliability of data reconstruction, enhances the generalization ability of the model and the ability to handle large-scale missing data, improves the efficiency and reliability of data preprocessing, adapts to actual business needs, and provides a modular system design. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0039] Figure 1 An overall flow chart of a method for reconstructing missing data based on an autoencoder provided for one embodiment of the present invention.

[0040] Figure 2 A system solution flow chart of a system for reconstructing missing data based on an autoencoder provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Example 1, reference Figure 1 , is the first embodiment of the present invention, which provides a method for reconstructing missing data based on an autoencoder, comprising: S1: Collect electricity market data indicators and pre-process the collected data.

[0044] The electricity market data collection indicators include collecting electricity load data, electricity price data, supply and demand relationship data, weather-related data, user power consumption data, power grid operation data, social and economic data, holidays and special events data, energy market data and electricity market policy data.

[0045] The weather-related data include temperature, humidity, wind speed and solar radiation intensity; the power grid operation data include power grid load rate, transmission line loss and transformer load condition.

[0046] The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods.

[0047] The anomaly detection formula based on the deep learning model is: , , in, is the comprehensive indicator of anomaly detection, N is the size of the data set, is the cth data point, is the variance of the data point distribution, is the mean of the data points, is the dimension of the data point, is the covariance matrix, is the loss function, is the output of the deep learning model after removing the cth data point, is the parameter of the deep learning model, P is the exponential function, is the normalization constant, is the transpose of the difference vector; when When it is 0, it means that the detected data point is an abnormal data point and is removed; when When it is 1, it indicates that the detected data point is a normal data point and the subsequent processing continues.

[0048] S2: Use the adaptive network architecture search algorithm to search and build the network structure of the autoencoder and LSTM, use the autoencoder to extract features from the preprocessed data, extract the potential features of the data through the encoder and decoder, and input the feature data into the LSTM network model.

[0049] The adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search and construct a network structure of an autoencoder and LSTM.

[0050] The formula of the adaptive network architecture search algorithm is: , in, is the optimization objective function of the adaptive network architecture search algorithm, is the reconstruction error function of the autoencoder, is the input data, α is the parameter of the autoencoder, is the prediction error function of LSTM, is the parameter of LSTM, M is the number of data samples, is the mean of the normal distribution, is the standard deviation of the normal distribution, is the integration variable.

[0051] The method for constructing the network structure of the autoencoder and LSTM includes defining the search space of the network architecture, selecting a neural architecture search algorithm, performing an iterative search in the search algorithm, and evaluating the network architecture in the current population according to an evaluation function in each iteration, and performing selection, crossover and mutation operations on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the search algorithm meets the performance requirements, the network architecture with the best evaluation results is selected as the final autoencoder and LSTM network structure.

[0052] The feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder.

[0053] The integrated periodic component analysis includes: performing Fourier transform on the preprocessed data, converting the time series data into frequency domain data, and capturing the periodic components in the data; the encoder maps the input data to a low-dimensional space and extracts the potential features of the data; the decoder reconstructs the low-dimensional features into the original data, and uses the periodic features extracted by the Fourier transform as the input of the autoencoder and captures the periodic components of the data; in the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data.

[0054] The Fourier transform formula is: , in, is the frequency domain data, is the time domain signal, b is the frequency, i is the imaginary unit, t is the time, is a complex exponential function.

[0055] S3: Generate missing electricity market data using a generative adversarial network, train the LSTM network model using the generated electricity market data, reconstruct the missing data using the trained model, and evaluate the reconstruction effect using actual data.

[0056] The generating the missing electricity market data includes generating the missing electricity market data using a generative adversarial network.

[0057] The generative adversarial network formula is expressed as: , in, is the estimated missing data, G is the generator network, is the generator parameter, is a noise sample, is the characteristic function, is the observable data, A is the number of observable data points, is the mean of the noise distribution, is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient of the generator output, and l is the exponential parameter of the exponential function.

[0058] The exponential parameter is expressed as: , in, is a noise sample, is the mean of the noise distribution, is the standard deviation of the noise distribution, and l is the exponential parameter of the exponential function.

[0059] The gradient of the generator output is expressed as: , Among them, n is the gradient of the generator output, G is the generator network, is the generator parameter, The training of the LSTM network model includes using a generative adversarial network to generate missing power market data to train the LSTM network model.

[0060] The LSTM network model includes LSTM units, and the calculation formula of LSTM units is: , , , , , , in, is the output of the forget gate, is the output of the input gate, is the output of the output gate, is the candidate cell state, is the cell state, is the hidden state, , , and is the weight, , , and is the bias, is the Sigmoid activation function, is the hyperbolic tangent activation function, is the element-wise product.

[0061] The training of the LSTM network model includes training the LSTM network using a loss function, and the loss function formula is expressed as: , Where L is the loss function, M is the number of samples in the batch, is the true value, is the predicted value, is the network parameter.

[0062] The evaluation of the reconstruction effect includes reconstructing the missing data using the trained model and evaluating the reconstruction effect using actual data.

[0063] The reconstruction of missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, the model outputting the predicted value of the missing data, denormalizing the predicted value to obtain the original data range, combining the predicted missing data with the known complete data to form a complete power market data set.

[0064] The anti-normalization formula is expressed as: , in, is the predicted data after denormalization, is the normalized prediction data, is the maximum value of the original data, is the minimum value of the original data.

[0065] The evaluation of the reconstruction effect includes dividing the original data set into a training set, a validation set and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the loss function value, accuracy and recall rate on the validation set meet the standards.

[0066] The accuracy is expressed as: , Among them, I is the accuracy, TP is the number of correctly predicted positive samples, is the number of correctly predicted negative samples, is the total number of samples in the validation set.

[0067] The recall rate is expressed as: , Among them, R is the recall rate, TP is the number of correctly predicted positive samples, and FN is the number of positive samples incorrectly predicted as negative samples.

[0068] When the accuracy is greater than or equal to 80% and the recall is greater than or equal to 70%, it means that the reconstructed missing data meets the normal level. When the accuracy is less than 80% or the recall is less than 70%, it means that the reconstructed missing data does not meet the normal level and the missing data should be regenerated until it meets the normal level.

[0069] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0070] Embodiment 2, the second embodiment of the present invention, is different from the previous embodiment in that: If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0072] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0073] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0074] Example 3, reference Figure 2 , is the third embodiment of the present invention, which provides a system for reconstructing missing data based on an autoencoder, comprising:

[0075] The data collection module 100 is used to collect power market data indicators.

[0076] The preprocessing module 200 is used to preprocess the collected data.

[0077] The feature extraction module 300 is used to extract features from the preprocessed data, integrate periodic component analysis through the encoder and decoder, and capture periodic components through Fourier transform. In the encoder part of the autoencoder, the original data and periodic features are fused and feature extraction is performed.

[0078] Reconstruct the LSTM module 400 to define the search space of the network architecture, select the neural architecture search algorithm, perform iterative search, iteratively evaluate the network architecture in the current population, and perform selection, crossover and mutation operations on the population based on the evaluation results, and dynamically adjust the search space until the search algorithm meets the performance requirements.

[0079] The model training module 500 is used to define the calculation formula of the LSTM unit, use the loss function for training, and use the trained model to reconstruct the missing data.

[0080] The model evaluation module 600 is used to obtain the original data range using the denormalized prediction value, combine the missing data obtained by prediction with the complete data, and evaluate the reconstruction effect by calculating the loss function value, accuracy and recall rate.

[0081] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for reconstructing missing data based on an autoencoder, characterized in that: include, Collect power market data indicators and pre-process the collected data; Use the adaptive network architecture search algorithm to search and build the network structure of the autoencoder and LSTM, use the autoencoder to extract features from the preprocessed data, extract the potential features of the data through the encoder and decoder, and input the feature data into the LSTM network model; A generative adversarial network is used to generate missing electricity market data, and the generated electricity market data is used to train the LSTM network model. The trained model is used to reconstruct the missing data, and the reconstruction effect is evaluated using actual data.

2. The method for reconstructing missing data based on an autoencoder according to claim 1, characterized in that: The collection of electricity market data indicators includes collecting electricity load data, electricity price data, supply and demand relationship data, weather-related data, user power consumption data, power grid operation data, social and economic data, holidays and special events data, energy market data and electricity market policy data; The weather-related data include temperature, humidity, wind speed and solar radiation intensity; the power grid operation data include power grid load rate, transmission line loss and transformer load; The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods.

3. The method for reconstructing missing data based on an autoencoder according to claim 2, characterized in that: The adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search for and construct a network structure of an autoencoder and an LSTM; The method for constructing the network structure of the autoencoder and LSTM includes defining the search space of the network architecture, selecting a neural architecture search algorithm, performing an iterative search in the search algorithm, and evaluating the network architecture in the current population according to an evaluation function in each iteration, and performing selection, crossover and mutation operations on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the search algorithm meets the performance requirements, the network architecture with the best evaluation results is selected as the final autoencoder and LSTM network structure.

4. The method for reconstructing missing data based on an autoencoder according to claim 3, characterized in that: The feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder; The integrated periodic component analysis includes: performing Fourier transform on the preprocessed data, converting the time series data into frequency domain data, and capturing the periodic components in the data; the encoder maps the input data into a low-dimensional space, extracts the potential features of the data, the decoder reconstructs the low-dimensional features into the original data, uses the periodic features extracted by the Fourier transform as the input of the autoencoder, and captures the periodic components of the data; in the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data; The Fourier transform formula is: ， in, is the frequency domain data, is the time domain signal, b is the frequency, i is the imaginary unit, t is the time, is a complex exponential function.

5. The method for reconstructing missing data based on an autoencoder according to claim 4, characterized in that: Generating missing electricity market data includes generating missing electricity market data using a generative adversarial network; The generative adversarial network formula is expressed as: ， in, is the estimated missing data, G is the generator network, is the generator parameter, is a noise sample, is the characteristic function, is the observable data, A is the number of observable data points, is the mean of the noise distribution, is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient of the generator output, and l is the exponential parameter of the exponential function; The exponential parameter is expressed as: ， in, is a noise sample, is the mean of the noise distribution, is the standard deviation of the noise distribution, l is the exponential parameter of the exponential function; The gradient of the generator output is expressed as: ， Among them, n is the gradient of the generator output, G is the generator network, is the generator parameter, is a noise sample.

6. The method for reconstructing missing data based on an autoencoder according to claim 5, characterized in that: The training of the LSTM network model includes using a generative adversarial network to generate missing electricity market data to train the LSTM network model; The LSTM network model includes LSTM units, and the calculation formula of LSTM units is: ， ， ， ， ， ， in, is the output of the forget gate, is the output of the input gate, is the output of the output gate, is the candidate cell state, is the cell state, is the hidden state, , , and is the weight, , , and is the bias, is the Sigmoid activation function, is the hyperbolic tangent activation function, is the element-wise product; The training of the LSTM network model includes training the LSTM network using a loss function, and the loss function formula is expressed as: ， Where L is the loss function, M is the number of samples in the batch, is the true value, is the predicted value, is the network parameter.

7. The method for reconstructing missing data based on an autoencoder according to claim 6, characterized in that: The evaluation of the reconstruction effect, This includes using the trained model to reconstruct missing data and evaluating the reconstruction effect using actual data; The reconstruction of missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, the model outputting the predicted value of the missing data, denormalizing the predicted value to obtain the original data range, combining the predicted missing data with the known complete data to form a complete power market data set; The anti-normalization formula is expressed as: ， in, is the predicted data after denormalization, is the normalized prediction data, is the maximum value of the original data, is the minimum value of the original data; The evaluation of the reconstruction effect includes dividing the original data set into a training set, a validation set, and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the loss function value, accuracy, and recall rate on the validation set meet the standards; The accuracy is expressed as: ， Among them, I is the accuracy, TP is the number of correctly predicted positive samples, is the number of correctly predicted negative samples, is the total number of samples in the validation set; The recall rate is expressed as: ， Among them, R is the recall rate, TP is the number of correctly predicted positive samples, and FN is the number of positive samples incorrectly predicted as negative; When the accuracy is greater than or equal to 80% and the recall is greater than or equal to 70%, it means that the reconstructed missing data meets the normal level. When the accuracy is less than 80% or the recall is less than 70%, it means that the reconstructed missing data does not meet the normal level and the missing data should be regenerated until it meets the normal level.

8. A system based on a missing data reconstruction method of an autoencoder, characterized in that: include, Data collection module, used to collect power market data indicators; A preprocessing module, used to preprocess the collected data; The feature extraction module is used to extract features from the preprocessed data. Through the encoder and decoder, the periodic component analysis is integrated and the periodic components are captured through Fourier transform. In the encoder part of the autoencoder, the original data and periodic features are fused and feature extraction is performed; Reconstruct the LSTM module to define the search space of the network architecture, select the neural architecture search algorithm, perform iterative search, iteratively evaluate the network architecture in the current population, and perform selection, crossover and mutation operations on the population based on the evaluation results, and dynamically adjust the search space until the search algorithm meets the performance requirements; Model training module, which is used to define the calculation formula of LSTM unit, use loss function for training, and use the trained model to reconstruct missing data; The model evaluation module is used to obtain the original data range using the denormalized prediction value, combine the missing data obtained by prediction with the complete data, and evaluate the reconstruction effect by calculating the loss function value, accuracy and recall rate.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for reconstructing missing data based on the autoencoder are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for reconstructing missing data based on an autoencoder are implemented.

Images