Missing Data Reconstruction Method, System, Device and Storage Medium Based on Autoencoder

Abstract

The present invention discloses a method, system, device and storage medium for reconstructing missing data based on an autoencoder, which relates to the technical field of power market data processing. It includes collecting power market data indicators, preprocessing the collected data, using an adaptive network architecture search algorithm to search for and construct the network structures of an autoencoder and an LSTM, using the autoencoder to extract features from the preprocessed data, extracting the latent features of the data through the encoder and decoder, inputting the feature data into the LSTM network model, using a generative adversarial network to generate missing power market data, using the generated power market data to train the LSTM network model, using the trained model to reconstruct the missing data, and evaluating the reconstruction effect through actual data. The present invention can effectively solve the problem of data missing and improve the integrity and reliability of power market data.

Description

Technical Field

[0001] The present invention relates to the technical field of power market data processing, and particularly to a method, system, device and storage medium for reconstructing missing data based on an autoencoder. Background Art

[0002] In the power market, the integrity and accuracy of data are of crucial importance. Power market data includes various aspects of information such as power load and price. These data not only support the daily operation of the power market but also serve as important bases for power system planning, operation optimization, and risk assessment. However, in actual operation, power market data often has missing values due to reasons such as equipment failures, communication interruptions, and human errors. If these missing data are not processed, they will have a serious impact on the analysis and prediction of the power market, thereby affecting the operation efficiency and stability of the power system. In recent years, with the development of deep learning technology, many studies have begun to attempt to apply deep learning to missing data processing. An autoencoder can effectively extract features of data and capture the latent features and structural information of the data by constructing an encoder-decoder structure. In dealing with missing data, the autoencoder can generate completed data similar to the original data structure by learning the low-dimensional representation of the data. However, when processing time series data, it is difficult for the autoencoder to fully consider the time dependence. Long Short-Term Memory (LSTM) network is a special recurrent neural network that can effectively capture the long-term dependence relationships in time series data. In power market data prediction, LSTM has shown superior performance. However, when using LSTM alone to process missing data, it often faces the problem of insufficient training data, resulting in poor generalization ability of the model. Although the autoencoder and LSTM each have their own advantages in dealing with missing data, the application of a single method often has limitations. Existing technologies usually have difficulty in simultaneously taking into account data feature extraction and the capture of time dependence relationships when facing data missing problems, resulting in insufficient accuracy and reliability of the reconstructed data. In view of the above deficiencies of the existing technologies, the present invention proposes a method for reconstructing missing power market data based on the combination of an autoencoder and LSTM. This method extracts data features through the autoencoder and uses the LSTM network to capture the long-term dependence relationships of the time series, realizing high-precision reconstruction of missing data, solving the deficiencies existing in the existing technologies, and significantly improving the accuracy and reliability of data reconstruction.

[0003] By combining the advantages of autoencoders and LSTMs, the present invention can effectively handle missing problems in electricity market data, improve the integrity and reliability of the data, and thus enhance the accuracy of electricity market analysis and prediction. Autoencoders can effectively extract the latent features of the data and improve the accuracy of data reconstruction; LSTMs can capture long-term dependencies in time series data, addressing the deficiency of traditional methods that ignore time dependencies. By combining the advantages of autoencoders and LSTMs, the present invention can still maintain a high reconstruction accuracy in the case of missing data and has strong generalization ability. In summary, based on the existing technology, the present invention proposes a new method for reconstructing missing data in the electricity market, which can effectively solve the problem of missing data, improve the integrity and reliability of electricity market data, and has important application value. Summary of the Invention

[0004] In view of the above existing problems, the present invention provides a method, system, device and storage medium for reconstructing missing data based on an autoencoder to solve the problem of missing data.

[0005] To solve the above technical problems, a method for reconstructing missing data based on an autoencoder is proposed, including,

[0006] Collect electricity market data indicators and preprocess the collected data; use an adaptive network architecture search algorithm to search for and construct the network structures of the autoencoder and LSTM, use the autoencoder to extract features from the preprocessed data, extract the latent features of the data through the encoder and decoder, and input the feature data into the LSTM network model; use a generative adversarial network to generate missing electricity market data, use the generated electricity market data to train the LSTM network model, use the trained model to reconstruct the missing data, and evaluate the reconstruction effect through actual data.

[0007] As a preferred solution of the method for reconstructing missing data based on an autoencoder according to the present invention, wherein: the collection of electricity market data indicators includes collecting electricity load data, electricity price data, supply-demand relationship data, weather-related data, user power consumption data, power grid operation data, social and economic data, holiday and special event data, energy market data, and electricity market policy data.

[0008] The weather-related data includes temperature, humidity, wind speed, and solar radiation intensity; the power grid operation data includes power grid load rate, transmission line loss, and transformer load conditions.

[0009] The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods.

[0010] The formula for anomaly detection using a deep learning model is:

[0011] ,

[0012] ,

[0013] wherein, is the comprehensive index for anomaly detection, N is the size of the dataset, is the c-th data point, is the variance of the data point distribution, is the mean of the data points, is the dimension of the data points, is the covariance matrix, is the loss function, is the output of the deep learning model after removing the c-th data point, is the parameter of the deep learning model, P is the exponential function, is the normalization constant, is the transpose of the difference vector; when is 0, it indicates that the detected data point is an abnormal data point and is removed; when is 1, it indicates that the detected data point is a normal data point and subsequent processing continues.

[0014] As a preferred solution of the method for reconstructing missing data based on an autoencoder according to the present invention, wherein: the adaptive network architecture search algorithm includes searching and constructing the network structures of the autoencoder and LSTM using the adaptive network architecture search algorithm.

[0015] The construction of the network structures of the autoencoder and LSTM includes defining the search space of the network architecture, selecting the neural architecture search algorithm. In the search algorithm, iterative search is performed, and in each iteration, the network architectures in the current population are evaluated according to the evaluation function, and selection, crossover, and mutation operations are performed on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the performance requirements of the search algorithm are met, the network architecture with the best evaluation result is selected as the final network structures of the autoencoder and LSTM.

[0016] As a preferred solution of the method for reconstructing missing data based on an autoencoder according to the present invention, wherein: the feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder.

[0017] The integrated periodic component analysis includes performing a Fourier transform on the preprocessed data to convert the time series data into frequency domain data and capture the periodic components in the data. The encoder maps the input data into a low-dimensional space to extract the latent features of the data, and the decoder reconstructs the low-dimensional features into the original data. The periodic features extracted by the Fourier transform are used as the input of the autoencoder to capture the periodic components in the data. In the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data.

[0018] The Fourier transform formula is:

[0019] ,

[0020] where, is the frequency domain data, is the time domain signal, b is the frequency, i is the imaginary unit, t is the time, is the complex exponential function.

[0021] As a preferred solution of the missing data reconstruction method based on the autoencoder according to the present invention, wherein: the generation of the missing power market data includes using a generative adversarial network to generate the missing power market data.

[0022] The formula of the generative adversarial network is expressed as:

[0023] ,

[0024] where, is the estimated missing data, G is the generator network, are the generator parameters, is the noise sample, is the feature function, is the observable data, A is the number of observable data points, is the mean of the noise distribution, is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient output by the generator, and l is the exponential parameter of the exponential function.

[0025] The exponential parameter is expressed as:

[0026] ,

[0027] where, is the noise sample, is the mean of the noise distribution, is the standard deviation of the noise distribution, and l is the exponential parameter of the exponential function.

[0028] The gradient output by the generator is expressed as:

[0029] ,

[0030] where n is the gradient output by the generator, G is the generator network, are the generator parameters, is the noise sample.

[0031] As a preferred solution of the missing data reconstruction method based on the autoencoder according to the present invention, wherein: the training LSTM network model includes training the LSTM network model with the missing power market data generated by the generative adversarial network.

[0032] The LSTM network model includes LSTM cells, and the calculation formula of the LSTM cells is:

[0033] ,

[0034] ,

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] where is the output of the forget gate, is the output of the input gate, is the output of the output gate, is the candidate cell state, is the cell state, is the hidden state, , , and are weights, , , and are biases, is the Sigmoid activation function, is the hyperbolic tangent activation function, is the element-wise product.

[0040] The training LSTM network model includes training the LSTM network with a loss function, and the loss function formula is expressed as:

[0041] ,

[0042] Among them, L is the loss function, M is the number of samples in the batch, is the true value, is the predicted value, are the network parameters.

[0043] As a preferred solution of the missing data reconstruction method based on the autoencoder according to the present invention, wherein: evaluating the reconstruction effect includes reconstructing the missing data using the trained model and evaluating the reconstruction effect through the actual data.

[0044] Reconstructing the missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, the model will output the predicted value of the missing data, performing inverse normalization on the predicted value to obtain the original data range, and combining the predicted missing data with the known complete data to form a complete power market data set.

[0045] The inverse normalization formula is expressed as:

[0046] ,

[0047] wherein, is the predicted data after inverse normalization, is the predicted data after normalization, is the maximum value of the original data, is the minimum value of the original data.

[0048] Evaluating the reconstruction effect includes splitting the original data set into a training set, a validation set and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the indicators of the loss function value, accuracy rate and recall rate on the validation set meet the standards.

[0049] The accuracy rate is expressed as:

[0050] ,

[0051] wherein, I is the accuracy rate, TP is the number of positive class samples correctly predicted, is the number of negative class samples correctly predicted, is the total number of samples in the validation set.

[0052] The recall rate is expressed as:

[0053] ,

[0054] wherein, R is the recall rate, TP is the number of positive class samples correctly predicted, FN is the number of positive class samples mispredicted as negative class samples.

[0055] Another object of the present invention is to provide a system for reconstructing missing data based on an autoencoder, which improves the integrity and reliability of power market data; by combining the advantages of an autoencoder and LSTM, the system of the present invention can effectively handle the missing problems in power market data, improve the integrity and reliability of the data, and thus enhance the accuracy of power market analysis and prediction.

[0056] As a preferred embodiment of the system for reconstructing missing data based on an autoencoder according to the present invention, it is characterized by including

[0057] A data acquisition module for collecting power market data indicators.

[0058] A preprocessing module for preprocessing the collected data.

[0059] A feature extraction module for extracting features from the preprocessed data. By integrating periodic component analysis through an encoder and a decoder, and capturing periodic components through Fourier transform, in the encoder part of the autoencoder, the original data and periodic features are fused and feature extraction is performed.

[0060] A reconstruction LSTM module for defining the search space of the network architecture, selecting a neural architecture search algorithm, performing iterative search, iteratively evaluating the network architectures in the current population, and performing selection, crossover, and mutation operations on the population according to the evaluation results, dynamically adjusting the search space until the search algorithm meets the performance requirements.

[0061] A model training module for defining the calculation formula of the LSTM unit, training using a loss function, and reconstructing missing data using the trained model.

[0062] A model evaluation module for obtaining the original data range using the anti-normalized predicted values, combining the predicted missing data with the complete data, and evaluating the reconstruction effect by calculating the loss function value, accuracy, and recall rate.

[0063] A computer device includes a memory and a processor. The memory stores a computer program. It is characterized in that when the processor executes the computer program, the steps of the method for reconstructing missing data based on an autoencoder are implemented.

[0064] A computer-readable storage medium stores a computer program. It is characterized in that when the computer program is executed by a processor, the steps of the method for reconstructing missing data based on an autoencoder are implemented.

[0065] Advantages of the present invention: The present invention combines the advantages of autoencoders and LSTM. The autoencoder can effectively capture the internal structure information of data by extracting the latent features of the data, improving the accuracy of feature extraction; the LSTM network can capture the long-term dependencies in time series data and effectively handle the dynamic changes of sequence data. Therefore, through the joint modeling of the autoencoder and LSTM, the accuracy of missing data reconstruction can be greatly improved, ensuring that the reconstructed data is closer to the actual situation. When dealing with missing data in the power market, the existing technologies often rely on a single statistical method or machine learning model, and are prone to problems such as overfitting or insufficient generalization ability when the data changes. By using the autoencoder for feature extraction and LSTM to capture time dependencies, the advantages of both are combined, significantly improving the generalization ability of the model on different datasets and enabling it to better adapt to the complexity and diversity of data changes in the power market. Traditional methods such as mean filling and interpolation are limited in dealing with large-scale missing data, while the present invention can still maintain a high reconstruction accuracy in the case of large-scale missing data through deep learning technology. The autoencoder can learn the latent features of the data and provide reliable feature representations even when there are many missing data; LSTM can utilize the historical information of the time series to generate reasonable prediction results, thus effectively coping with the problem of large-scale data missing. By combining the advantages of the autoencoder and LSTM, a new method for reconstructing missing data in the power market is proposed, significantly improving the accuracy and reliability of data reconstruction, enhancing the generalization ability of the model and the ability to handle large-scale missing data, improving the efficiency and reliability of data preprocessing, adapting to the actual business needs, and providing a modular system design. Brief Description of the Drawings

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, where:

[0067] Figure 1 It is the overall flowchart of the method for reconstructing missing data based on autoencoder provided by an embodiment of the present invention.

[0068] Figure 2 It is the system scheme flowchart of the system for reconstructing missing data based on autoencoder provided by an embodiment of the present invention. Detailed Embodiments

[0069] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0070] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0071] Example 1, referring to Figure 1 , which is the first embodiment of the present invention. This embodiment provides a method for reconstructing missing data based on an autoencoder, including:

[0072] S1: Collect power market data indicators and preprocess the collected data.

[0073] The collection of power market data indicators includes collecting power load data, power price data, supply-demand relationship data, weather-related data, user power consumption data, grid operation data, social and economic data, holiday and special event data, energy market data, and power market policy data.

[0074] The weather-related data includes temperature, humidity, wind speed, and solar radiation intensity; the grid operation data includes grid load rate, transmission line loss, and transformer load condition.

[0075] The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods.

[0076] The formula for anomaly detection using a deep learning model is:

[0077] ,

[0078] ,

[0079] where, is the comprehensive index of anomaly detection, N is the size of the dataset, is the c-th data point, is the variance of the data point distribution, is the mean of the data point, is the dimension of the data point, is the covariance matrix, is the loss function, is the output of the deep learning model after removing the c-th data point, are the parameters of the deep learning model, P is the exponential function, is the normalization constant, is the transpose of the difference vector; when is 0, it indicates that the detected data point is an abnormal data point and is removed; when is 1, it indicates that the detected data point is a normal data point and subsequent processing continues.

[0080] S2: Use the adaptive network architecture search algorithm to search for and construct the network structures of the autoencoder and LSTM. Use the autoencoder to extract features from the preprocessed data, extract the latent features of the data through the encoder and decoder, and input the feature data into the LSTM network model.

[0081] The adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search for and construct the network structures of the autoencoder and LSTM.

[0082] The formula of the adaptive network architecture search algorithm is:

[0083] ,

[0084] where, is the optimization objective function of the adaptive network architecture search algorithm, is the reconstruction error function of the autoencoder, is the input data, α are the parameters of the autoencoder, is the prediction error function of LSTM, are the parameters of LSTM, M is the number of data samples, is the mean of the normal distribution, is the standard deviation of the normal distribution, is the integration variable.

[0085] The construction of the network structures of the autoencoder and LSTM includes defining the search space of the network architecture, selecting the neural architecture search algorithm. In the search algorithm, iterative search is carried out, and in each iteration, the network architectures in the current population are evaluated according to the evaluation function, and selection, crossover and mutation operations are performed on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the performance requirements of the search algorithm are met, the network architecture with the best evaluation result is selected as the final network structures of the autoencoder and LSTM.

[0086] The feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder.

[0087] The integrated periodic component analysis includes performing a Fourier transform on the preprocessed data, converting the time series data into frequency domain data, and capturing the periodic components in the data. The encoder maps the input data to a low-dimensional space to extract the latent features of the data. The decoder reconstructs the low-dimensional features into the original data. The periodic features extracted by the Fourier transform are used as the input of the autoencoder to capture the periodic components in the data. In the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data.

[0088] The Fourier transform formula is:

[0089] ,

[0090] where, is the frequency domain data, is the time domain signal, b is the frequency, i is the imaginary unit, t is the time, is the complex exponential function.

[0091] S3: Use a generative adversarial network to generate the missing electricity market data, use the generated electricity market data to train the LSTM network model, use the trained model to reconstruct the missing data, and evaluate the reconstruction effect through the actual data.

[0092] The generation of the missing electricity market data includes using a generative adversarial network to generate the missing electricity market data.

[0093] The generative adversarial network formula is expressed as:

[0094] ,

[0095] where, is the estimated missing data, G is the generator network, is the generator parameter, is the noise sample, is the feature function, is the observable data, A is the number of observable data points, is the mean of the noise distribution, is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient output by the generator, and l is the exponential parameter of the exponential function.

[0096] The exponential parameter is expressed as:

[0097] ,

[0098] where, is the noise sample, is the mean of the noise distribution, is the standard deviation of the noise distribution, and l is the exponential parameter of the exponential function.

[0099] The gradient output by the generator is expressed as:

[0100] ,

[0101] where n is the gradient output by the generator, G is the generator network, is the generator parameter, is the noise sample. The training of the LSTM network model includes training the LSTM network model with the missing power market data generated by the generative adversarial network.

[0102] The LSTM network model includes LSTM cells, and the calculation formula of the LSTM cell is:

[0103] ,

[0104] ,

[0105] ,

[0106] ,

[0107] ,

[0108] ,

[0109] where, is the output of the forget gate, is the output of the input gate, is the output of the output gate, is the candidate cell state, is the cell state, is the hidden state, 、 、 and are weights, 、 、 and are biases, is the Sigmoid activation function, is the hyperbolic tangent activation function, is the element-wise product.

[0110] The training of the LSTM network model includes training the LSTM network using the loss function, and the formula of the loss function is expressed as:

[0111] ,

[0112] Among them, L is the loss function, M is the number of samples in a batch, is the true value, is the predicted value, and

[0113] The evaluation of the reconstruction effect includes using the trained model to reconstruct the missing data and evaluating the reconstruction effect through the actual data.

[0114] The reconstruction of the missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, and the model will output the predicted values of the missing data. The predicted values are de-normalized to obtain the original data range, and the predicted missing data is combined with the known complete data to form a complete power market dataset.

[0115] The de-normalization formula is expressed as:

[0116] where

[0117] is the predicted data after de-normalization, is the predicted data after normalization, is the maximum value of the original data, is the minimum value of the original data. is the minimum value of the original data.

[0118] The evaluation of the reconstruction effect includes splitting the original dataset into a training set, a validation set, and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the indicators of the loss function value, accuracy, and recall rate on the validation set meet the standards.

[0119] The accuracy is expressed as:

[0120] where

[0121] I is the accuracy, TP is the number of correctly predicted positive class samples, is the number of correctly predicted negative class samples, is the total number of samples in the validation set.

[0122] The recall rate is expressed as:

[0123] where

[0124] R is the recall rate, TP is the number of correctly predicted positive class samples, and FN is the number of positive class samples mispredicted as negative class samples.

[0125] When the accuracy rate is greater than or equal to 80% and the recall rate is greater than or equal to 70%, it indicates that the reconstructed missing data meets the normal level. When the accuracy rate is less than 80% or the recall rate is less than 70%, it indicates that the reconstructed missing data does not meet the normal level, and the missing data should be regenerated until it meets the normal level.

[0126] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

[0127] Embodiment 2, the second embodiment of the present invention, which is different from the previous embodiment in that:

[0128] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0129] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0130] More specific examples (nonexhaustive list) of computer-readable media include the following: electrical connections (electronic devices) having one or more wirings, portable computer disk cartridges (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0131] It should be understood that the various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0132] Example 3, referring to Figure 2 , is the third embodiment of the present invention. This embodiment provides a missing data reconstruction system based on an autoencoder, including

[0133] A data acquisition module 100 for collecting power market data metrics.

[0134] A preprocessing module 200 for preprocessing the collected data.

[0135] A feature extraction module 300 for extracting features from the preprocessed data. Through an encoder and a decoder, it integrates periodic component analysis and captures periodic components through Fourier transform. In the encoder part of the autoencoder, it fuses the original data and periodic features and performs feature extraction.

[0136] A reconstruction LSTM module 400 for defining the search space of the network architecture, selecting a neural architecture search algorithm, performing iterative search, iteratively evaluating the network architectures in the current population, and performing selection, crossover, and mutation operations on the population according to the evaluation results, dynamically adjusting the search space until the search algorithm meets the performance requirements.

[0137] A model training module 500 for defining the calculation formula of the LSTM unit, training using a loss function, and reconstructing missing data using the trained model.

[0138] The model evaluation module 600 is used to obtain the original data range by using the anti-normalized predicted values, combine the predicted missing data with the complete data, and evaluate the reconstruction effect by calculating the loss function value, accuracy, and recall rate.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A method for reconstructing missing data based on an autoencoder, characterized in that: including Collecting power market data metrics and preprocessing the collected data; Using an adaptive network architecture search algorithm to search for and construct the network structures of an autoencoder and LSTM, using the autoencoder to extract features from the preprocessed data, extracting the latent features of the data through the encoder and decoder, and inputting the latent feature data into the LSTM network model; Using a generative adversarial network to generate missing power market data, using the generated power market data to train the LSTM network model, using the trained model to reconstruct the missing data, and evaluating the reconstruction effect through actual data; The collecting of power market data metrics includes collecting power load data, power price data, supply-demand relationship data, weather-related data, user power consumption data, power grid operation data, social and economic data, holiday and special event data, energy market data, and power market policy data; The weather-related data includes temperature, humidity, wind speed, and solar radiation intensity; the power grid operation data includes power grid load rate, transmission line loss, and transformer load condition; The preprocessing includes data cleaning, removing outliers using anomaly detection based on a deep learning model, and data normalization based on statistical methods; The formula for anomaly detection using a deep learning model is: Among them, Ω(x) is the comprehensive index for anomaly detection, N is the size of the dataset, x c is the c-th data point, σ 2 is the variance of the data point distribution, μ is the mean of the data points, d is the dimension of the data points, Σ is the covariance matrix, is the loss function, f(x c ; θ) is the output of the deep learning model after removing the c-th data point, θ is the parameter of the deep learning model, P is the exponential function, is the normalization constant, (x c -μ) T is the transpose of the difference vector; when Ω(x) is 0, it means the detected data point is an abnormal data point and is removed; when Ω(x) is 1, it means the detected data point is a normal data point and subsequent processing continues; The adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search for and construct the network structures of an autoencoder and LSTM; The adaptive network architecture search algorithm includes using the adaptive network architecture search algorithm to search for and construct the network structures of an autoencoder and LSTM; The formula of the adaptive network architecture search algorithm is: Among them, F is the optimization objective function of the adaptive network architecture search algorithm, and g(y j , α) is the reconstruction error function of the autoencoder, where y j is the input data, α is the parameter of the autoencoder, h(y j , β) is the prediction error function of the LSTM, β is the parameter of the LSTM, M is the number of data samples, γ is the mean of the normal distribution, ε is the standard deviation of the normal distribution, and y is the integration variable; The construction of the network structures of the autoencoder and LSTM includes defining the search space of the network architecture, selecting a neural architecture search algorithm. In the search algorithm, iterative search is performed, and in each iteration, the network architectures in the current population are evaluated according to the evaluation function, and selection, crossover, and mutation operations are performed on the population according to the evaluation results. During the search process, the search space is dynamically adjusted according to the feedback and evaluation results of the search algorithm. When the performance requirements of the search algorithm are met, the network architecture with the best evaluation result is selected as the final network structures of the autoencoder and LSTM; The feature extraction includes integrating periodic component analysis into the autoencoder for data extraction and adding regularization constraints to the autoencoder; The integration of periodic component analysis includes performing Fourier transform on the preprocessed data, converting the time series data into frequency domain data, and capturing the periodic components in the data. The encoder maps the input data to a low-dimensional space to extract the latent features of the data, and the decoder reconstructs the low-dimensional features into the original data. The periodic features extracted by the Fourier transform are used as the input of the autoencoder to capture the periodic components in the data. In the encoder part of the autoencoder, the original data and the periodic features are fused to form new input data; The formula for Fourier transform is: where f(a) is the frequency-domain data, a(t) is the time-domain signal, b is the frequency, i is the imaginary unit, t is the time, and e -i2πzt is the complex exponential function; The generation of missing power market data includes using a generative adversarial network to generate missing power market data; The generative adversarial network is represented by the formula: Among them, D(z) is the estimated missing data, G is the generator network, τ is the generator parameter, z j is the noise sample, f k is the feature function, y k is the observable data, A is the number of observable data points, σ z is the standard deviation of the noise distribution, B is the number of samples generated by the generator, n is the gradient output by the generator, and l is the exponential parameter of the exponential function; The exponential parameter is represented as: where z j is the noise sample, μ z is the mean of the noise distribution, σ z is the standard deviation of the noise distribution, and l is the exponential parameter of the exponential function; The gradient output by the generator is expressed as: where n is the gradient output by the generator, G is the generator network, τ is the generator parameter, and z j is a noise sample.

2. The method for reconstructing missing data based on an autoencoder according to claim 1, characterized in that: The training of the LSTM network model includes training the LSTM network model with the missing power market data generated by the generative adversarial network. The LSTM network model includes LSTM units, and the calculation formula of the LSTM unit is: f t = σ(W f · [h t-1 , x t + b f ) i t = σ(W i ·[h t-1 , x t + b i ) o t = σ(W o · [h t-1 , x t + b o ) h t = o t ⊙tanh(C t ) Among them, f t is the output of the forget gate, i t is the output of the input gate, o t is the output of the output gate, is the candidate cell state, C t is the cell state, h t is the hidden state, W f 、W i 、W C and W o are weights, b f 、b i 、b C and b o are biases, σ is the Sigmoid activation function, tanh is the hyperbolic tangent activation function, and ⊙ is the element-wise product; The training of the LSTM network model includes training the LSTM network using a loss function, and the formula of the loss function is expressed as: Among them, L is the loss function, M is the number of samples in the batch, g m is the true value, is the predicted value, and ω is the network parameter.

3. The method for reconstructing missing data based on an autoencoder according to claim 2, wherein: Evaluating the reconstruction effect includes reconstructing the missing data using the trained model and evaluating the reconstruction effect through the actual data. Reconstructing the missing data includes inputting the preprocessed time series data into the LSTM network, passing the input data to the trained LSTM network model, the model will output the predicted values of the missing data, performing inverse normalization on the predicted values to obtain the original data range, and combining the predicted missing data with the known complete data to form a complete power market data set. The inverse normalization formula is expressed as: E pred = E norm × (E max - E min ) + E min Among them, E pred is the predicted data after anti-normalization, and E norm is the predicted data after normalization, and E max is the maximum value of the original data, and E min is the minimum value of the original data; Evaluating the reconstruction effect includes splitting the original data set into a training set, a validation set, and a test set, and evaluating the reconstruction effect of the missing data by calculating whether the indicators of the loss function value, accuracy, and recall rate on the validation set meet the standards. The accuracy is expressed as: where I is the accuracy, TP is the number of positive class samples correctly predicted, TN is the number of negative class samples correctly predicted, and TS is the total number of samples in the validation set. The recall rate is expressed as: where R is the recall rate, TP is the number of positive class samples correctly predicted, and FN is the number of positive class samples mispredicted as negative class samples. When the accuracy is greater than or equal to 80% and the recall rate is greater than or equal to 70%, it indicates that the reconstructed missing data meets the normal level. When the accuracy is less than 80% or the recall rate is less than 70%, it indicates that the reconstructed missing data does not meet the normal level, and the missing data should be regenerated until it meets the normal level.

4. A system adopting a method for reconstructing missing data based on an autoencoder as described in any one of claims 1 to 3, characterized in that: including a data acquisition module for collecting power market data indicators; a preprocessing module for preprocessing the collected data; a feature extraction module for extracting features from the preprocessed data, integrating periodic component analysis through an encoder and a decoder, capturing periodic components through Fourier transform, and fusing the original data and periodic features in the encoder part of the autoencoder for feature extraction; a reconstruction LSTM module for defining the search space of the network architecture, selecting a neural architecture search algorithm, performing iterative search, iteratively evaluating the network architectures in the current population, and performing selection, crossover, and mutation operations on the population according to the evaluation results, dynamically adjusting the search space until the search algorithm meets the performance requirements; a model training module for defining the calculation formula of the LSTM unit, training using a loss function, and reconstructing the missing data using the trained model; a model evaluation module for obtaining the original data range using the inverse normalization predicted values, combining the predicted missing data with the complete data, and evaluating the reconstruction effect by calculating the loss function value, accuracy, and recall rate.

5. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for reconstructing missing data based on an autoencoder according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for reconstructing missing data based on an autoencoder according to any one of claims 1 to 3.

Images