New energy station multi-source heterogeneous data synchronization and reconstruction system
By adopting multi-source heterogeneous data synchronization and reconstruction systems in new energy stations, combining deep learning and data security analysis technology, the problems of data synchronization and security verification are solved, and efficient data synchronization and secure storage are achieved, improving the efficiency of data synchronization and safe storage. The real-time and intelligent level of data processing.
Patent Information
- Application Number
- CN202410962539.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-17
AI Technical Summary
Synchronization and security verification of multi-source heterogeneous data in new energy stations is difficult to achieve, resulting in the impact of data leakage risks and overall security.
The data acquisition and recording unit, data synchronization processing unit, data security analysis unit and data reconstruction and acquisition unit are adopted, combining long-term memory networks, information entropy detection, genetic algorithms, Huffman encoding, LZW compression algorithm, distributed database, access control list, Sparse representation and iterative reconstruction algorithms ensure data synchronization, security and integrity.
It realizes efficient synchronization, secure storage and intelligent reconstruction of data, improves data management efficiency, ensures real-time, security and intelligence of data, and provides technical support for efficient operation and maintenance of the site.
Smart Images

Figure CN119066122A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-source data synchronization, and in particular to a multi-source heterogeneous data synchronization and reconstruction system for a new energy station. Background Art
[0002] With the rapid development of new energy technologies, new energy stations, as important carriers of renewable energy, face many challenges in their operation and management. Among them, how to effectively integrate data from different sources with different structures and formats to support intelligent decision-making and optimized operation of stations has become an urgent problem to be solved. The existing multi-source heterogeneous data synchronization and reconstruction system of new energy stations adopts advanced time synchronization technology to ensure that data from different sources can be collected and processed under a unified time reference.
[0003] Based on the existing technology, although data from different sources with different structures and formats can be synchronized to the system in real time and accurately, due to the complexity of the data sources, when different source data are collected and processed centrally, due to the different environments in which different source data are located, the servers providing different data sources are also different. If there is a security risk in one of the data source servers, it may cause other data sources to be leaked, thereby further affecting the safety of the entire new energy site. Summary of the invention
[0004] In view of the above existing problems, the present invention is proposed.
[0005] Therefore, the present invention provides a multi-source heterogeneous data synchronization and reconstruction system for a new energy station, and the system solves data synchronization, security and integrity verification, efficient reconstruction of complex data structures, and target data acquisition.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] In the first aspect, an embodiment of the present invention provides a multi-source heterogeneous data synchronization and reconstruction system for a new energy station, which includes a data acquisition and recording unit for acquiring log record data and status parameters corresponding to each data source in the new energy station; a data synchronization processing unit for deeply analyzing and processing the update status of the log record data, using a long short-term memory network to enhance time series analysis, and accurately synchronizing the log records to a unified time reference; a data security analysis unit for storing the log record data after the synchronization conversion is completed to a database, verifying the security of the log record data through status parameters, and automatically sending an early warning when a risk is detected; a data reconstruction acquisition unit for performing reconstruction processing operations on the verified log record data, and obtaining target data based on the reconstruction results.
[0008] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, wherein: the data acquisition and recording unit is used to obtain the log record data and state parameters corresponding to each data source in the new energy station, and the specific steps are as follows;
[0009] Clarify the types of data sources within new energy sites, including: wind turbines, photovoltaic power generation systems, energy storage systems, meteorological stations, SCADA systems, relay protection equipment, communication equipment, security systems, environmental monitoring systems, and grid access equipment;
[0010] Deploy edge computing devices in the site and install data collection software, and configure connection parameters with various data sources;
[0011] Based on the deployed edge computing devices, design the data flow architecture to ensure real-time data transmission from edge devices to the central server;
[0012] Use Huffman coding and LZW compression algorithms to reduce data transmission volume, speed up transmission, and reduce network bandwidth;
[0013] Use a distributed database system to store log data through the transmitted data;
[0014] Implement data redundancy backup strategy;
[0015] Before data is stored in the database, data integrity check is performed;
[0016] Check the data that does not meet the standards, mark it, and design a retry mechanism to retrieve the data;
[0017] Based on the above operations, set up access control lists in the database to ensure that only authorized users can access specific data;
[0018] Finally, implement the log audit function on the data to record all data access behaviors.
[0019] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the update status of the log record data is deeply analyzed and processed, and the specific steps are as follows:
[0020] Monitor the update status of log record data and record the time stamp corresponding to each update;
[0021] Combine the update status of each log record data with the synchronous response principle to record the corresponding change time mark information, and use information entropy to detect the change time mark information. The expression is:
[0022]
[0023] Among them, H(X) represents the information entropy of random variable X, n represents the number of all possible values of random variable X, that is, the size of the sample space of X, i is the index variable used to traverse the possible values of random variable X, and x i represents the i-th possible value of the random variable X, P(x i ) indicates that the random variable X takes the i-th value x i The probability of log 2 P(x i ) means the logarithmic function with base 2 is applied to P(x i );
[0024] Genetic algorithm is used to optimize the information entropy results and construct the deviation data characteristic response function, which is expressed as:
[0025] y = f(Δt; θ);
[0026] Where y is the feature response, f is the function itself, and the feature response is predicted based on the input time deviation. Δt represents the time deviation, that is, the difference between the actual update time and the expected update time of the log record data, and θ is the model parameter.
[0027] Combine the change time marking rule and the deviation data characteristic response function to extract the synchronization time deviation of the log record data;
[0028] The collected data will be processed to remove outliers and missing values;
[0029] Based on the information entropy detection results, the log record data is divided into normal data and deviation data, and a frequency domain model is defined to determine the membership mean of the deviation data;
[0030] The initial frequency mean and the corresponding standard deviation of the deviation data are set in advance, and the set results are combined with the genetic simulation technology to update and smooth the deviation data to obtain the deviation data characteristic model;
[0031] After setting the deviation data training set and deviation sample category, feature selection is performed on the deviation data to obtain the center point of the deviation data;
[0032] The center point of the deviation data is updated through the minimum square difference function value mechanism, the spectral density function of the deviation data is calculated, and the spectral density function is used as the deviation data feature to obtain the deviation data feature response function.
[0033] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the long short-term memory network model is used to enhance time series analysis and accurately synchronize log records to a unified time reference. The specific steps are as follows:
[0034] Based on the deviation data feature response function, the processed data features are obtained;
[0035] Use Z-score normalization to scale all data features to the same range. The expression is:
[0036]
[0037] Among them, x' is the eigenvalue after standardization, μ is the mean value of the feature, and σ is the standard deviation of the feature;
[0038] Based on the standardized data, a long short-term memory network LSTM recurrent neural network model is constructed;
[0039] The attention mechanism is added to the original long short-term memory network LSTM model, and the expression is:
[0040] a t =softmax(W a [h t ;x t ]+b a );
[0041] Among them, a t is the attention weight at time step t, W a is the attention weight matrix, h t is the hidden state at time step t, x t is the input vector at time step t, b a is the attention bias term;
[0042] Based on the unit state, the attention weight a t Applied to cell state c t , we get the unit state update with attention, expressed as:
[0043] c t =f t ⊙c t-1 +i t ⊙a t ⊙tanh(W xc x t +W hc h t-1 +b c );
[0044] Among them, c t represents the unit state, tanh is the hyperbolic tangent function, a t Representing the attention weight at time step t, ⊙ is the Hadamard product, element-wise multiplication, f t It is the forget gate, t is the input gate, W xc ,W hc is the weight matrix, b cis the bias term;
[0045] Add bidirectional processing data to the hidden state of the long short-term memory network LSTM model, and the expression is:
[0046]
[0047] Among them, h t represents the final hidden state, h f t and h b t They are the hidden states of the forward and backward long short-term memory networks LSTM respectively;
[0048] By using the Adam optimizer to automatically adjust the learning rate of the long short-term memory network LSTM model, the efficiency and effect of model training can be improved. The expression is:
[0049]
[0050] Where m t is the first moment estimate of the gradient, v t is the second-order moment estimate of the gradient, β 1 is the exponential decay rate of the first-order moment, β 2 is the exponential decay rate of the second-order moment, represents the gradient of the loss function J with respect to the parameter θ, is the bias-corrected first-order moment estimate, is the bias-corrected second-order moment estimate, α is the learning rate, ∈ refers to the minimum value, and θ represents the model parameters at time step t;
[0051] Train the created long short-term memory network LSTM model, use the historical log record data as input, and predict the log record data at the next moment;
[0052] Use the trained LSTM model to predict the log data at future time points, then compare the predicted data with the actual data and adjust the timestamp of the predicted data to ensure that all data is aligned to a unified and synchronized time base.
[0053] Then, the unified synchronization time base is used to build a data synchronizer through the agent concept, and the log record data is synchronously converted according to the affiliation;
[0054] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the unified synchronization time reference is used to construct a data synchronizer through the agent concept, and the log record data is synchronously converted according to the belonging relationship. The specific steps are as follows:
[0055] The toolkit is used to completely map and convert the unified time base into an object-oriented form and import it into the binary tree, and the agent concept is introduced into each branch node in the binary tree;
[0056] The proxy concept is used to extract the proxy keywords corresponding to the unified time base, and the binary tree is converted into a set of relational patterns based on the proxy keywords;
[0057] The relationship pattern set is used as a parameter of the synchronization conversion process to complete the creation of the data synchronizer, and the log record data that does not belong to the unified time base is passed to the data synchronizer based on the belonging relationship result;
[0058] Use a data synchronizer to convert the time zone of the log data that does not belong to the unified time base to the unified time base, and retain the time zone information of the original data;
[0059] After converting all log record data to the same time base, scan the file suffix of the log record data;
[0060] The file extension is compared with the pre-set extension in the classification matrix, and the files belonging to the same extension table classification are classified as the same type of files according to the comparison result.
[0061] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the file suffix name is compared with the pre-set suffix name by classification matrix, and the files belonging to the same suffix name table classification are classified into the same type of files according to the comparison result. The specific steps are as follows:
[0062] The file suffix and the pre-set suffix are represented as weighted co-occurrence graphs, and the similarity between the names is obtained by name embedding to construct a suffix similarity matrix;
[0063] The weighted co-occurrence graph is enhanced using the suffix similarity matrix, suffix nodes are added based on the enhanced result, and the suffix nodes are multiplied by the suffix similarity matrix to obtain an enhanced node vector;
[0064] The graph kernel calculation based on edge matching is used to enhance the similarity between node vectors, and the kernel matrix is constructed using the similarity values. The kernel matrix is fed into the preset classifier to compare the classification matrices of the suffix names.
[0065] Based on the classification comparison result, the log record data belonging to the same suffix name are classified into the same type.
[0066] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the data security analysis unit has the following specific steps:
[0067] A log record data storage module is used to set type folders according to type classification results and store log record data in corresponding folders;
[0068] The security analysis preprocessing module is used to compare the state parameters with the historical state parameters to obtain the parameters affecting data security, where the state parameters include the network association address and domain name;
[0069] The initial safety analysis module is used to define the safety factor calculation formula using the state parameters to obtain the safety factor of the log record data, and compare the safety factor with the factor threshold to determine the risk existence state;
[0070] The secondary safety analysis module is used to define the secondary safety factor calculation formula according to the state parameters, conduct secondary safety analysis on the log record data with risks in combination with the risk existence status judgment results, and send an early warning based on the analysis results.
[0071] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the calculation formula of the safety factor is:
[0072]
[0073] The calculation formula of the secondary safety factor is:
[0074]
[0075] Among them, σ i represents the security factor of the i-th group of log data, I represents the impact parameter of the network association address on the security of log data, D represents the impact parameter of the domain name on the security of log data, e represents a constant, α and β represent the weight coefficients of I and D respectively;
[0076] ρ i represents the quadratic safety factor of the i-th group of log records, n j Indicates the total number of log records provided by the network association address in the jth historical time period, n j-1 Indicates the total number of log records provided by the network association address in the j-1th historical time period, m j represents the total number of log records provided by the domain name in the jth historical time period, m j-1 Indicates the total number of log records provided by the domain name in the j-1th historical time period.
[0077] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station of the present invention, the data reconstruction acquisition unit has the following specific steps:
[0078] The log data sparse representation module is used to perform sparse representation on the verified log data using Fourier transform and select the data mask tool as the observation matrix;
[0079] A data reconstruction operation processing module is used to combine the iterative reconstruction algorithm with the splitting iterative algorithm to form a dual reconstruction iteration, and reconstruct the sparse representation based on the reconstruction control criterion condition;
[0080] The target data acquisition module is used to use the observation matrix to determine the integrity of the reconstruction result to obtain the reconstructed log record data, and to obtain the target data from the log record data based on the target data acquisition criterion.
[0081] As a preferred solution of the multi-source heterogeneous data synchronization and reconstruction system of the new energy station described in the present invention, the iterative reconstruction algorithm is combined with the splitting iterative algorithm to form a dual reconstruction iteration, and the sparse representation is reconstructed based on the reconstruction control criterion condition. The specific steps are:
[0082] The observation matrix is used to extract reconstruction information from the sparse representation, and the initial solution and the maximum number of iterations of the iterative reconstruction algorithm are set according to the reconstruction information;
[0083] Based on the reconstruction information and the initial solution, the current solution is updated within the range of iteration times to perform external iteration operations, and it is determined whether the current solution meets the preset stop condition. If so, the external iteration is stopped; if not, the internal iteration is performed;
[0084] The classification iterative algorithm is used to perform internal iterative reconstruction processing on the sparse representation to obtain the reconstruction processing result, and the reconstruction processing result is optimized based on the non-convex optimization algorithm to obtain the optimal reconstruction result.
[0085] The beneficial effects of the present invention are:
[0086] The present invention significantly improves the data management efficiency through efficient data acquisition, in-depth analysis and processing, secure storage and intelligent reconstruction. Specifically, the data acquisition stage adopts coding compression and real-time transmission strategies to ensure data integrity and security; data synchronization processing enhances time series analysis through long short-term memory networks to achieve accurate data synchronization and improve system response speed; the data security analysis unit effectively identifies and warns of potential risks to ensure data security; data reconstruction acquisition is optimized through intelligent algorithms to improve data processing accuracy and efficiency. Overall, the system greatly enhances the real-time, security and intelligence level of data processing at new energy stations, providing solid technical support for efficient operation and maintenance of stations. BRIEF DESCRIPTION OF THE DRAWINGS
[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0088] Figure 1 This is a flow chart of the multi-source heterogeneous data synchronization and reconstruction system for new energy stations in Example 1.
[0089] Figure 2 This is a flowchart of file name classification and normalization in Example 1. DETAILED DESCRIPTION
[0090] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0091] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0092] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0093] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a new energy station multi-source heterogeneous data synchronization and reconstruction system, comprising the following steps:
[0094] S1. A data acquisition and recording unit is used to obtain log record data and status parameters corresponding to each data source in the new energy station.
[0095] Furthermore, the data source types within the new energy sites are clarified, including: wind turbines, photovoltaic power generation systems, energy storage systems, meteorological stations, SCADA systems, relay protection equipment, communication equipment, security systems, environmental monitoring systems, and grid access equipment;
[0096] Deploy edge computing devices in the site and install data collection software, and configure connection parameters with various data sources;
[0097] Based on the deployed edge computing devices, design the data flow architecture to ensure real-time data transmission from edge devices to the central server;
[0098] In order to reduce the consumption of data transmission, Huffman coding and LZW compression algorithm are used to reduce the data transmission volume, speed up the transmission speed, and reduce the network bandwidth;
[0099] Huffman coding assigns shorter codes to characters with high frequency and longer codes to characters with low frequency, which can reduce the overall code length and achieve data compression;
[0100] The LZW compression algorithm achieves data compression by establishing a string table to store repeated strings and replacing longer strings with shorter codes;
[0101] Use a distributed database system to store log data through the transmitted data;
[0102] In order to prevent data loss, implement data redundancy backup strategy;
[0103] Data redundancy backup strategy is a data protection measure that creates and stores multiple independent copies of data to ensure that when the primary data source fails, is damaged, or the data is lost, the data can be restored from the backup to maintain business continuity and data integrity. In the context of new energy stations, data redundancy backup strategy is particularly important because it involves the operating data of multiple key systems and equipment, which is essential for monitoring, controlling, and optimizing the operation of the station;
[0104] Before data is stored in the database, data integrity check is performed to ensure data quality;
[0105] Check the data that does not meet the standards, mark them, and design a retry mechanism to retrieve the data to reduce missing data;
[0106] Based on the above operations, set up an access control list in the database to ensure that only authorized users can access specific data to ensure data security;
[0107] Finally, implement log auditing function on data to record all data access behaviors, so as to facilitate the tracking and analysis of potential security threats;
[0108] The log audit function refers to the process of deploying and using a log audit system in an IT system to record, monitor, analyze and report all system operations and user activities.
[0109] It should be noted that a reliable, secure and efficient data processing system is established to adapt to the special needs of new energy sites and support their operation, maintenance and decision-making processes. Through these steps, the accuracy and integrity of the data can be ensured while protecting the data from unauthorized access and potential attacks.
[0110] S2, data synchronization processing unit, is used to deeply analyze and process the update status of log record data, use long short-term memory network to enhance time series analysis, and accurately synchronize log records to a unified time base.
[0111] Monitor the update status of log record data and record the time stamp corresponding to each update;
[0112] Combine the update status of each log record data with the synchronous response principle to record the corresponding change time mark information, and use information entropy to detect the change time mark information. The expression is:
[0113]
[0114] Among them, H(X) represents the information entropy of random variable X, n represents the number of all possible values of random variable X, that is, the size of the sample space of X, i is the index variable used to traverse the possible values of random variable X, and x i represents the i-th possible value of the random variable X, P(x i ) indicates that the random variable X takes the i-th value x i The probability of log 2 P(x i ) means the logarithmic function with base 2 is applied to P(x i );
[0115] Specifically, when the update status is combined with the synchronization response principle to obtain the change time mark of each log record data, and the synchronization time deviation is calculated based on the change time mark result, the update status of each log record data can be combined with the synchronization response principle to record the corresponding change time mark information, and the information entropy is used to detect the change time mark information, and the genetic simulation technology is used to perform update and smoothing processing on the information entropy detection result, and the center of the change time mark information is updated through the function value minimum mechanism to obtain the deviation data characteristic response function, and the deviation data characteristic response function is used as the basis for judging the synchronization time deviation, and combined with the change time mark law to extract the synchronization time deviation of the log record data;
[0116] Among them, the update status of each log record data is combined with the synchronization response principle to record the corresponding change time mark information, and when the information entropy is used to detect the change time mark information, the update status can be ensured to be synchronized with the status of other components in the system or network according to the synchronization response principle. For example, in a distributed system, it is ensured that all nodes receive and record the same update information, and additional information related to update synchronization is recorded, such as synchronization delay, the number of synchronization failure attempts, etc. These are all important data for evaluating system efficiency, and the entropy value of the time mark information is calculated using the information entropy formula. It should be noted that information entropy is a measure of the uncertainty of random variables. A high entropy value usually indicates that the changes in the data are more frequent or random. According to the calculation results of the information entropy, the change time mark information is analyzed;
[0117] Genetic algorithm is used to optimize the information entropy results and construct the deviation data characteristic response function, which is expressed as:
[0118] y = f(Δt; θ);
[0119] Where y is the feature response, f is the function itself, and the feature response is predicted based on the input time deviation. Δt represents the time deviation, that is, the difference between the actual update time and the expected update time of the log record data, and θ is the model parameter.
[0120] Genetic algorithm is a heuristic global optimization search algorithm that searches for the optimal solution in the solution space through an iterative process;
[0121] Combine the change time marking rule and the deviation data characteristic response function to extract the synchronization time deviation of the log record data;
[0122] The collected data will be processed to remove outliers and missing values;
[0123] Based on the information entropy detection results, the log record data is divided into normal data and deviation data, and a frequency domain model is defined to determine the membership mean of the deviation data;
[0124] The initial frequency mean and the corresponding standard deviation of the deviation data are set in advance, and the set results are combined with the genetic simulation technology to update and smooth the deviation data to obtain the deviation data characteristic model;
[0125] After setting the deviation data training set and deviation sample category, feature selection is performed on the deviation data to obtain the center point of the deviation data;
[0126] The center point of the deviation data is updated through the minimum square difference function value mechanism, the spectral density function of the deviation data is calculated, and the spectral density function is used as the deviation data feature to obtain the deviation data feature response function.
[0127] Specifically, when the information entropy detection result is updated and smoothed by using the genetic simulation technology, and the center of the change time mark information is updated through the function value minimum mechanism to obtain the deviation data characteristic response function, the log record data can be divided into normal data and deviation data based on the information entropy detection result, and a frequency domain model is defined to determine the membership mean of the deviation data, and the initial frequency mean and the corresponding standard deviation of the deviation data are pre-set, and the setting result is combined with the genetic simulation technology to update and smooth the deviation data to obtain the deviation data characteristic model, and after setting the deviation data training set and the deviation sample category, the deviation data is feature selected to obtain the center point of the deviation data, and the center point of the deviation data is updated through the square difference function value minimum mechanism, the spectral density function of the deviation data is calculated, and the spectral density function is used as the deviation data feature to obtain the deviation data characteristic response function;
[0128] Among them, the deviation data characteristic response function is used as the basis for judging the synchronization time deviation. When the synchronization time deviation of the log record data is extracted in combination with the change time mark law, the necessary change time mark information is collected from the log record, including the timestamp and related synchronization data of each event, and the sorted change time mark data is input into the deviation data characteristic response function, and the synchronization time deviation of each event is calculated. The output of the response function is analyzed to identify those events with significant time deviations, and the regularity of the change time mark, such as periodicity, frequency, etc., is analyzed to help explain the cause of the time deviation. In combination with the regularity of the time mark and the result of the deviation data characteristic response, the key synchronization time deviation information is extracted;
[0129] Furthermore, based on the deviation data feature response function, the processed data features are obtained;
[0130] Use Z-score normalization to scale all data features to the same range. The expression is:
[0131]
[0132] Among them, x' is the eigenvalue after standardization, μ is the mean value of the feature, and σ is the standard deviation of the feature;
[0133] Based on the standardized data, a recurrent neural network LSTM model is constructed;
[0134] The LSTM model of the recurrent neural network includes: input gate, forget gate, output gate, unit state and hidden layer state;
[0135] The input gate expression is:
[0136] i t =σ(W xi x t +W hi ht-1 +b i );
[0137] Among them, i t represents the input gate, σ is the Sigmoid activation function, which is used to control the inflow and outflow of information, and W xi , W hi is the weight matrix used to linearly transform the input vector x t and the hidden state h at the previous time step t-1 ,h t-1 is the hidden state at the previous time step t-1, which is equivalent to the output of the LSTM unit, x t is the input vector of the current time step t, which contains all the features passed in from the outside, b i is the bias term, corresponding to the calculation of the input gate;
[0138] The forget gate expression is:
[0139] f t =σ(W xf x t +W hf h t-1 +b f );
[0140] Among them, f t W stands for forget gate. xf is the weight matrix used to linearly transform the input vector x t and the hidden state h at the previous time step t-1 , b f is the bias term, corresponding to the calculation of the forget gate;
[0141] The output gate expression is:
[0142] o t =σ(W xo x t +W ho h t-1 +b o );
[0143] Among them, t Represents the output gate, W xo is the weight matrix used to linearly transform the input vector x t and the hidden state h at the previous time step t-1 , b o is the bias term, corresponding to the calculation of the output gate;
[0144] The unit state expression is:
[0145] c t =f t ⊙c t-1+i t ⊙tanh(W xc x t +W hc h t-1 +b c );
[0146] Among them, c t represents the cell state, tanh is the hyperbolic tangent function, which is usually used to update the cell state, ⊙ is the Hadamard product, element-level multiplication, used to control the switch of information flow, and f t It is the forget gate, t is the input gate, W xc ,W hc is the weight matrix used to linearly transform the input vector x t and the hidden state h at the previous time step t-1 , b c is the bias term, corresponding to the calculation of the cell state;
[0147] The hidden layer state expression is:
[0148] h t =o t ⊙tanh (c t );
[0149] Among them, h t represents the hidden layer state, o t is the input layer;
[0150] Then, the attention mechanism is added to the original recurrent neural network model LSTM model to improve the prediction accuracy. The expression is:
[0151] a t =softmax(W a [h t ;x t ]+b a );
[0152] Among them, a t is the attention weight at time step t, which determines how much attention the model should allocate to the current input and its context when processing the input of the current time step, W a is the attention weight matrix used to extract the hidden state h t and the input vector x t Extract important features from the t is the hidden state at time step t, which carries information from the beginning of the sequence to the current time step, x t is the input vector at time step t, which contains the input information of the current time step, b ais the attention bias term, which is used to introduce bias when calculating the attention weight, and softmax is the activation function, which is used to vectorize and normalize the attention weight;
[0153] Based on the unit state, the attention weight a t Applied to cell state c t , we get the unit state update with attention, expressed as:
[0154] c t =f t ⊙c t-1 +i t ⊙a t ⊙tanh(W xc x t +W hc h t-1 +b c );
[0155] Among them, c t represents the cell state, tanh is the hyperbolic tangent function, which is usually used to update the cell state, ⊙ is the Hadamard product, element-level multiplication, used to control the switch of information flow, and f t It is the forget gate, t is the input gate, a t Represents the attention weight at time step t, which is used to adjust the degree to which new information is stored in the unit state, W xc ,W hc is the weight matrix used to linearly transform the input vector x t and the hidden state h at the previous time step t-1 , b c is the bias term, corresponding to the calculation of the cell state;
[0156] Add bidirectional processing data to the hidden state of the recurrent neural network LSTM to capture sequence context information. The expression is:
[0157]
[0158] Among them, h t represents the final hidden state, h f t and h b t They are the hidden states of the forward and backward recurrent neural network LSTM models respectively;
[0159] By using the Adam optimizer to automatically adjust the learning rate of the recurrent neural network LSTM model, the efficiency and effect of model training can be improved. The expression is:
[0160]
[0161] Where m t is the first-order moment estimate of the gradient, which represents the moving average of the gradient, v t is the second-order moment estimate of the gradient, which represents the moving average of the square of the gradient, β 1 is the exponential decay rate of the first-order moment, β 2 is the exponential decay rate of the second-order moment, represents the gradient of the loss function J with respect to the parameter θ, is the bias-corrected first-order moment estimate, which is used to eliminate the initial bias. is the second-order moment estimate after bias correction, which is also used to eliminate the initial bias. α is the learning rate, which determines the step size of parameter update. ∈ refers to the minimum value, which is used to prevent division by zero. θ represents the model parameters at time step t, which is updated by the Adam optimizer.
[0162] Train the created recurrent neural network LSTM model, use the historical log record data as input, and predict the log record data at the next moment;
[0163] Use the trained LSTM model to predict log data at future time points, then compare the predicted data with the actual data and adjust the timestamp of the predicted data to ensure that all data is aligned to a unified time base.
[0164] Based on the recurrent neural network LSTM model, in order to improve the prediction accuracy, the model is optimized, the best hyperparameter combination is found through grid search, and the mean square error indicator is used to evaluate the performance of the model to ensure effectiveness and reliability;
[0165] Then, the unified synchronization time base is used to build a data synchronizer through the agent concept, and the log record data is synchronously converted according to the affiliation;
[0166] Furthermore, the unified synchronization time base is used to construct a data synchronizer through the agent concept, and the log record data is synchronously converted according to the affiliation;
[0167] The proxy concept is used to extract the proxy keywords corresponding to the unified time base, and the binary tree is converted into a set of relational patterns based on the proxy keywords;
[0168] The relationship pattern set is used as a parameter of the synchronization conversion process to complete the creation of the data synchronizer, and the log record data that does not belong to the unified time base is passed to the data synchronizer based on the belonging relationship result;
[0169] Use a data synchronizer to convert the time zone of the log data that does not belong to the unified time base to the unified time base, and retain the time zone information of the original data;
[0170] After converting all log record data to the same time base, scan the file suffix of the log record data;
[0171] Compare the file extension with the pre-set extension in a classification matrix, and classify the files classified in the same extension table into the same type of files according to the comparison result;
[0172] Furthermore, the file suffix name and the pre-set suffix name are represented as weighted co-occurrence graphs, and the similarity between the names is obtained by using name embedding to construct a suffix name similarity matrix;
[0173] The weighted co-occurrence graph is enhanced using the suffix similarity matrix, suffix nodes are added based on the enhanced result, and the suffix nodes are multiplied by the suffix similarity matrix to obtain an enhanced node vector;
[0174] The graph kernel calculation based on edge matching is used to enhance the similarity between node vectors, and the kernel matrix is constructed using the similarity values. The kernel matrix is fed into the preset classifier to compare the classification matrices of the suffix names.
[0175] Based on the classification comparison result, the log record data belonging to the same suffix name are classified into the same type;
[0176] Specifically, when combining the proxy concept with the unified time base to construct a data synchronizer, and performing synchronous conversion processing on the log record data according to the belonging relationship, the toolkit can be used to completely map the unified time base and convert it into an object-oriented form and import it into the binary tree, and analyze each branch node in the binary tree to introduce the proxy concept, use the proxy concept to extract the proxy keyword corresponding to the unified time base, convert the binary tree into a set of relational patterns based on the proxy keywords, and use the relational pattern set as a parameter of the synchronous conversion process to complete the creation of the data synchronizer, and based on the belonging relationship result, the log record data that does not belong to the unified time base is passed to the data synchronizer, and the time zone of the time zone to which the log record data belongs that does not belong to the unified time base is converted into the unified time base by using the data synchronizer, and the time zone information of the original data is retained. After all the log record data are converted to the same time base, the file suffix of the log record data is scanned; the file suffix is compared with the pre-set suffix in the classification matrix, and the files classified in the same suffix table are classified into the same type of files according to the comparison result;
[0177] It should be noted that when comparing the file suffix name with the pre-set suffix name by a classification matrix, and classifying the files classified into the same suffix name table as files of the same type according to the comparison result, the file suffix name and the pre-set suffix name can be represented as weighted co-occurrence graphs respectively, and the similarity between the names is obtained by name embedding to construct a suffix name similarity matrix, the weighted co-occurrence graph is enhanced by using the suffix name similarity matrix, suffix name nodes are added based on the enhanced result, and the suffix name nodes are multiplied by the suffix name similarity matrix to obtain an enhanced node vector, the similarity between the enhanced node vectors is calculated by using an edge matching-based graph kernel, and a kernel matrix is constructed using the similarity values, the kernel matrix is fed into a preset classifier to compare the suffix names by a classification matrix, and the log record data belonging to the same suffix name is classified into the same type based on the classification comparison result;
[0178] Specifically, the suffix table classification matrix is shown in Table 1 below:
[0179] Table 1 Suffix table classification matrix
[0180]
[0181] It should be noted that by integrating advanced data analysis technology and data synchronization processing under a unified time base, not only the efficiency and accuracy of data processing are improved, but also intelligent classification and management of data are realized, providing a solid technical guarantee for subsequent data analysis and application.
[0182] S3, data security analysis unit, is used to store the log record data after synchronous conversion into the database, verify the security of the log record data through status parameters, and automatically send an early warning when a risk is detected.
[0183] Furthermore, the data security analysis unit includes:
[0184] A log record data storage module is used to set type folders according to type classification results and store log record data in corresponding folders;
[0185] The security analysis preprocessing module is used to compare the state parameters with the historical state parameters to obtain the parameters affecting data security, where the state parameters include the network association address and domain name;
[0186] The initial safety analysis module is used to define the safety factor calculation formula using the state parameters to obtain the safety factor of the log record data, and compare the safety factor with the factor threshold to determine the risk existence state;
[0187] The secondary safety analysis module is used to define the secondary safety factor calculation formula according to the state parameters, conduct secondary safety analysis on the log record data with risks in combination with the risk existence state judgment results, and send warnings based on the analysis results;
[0188] The process of obtaining the parameters affecting data security is as follows:
[0189] When the IP address of the data source server corresponding to the i-th group of log record data provided is the same as the IP address of the data source server corresponding to the i-th group of log record data provided last time, I=0; when the IP address of the data source server corresponding to the i-th group of log record data provided last time is different but has appeared in historical data, I=0.5; when the IP address of the data source server corresponding to the i-th group of log record data provided first appears, I=1;
[0190] When the DNS of the data source server corresponding to the i-th group of log record data is the same as the DNS of the data source server corresponding to the i-th group of log record data provided last time, D=0; when the DNS of the data source server corresponding to the i-th group of log record data is different from the DNS of the data source server corresponding to the i-th group of log record data provided last time but has appeared in historical data, D=0.5; when the DNS of the data source server corresponding to the i-th group of log record data appears for the first time, D=1;
[0191] It should be noted that the working process of the data security analysis unit 3 includes:
[0192] Step 1: Select Coordinated Universal Time (UTC) as the unified time base;
[0193] Step 2: It is stipulated that the log records of all data sources contain timestamps, and the format of the standardized timestamps is ISO8601 format. The timestamps in the log records of all data sources are traversed to determine whether the time in the time zone to which the data source belongs is the UTC time base. If it is the UTC time base, proceed to step 4; if not, proceed to step 3.
[0194] Step 3: Convert the time zone of the data source that is not based on the UTC time base to the UTC time base, and retain the time zone information of the original data for easy backtracking, and proceed to step 4;
[0195] Step 4: After converting all data sources to the same time base, traverse the file names of all data, store files of the same type in the same folder, and further analyze the security of all data sources;
[0196] Step 5: If the security of all data sources is not a problem, the data reconstruction acquisition unit 4 can be used to reconstruct data from different sources under the same time standard;
[0197] At the same time, the work process of further analyzing the security of all data in step 4 includes:
[0198] The first safety factor of the i-th group of log record data is calculated by the safety factor calculation formula, and the first safety factor is compared with the first safety factor threshold. If the first safety factor is greater than or equal to the first safety factor threshold, it is judged that there is a security risk in the i-th group of log record data, and the security of the i-th group of log record data needs to be further judged. If the first safety factor is less than the first safety factor threshold, it is judged that the current status of the i-th group of log record data is normal.
[0199] It should be explained that further judging the security of the i-th group of log record data includes the following steps:
[0200] Get the IP address of the data source server that provides the i-th group of log records corresponding to the total number of data sources n provided in the j-th historical time period j , get the IP address of the data source server corresponding to the i-th group of log records and the total number of data sources n provided in the j-1th historical time period j-1 , obtain the total number of data sources m provided by the DNS of the data source server corresponding to the i-th group of log records in the j-th historical time period j , obtain the total number of data sources provided by the DNS of the data source server corresponding to the i-th group of log record data in the j-1th historical time period;
[0201] The second safety factor is calculated using the above data, and the second safety factor is compared with the second safety factor threshold. If the second safety factor is greater than or equal to the second safety factor threshold, it is judged that there is a security risk in the i-th group of log record data, and an early warning is immediately issued. If the second safety factor is less than the second safety factor threshold, it is judged that the current status of the i-th group of log record data is normal.
[0202] The calculation formula of safety factor is:
[0203]
[0204] The calculation formula of the secondary safety factor is:
[0205]
[0206] Among them, σ i represents the security factor of the i-th group of log data, I represents the impact parameter of the network association address on the security of log data, D represents the impact parameter of the domain name on the security of log data, e represents a constant, and its value is approximately 2.718, α and β represent the weight coefficients of I and D respectively;
[0207] ρ i represents the quadratic safety factor of the i-th group of log records, n jIndicates the total number of log records provided by the network association address in the jth historical time period, n j-1 Indicates the total number of log records provided by the network association address in the j-1th historical time period, m j represents the total number of log records provided by the domain name in the jth historical time period, m j-1 Indicates the total number of log records provided by the domain name in the j-1th historical time period.
[0208] S4, a data reconstruction acquisition unit, used to perform a reconstruction processing operation on the verified log record data, and acquire target data based on the reconstruction result.
[0209] Furthermore, the data reconstruction acquisition unit includes:
[0210] The log data sparse representation module is used to perform sparse representation on the verified log data using Fourier transform and select the data mask tool as the observation matrix;
[0211] A data reconstruction operation processing module is used to combine the iterative reconstruction algorithm with the splitting iterative algorithm to form a dual reconstruction iteration, and reconstruct the sparse representation based on the reconstruction control criterion condition;
[0212] A target data acquisition module is used to determine the integrity of the reconstruction result using the observation matrix to obtain the reconstructed log record data, and to obtain the target data from the log record data based on the target data acquisition criterion;
[0213] Specifically, when Fourier transform is used to perform sparse representation on the verified log record data, and a data masking tool is selected as the observation matrix, necessary preprocessing, such as denoising and standardization, can be performed on the log record data to facilitate effective Fourier transform, and Fourier transform is performed on the preprocessed log record data to convert the data from the time domain to the frequency domain; the transformation results are analyzed to identify the main frequency components in the data, which usually represent the main signals of the data; based on the results of the frequency analysis, important frequency components are selected for constructing a sparse representation, and a sparse representation matrix is constructed using the selected frequency components, which will be used to describe the main characteristics of the data; at the same time, a data masking tool is selected or designed as the observation matrix, and the observation matrix is used to mask the original sparse representation, so as to perform specific data compression or feature extraction;
[0214] Specifically, the iterative reconstruction algorithm is combined with the splitting iterative algorithm to form a dual reconstruction iteration, and when the sparse representation is reconstructed based on the reconstruction control criterion condition, the observation matrix can be used to extract the reconstruction information from the sparse representation, and the initial solution and the maximum number of iterations of the iterative reconstruction algorithm are set according to the reconstruction information; based on the reconstruction information and the initial solution, the current solution is updated within the range of the iteration number to perform an external iteration operation, and it is determined whether the current solution meets the preset stop condition. If it meets the condition, the external iteration is stopped, and if it does not meet the condition, the internal iteration is performed; the classification iterative algorithm is used to perform internal iterative reconstruction processing on the sparse representation to obtain the reconstruction processing result, and the reconstruction processing result is optimized based on the non-convex optimization algorithm to obtain the optimal reconstruction result;
[0215] When the reconstruction information is extracted from the sparse representation using the observation matrix and the initial solution and the maximum number of iterations of the iterative reconstruction algorithm are set according to the reconstruction information, important information can be extracted from the data processed by the observation matrix, and this information will be used as the input of the iterative reconstruction algorithm; based on the extracted reconstruction information, an initial solution is selected, where the initial solution can be an all-zero matrix, a random matrix, or an approximate solution obtained based on a certain heuristic method, and its selection depends on the specific problem and the expected solution efficiency, and a reasonable initial solution can significantly improve the convergence speed of the algorithm and the quality of the final solution; based on the complexity of the selected iterative reconstruction algorithm and the expected computing resources, a reasonable number of iterations is evaluated, and a maximum number of iterations is set to prevent the algorithm from running too long or falling into an invalid loop;
[0216] Based on the reconstruction information and the initial solution, the current solution is updated within the range of the number of iterations to perform external iteration operations. When judging whether the current solution meets the preset stop condition, the previously set initial solution is loaded into the iterative algorithm as the starting point of the iteration, and the parameters that need to be monitored during the iteration are determined, such as the error threshold, the maximum number of iterations, etc.; in each iteration, the current solution is updated according to the reconstruction algorithm rules. The update may be based on gradient descent, projection method, etc. In the process of updating the solution, the reconstruction information previously extracted from the sparse representation is used to guide the update of the solution to ensure that the solution is optimized in the right direction; after each iteration, the error between the current solution and the target solution is calculated. The error can be a norm difference, a relative error, etc. The calculated error is compared with the preset stop threshold. If the error is less than or equal to the threshold, it is considered that a satisfactory solution has been reached. Track whether the number of iterations has reached the preset maximum number. If it has, the iteration needs to be stopped even if the error condition is not met to avoid over-calculation;
[0217] The classification iterative algorithm is used to perform internal iterative reconstruction processing on the sparse representation to obtain the reconstruction processing result, and the reconstruction processing result is optimized based on the non-convex optimization algorithm to obtain the optimal reconstruction result. The parameters required by the iterative algorithm are set, including the learning rate, the regularization parameter, the maximum number of iterations, etc., and the initial solution based on the appropriate basis is selected, which is usually a random solution or an initial solution based on prior knowledge. In each optimization iteration, the classification iterative algorithm (such as the algorithm gradient descent method, the projection gradient method, etc.) is used to update and optimize the current solution, and the optimal solution is gradually approached based on the reconstruction information. In each iteration, the reconstructed solution is updated based on the current solution, and the error between the current solution and the target solution is calculated to judge the convergence. If the error is less than the preset threshold or reaches the maximum number of iterations, the iteration is stopped to obtain the preliminary reconstruction result, and a non-convex optimization model is established. In the non-convex optimization process, the optimization algorithm is used to further adjust and optimize the preliminary reconstruction result to seek the global optimal solution, and the optimized reconstruction result is evaluated to ensure that its quality and accuracy meet expectations, and the optimal reconstruction result is recorded and output as the final reconstruction processing result;
[0218] Specifically, when using the observation matrix to judge the integrity of the reconstruction result and obtain the target data from the reconstructed log record data, the observation matrix is first used to evaluate the reconstruction result to ensure that the reconstructed data maintains the key features and structure of the original data. By comparing the data before and after the observation matrix is processed, it is judged whether the reconstructed data is complete, that is, whether key information is missing or distorted in the data, and the quality of the reconstructed data is evaluated to ensure that the data quality meets the needs of subsequent processing and analysis; if the data does not pass the integrity test, it is necessary to return to the reconstruction step, adjust the parameters or algorithm, and perform data reconstruction again. According to the needs and goals of the analysis, a set of clear data acquisition criteria are formulated. The criteria may be based on specific features of the data, such as timestamps, event types, keywords, etc., and the formulated criteria are used to filter and extract the required target data from the reconstructed log record data;
[0219] It should be noted that by integrating key technologies such as data storage, security analysis, sparse representation and reconstruction, and target data acquisition, not only the secure storage and security analysis of log data are achieved, but also the integrity and accuracy of the data are ensured through efficient sparse representation and reconstruction technology, providing a solid technical guarantee for subsequent data analysis and application. This series of steps and methods not only improves the level of intelligent data processing, but also greatly enhances data security and reconstruction efficiency, laying a solid foundation for data-driven decision-making and system optimization.
[0220] In summary, the present invention realizes efficient, safe and intelligent data processing of new energy stations through key steps such as deep data analysis, data security assurance, efficient data structuring and classification, data reconstruction and target data acquisition, provides solid technical guarantee for subsequent data analysis and application, significantly improves decision-making efficiency, system security and intelligent management level, promotes optimal resource allocation and cost control, and injects strong scientific and technological impetus into the sustainable development of new energy stations.
[0221] Furthermore, the beneficial effects of the present invention also include:
[0222] 1. The present invention first stores data from different sources and obtains status parameters of different source data. Based on the status parameters, the security of the server providing the data source is determined to ensure that information from other data sources will not be leaked. At the same time, time synchronization technology is used to ensure that data from different sources can be processed under a unified time reference, thereby ensuring data consistency and accuracy.
[0223] 2. The present invention selects Coordinated Universal Time as the unified time base, and converts all data sources into the unified time base and standardized timestamp format. At the same time, it traverses the file names of all data, selects a suitable database to store files of the same type, further analyzes the security of all data, and reconstructs the data when the security of all data sources is ensured to ensure the consistency and correctness of all data.
[0224] 3. The present invention captures the update status of data through a real-time data monitor based on particle simulation technology, which can ensure the real-time update and accuracy of data. At the same time, it captures the change time mark of each log record data through the synchronous response principle and calculates the synchronization time deviation, which can ensure the temporal consistency of data collected from different sources or at different times. The agent concept is combined with a unified time base to construct a data synchronizer to ensure the effect of data synchronization.
[0225] 4. The present invention utilizes Fourier transform to perform sparse representation on log record data, which can effectively extract key features in the data. At the same time, a data masking tool is selected as the observation matrix, so that different masking strategies can be selected according to different needs, thereby providing the ability to customize data processing, and combining the iterative reconstruction algorithm with the splitting iterative algorithm to form a dual reconstruction iteration, which can more effectively handle complex reconstruction tasks. Finally, the observation matrix is used to judge the integrity of the reconstruction result, thereby ensuring that the reconstructed data retains the same key information as the original data, avoiding data loss or distortion that may occur during the reconstruction process.
[0226] Example 2, referring to Table 1, is the second example of the present invention. To further verify the advancement of the present invention, experimental simulation data of a multi-source heterogeneous data synchronization and reconstruction system for a new energy station are provided.
[0227] First, edge computing equipment was deployed in the station and data collection software was installed. Connection parameters with various data sources were configured and data flow architecture was designed to ensure real-time data transmission to the central server. Efficient coding and compression algorithms were used to reduce the volume of data transmission, and a distributed database system was used to store log data. Data redundancy backup strategies, data integrity checks, retry mechanisms, and access control lists were implemented to ensure data accuracy and security.
[0228] The data synchronization processing unit uses the LSTM model to deeply analyze and process the update status of log data, and uses information entropy detection and genetic algorithm optimization to accurately synchronize log records to a unified time base. In the data security analysis unit, the agent concept and relationship model set are used to achieve efficient structuring and refined classification of data, ensuring the secure storage and status verification of data;
[0229] In the data reconstruction acquisition unit, efficient data reconstruction processing is achieved through Fourier transform and dual reconstruction iterative algorithm, and based on the target data acquisition criteria, the target data is accurately acquired from the log record data, ensuring the integrity and accuracy of the data.
[0230]
[0231] It can be seen from the data in the above table that compared with the traditional data processing method, the present invention has significant advantages in data processing time and data integrity. When the amount of data is 10,000, the processing time of the traditional method is 60 seconds, while the present invention is only 20 seconds, which shortens the processing time by two thirds. When the amount of data increases to 50,000 and 100,000, this advantage is more obvious, and the processing time is shortened by 75% and 75% respectively. At the same time, the data integrity reaches 98% and 99% respectively, which is much higher than the traditional method.
[0232] In addition, the comparison of data synchronization errors shows that the present invention can control the synchronization error at a relatively low level, with a maximum of only 10 milliseconds, and as the amount of data increases, the error actually decreases, proving the high efficiency and stability of the present invention in large-scale data processing.
[0233] In summary, the present invention has shown significant innovation and advantages in data processing time, data integrity, data synchronization error, etc., especially in large-scale data processing scenarios, which can significantly improve data processing efficiency and data quality, and provide strong technical support for the operation, maintenance and decision-making of new energy stations. Through the application of key technologies such as LSTM model, efficient data structuring, data security analysis and data reconstruction acquisition, the present invention not only improves the intelligent level of data processing, but also greatly enhances data security and reconstruction efficiency, laying a solid foundation for data-driven decision-making and system optimization.
[0234] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A new energy station multi-source heterogeneous data synchronization and reconstruction system, characterized in that: include: A data acquisition and recording unit is used to obtain log record data and status parameters corresponding to each data source in the new energy station; Data synchronization processing unit, used to deeply analyze and process the update status of log record data, use long short-term memory network to enhance time series analysis, and accurately synchronize log records to a unified time base; A data security analysis unit is used to store the log record data after synchronous conversion into a database, verify the security of the log record data through status parameters, and automatically send an early warning when a risk is detected; The data reconstruction acquisition unit is used to perform a reconstruction processing operation on the verified log record data and acquire target data based on the reconstruction result.
2. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 1, characterized in that: The specific steps of obtaining the log record data and status parameters corresponding to each data source in the new energy station are as follows: Clarify the types of data sources within new energy sites, including: wind turbines, photovoltaic power generation systems, energy storage systems, meteorological stations, SCADA systems, relay protection equipment, communication equipment, security systems, environmental monitoring systems, and grid access equipment; Deploy edge computing devices in the site and install data collection software, and configure connection parameters with various data sources; Based on the deployed edge computing devices, design the data flow architecture to ensure real-time data transmission from edge devices to the central server; Use Huffman coding and LZW compression algorithms to reduce data transmission volume, speed up transmission, and reduce network bandwidth; Use a distributed database system to store log data through the transmitted data; Implement data redundancy backup strategy; Before data is stored in the database, data integrity check is performed; Check the data that does not meet the standards, mark it, and design a retry mechanism to retrieve the data; Based on the above operations, set up access control lists in the database to ensure that only authorized users can access specific data; Finally, implement the log audit function on the data to record all data access behaviors.
3. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 2 is characterized in that: The detailed steps of deeply analyzing and processing the update status of log record data are as follows: Monitor the update status of log record data and record the time stamp corresponding to each update; Combine the update status of each log record data with the synchronous response principle to record the corresponding change time mark information, and use information entropy to detect the change time mark information. The expression is: Among them, H(X) represents the information entropy of random variable X, n represents the number of all possible values of random variable X, that is, the size of the sample space of X, i is the index variable used to traverse the possible values of random variable X, and x i represents the i-th possible value of the random variable X, and P(xi) represents the i-th value x taken by the random variable X. i The probability of log2P(x i ) means the logarithmic function with base 2 is applied to P(x i ); Genetic algorithm is used to optimize the information entropy results and construct the deviation data characteristic response function, which is expressed as: y = f(Δt; 0); Where y is the feature response, f is the function itself, and the feature response is predicted based on the input time deviation. Δt represents the time deviation, that is, the difference between the actual update time and the expected update time of the log record data, and θ is the model parameter. Combine the change time marking rule and the deviation data characteristic response function to extract the synchronization time deviation of the log record data; The collected data will be processed to remove outliers and missing values; Based on the information entropy detection results, the log record data is divided into normal data and deviation data, and a frequency domain model is defined to determine the membership mean of the deviation data; The initial frequency mean and the corresponding standard deviation of the deviation data are set in advance, and the set results are combined with the genetic simulation technology to update and smooth the deviation data to obtain the deviation data characteristic model; After setting the deviation data training set and deviation sample category, feature selection is performed on the deviation data to obtain the center point of the deviation data; The center point of the deviation data is updated through the minimum square difference function value mechanism, the spectral density function of the deviation data is calculated, and the spectral density function is used as the deviation data feature to obtain the deviation data feature response function.
4. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 3 is characterized in that: The use of the long short-term memory network model to enhance time series analysis and accurately synchronize log records to a unified time base is as follows: Based on the deviation data feature response function, the processed data features are obtained; Use Z-score normalization to scale all data features to the same range. The expression is: Among them, x' is the eigenvalue after standardization, μ is the mean value of the feature, and σ is the standard deviation of the feature; Based on the standardized data, a long short-term memory network LSTM recurrent neural network model is constructed; The attention mechanism is added to the original long short-term memory network LSTM model, and the expression is: a t =softmax(W a [h t ;x t ]+b a ); Among them, a t is the attention weight at time step t, W a is the attention weight matrix, h t is the hidden state at time step t, x t is the input vector at time step t, b a is the attention bias term; Based on the unit state, the attention weight a t Applied to cell state c t , we get the unit state update with attention, expressed as: c t =f t ⊙c t-1 +i t ⊙a t ⊙tanh(W cx x t +W hc h t-1 +b c ); Among them, c t represents the unit state, tanh is the hyperbolic tangent function, a t Representing the attention weight at time step t, ⊙ is the Hadamard product, element-wise multiplication, f t It is the forget gate, t is the input gate, W xc ,W hc is the weight matrix, b c is the bias term; Add bidirectional processing data to the hidden state of the long short-term memory network LSTM model, and the expression is: Among them, h t represents the final hidden state, h f t and h b t They are the hidden states of the forward and backward long short-term memory networks LSTM respectively; By using the Adam optimizer to automatically adjust the learning rate of the long short-term memory network LSTM model, the efficiency and effect of model training can be improved. The expression is: Where m t is the first-order moment estimate of the gradient, vt is the second-order moment estimate of the gradient, β1 is the exponential decay rate of the first-order moment, β2 is the exponential decay rate of the second-order moment, represents the gradient of the loss function J with respect to the parameter θ, is the bias-corrected first-order moment estimate, is the bias-corrected second-order moment estimate, α is the learning rate, ∈ refers to the minimum value, and θ represents the model parameters at time step t; Train the created long short-term memory network LSTM model, use the historical log record data as input, and predict the log record data at the next moment; Use the trained LSTM model to predict the log data at future time points, then compare the predicted data with the actual data and adjust the timestamp of the predicted data to ensure that all data is aligned to a unified and synchronized time base. The unified synchronization time benchmark is then used to construct a data synchronizer through the agent concept, and the log record data is synchronously converted according to the affiliation.
5. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 4, characterized in that: The unified synchronization time base is used to construct a data synchronizer through the agent concept, and the log record data is synchronously converted according to the belonging relationship. The specific steps are as follows: The toolkit is used to completely map and convert the unified time base into an object-oriented form and import it into the binary tree, and the agent concept is introduced into each branch node in the binary tree; The proxy concept is used to extract the proxy keywords corresponding to the unified time base, and the binary tree is converted into a set of relational patterns based on the proxy keywords; The relationship pattern set is used as a parameter of the synchronization conversion process to complete the creation of the data synchronizer, and the log record data that does not belong to the unified time base is passed to the data synchronizer based on the belonging relationship result; Use a data synchronizer to convert the time zone of the log data that does not belong to the unified time base to the unified time base, and retain the time zone information of the original data; After converting all log record data to the same time base, scan the file suffix of the log record data; The file extension is compared with the pre-set extension in the classification matrix, and the files belonging to the same extension table classification are classified as the same type of files according to the comparison result.
6. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 5, characterized in that: The specific steps of comparing the file extension with the preset extension by classification matrix and classifying the files classified in the same extension table into the same type of files according to the comparison result are as follows: The file suffix and the pre-set suffix are represented as weighted co-occurrence graphs, and the similarity between the names is obtained by name embedding to construct a suffix similarity matrix; The weighted co-occurrence graph is enhanced using the suffix similarity matrix, suffix nodes are added based on the enhanced result, and the suffix nodes are multiplied by the suffix similarity matrix to obtain an enhanced node vector; The graph kernel calculation based on edge matching is used to enhance the similarity between node vectors, and the kernel matrix is constructed using the similarity values. The kernel matrix is fed into the preset classifier to compare the classification matrices of the suffix names. Based on the classification comparison result, the log record data belonging to the same suffix name are classified into the same type.
7. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 6, characterized in that: The data security analysis unit comprises: A log record data storage module is used to set type folders according to type classification results and store log record data in corresponding folders; The security analysis preprocessing module is used to compare the state parameters with the historical state parameters to obtain the parameters affecting data security, where the state parameters include the network association address and domain name; The initial safety analysis module is used to define the safety factor calculation formula using the state parameters to obtain the safety factor of the log record data, and compare the safety factor with the factor threshold to determine the risk existence state; The secondary safety analysis module is used to define the secondary safety factor calculation formula according to the state parameters, conduct secondary safety analysis on the log record data with risks in combination with the risk existence status judgment results, and send an early warning based on the analysis results.
8. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 7, characterized in that: The calculation formula of the safety factor is: The calculation formula of the secondary safety factor is: Among them, σ i represents the security factor of the i-th group of log data, I represents the impact parameter of the network association address on the security of log data, D represents the impact parameter of the domain name on the security of log data, e represents a constant, α and β represent the weight coefficients of I and D respectively; ρ i represents the quadratic safety factor of the i-th group of log records, n j Indicates the total number of log records provided by the network association address in the jth historical time period, n j-1 Indicates the total number of log records provided by the network association address in the j-1th historical time period, m j represents the total number of log records provided by the domain name in the jth historical time period, m j-1 Indicates the total number of log records provided by the domain name in the j-1th historical time period.
9. A new energy station multi-source heterogeneous data synchronization and reconstruction system according to claim 8, characterized in that: The data reconstruction acquisition unit comprises: The log data sparse representation module is used to perform sparse representation on the verified log data using Fourier transform and select the data mask tool as the observation matrix; A data reconstruction operation processing module is used to combine the iterative reconstruction algorithm with the splitting iterative algorithm to form a dual reconstruction iteration, and reconstruct the sparse representation based on the reconstruction control criterion condition; The target data acquisition module is used to use the observation matrix to determine the integrity of the reconstruction result to obtain the reconstructed log record data, and to obtain the target data from the log record data based on the target data acquisition criterion.
10. The multi-source heterogeneous data synchronization and reconstruction system for new energy stations according to claim 9, characterized in that: The iterative reconstruction algorithm is combined with the splitting iterative algorithm to form a dual reconstruction iteration, and the sparse representation is reconstructed based on the reconstruction control criterion condition. The specific steps are: The observation matrix is used to extract reconstruction information from the sparse representation, and the initial solution and the maximum number of iterations of the iterative reconstruction algorithm are set according to the reconstruction information; Based on the reconstruction information and the initial solution, the current solution is updated within the range of iteration times to perform external iteration operations, and it is determined whether the current solution meets the preset stop condition. If so, the external iteration is stopped; if not, the internal iteration is performed; The classification iterative algorithm is used to perform internal iterative reconstruction processing on the sparse representation to obtain the reconstruction processing result, and the reconstruction processing result is optimized based on the non-convex optimization algorithm to obtain the optimal reconstruction result.
Citation Information
Patent Citations
New energy station safety intelligent strategy calculation control system
CN115967519A
Log anomaly detection system
WO2022047658A1
Cited By
Distributed integrated power supply power supply management system and method
CN120546246A
Distributed integrated power supply management system and method
CN120546246B
Multi-data fusion method and system based on edge calculation
CN120974414A
