Electric power spot market and OMS data verification method and system based on file transmission

Abstract

The invention discloses an electric power spot market and OMS data verification method based on file transmission, and relates to the field of electric power spot market and OMS data verification, and the method comprises the steps: obtaining electric power spot market data and OMS data, carrying out the cleaning, classification and anomaly detection through a long short-term memory (LSTM) network algorithm, generating a data verification rule set, and carrying out the verification of the data verification rule set. And establishing a multi-dimensional data verification model, performing deep fusion and cross validation, generating a verification report, and optimizing a data verification rule set. The method comprises the following steps: acquiring electric power spot market and OMS data, performing cleaning, classification and anomaly detection by using an LSTM algorithm, and generating a data verification rule set; a multi-dimensional data verification model is constructed by applying a graph neural network algorithm, deep fusion and cross validation are realized, a verification report is generated, a data verification rule set is optimized by utilizing a support vector machine algorithm, efficient verification and quality control of data are realized, and data accuracy and reliability are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data verification between a power spot market and an operation management system (OMS), and in particular to a method and system for verifying data between a power spot market and an OMS based on file transmission. Background Art

[0002] In the electricity spot market, the accuracy and integrity of transaction data are crucial to the normal operation of the market. However, due to the large amount of transaction data in the electricity spot market and the involvement of multiple participants, errors and omissions are prone to occur during the collection, transmission and processing of data. Therefore, an effective method is needed to verify and check the electricity spot market data to ensure the accuracy and integrity of the data.

[0003] In the prior art, data transmission between the electricity spot market and the operation management system (OMS) usually relies on manual reconciliation and data comparison, which is not only labor-intensive but also prone to human errors. In addition, due to the real-time and dynamic nature of electricity spot market transactions, manual reconciliation methods are difficult to meet the market's requirements for timely data verification.

[0004] In order to improve the efficiency and accuracy of data verification, a data verification method and system for the power spot market and OMS based on file transfer is proposed. This method realizes consistency verification of the data of the power spot market and the operation data of the OMS by automating file transfer and comparison. By using advanced algorithms and technologies, inconsistencies in the data can be quickly and accurately detected and identified, and corresponding verification reports can be generated; this not only greatly reduces the workload of manual reconciliation, but also improves the timeliness and accuracy of data verification, providing a strong guarantee for the smooth operation of the power spot market. Summary of the invention

[0005] In view of the problems that the existing data verification methods have a large manual reconciliation workload, are prone to human errors, and cannot meet the timeliness requirements of data verification, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to improve the efficiency and accuracy of the verification of the power spot market and the OMS data.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In the first aspect, an embodiment of the present invention provides a method for verifying electricity spot market and OMS data based on file transfer, which includes obtaining electricity spot market data and OMS data, and using a long short-term memory network (LSTM) algorithm to clean, classify and detect anomalies, and generate a data verification rule set; based on the data verification rule set, a multidimensional data verification model is established using a graph neural network algorithm, and deep fusion and cross-validation are performed; based on the verification results of the multidimensional data verification model, a verification report is generated, and the data verification rule set is optimized using a support vector machine algorithm.

[0009] As a preferred solution of the power spot market and OMS data verification method for file transmission of the present invention, the generation process of the data verification rule set includes the following steps: extracting transaction data from the power spot market system and extracting operation data from the operation management system OMS; and extracting the transaction data and operation data X = {x1, x1, ..., x n} for preprocessing, where x n The nth data point is extracted according to the pre-processed transaction data and operation data; the long short-term memory network LSTM model is constructed by setting the input features, the number of hidden layers and the number of neurons; the long short-term memory network LSTM model is trained with historical data and the time series features of the historical data; the trained long short-term memory network LSTM model is used to identify and correct abnormal values ​​of the feature data; the feature data is classified based on the prediction results of the long short-term memory network LSTM model; the abnormal points and abnormal trends in the feature data are detected by comparing the predicted values ​​of the long short-term memory network LSTM model with the actual values; the feature data that fails the verification is marked, the data sender is notified to make corrections, and a data verification rule set is generated; at the same time, a data feedback mechanism is established to promptly feedback the verification results to the data sender, analyze the verification log, and determine the data quality.

[0010] As a preferred solution of the power spot market and OMS data verification method for file transmission of the present invention, the classification includes normal data, suspicious data and abnormal data; the data verification rule set includes general verification rules, single verification rules and comprehensive verification rules; the specific formula of the long short-term memory network LSTM model is as follows:

[0011]

[0012] Among them, Q is the quality indicator of the data verification rule set, N is the number of data samples, and f L is the output function of the long short-term memory network LSTM model, x i is the ith input data point, h i is the i-th hidden state, γ is the weight coefficient of anomaly detection, erf is the error function, yi is the actual value, is the predicted value of the long short-term memory network LSTM model, T is the total time length of data collection, μ is the mean of the time series, λ is the time weight coefficient, β is the benchmark value of the rule set complexity, and σ is the standard deviation of the prediction error.

[0013] when And f L (x i ,h i ) is greater than or equal to the preset threshold, the ith data point is marked as abnormal, and the data sender is notified to check and correct it; if the corrected data is re-entered into the system and the Q value is improved by more than 20%, the correction is determined to be effective, and positive feedback is sent to the data sender; if the corrected data causes the Q value to drop by more than 10%, the correction process is reviewed; if the data sender corrects the same data point three times and still fails to pass the verification, the data point is marked as difficult to correct and an on-site inspection is carried out.

[0014] When n consecutive data points are marked as abnormal data, it is determined to be an abnormal trend, and the data sender is notified to urgently check the data collection system; if the abnormal rate of a certain type of data exceeds 10% of the total samples, a quality warning for this type of data is sent to the data sender, and the data is collected again.

[0015] As a preferred solution of the power spot market and OMS data verification method for file transmission described in the present invention, wherein: the general verification rule is to check the file format and date format of the data; the single verification rule is to verify the load data; the comprehensive verification rule is the mutual verification between several data; it also includes receiving the information of the data verification rule set and converting it into a format for graph neural network processing; using the graph convolutional network GCN algorithm to build a multidimensional data verification model, and representing the data points as nodes in the graph neural network, and the relationship between the data is represented as edges, wherein the nodes contain multidimensional features and the edges protect the correlation between the data points; training the multidimensional data verification model by using labeled data, and adjusting the model hyperparameters by using the cross-validation method; using the trained graph neural network model, deeply integrating the data features of different dimensions, and verifying the new data at the same time; comparing and integrating the verification results of the multidimensional data verification model with the existing data verification rule set, generating a verification report, and optimizing and updating the data verification rule set.

[0016] As a preferred solution of the power spot market and OMS data verification method for file transmission described in the present invention, the verification includes the following steps: corresponding to the general verification rules, preliminary analysis of the marked data, obtaining basic information of the data, and caching the data content in the data queue to be verified; reading the configuration information of the data verification rule set from the database, and loading the corresponding verification dynamic library by the data verification engine to verify the data according to the priority order of the rules; when a verification rule fails to pass the verification and its corresponding alarm level is error, stop the subsequent verification of the current data, and directly enter the verification process of the next data in the data queue to be verified; if the alarm level is alarm, the verification engine continues to execute the verification rules until the next failed error level verification rule or all rules have been verified.

[0017] As a preferred solution of the power spot market and OMS data verification method for file transmission described in the present invention, wherein: the basic information includes data type, data date and data size; it also includes, for a single verification rule, a verification engine checks whether the data conforms to the preset file format and date format; if the inspection data passes the verification, it is marked as valid and enters the verification of the next inspection data; if the data fails the verification, the abnormality is recorded and manual intervention is performed; for a comprehensive verification rule, the verification engine determines whether all the data currently required to be inspected are complete; if all the inspection data are complete, comprehensive verification is continued, and multi-dimensional data correlation analysis and logic verification are performed; if any data is not complete, the verification process exits, the current verification is stopped, and the comprehensive verification rule is restarted after waiting for all relevant data to arrive.

[0018] As a preferred solution of the power spot market and OMS data verification method for file transmission described in the present invention, the data feedback mechanism includes the following steps: generating alarm information based on the verification results and distinguishing alarms of different levels; recording the verification results after each data verification is completed and generating a verification report; when a certain verification rule fails and the alarm level is an error, the alarm notification mechanism is immediately triggered and sent to the relevant person in charge; based on the information in the verification report, the cause is analyzed and the data is re-collected, and the data is resubmitted for verification, and the support vector machine algorithm is used to optimize and improve the data verification rule set and feedback mechanism; the verification report includes the execution status of the verification rules, failed verification information and warning level.

[0019] In the second aspect, an embodiment of the present invention provides an electricity spot market and OMS data verification system based on file transfer, which includes: a generation module, used to obtain electricity spot market data and OMS data, and use a long short-term memory network LSTM algorithm to clean, classify and detect anomalies to generate a data verification rule set; an establishment module, based on the data verification rule set, uses a graph neural network algorithm to establish a multi-dimensional data verification model, and performs deep fusion and cross-validation; an optimization module, used to generate a verification report based on the verification results of the multi-dimensional data verification model, and use a support vector machine algorithm to optimize the data verification rule set.

[0020] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the method for verifying electricity spot market and OMS data for file transmission as described in the first aspect of the present invention are implemented.

[0021] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the method for verifying the electricity spot market and OMS data for file transmission as described in the first aspect of the present invention are implemented.

[0022] The beneficial effects of the present invention are as follows: the present invention obtains electricity spot market and OMS data, uses the LSTM algorithm to perform cleaning, classification and anomaly detection, and generates a data verification rule set; applies the graph neural network algorithm to build a multi-dimensional data verification model to achieve deep fusion and cross-validation; generates a verification report based on the verification results, and uses the support vector machine algorithm to optimize the data verification rule set to achieve efficient data verification and quality control, ensure data accuracy and reliability, and improve the intelligence level and data management capabilities of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0024] Figure 1 This is a flow chart of the method for verifying power spot market and OMS data during file transmission in Example 1.

[0025] Figure 2 This is a computer equipment diagram of the power spot market and OMS data verification method for file transmission in Example 2. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0027] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from the description, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0029] Example 1

[0030] Reference Figure 1 , which is the first embodiment of the present invention, and provides a method for verifying data between the power spot market and OMS based on file transmission, including:

[0031] S1: Obtain electricity spot market data and OMS data, use the long short-term memory network (LSTM) algorithm to clean, classify and detect anomalies, and generate a data verification rule set.

[0032] Specifically, the process of generating the data verification rule set includes the following steps: extracting transaction data from the power spot market system and extracting operation data from the operation management system OMS; and performing a validation on the extracted transaction data and operation data X = {x1, x1, ..., x n} for preprocessing, where x n is the nth data point; extract feature data based on the preprocessed transaction data and operation data; construct a long short-term memory network LSTM model by setting input features, the number of hidden layers and the number of neurons; preprocessing includes data format unification, missing value processing and noise filtering; feature data include transaction volume and price changes in transaction data and equipment load changes and failure frequency in operation data.

[0033] Furthermore, the specific formula of the long short-term memory network LSTM model is as follows:

[0034]

[0035] Among them, Q is the quality indicator of the data verification rule set, N is the number of data samples, and f L is the output function of the long short-term memory network LSTM model, x i is the ith input data point, hi is the i-th hidden state, γ is the weight coefficient of anomaly detection, erf is the error function, y i is the actual value, is the predicted value of the long short-term memory network LSTM model, T is the total time length of data collection, μ is the mean of the time series, λ is the time weight coefficient, β is the benchmark value of the rule set complexity, and σ is the standard deviation of the prediction error.

[0036] Furthermore, the long short-term memory network LSTM model is trained with historical data and the time series characteristics of the historical data; the trained long short-term memory network LSTM model is used to identify and correct abnormal values ​​of the feature data; the feature data is classified based on the prediction results of the long short-term memory network LSTM model; the abnormal points and abnormal trends in the feature data are detected by comparing the predicted values ​​and actual values ​​of the long short-term memory network LSTM model; the feature data that fails the verification is marked, the data sender is notified to make corrections, and a data verification rule set is generated; at the same time, a data feedback mechanism is established to promptly feedback the verification results to the data sender, analyze the verification log, and determine the data quality.

[0037] Specifically, the classification includes normal data, suspicious data and abnormal data; the data verification rule set includes general verification rules, single verification rules and comprehensive verification rules; when And f L (x i ,h i ) is greater than or equal to the preset threshold, the ith data point is marked as abnormal, and the data sender is notified to check and correct it; if the corrected data is re-entered into the system and the Q value is improved by more than 20%, the correction is determined to be effective, and positive feedback is sent to the data sender; if the corrected data causes the Q value to drop by more than 10%, the correction process is reviewed; if the data sender corrects the same data point three times and still fails to pass the verification, the data point is marked as difficult to correct and an on-site inspection is carried out.

[0038] Furthermore, when n consecutive data points are marked as abnormal data, it is determined to be an abnormal trend, and the data sender is notified to urgently check the data collection system; if the abnormal rate of a certain type of data exceeds 10% of the total samples, a quality warning for this type of data is sent to the data sender, and the data is collected again.

[0039] Furthermore, the general verification rule is to check the file format and date format of the data; the single verification rule is to verify the load data; and the comprehensive verification rule is to verify the mutual verification between several data.

[0040] S2: Based on the data verification rule set, a multi-dimensional data verification model is established using the graph neural network algorithm to perform deep fusion and cross-validation.

[0041] Specifically, the information of the data verification rule set is received and converted into a format for graph neural network processing; a multidimensional data verification model is constructed using the graph convolutional network (GCN) algorithm, and data points are represented as nodes in the graph neural network, and the relationship between data is represented as edges, where nodes contain multidimensional features and edges protect the correlation between data points; the multidimensional data verification model is trained by using labeled data, and the model hyperparameters are adjusted using the cross-validation method; the trained graph neural network model is used to deeply fuse data features of different dimensions, and new data is verified at the same time.

[0042] Furthermore, corresponding to the general verification rules, the marked data is preliminarily analyzed to obtain the basic information of the data, and the data content is cached in the data queue to be verified; the configuration information of the data verification rule set is read from the database, and the data verification engine loads the corresponding verification dynamic library to verify the data according to the priority order of the rules; when a verification rule fails to pass the verification and its corresponding alarm level is error, the subsequent verification of the current data is stopped, and the verification process of the next data in the data queue to be verified is directly entered; if the alarm level is alarm, the verification engine continues to execute the verification rules until the next failed error level verification rule or all rules have been verified; the basic information includes data type, data date and data size.

[0043] Furthermore, for a single verification rule, the verification engine checks whether the data conforms to the preset file format and date format; if the verification data passes the verification, it is marked as valid and proceeds to the next verification data; if the data fails the verification, the exception is recorded and manual intervention is performed.

[0044] Specifically, for the comprehensive verification rules, the verification engine determines whether all the data that need to be verified are present; if all the verification data are present, the comprehensive verification continues, and multi-dimensional data correlation analysis and logic verification are performed; if any data is not present, the verification process exits, stops the current verification, and waits until all relevant data are present before restarting the comprehensive verification rules.

[0045] Furthermore, the verification results of the multi-dimensional data verification model are compared and integrated with the existing data verification rule set, a verification report is generated, and the data verification rule set is optimized and updated.

[0046] S3: Generate a verification report based on the verification results of the multi-dimensional data verification model, and optimize the data verification rule set using a support vector machine algorithm.

[0047] Specifically, alarm information is generated based on the verification results, and different levels of alarms are distinguished; the verification results after each data verification are completed are recorded, and a verification report is generated.

[0048] Furthermore, when a verification rule fails and the alarm level is an error, the alarm notification mechanism is immediately triggered and sent to the relevant person in charge; based on the information in the verification report, the cause is analyzed and the data is recollected and resubmitted for verification, and the support vector machine algorithm is used to optimize and improve the data verification rule set and feedback mechanism; the verification report includes the execution status of the verification rules, verification failure information and the alarm level.

[0049] Furthermore, the present embodiment also provides a power spot market and OMS data verification system based on file transfer, including: a generation module, used to obtain power spot market data and OMS data, and use a long short-term memory network LSTM algorithm for cleaning, classification and anomaly detection to generate a data verification rule set; an establishment module, based on the data verification rule set, using a graph neural network algorithm to establish a multi-dimensional data verification model, and perform deep fusion and cross-validation; an optimization module, used to generate a verification report based on the verification results of the multi-dimensional data verification model, and use a support vector machine algorithm to optimize the data verification rule set.

[0050] In summary, the present invention obtains electricity spot market and OMS data, uses the LSTM algorithm for cleaning, classification and anomaly detection, and generates a data verification rule set; applies the graph neural network algorithm to build a multi-dimensional data verification model to achieve deep fusion and cross-validation; generates a verification report according to the verification results, and uses the support vector machine algorithm to optimize the data verification rule set to achieve efficient data verification and quality control, ensure data accuracy and reliability, and improve the intelligence level and data management capabilities of the system.

[0051] Example 2

[0052] Reference Figure 2 , is an embodiment of the present invention, which is different from the previous embodiment in that:

[0053] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0054] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0055] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0056] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0057] Example 3

[0058] Referring to Table 1, which is the second embodiment of the present invention, this embodiment provides a method for verifying data between the electricity spot market and the OMS based on file transmission. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0059] Specifically, a 30-day test was conducted at a provincial power dispatching center. The test environment included a simulated power spot market system and an operation management system (OMS), and data was exchanged between the two systems through a secure isolation gateway.

[0060] Furthermore, 30 days of historical data, including generator output, load demand, transmission line power flow, and equipment maintenance plan, are collected from the power spot market system and the OMS system respectively. These data are preprocessed and standardized to form the initial data set X. Subsequently, the long short-term memory network (LSTM) is used to model the time series data and identify potential abnormal patterns. For example, when the output curve of a generator set shows a sudden fluctuation that does not conform to the historical pattern, the system will mark it as a potential abnormal point. An autoencoder is used for anomaly detection. The standardized data is input into the autoencoder, and abnormal data points are identified by comparing the reconstruction error. The reconstruction error threshold is set to 0.1. When the reconstruction error of a data point exceeds this threshold, it is marked as abnormal. Based on the detected anomalies and domain expert knowledge, an initial verification rule set R0 is generated, which contains 20 rules.

[0061] Furthermore, a graph structure G is constructed, where nodes represent data points and edges represent the relationship between data points. When the similarity between two data points exceeds 0.8, an edge is added between them. The graph convolutional network (GCN) is used to extract features from this graph structure. After 5 iterations, the change in the feature matrix H is less than 1e-4, and the iteration is stopped. After the feature extraction is completed, the attention mechanism is applied for feature fusion. The attention weight threshold is set to 0.1. When the attention weight of a feature exceeds this threshold, it is marked as a key feature.

[0062] Specifically, to evaluate the performance of the model, a 5-fold cross validation was performed. The mean square error (MSE) of each validation was recorded. When the average MSE was less than 0.05, the model training was considered complete. The reinforcement learning algorithm was used to optimize the validation rules. After each iteration, the data quality score Q was calculated, including the three dimensions of accuracy, completeness, and consistency. When the improvement of the Q value was less than 1%, the optimization process was stopped.

[0063] Furthermore, as shown in Table 1, by analyzing the above data, in terms of improving the detection rate, the method of the present invention has significantly improved the detection rate in various types of abnormality detection. Taking the abnormal output of the generator set as an example, the detection rate is increased from 78.5% of the traditional method to 94.2%, an increase of 15.7 percentage points. This means that the use of this method greatly reduces safety hazards; in reducing the false alarm rate, at the same time, the method of the present invention significantly reduces the false alarm rate. Taking the load demand forecast deviation as an example, the false alarm rate is reduced from 12.7% of the traditional method to 4.2%, a reduction of 8.5 percentage points. This greatly improves the reliability of the system and reduces unnecessary manual intervention and waste of resources.

[0064] Table 1 Test data table

[0065]

[0066] Furthermore, in terms of overall performance improvement, it can be seen from the table that the method of the present invention has achieved significant performance improvement in all test types. Whether it is the power generation side, the power consumption side or the transmission network, the method has shown excellent adaptability and stability. Efficiency improvement: Although the method of the present invention introduces more complex algorithms, the data processing time remains within an acceptable range, and in most cases does not exceed 2 minutes. Considering the significant improvement in performance, this time overhead is completely acceptable.

[0067] It is particularly noteworthy that this method also performs well in dealing with cross-regional power trading anomalies, with the detection rate increased by 16.8 percentage points and the false alarm rate reduced by 10.7 percentage points. This shows that this method is not only suitable for traditional power system monitoring, but can also effectively cope with emerging market-oriented trading scenarios. From the consistency of various indicators, the method of the present invention maintains stable high performance in different types of anomaly detection. The detection rate is always maintained above 93%, and the false alarm rate is controlled below 6%. This stability is crucial to the safe operation of the power system.

[0068] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for verifying power spot market and OMS data based on file transmission, characterized in that: include, Obtain power spot market data and OMS data, use the long short-term memory network (LSTM) algorithm to clean, classify and detect anomalies, and generate a data verification rule set; Based on the data verification rule set, a multi-dimensional data verification model is established using the graph neural network algorithm to perform deep fusion and cross-validation; A verification report is generated based on the verification results of the multi-dimensional data verification model, and a support vector machine algorithm is used to optimize the data verification rule set.

2. The method for verifying the power spot market and OMS data in file transmission according to claim 1, characterized in that: The process of generating the data verification rule set includes the following steps: Extract transaction data from the power spot market system and operation data from the operation management system OMS; and extract the transaction data and operation data X = {x1, x1, ..., x n } for preprocessing, where x n is the nth data point; Extract feature data based on pre-processed transaction data and operation data; By setting the input features, the number of hidden layers and the number of neurons, a long short-term memory network LSTM model is constructed; Use historical data to train the LSTM model, the time series characteristics of historical data; Use the trained long short-term memory network LSTM model to identify feature data and correct outliers; Classify the feature data based on the prediction results of the long short-term memory network LSTM model; By comparing the predicted values ​​of the LSTM model with the actual values, abnormal points and abnormal trends in the feature data are detected; Mark the feature data that fails the verification, notify the data sender to make corrections, and generate a data verification rule set; At the same time, a data feedback mechanism is established to promptly feed back the verification results to the data sender, analyze the verification log, and determine the data quality.

3. The method for verifying the power spot market and OMS data in file transmission according to claim 2, characterized in that: The classification includes normal data, suspicious data and abnormal data; the data verification rule set includes general verification rules, single verification rules and comprehensive verification rules; the specific formula of the long short-term memory network LSTM model is as follows: Among them, Q is the quality indicator of the data verification rule set, N is the number of data samples, and f L is the output function of the long short-term memory network LSTM model, x i is the ith input data point, h i is the i-th hidden state, γ is the weight coefficient of anomaly detection, erf is the error function, y i is the actual value, is the predicted value of the long short-term memory network LSTM model, T is the total time length of data collection, μ is the mean of the time series, λ is the time weight coefficient, β is the benchmark value of the rule set complexity, and σ is the standard deviation of the prediction error; when And f L (x i ,h i ) is greater than or equal to the preset threshold, the i-th data point is marked as abnormal, and the data sender is notified to check and correct it; if the corrected data is re-entered into the system and the Q value increases by more than 20%, the correction is determined to be effective, and positive feedback is sent to the data sender; if the corrected data causes the Q value to drop by more than 10%, the correction process is reviewed; if the data sender corrects the same data point three times and still fails to pass the verification, the data point is marked as difficult to correct and an on-site inspection is carried out; When n consecutive data points are marked as abnormal data, it is determined to be an abnormal trend, and the data sender is notified to urgently check the data collection system; if the abnormal rate of a certain type of data exceeds 10% of the total samples, a quality warning for this type of data is sent to the data sender, and the data is collected again.

4. The method for verifying the power spot market and OMS data in file transmission according to claim 3, characterized in that: The general verification rule is to check the file format and date format of the data; the single verification rule is to verify the load data; the comprehensive verification rule is to verify the mutual verification between several data; include, Receive information about the data validation rule set and convert it into a format that can be processed by the graph neural network; A multi-dimensional data verification model is constructed using the graph convolutional network (GCN) algorithm, and data points are represented as nodes in the graph neural network, and the relationship between data is represented as edges. Nodes contain multi-dimensional features, and edges protect the correlation between data points. Train the multi-dimensional data validation model using labeled data and adjust the model hyperparameters using cross-validation. Use the trained graph neural network model to deeply integrate data features of different dimensions and verify new data at the same time; Compare and integrate the verification results of the multi-dimensional data verification model with the existing data verification rule set, generate a verification report, and optimize and update the data verification rule set.

5. The method for verifying the power spot market and OMS data in file transmission according to claim 4, characterized in that: The verification comprises the following steps: Corresponding to the general verification rules, the marked data is preliminarily analyzed to obtain the basic information of the data, and the data content is cached in the queue of data to be verified; Read the configuration information of the data verification rule set from the database, and according to the priority order of the rules, the data verification engine loads the corresponding verification dynamic library to verify the data; When a verification rule fails to pass the verification and its corresponding alarm level is error, the subsequent verification of the current data is stopped and the verification process of the next data in the queue of data to be verified is directly entered; If the alarm level is warning, the verification engine continues to execute the verification rules until the next verification rule of the error level fails or all rules have been verified.

6. The method for verifying the power spot market and OMS data in file transmission according to claim 5, characterized in that: The basic information includes data type, data date and data size; include, For a single check rule, the check engine checks whether the data conforms to the preset file format and date format; if the check data passes the check, it is marked as valid and proceeds to check the next check data; If the data fails verification, the exception is recorded and manual intervention is performed; For comprehensive verification rules, the verification engine determines whether all the data currently to be verified are complete; If all the verification data are available, comprehensive verification will continue, and multi-dimensional data correlation analysis and logic verification will be performed; if any data is not available, the verification process will exit, the current verification will be stopped, and the comprehensive verification rules will be restarted after all relevant data are available.

7. The method for verifying the power spot market and OMS data in file transmission according to claim 2, characterized in that: The data feedback mechanism comprises the following steps: Generate alarm information based on the verification results and distinguish alarms of different levels; Record the verification results after each data verification is completed and generate a verification report; When a verification rule fails and the alarm level is an error, the alarm notification mechanism is immediately triggered and sent to the relevant person in charge; According to the information in the verification report, analyze the reasons and re-collect data, resubmit the data for verification, and use the support vector machine algorithm to optimize and improve the data verification rule set and feedback mechanism; The verification report includes the execution status of the verification rules, verification failure information and warning level.

8. A power spot market and OMS data verification system based on file transmission, and a power spot market and OMS data verification method based on file transmission according to any one of claims 1 to 7, characterized in that: include, The generation module is used to obtain power spot market data and OMS data, and use the long short-term memory network (LSTM) algorithm to perform cleaning, classification, and anomaly detection to generate a data verification rule set; Establish a module, based on the data verification rule set, use the graph neural network algorithm to establish a multi-dimensional data verification model, and perform deep fusion and cross-validation; The optimization module is used to generate a verification report according to the verification results of the multi-dimensional data verification model, and optimize the data verification rule set using a support vector machine algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for verifying the data of the power spot market and OMS for file transmission according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for verifying the data of the power spot market and OMS for file transmission described in any one of claims 1 to 7 are implemented.

Images