Power market settlement statistical data backtracking method and system based on data link

Abstract

The invention discloses an electricity market settlement statistical data backtracking method and system based on a data link. The method comprises the following steps: converting electricity market settlement business data into electricity market settlement index data; dividing the power market settlement index data into time sequence data of different statistical dimensions according to multi-dimensional analysis requirements; dividing the time sequence data into cold data and hot data according to the data popularity, and correspondingly storing the cold data and the hot data to a cold data layer and a hot data layer; when the data popularity changes, data migration is carried out on data in the cold data layer and the hot data layer; and generating a corresponding offset index according to the data offset before and after data migration, and realizing power market settlement business data tracing through the offset index. According to the method, the complete data link and the efficient storage engine are designed, the data link covers data extraction, temporary storage, migration and storage, power market index data generation, extraction and traceability can be accurately achieved, and the efficiency and integrity of power market settlement are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power market settlement data processing, and particularly relates to a method and system for tracing back power market settlement statistical data based on a data link. Background Technique

[0002] In the context of the power market, the efficient operation of settlement services is crucial for monitoring the operation of the power market and promoting the sustainable development of the energy industry. In view of the increasing frequency of statistical analysis of short-cycle settlement indicators in the power settlement market, the traditional method of relying on business personnel to manually trigger statistics and manually verify data can no longer meet the new requirements. At the same time, with the gradual deepening of the power marketization construction, the volume of settlement data has increased significantly, and the design of the data model needs to consider computing performance and efficiency.

[0003] In actual settlement operations, the data sources in the power market are diverse. In addition to the power trading platform itself, there are also the spot system and metering system from dispatching, the power consumption acquisition system from marketing, and price data from finance, etc. When these preposed data change, on the one hand, it will cause data changes in the power trading system, and on the other hand, it will also affect the results of statistical analysis. The trading platform itself may also reprocess its business due to market parameter adjustments, resulting in the redefinition of settlement indicators and changes in the statistical analysis results of the indicators. These problems together require an efficient system that can quickly trace back and locate the original business data when the statistical indicator data changes, verify the original data, and trigger automatic restatistics.

[0004] Facing the dual challenges of data complexity and timeliness in power market settlement operations, researchers are committed to developing automated processing and intelligent monitoring technologies. These technologies include real-time processing of data streams from multiple heterogeneous data sources, and a dynamic statistical analysis system based on an event-triggered mechanism to ensure real-time accuracy of statistical analysis when data changes. At the same time, an intelligent data tracing and verification framework is also being built, using metadata management and graph database technology to track the data flow, quickly locate and correct the data that affects the results. These innovations aim to improve the automation of settlement operations, ensure data quality, accelerate decision-making, and support the stable operation of the power market and the sustainable development of the energy industry.

[0005] However, the existing technologies lack a hierarchical storage mechanism for cold and hot data and an effective traceability mechanism. When dealing with large-scale power market data, it is difficult to effectively reduce storage costs and achieve rapid data traceability while ensuring data access speed. Different data types and access frequencies require different storage media and engines to optimize performance and cost. The existing technologies do not fully consider this, which may lead to waste of storage resources or performance bottlenecks. At the same time, the existing technologies do not have a complete data link design, making it difficult to ensure data integrity, consistency, and traceability. Therefore, a mechanism that can automatically identify data heat and perform hierarchical storage is needed, while ensuring that the original data can be quickly traced when problems occur with the data. A scientific storage resource allocation strategy also needs to be formulated, including selecting appropriate storage media and engines to store data based on factors such as data type, access frequency, and performance requirements. In addition, a complete data link from data extraction, staging, migration to storage needs to be designed to promptly detect and correct errors during the data processing process and ensure the accuracy of the final result. Summary of the Invention

[0006] The purpose of the present invention is to address the above problems in the existing technologies and provide a method and system for power market settlement statistical data traceback based on a data link. By designing a complete data link and an efficient storage engine, the generation, extraction, and traceability of power market index data are accurately achieved, improving the efficiency and integrity of power market settlement.

[0007] To achieve the above purpose, the present invention has the following technical solutions: In the first aspect, a method for power market settlement statistical data traceback based on a data link is provided, including: Converting power market settlement service data into power market settlement index data; Dividing the power market settlement index data into time-series data of different statistical dimensions according to multi-dimensional analysis requirements; Determining data heat according to the time sequence corresponding to the time-series data, dividing the time-series data into cold data and hot data according to data heat, and storing them in the cold data layer and the hot data layer respectively; When the data heat changes, data migration is performed on the data in the cold data layer and the hot data layer, migrating the latest data to the hot data layer and migrating the data in the hot data layer to the cold data layer; Generating a corresponding offset index according to the data offset before and after data migration, and realizing the traceability of power market settlement service data through the offset index.

[0008] As a preferred solution, after the power market settlement service data is generated, a time-series tag is added to the power market settlement service data; a cache time is set, and data migration is triggered when the cache time is exceeded.

[0009] As a preferred solution, the step of converting the power market settlement service data into power market settlement index data includes: inputting the power market settlement service data into the index statistical platform, encapsulating the index calculation formula into a script through code, editing the index calculation formula according to requirements, and calculating the power market settlement service data through the index calculation formula to obtain the power market settlement index data.

[0010] As a preferred solution, the multi-dimensional analysis requirements include any one or a combination of multiple of the trading cycle, power source type, and component dimension; Corresponding data tables are established for storing the power market settlement index data of different statistical dimensions.

[0011] As a preferred solution, the step of migrating the data in the cold data layer and the hot data layer when the data heat changes, migrating the latest data to the hot data layer, and migrating the data in the hot data layer to the cold data layer. When the power market settlement service data is cached in the cache database, a high-performance storage medium is used, and only the latest data is retained. After the set cache time ends, the original data is cleared from the high-performance storage medium and migrated to a low-performance storage medium.

[0012] As a preferred solution, the step of classifying the time-series data into cold data and hot data according to the data heat and storing them in the cold data layer and the hot data layer respectively. According to the time-series label of the time-series data, the latest power market settlement service data is marked as hot data, and other power market settlement service data is marked as cold data; the cold data is stored in a database different from the hot data, and the power market settlement service data is divided into multiple data blocks for storage according to time series.

[0013] As a preferred solution, the step of migrating the data in the cold data layer and the hot data layer when the data heat changes, migrating the latest data to the hot data layer, and migrating the data in the hot data layer to the cold data layer. The power market settlement index data is replicated in real time and synchronized to the cache database. During the index query process, a cache query is performed; when the set cache time ends, the power market settlement service data in the cache database is replicated and migrated, and is replicated in real time to the structured database. The daily power market settlement service data is divided into 24 data blocks by hour and stored as cold data in a low-performance storage medium.

[0014] As a preferred solution, in the step of migrating data in the cold data layer and the hot data layer when the data heat changes, moving the latest data to the hot data layer and moving the data in the hot data layer to the cold data layer, in addition to storing cached data, the hot data layer also stores an offset index corresponding to each data set in the cold data layer, which is stored in the structured database; for the data files in the structured database, an index file is established for the power market settlement service data divided into 24 data blocks per hour per day, storing the offset of each data block, and the index file is stored in a high-performance storage medium.

[0015] As a preferred solution, in the step of tracing the power market settlement service data through the offset index, after the power market settlement service data is migrated to the cold data layer, the data is chunked with time series tags, and a corresponding offset index is generated for each chunk of data set and stored in the index file; when the power market settlement index data generated by the power market settlement service data is abnormal or changes, by virtue of the corresponding relationship between the index file and the original power market settlement service data, the data block used to calculate the corresponding index data in the cold data layer is called and copied through the index file to obtain the corresponding power market settlement service data, completing the tracing.

[0016] In a second aspect, a power market settlement statistical data traceback system based on a data link is provided, including: A settlement service data conversion module, configured to convert power market settlement service data into power market settlement index data; A settlement index data dimension division module, configured to divide the power market settlement index data into time series data of different statistical dimensions according to the multi-dimensional analysis requirements; A data heat division and storage module, configured to determine the data heat according to the time sequence corresponding to the time series data, divide the time series data into cold data and hot data according to the data heat, and store them in the cold data layer and the hot data layer respectively; A data migration module, configured to migrate data in the cold data layer and the hot data layer when the data heat changes, moving the latest data to the hot data layer and moving the data in the hot data layer to the cold data layer; An offset index module, configured to generate a corresponding offset index according to the data offset before and after data migration, and realize the tracing of power market settlement service data through the offset index.

[0017] As a preferred solution, when the power market settlement service data is generated, the settlement service data conversion module adds a time series tag to the power market settlement service data; sets a cache time, and triggers data migration when the cache time is exceeded.

[0018] As a preferred solution, when the settlement business data conversion module converts the power market settlement business data into power market settlement index data, it inputs the power market settlement business data into the index statistics platform, encapsulates the index calculation formula into a script through code, edits the index calculation formula according to requirements, and calculates the power market settlement index data through the index calculation formula for the power market settlement business data.

[0019] As a preferred solution, when the settlement index data dimension division module divides the power market settlement index data into time-series data of different statistical dimensions according to the multi-dimensional analysis requirements, the multi-dimensional analysis requirements include any one or more combinations of trading cycle, power source type, and component dimension; corresponding data tables are established for storing the power market settlement index data of different statistical dimensions.

[0020] As a preferred solution, when the power market settlement business data is cached in the cache database, the data migration module uses a high-performance storage medium, only retains the latest data, and clears the original data from the high-performance storage medium and migrates it to a low-performance storage medium after the set cache time ends.

[0021] As a preferred solution, the data heat division storage module marks the latest power market settlement business data as hot data and other power market settlement business data as cold data according to the time-series tags of the time-series data; stores the cold data in a database different from the hot data, and divides the power market settlement business data into multiple data blocks for storage according to time series.

[0022] As a preferred solution, the data migration module performs real-time replication on the power market settlement index data, synchronizes the power market settlement index data to the cache database, and performs cache query during the index query process; at the end of the set cache time, replicates and migrates the power market settlement business data in the cache database, and real-time replicates it to the structured database, divides the daily power market settlement business data into 24 data blocks by hour, and stores it as cold data in a low-performance storage medium; In addition to storing cache data, the hot data layer also stores the offset index corresponding to each data set in the cold data layer, which is stored in the structured database; for the data files in the structured database, an index file is established for the power market settlement business data divided into 24 data blocks by hour every day, stores the offset of each data block, and stores the index file in a high-performance storage medium.

[0023] As a preferred solution, after the power market settlement service data is migrated to the cold data layer, the offset index module divides the data into blocks according to the time series tags, generates corresponding offset indexes for each data set block and stores them in the index file; when the power market settlement index data generated by the power market settlement service data is abnormal or changed, by virtue of the corresponding relationship between the index file and the original power market settlement service data, the data blocks used to calculate the corresponding index data in the cold data layer are called and copied through the index file to obtain the corresponding power market settlement service data, completing the traceability.

[0024] In a third aspect, an electronic device is provided, including a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the power market settlement statistical data traceback method based on the data link as described in the first aspect.

[0025] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the power market settlement statistical data traceback method based on the data link as described in the first aspect is implemented.

[0026] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: The present invention converts the power market settlement service data into power market settlement index data, and divides the power market settlement index data into time series data of different statistical dimensions for storage respectively according to the multi-dimensional analysis requirements of the actual power market settlement service. A complete data link is constructed based on the time series index. The time series data is divided into cold data and hot data according to the data heat, and is correspondingly stored in the cold data layer and the hot data layer. When the data heat changes, data migration is performed on the data in the cold data layer and the hot data layer, migrating the data from one storage medium to another, generating corresponding offset indexes according to the data offsets before and after the data migration. When the power market settlement index data is abnormal or updated, the power market settlement service data traceability can be realized through the offset index. The data link constructed by the present invention covers data extraction, temporary storage, migration and storage, and can efficiently realize the generation, extraction and traceability of the power market settlement index data, improving the efficiency and integrity of the power market settlement.

[0027] It can be understood that the beneficial effects of the second to fourth aspects can refer to the relevant descriptions in the first aspect above, and will not be repeated here. Description of the Drawings

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0029] Figure 1 Schematic diagram of the principle of the method for tracing back the statistical data of power market settlement based on the data link in the embodiments of the present invention; Figure 2 Schematic diagram of the cache structure of the CACAHE format file in the embodiments of the present invention; Figure 3 Schematic diagram of the structure of the DATA format file in the embodiments of the present invention; Figure 4 Schematic diagram of the structure of the INDEX index file in the embodiments of the present invention; Figure 5 Flowchart of the operation of the method for tracing back the statistical data of power market settlement based on the data link in the embodiments of the present invention. Detailed implementation manners

[0030] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0031] The embodiments of the present invention propose a method for tracing back the statistical data of power market settlement based on the data link, which uses a cache database to real-time copy the original power market settlement business data, processes the original power market settlement business data to obtain power market settlement index data. After a specific time, the power market settlement business data in the cache database is migrated to the structure database, and relevant indexes are established for the power market settlement business data, and tracing is achieved through the indexes, and a suitable storage medium and storage engine are selected for the above structure database. Thus, a complete data link is formed, and the data link covers data extraction, temporary storage, migration, and storage. If the power market settlement index data appears abnormal, according to the market settlement index data table with the finest granularity, the corresponding original business data can be called using the index file to complete the tracing of the power market settlement business data.

[0032] Please refer to Figure 1 , the method for tracing back the statistical data of power market settlement based on the data link in the embodiments of the present invention includes the following steps: S1. Convert the power market settlement business data into power market settlement index data; S2. Divide the power market settlement index data into time series data of different statistical dimensions according to the multi - dimensional analysis requirements; S3. Determine the data heat according to the time sequence of the time series data, divide the time series data into cold data and hot data according to the data heat, and store them in the cold data layer and the hot data layer respectively; S4. When the data heat changes, perform data migration on the data in the cold data layer and the hot data layer, migrate the latest data to the hot data layer, and migrate the data in the hot data layer to the cold data layer; S5. Generate a corresponding offset index according to the data offset before and after the data migration, and realize the traceability of the power market settlement business data through the offset index.

[0033] The power market settlement statistical data traceability method based on the data link in the embodiment of the present invention is divided into several steps: data migration, cold - hot data stratification, index traceability, data conversion, and dimension division. Among them: Data migration is an operation in which data is migrated from one storage medium to another; Cold - hot data stratification means that according to a certain standard, part of the data is marked as cold data and stored in a storage medium with lower performance, and another part of the data is marked as hot data and stored in a high - performance storage medium; Index traceability refers to a mechanism that realizes querying the index file to trace back to the original data by storing the positioning of the original data in the index file; Data conversion means that after the original data is input, through a series of data processing, stratification and other operations, new data is obtained and stored in a new database; Dimension division means that the settlement index data is divided into different statistical dimension index calculation methods according to business requirements, and the statistical data of the same index in different dimensions adopts a hierarchical storage strategy.

[0034] In a possible implementation manner, after the power market settlement business data is generated, a time series tag is added to the power market settlement business data and cached in a CACAHE - format file in the cache database.

[0035] Set a cache time. When the cache time is exceeded, data migration is triggered, and the power market settlement business data cached in the CACAHE - format file is migrated to a DATA - format file in the MySQL database.

[0036] In a possible implementation, the process of converting the power market settlement service data into power market settlement index data in step S1 includes the input of power market settlement service data, the editing of index formulas, and the calculation of power market settlement index data. The power market settlement service data is input into the index statistical platform, the index calculation formula is encapsulated into a script through code, the index calculation formula is edited according to requirements, the power market settlement service data is calculated through the index calculation formula to obtain the power market settlement index data, and then output to the corresponding business index database, constituting a complete data conversion process.

[0037] In a possible implementation, the multi-dimensional analysis requirements in step S2 include any one or a combination of multiple ones among the trading cycle, power source type, and component dimension. The statistical results with different granularities are stored in layers. After the power market settlement index data is output, according to business requirements, the power market settlement index data of different dimensions is output, and appropriate data tables are established for the power market settlement index data of multiple markets with different statistical dimensions for storage.

[0038] In a possible implementation, in step S3, according to the time series label of the time series data, the power market settlement service data of the latest day is marked as hot data, and the power market settlement service data except the latest day is marked as cold data. The cold data is stored in a database different from the hot data, using files in the DATA format. The power market settlement service data is divided into multiple data blocks according to the time series and stored using a storage medium with lower cost (such as in a disk array). Except for the data of the current day, the query frequency of the power market settlement service data is relatively low. The power market settlement service data of the current day is also stored in the hot data layer in the form of power market settlement index data after data processing and serves as the main basis for subsequent settlement statistical analysis.

[0039] In a possible implementation, in step S4, when the power market settlement service data is cached in the CACHE cache file format of the cache database, a high-performance storage medium is used, and only the data of the latest day is retained. After the end of the latest day, the power market settlement service data is emptied from the initial storage medium and migrated to a low-performance storage medium, ensuring that the latest data can be stored in the high-performance storage medium and data migration is performed when the data loses timeliness, without occupying the storage space of the high-performance storage medium, which helps to improve the query efficiency when querying the latest data and save storage costs.

[0040] In a possible implementation, in step S5, after the power market settlement service data is migrated to the cold data layer, the data is chunked with time series tags, and an offset index corresponding to each dataset is generated and stored in the INDEX index file; when the power market settlement index data generated by the power market settlement service data is abnormal or changed, by virtue of the corresponding relationship between the INDEX index file and the original power market settlement service data, the data blocks used for calculating this index data are called and copied from the cold data layer through the INDEX index file to obtain the corresponding power market settlement service data, completing the traceability.

[0041] In the data backtracking method for power market settlement statistical data based on the data link in the embodiment of the present invention, the four links of data migration, cold and hot data layering, index traceability, and data conversion are closely connected. After the latest original service data is generated, a time series tag is added and cached in the CACAHE format file in the cache database, and a specific cache time is set. When the set time is exceeded, a data migration transaction mechanism is triggered to migrate the original service data to the DATA format file in the MySQL database. The storage layer after the original data migration is marked as the cold data layer, and the cached data is stored in the hot data layer. The cold and hot data layering is achieved by using storage media, databases, and file formats with different performances. High-performance storage media are used for hot data, and low-performance storage media (such as disk arrays) are used for cold data. After the cached data is migrated to the cold data layer, it is divided into 24 data according to the time series tags, corresponding to 1 hour of data respectively. In addition to storing the cached data, the hot data layer also stores the offset index corresponding to each dataset in the cold data layer, stored in the INDEX format file in the structured database. Through the offset index, data traceability can be realized when the settlement statistical index data is abnormal or the index is updated, directly tracing back to the original service data in the cold data layer. The settlement index data is copied from the original service data to the index statistical platform and the index data is obtained after a series of data conversion processes.

[0042] The embodiment of the present invention adopts a hierarchical storage mechanism based on cold and hot data. According to factors such as the access frequency and importance of the data, the data is divided into hot data and cold data and stored on storage devices with different performances and costs respectively. Storing the hot data on high-performance storage devices can ensure the rapid access and processing of the data, meeting the requirements of application scenarios with high concurrency and high real-time requirements. Storing the cold data on low-cost storage devices, such as hard disks or cloud storage services, can greatly save storage space and reduce the enterprise's data storage cost. Optimize the utilization of storage resources. Through the separation of cold and hot data, the storage resources can be utilized more effectively, avoiding waste of storage resources and improving the overall utilization rate of storage resources.

[0043] The embodiments of the present invention adopt an offset index. In the technical field of data storage and processing, an offset index is a mechanism for locating the position of data. Simply put, it is a numerical value representing the distance from a certain starting point to the target data position. This starting point can be the beginning of a file, the beginning of a data block, or other specified reference positions. Through the offset index, specific data elements can be quickly and accurately located, which is very useful when dealing with a large amount of data, such as database records, log files, or large binary files. Knowing the offset, the program can directly jump to the corresponding data position for reading or writing operations without having to sequentially search the entire data set from the beginning.

[0044] The embodiments of the present invention adopt the CACHE cache file format. The cache structure of the CACHE format file is as Figure 2 shown. The CACHE cache file stores the original business data of different power markets for the latest one day. It is a two-dimensional array structure. The number of rows of the array corresponds to the business types. Each row of the array is the time-series data of a business for the latest one day. The time-series data of each business is sorted in the order of time stamps. When storing the original business data of different power markets for the next day, the data in the current CACHE cache file structure is stored in the DATA data file, the content of the CACHE cache data file is cleared, and the business data for the next day is written into the CACHE. By repeating this method, the CACHE space is reused to ensure that the data in the CACHE is always the latest.

[0045] Please refer to Figure 3 , the embodiments of the present invention adopt the DATA data file format. The DATA data file is used to store the original business data migrated from the CACHE cache file. The DATA data file splits the data of each business for one day in the CACHE cache file into 24 data blocks, and each data block stores the data of that business for 1 hour.

[0046] Please refer to Figure 4 , the embodiments of the present invention adopt the INDEX index file format. The INDEX index file is used to index the data blocks in the DATA data file. For the basic unit of the data of each business for 1 hour in the DATA data file, the INDEX index file is indexed through the file offset (offset). The offset is stored in a two-dimensional form, with the hour number sequence as the horizontal axis and the label point sequence as the vertical axis. Each element offsetij in the matrix represents the offset of the data block corresponding to the hth data of the ith business in the DATA data file.

[0047] In the prior art, for example, the literature "Method for Expanding Power Time-Series Database Cluster Based on Backtracking Search" proposes a method for expanding a power time-series database cluster based on backtracking search. First, a search topology structure is constructed through an adjacency matrix, and the backtracking search algorithm is used to perform hierarchical backtracking to clarify the target, narrow the scope, add constraint conditions, and judge relationships based on a switch matrix to achieve expansion; at the same time, backtracking simplification rules are formulated to improve the response efficiency, and the influence of the data trajectory jitter coefficient on reliability is considered. Experiments show that the response time of this method is less than 0.5 s, and the cluster efficiency exceeds 98%, which is better than the traditional method and can improve the data search efficiency and accuracy. Another example is the literature "Method and System for Managing Hot Data in a Time-Series Storage Engine", which aims to solve the problems of increased storage costs and affected read performance caused by cold data. The method includes defining parameters such as ACTIVE_TIME to divide hot and cold data, optimizing the timing of blocks entering the recycling queue, and handling situations such as insufficient memory. The system includes parameter configuration and storage design modules to execute this method. This invention can optimize memory usage and improve query performance, and has important application value. However, the prior art lacks hierarchical storage of hot and cold data and an effective traceability mechanism. When dealing with large-scale power market data, it is difficult to effectively reduce the storage cost and achieve rapid traceability of data while ensuring the data access speed. Different data types and access frequencies require different storage media and engines to optimize performance and cost. The prior art does not fully consider this point, which may lead to waste of storage resources or performance bottlenecks. In addition, without a complete data link design in the prior art, it is difficult to ensure the integrity, consistency, and traceability of data.

[0048] The embodiment of the present invention provides a method for tracing back power market settlement statistical data based on a data link. The method applies a cache database to real-time replicate the original power market settlement business data, processes the original power market settlement business data to obtain power market settlement index data. After a specific time, the original power market settlement business data in the cache database is migrated to a structured database, and relevant indexes are established for the original power market settlement business data. Tracing is achieved through the indexes, and appropriate storage media and storage engines are selected for the above databases. The above forms a complete data link, and the data link covers data extraction, temporary storage, migration, and storage. If an abnormality occurs in the settlement index data, the corresponding original power market settlement business data can be called using the index file according to the market settlement index data table with the finest granularity to complete the traceability of the power market settlement business data.

[0049] Please refer to Figure 5 , another embodiment of the present invention provides a method for tracing back power market settlement statistical data based on a data link, including the following steps: 1. Copy the business data of the latest day of the original business data to the CACHE cache file of the cache database and store it as hot data in a high-performance storage medium (such as SSD).

[0050] 2. Input, transform, and perform formula editing on the business data of the latest day in the CACHE cache file to obtain the daily table of power market settlement data at the finest time granularity. Then create a transaction regulator to generate the monthly and annual tables of power market settlement data sequentially at specific times. Store the power market settlement index data as hot data in a structured database (such as Mysql), store it as hot data in a high-performance storage medium (such as SSD), and apply the InnoDB and time-series data storage engines to the structured database.

[0051] 3. Perform real-time replication on the power market settlement index data and synchronize the data to the cache database Redis. During the index query process in the application layer, perform cache queries to improve the efficiency of index queries.

[0052] 4. At the end of the latest day, copy and migrate the business data in the CACHE cache file of the cache database and real-time copy it to the DATA data file in the structured database (such as Mysql). Divide the daily business data into 24 data blocks by hour and store them as cold data in a low-performance storage medium (such as a disk array), and apply the MyISAM and time-series data storage engines to the structured database.

[0053] 5. Establish an INDEX index file for the 24 data blocks of the daily business data in the DATA data file of the structured database, and store the offset of each data block. Store the INDEX index file in a high-performance storage medium (such as SSD), and apply the InnoDB storage engine to the structured database.

[0054] 6. When abnormal index data occurs, perform a backtrace of the power market settlement index data according to the transaction regulator, backtrace to the power market settlement index data table at the finest granularity, determine the specific time period of the business data, and then call the INDEX index file to index and backtrace to the corresponding data block of the business data in the DATA data file according to the offset of this specific time.

[0055] In some possible implementation manners, in the backtrace mechanism of the exception handling process, although the INDEX index file and the transaction regulator can efficiently complete data backtrace, there are also other alternative solutions to achieve this purpose. The source and change process of abnormal data can be traced by analyzing the logs, so as to locate the specific business data block.

[0056] Another embodiment of the present invention also proposes a power market settlement statistical data backtrace system based on a data link, including: A settlement business data conversion module, configured to convert power market settlement business data into power market settlement index data; The settlement index data dimension division module is used to divide the power market settlement index data into time series data of different statistical dimensions according to the multi-dimensional analysis requirements; The data heat division and storage module is used to determine the data heat according to the time sequence corresponding to the time series data, divide the time series data into cold data and hot data according to the data heat, and store them in the cold data layer and the hot data layer respectively; The data migration module is used to migrate the data in the cold data layer and the hot data layer when the data heat changes, migrate the latest data to the hot data layer, and migrate the data in the hot data layer to the cold data layer; The offset index module is used to generate the corresponding offset index according to the data offset before and after the data migration, and realize the traceability of the power market settlement business data through the offset index.

[0057] In a possible implementation manner, when the power market settlement business data is generated, the settlement business data conversion module adds a time series label to the power market settlement business data and caches it in a CACAHE format file in the cache database; sets the cache time, and triggers data migration when the cache time is exceeded, and migrates the power market settlement business data cached in the CACAHE format file to a DATA format file in the MySQL database.

[0058] In a possible implementation manner, when the settlement business data conversion module converts the power market settlement business data into the power market settlement index data, it inputs the power market settlement business data into the index statistical platform, encapsulates the index calculation formula into a script through code, edits the index calculation formula according to the requirements, and the power market settlement business data is calculated through the index calculation formula to obtain the power market settlement index data.

[0059] In a possible implementation manner, when the settlement index data dimension division module divides the power market settlement index data into time series data of different statistical dimensions according to the multi-dimensional analysis requirements, the multi-dimensional analysis requirements include any one or more combinations of the trading cycle, power source type, and component dimension; corresponding data tables are established for the power market settlement index data of different statistical dimensions for storage.

[0060] In a possible implementation manner, when the data migration module migrates the data in the cold data layer and the hot data layer, when the power market settlement business data is cached in a CACAHE format file in the cache database, a high-performance storage medium is used to only retain the data of the latest day. After the end of the latest day, the original data is cleared from the initial storage medium and migrated to a low-performance storage medium.

[0061] In a possible implementation, the data heat division storage module marks the electricity market settlement service data of the latest day as hot data according to the time series tags of the time series data, and marks the electricity market settlement service data except for the latest day as cold data; stores the cold data in a database different from the hot data, uses files in the DATA format, and divides the electricity market settlement service data into multiple data blocks according to the time series for storage.

[0062] In a possible implementation, the data migration module performs real-time replication on the electricity market settlement index data, synchronizes the electricity market settlement index data to the cache database Redis, and performs cache queries during the index query in the application layer; at the end of the latest day, replicates and migrates the electricity market settlement service data in the cache file of the cache database CACHE to the DATA format file in the structured database in real time, divides the electricity market settlement service data of each day into 24 data blocks by hour, stores it as cold data in a low-performance storage medium, and applies the MyISAM and time series data storage engines to the structured database; in addition to storing cache data in the hot data layer, it also stores the offset index corresponding to each data set in the cold data layer, stored in the INDEX format file of the structured database; establishes an INDEX index file for the electricity market settlement service data divided into 24 data blocks by hour every day in the DATA data file of the structured database, stores the offset of each data block, stores the INDEX index file in a high-performance storage medium, and applies the InnoDB storage engine to the structured database.

[0063] In a possible implementation, after the electricity market settlement service data is migrated to the cold data layer, the offset index module divides the data into blocks according to the time series tags, generates a corresponding offset index for each block of data sets and stores it in the INDEX index file; when the electricity market settlement index data generated by the electricity market settlement service data is abnormal or changes, with the help of the corresponding relationship between the INDEX index file and the original electricity market settlement service data, calls and replicates the data blocks used to calculate this index data in the cold data layer through the INDEX index file to obtain the corresponding electricity market settlement service data and complete the traceability.

[0064] Another embodiment of the present invention also proposes an electronic device, including a processor and a memory, where the processor is used to execute a computer program stored in the memory to implement the method for tracing back the electricity market settlement statistical data based on the data link.

[0065] Another embodiment of the present invention also proposes a computer-readable storage medium, where the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, it implements the method for tracing back the electricity market settlement statistical data based on the data link.

[0066] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory, random access memory, electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For the sake of convenience of explanation, only the parts related to the embodiments of the present invention are shown above. For the specific technical details not disclosed, please refer to the method part of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in a storage device formed by various electronic devices, and can implement the execution process recorded in the method of the embodiments of the present invention.

[0067] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0068] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0069] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions in Figure 1 one process or multiple processes and / or blocks Figure 1The functions specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in one Figure 1 one process or more processes and / or boxes Figure 1 or more boxes.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the specific implementation manners of the present invention can still be modified or equivalently replaced, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A method for retrospectively analyzing power market settlement statistical data based on a data link, characterized in that, Including: Converting the power market settlement business data into power market settlement index data; Dividing the power market settlement index data into time-series data of different statistical dimensions according to the multi-dimensional analysis requirements; Determining the data heat according to the time sequence corresponding to the time-series data, classifying the time-series data into cold data and hot data according to the data heat, and storing them in the cold data layer and the hot data layer respectively; When the data heat changes, perform data migration on the data in the cold data layer and the hot data layer, migrate the latest data to the hot data layer, and migrate the data in the hot data layer to the cold data layer; Generate a corresponding offset index according to the data offset before and after the data migration, and realize the traceability of the power market settlement business data through the offset index.

2. The method for tracing back power market settlement statistical data based on a data link according to claim 1, wherein After the power market settlement business data is generated, add a time-series label to the power market settlement business data; set a cache time, and trigger data migration when the cache time is exceeded.

3. The method for backtracking power market settlement statistical data based on a data link according to claim 1, wherein The steps of converting the power market settlement business data into power market settlement index data include: inputting the power market settlement business data into the index statistical platform, encapsulating the index calculation formula into a script through code, editing the index calculation formula according to the requirements, and calculating the power market settlement index data through the index calculation formula for the power market settlement business data.

4. The method for backtracking power market settlement statistical data based on a data link according to claim 1, characterized in that, The multi-dimensional analysis requirements include any one or more combinations of trading cycle, power source type and component dimension; Establish corresponding data tables for storing the power market settlement index data of different statistical dimensions.

5. The method for tracing back power market settlement statistical data based on a data link according to claim 1, characterized in that In the step of performing data migration on the data in the cold data layer and the hot data layer when the data heat changes, migrating the latest data to the hot data layer and migrating the data in the hot data layer to the cold data layer, when the power market settlement business data is cached in the cache database, use a high-performance storage medium, only retain the latest data, and after the set cache time ends, empty the original data from the high-performance storage medium and migrate it to a low-performance storage medium.

6. The method for tracing back the electricity market settlement statistical data based on the data link according to claim 1, characterized in that, In the step of classifying the time-series data into cold data and hot data according to the data heat and storing them in the cold data layer and the hot data layer respectively, mark the latest power market settlement business data as hot data according to the time-series label of the time-series data, and mark other power market settlement business data as cold data; store the cold data in a different database from the hot data, and divide the power market settlement business data into multiple data blocks for storage according to the time sequence.

7. The method for backtracking power market settlement statistical data based on a data link according to claim 1, wherein In the step of performing data migration on the data in the cold data layer and the hot data layer when the data heat changes, migrating the latest data to the hot data layer and migrating the data in the hot data layer to the cold data layer, perform real-time replication on the power market settlement index data, synchronize the power market settlement index data to the cache database, and perform cache query during the index query process; When the set cache time ends, copy and migrate the power market settlement business data in the cache database, and real-time copy it to the structure database. Divide the daily power market settlement business data into 24 data blocks by hour and store them as cold data in a low-performance storage medium.

8. The method for tracing back the power market settlement statistical data based on the data link according to claim 1, wherein When the data heat changes, the data in the cold data layer and the hot data layer is migrated. The latest data is migrated to the hot data layer, and the data in the hot data layer is migrated to the cold data layer. In addition to storing cache data, the hot data layer also stores the offset index corresponding to each data set in the cold data layer, which is stored in the structured database. For the data files in the structured database, an index file is established for the electricity market settlement business data divided into 24 data blocks per hour and per day, and the offset of each data block is stored. The index file is stored in a high-performance storage medium.

9. The method for tracing power market settlement statistical data based on a data link according to claim 1, wherein In the step of realizing the traceability of the electricity market settlement business data through the offset index, after the electricity market settlement business data is migrated to the cold data layer, the data is segmented by time series tags, and an offset index corresponding to each data set is generated and stored in the index file. When the electricity market settlement index data generated by the electricity market settlement business data is abnormal or changes, relying on the corresponding relationship between the index file and the original electricity market settlement business data, the data block used to calculate the corresponding index data in the cold data layer is called and copied through the index file to obtain the corresponding electricity market settlement business data, completing the traceability.

10. A power market settlement statistical data backtracking system based on a data link, characterized in that, It includes: A settlement business data conversion module for converting the electricity market settlement business data into electricity market settlement index data. A settlement index data dimension division module for dividing the electricity market settlement index data into time series data of different statistical dimensions according to the multi-dimensional analysis requirements. A data heat division and storage module for determining the data heat according to the time sequence corresponding to the time series data, dividing the time series data into cold data and hot data according to the data heat, and storing them in the cold data layer and the hot data layer respectively. A data migration module for migrating the data in the cold data layer and the hot data layer when the data heat changes, migrating the latest data to the hot data layer, and migrating the data in the hot data layer to the cold data layer. An offset index module for generating corresponding offset indexes according to the data offsets before and after data migration, and realizing the traceability of the electricity market settlement business data through the offset indexes.

11. The power market settlement statistical data traceback system based on a data link according to claim 10, wherein, When the electricity market settlement business data is generated, the settlement business data conversion module adds time series tags to the electricity market settlement business data; sets the cache time, and triggers data migration when the cache time is exceeded.

12. The power market settlement statistical data traceback system based on a data link according to claim 10, wherein, When the settlement business data conversion module converts the electricity market settlement business data into electricity market settlement index data, it inputs the electricity market settlement business data into the index statistical platform, encapsulates the index calculation formula into a script through code, edits the index calculation formula according to the requirements, and the electricity market settlement business data calculates the electricity market settlement index data through the index calculation formula.

13. The power market settlement statistical data backtracking system based on a data link according to claim 10, wherein When the settlement index data dimension division module divides the electricity market settlement index data into time series data of different statistical dimensions according to the multi-dimensional analysis requirements, the multi-dimensional analysis requirements include any one or more combinations of trading cycle, power source type and component dimension; corresponding data tables are established for the electricity market settlement index data of different statistical dimensions for storage.

14. The power market settlement statistical data backtracking system based on a data link according to claim 10, wherein When the power market settlement service data is cached in the cache database, the data migration module uses a high-performance storage medium, retains only the latest data, and clears the original data from the high-performance storage medium and migrates it to a low-performance storage medium after the set cache time ends.

15. The power market settlement statistical data backtracking system based on a data link according to claim 10, wherein The data heat division storage module marks the latest power market settlement service data as hot data and other power market settlement service data as cold data according to the time series tags of the time series data; stores the cold data in a database different from the hot data, and divides the power market settlement service data into multiple data blocks for storage according to the time series.

16. The power market settlement statistical data retrospective system based on a data link according to claim 10, wherein, The data migration module performs real-time replication on the power market settlement index data, synchronizes the power market settlement index data to the cache database, and performs cache query during the index query process. When the set cache time ends, copy and migrate the power market settlement service data in the cache database, and replicate it in real time to the structure database. Divide the daily power market settlement service data into 24 data blocks per hour and store it as cold data in a low-performance storage medium. In addition to storing cache data, the hot data layer also stores the offset index corresponding to each data set in the cold data layer, which is stored in the structure database. For the data files in the structure database, an index file is established for the power market settlement service data divided into 24 data blocks per hour per day, the offset of each data block is stored, and the index file is stored in a high-performance storage medium.

17. The power market settlement statistical data backtracking system based on a data link according to claim 10, wherein When the power market settlement service data is migrated to the cold data layer, the offset index module divides the data into blocks according to the time series tags, generates a corresponding offset index for each block of data sets and stores it in the index file. When the power market settlement index data generated by the power market settlement service data is abnormal or changes, with the help of the corresponding relationship between the index file and the original power market settlement service data, call and copy the data blocks used to calculate the corresponding index data in the cold data layer through the index file to obtain the corresponding power market settlement service data and complete the traceability.

18. An electronic device, characterized in that, It includes a processor and a memory. The processor is used to execute the computer program stored in the memory to implement the power market settlement statistical data traceback method based on the data link as described in any one of claims 1 to 9.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, it implements the power market settlement statistical data traceback method based on the data link as described in any one of claims 1 to 9.

Images