Distributed archiving method and system based on power transaction business data

Abstract

The invention discloses a distributed filing method and system based on power transaction business data, and belongs to the field of power automation. The method comprises the steps of power transaction business data archiving task creation, power transaction business data archiving task scheduling and power transaction business data archiving service. According to the method, a library table structure of a power transaction service system is not changed, scattered archiving is directly carried out on sub-library and sub-table data, the complex and tedious data archiving preparation process is avoided, and the power transaction service data archiving efficiency is improved. Power transaction service single-table data is dispersed into a plurality of single-page data object files, parallel archiving storage and archived data verification are performed, the characteristics of extensible storage capacity, high read-write efficiency and the like of an object storage technology are fully played, and meanwhile, the archiving process state of the archived data files is monitored; according to the method, the power transaction business data archiving failure task or archiving interruption task can be restarted, so that the loss of archived data files is avoided, and the robustness of data archiving is improved.

Description

Technical Field

[0001] The present invention belongs to the field of electric power automation and relates to a distributed archiving method and system based on electric power transaction business data. Background Art

[0002] With the in-depth development of the power market, the current power trading institution trading platform has accumulated a large amount of business data, such as settlement business data. With the development of the power spot market, the market will generate hundreds of thousands of settlement and clearing data every day. The rapid growth of settlement data has posed challenges to the system resources and operating efficiency of the trading platform. Through timely data archiving of power trading business data, it can meet the needs of standardized management of business data of trading institutions. By archiving power trading business data and preserving it for a long time (such as more than 10 years) as a business voucher, it can not only meet the needs of external audit and supervision, but also provide reliable multi-version data for analysis and decision-making, and enhance the application value of archived data; at the same time, by timely archiving power trading business data, it can also control the scale of online data of the trading platform, avoid data with too long a storage period on the trading platform, avoid occupying platform system resources, and promote efficient and stable operation of the trading platform.

[0003] However, the existing power trading platform has a large number of business tables, including market services, transactions, contracts, plans, settlements, etc. The data scale is huge, and archiving business data will inevitably take a long time, and the risk of archiving service interruption is high. At the same time, it also increases the difficulty of archiving data retrieval and query, which restricts the efficient use of archived data. Therefore, how to improve the efficiency of archiving large-scale business data in power trading has become a practical problem. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of low efficiency in archiving large-scale business data of power trading in the prior art, and to provide a distributed archiving method and system based on power trading business data.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: The present invention proposes a distributed archiving method based on power trading business data, comprising: Create a power trading business data archiving task; Scheduling power transaction business data archiving tasks according to power transaction business data archiving tasks; Based on the power trading business data archiving task scheduling, the power trading business data archiving execution service is carried out to realize the distributed archiving of business data.

[0006] Preferably, the creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

[0007] Preferably, the scheduling of power transaction business data archiving tasks according to the power transaction business data archiving tasks is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

[0008] Preferably, while the archiving task is being executed, the execution progress and status of the archiving batch by the data archiving execution service and the execution progress and status of each business data archiving task corresponding to the archiving batch are timely monitored according to the archiving task management service process; Among them, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to 1 to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

[0009] Preferably, the power trading business data archiving execution service based on power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

[0010] Preferably, the object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

[0011] The present invention proposes a distributed archiving system based on power trading business data, comprising: A task creation module, wherein the task creation module is used to create a power transaction business data archiving task; A task scheduling module, wherein the task scheduling module is used to schedule the power trading business data archiving task according to the power trading business data archiving task; The task execution module is used to perform power trading business data archiving execution services based on power trading business data archiving task scheduling, thereby realizing decentralized archiving of business data.

[0012] Preferably, the creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

[0013] Preferably, the scheduling of power transaction business data archiving tasks according to the power transaction business data archiving tasks is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

[0014] Preferably, while the archiving task is being executed, the execution progress and status of the archiving batch by the data archiving execution service and the execution progress and status of each business data archiving task corresponding to the archiving batch are timely monitored according to the archiving task management service process; Among them, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to 1 to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

[0015] Preferably, the power trading business data archiving execution service based on power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

[0016] Preferably, the object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

[0017] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a distributed archiving method based on power trading business data when executing the computer program.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a decentralized archiving method based on power trading business data.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a distributed archiving method based on power trading business data, which includes power trading business data archiving task creation, power trading business data archiving task scheduling and power trading business data archiving service. Specifically, by creating power trading business data archiving tasks, data archiving work has specific goals and plans. It clarifies which data needs to be archived, when to archive, and the priority of archiving, which helps to improve the pertinence and orderliness of data management. Scheduling according to archiving tasks can reasonably allocate system resources to avoid resource conflicts and waste. According to the characteristics and requirements of different tasks, they are allocated to appropriate time and processing nodes, which improves the efficiency of data archiving. The execution service can be optimized according to the characteristics of power trading business data to improve the storage quality and retrieval efficiency of data. Compared with the existing big data archiving technology, it does not change the library and table structure of the power trading business system, and directly carries out distributed archiving for the sub-library and sub-table data, avoiding the complex and cumbersome data archiving preparation process and improving the archiving efficiency of power trading business data. Secondly, the single-table data of the power trading business is dispersed into several single-page data object files, and archived storage and archived data verification are performed in parallel, giving full play to the characteristics of object storage technology such as expandable storage capacity and high read and write efficiency. At the same time, the archiving process status of the archived data file is monitored, and the power trading business data archiving failure task or archiving interruption task can be restarted to avoid the loss of archived data files and improve the robustness of data archiving. The present invention can effectively form a data archive for the entire life cycle of the power trading business, provide data support for operational analysis, and is easy to access, which helps to assist the development of existing power trading business and improve the value of archived data analysis and decision-making.

[0020] The present invention proposes a distributed archiving system based on power trading business data, which divides the system into task creation module, task scheduling module and task execution module to obtain the optimal solution to the unit combination problem, thereby accelerating the archiving efficiency of large-scale business data of power trading. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of the distributed archiving method based on power trading business data of the present invention.

[0023] Figure 2 This is a diagram of the data archiving task management service and task execution service flow and interaction process of the present invention.

[0024] Figure 3 This is the hierarchical management organizational structure for archiving power trading business data of the present invention.

[0025] Figure 4 This is a diagram of a distributed archiving system based on power trading business data according to the present invention.

[0026] Figure 5 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0030] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0031] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings: Example 1 The present invention proposes a distributed archiving method based on power trading business data, such as Figure 1 As shown, including: S1. Create a power trading business data archiving task; The creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

[0033] S2. Scheduling power transaction business data archiving tasks according to power transaction business data archiving tasks; The scheduling of the power trading business data archiving task according to the power trading business data archiving task is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

[0034] While the archiving task is being executed, the execution progress and status of the archiving batch by the data archiving execution service and the execution progress and status of each business data archiving task corresponding to the archiving batch are monitored in a timely manner according to the archiving task management service process; wherein, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to "1" to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

[0035] S3. Perform power trading business data archiving execution services based on power trading business data archiving task scheduling to achieve distributed archiving of business data.

[0036] The power trading business data archiving execution service based on the power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

[0037] The object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

[0038] The method proposed in the present invention specifically includes power trading business data archiving task creation, power trading business data archiving task scheduling and power trading business data archiving execution service, data archiving task management service and task execution service process and interaction process as shown in FIG. Figure 2 shown.

[0039] The creation of power trading business data archiving tasks includes: 1) Create a batch sequence for archiving power trading business data. The creation of power trading business data archiving batches first classifies and creates power trading business data archives according to the type of power trading business, such as power trading settlement archives, trading clearing archives, trading execution archives and other archive classifications. Secondly, it is also necessary to create batch serial numbers to meet the archiving data version management. The creation of trading business data archive batch sequences includes setting business data archive categories (including market service archives, market clearing data archives, market settlement data archives and other categories), data archive batch numbers, data archive batch names, associated business start dates, associated business end dates, file creator names, file creation dates, notes, etc., and writes archive batch related information into the archive batch information management table.

[0040] 2) Configure the archiving task set corresponding to the archiving batch sequence. Each archiving batch of an archive classification contains multiple data types. For example, the archiving of settlement business data includes the archiving of settlement model data, the archiving of settlement component data, the archiving of metering data, etc., so it is necessary to establish an archiving task set for each archiving batch. Specifically, configure all data tables of the power trading business system associated with the data archive business category of the archiving batch sequence, create data archiving tasks for each data table, and form an archiving task set. By archiving the power trading business data in a dispersed manner and breaking it down into small pieces, it is helpful to simplify the data preparation process, avoid complex multi-table queries from being aggregated into large data file archiving, and improve archiving efficiency. Specifically, the configuration is completed by writing the archiving task number, archiving task name, source table number, source table name, archiving task creation system time and other parameters corresponding to the archiving batch number into the archiving task information management table.

[0041] 3) Set the execution parameters of the archiving task. The archiving execution parameter settings for each archiving task include the archiving data source parameter settings, archiving parameter settings, and archiving task scheduling parameter settings. The data source setting parameters mainly set the archiving fields for each single data source table in the archiving task, so that the archiving execution module can perform data archiving processing on the relevant records in the single data table according to the archiving flag field of the single data table; the archiving setting parameters include the archiving source data filtering condition parameters and the archiving target storage parameter composition. The archiving source data filtering condition parameters set the relevant record collection scope in the source table for the archiving execution service, such as time filtering conditions, data type filtering conditions, data record quantity filtering conditions, incremental / full filtering conditions, etc.; the archiving target storage parameters set the target storage space, directory path, etc. for data archiving for the archiving execution service. The archiving task scheduling parameters set the archiving task trigger start mode and execution priority order under each archiving batch for the archiving execution service.

[0042] The scheduling of power trading business data archiving tasks is specifically divided into three steps: 1) When the archive batch and archive task are created, the archive task management service process will store the batch sequence to be archived and the corresponding task set in the archive task queue according to the task priority, and the archive task execution service process will actively poll the execution progress and status of the archive batch tasks in the queue, find the batch sequence in the state of being archived, and apply for the archive token issued by the archive task management service to execute data archiving, so as to facilitate the horizontal expansion of the archive execution service. At the same time, the archive task management service process updates the execution status of the archive task queue to avoid other archive execution services from repeatedly applying for archive tokens.

[0043] 2) While the archiving task is being executed, the archiving batch and the archiving task execution process are monitored. The archiving task management service process timely monitors the execution progress and execution status of the archiving batch by the data archiving execution service and the execution progress and execution status of each business data archiving task corresponding to the archiving batch. The archiving task execution progress monitoring records the number of files and records that have been successfully archived for each archiving task, the number of files and records that are being processed, and the number of failed files and records. The archiving task execution status monitoring records the task status of each archiving task (0 indicates not started, 1 indicates in progress), and the task completion status (0 for success, 1 for failure). Similarly, the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed. The archiving batch task execution status monitoring records the task execution status of the archiving batch (0 indicates not started, 1 indicates in progress), and the task completion status (1 for success, 0 for failure). When all archiving tasks corresponding to the archiving batch number are successfully completed, the archiving status of the archiving batch sequence is set to "1" to indicate that all archiving tasks under the batch have been completed. When the archiving task corresponding to the archiving batch number fails to archive, a warning is issued.

[0044] 3) Update and maintenance of the archiving task queue. When the archiving task is in a successful state, the archiving task management service removes the archiving task from the task queue and adds the newly added archiving task to the task queue.

[0045] The data archiving execution service for power trading business includes reading archiving configuration, executing archiving tasks, and monitoring archiving results. The specific contents are as follows: 1) According to the archive source data screening condition configuration parameters of the archive task, archive data records are collected from the single table of the trading platform business library, loaded into the server memory, and further encapsulated into an object data set. The first step is to obtain the archive source data time screening condition data source configuration parameters, and extract archive data from the single table of the source business database corresponding to the archive task according to this parameter; the second step is to obtain the incremental / full screening condition data source configuration parameters. If it is a full configuration parameter, all extracted data is directly encapsulated into a full object set; if it is an incremental configuration parameter, the previous version of archive data is downloaded from the archive OSS storage server and loaded into the archive execution server memory, compared with the currently extracted archive data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

[0046] 2) According to the number of archived data objects in the collection, formulate a paging archive upload strategy to generate one or more archived object data files. The paging archive upload strategy is as follows: obtain the quantity size information of the archived object data corresponding to the object collection, and compare the archived object size information with the object paging upload object data threshold. If it is greater than the upload object paging threshold, the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named. At the same time, the association relationship between the data archiving task and the archived object data file is established. Otherwise, the archived data object collection is directly converted into an archived object data file.

[0047] 3) The generated archive object data file records not only the source table data content, but also the metadata of the data file, including data file encoding method, source database type, source database single table screening conditions (data start date, data end date, incremental / full), total data file page number, current data file page number, number of data file rows, data file size, number of fields, data file feature code (summary information); further records the attributes of each field, including: affiliated source table id, affiliated source table name, field English name, field Chinese name, field type, character length (number of characters), field length (number of bytes), precision (decimal number precision, timestamp precision), whether it is empty, whether it is a primary key, comments, whether it contains Chinese, and whether to generate summary information.

[0048] 4) Create a data upload thread for each archive object data file, and perform serialization and transmission to the OSS object storage server according to the archive configuration parameters to improve the efficiency of archive file transmission. Obtain the configuration parameters such as the archive OSS storage space, storage file path, archive object data file name, etc., and use them to serialize and upload to the OSS object server for persistent storage; then write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archiving status table. Among them, the archive files in the OSS storage space are organized in layers according to the archiving scenario-archiving category-archiving object-archiving object group-archiving object file-archiving object attachment (such as attachment Figure 3 ), which facilitates the retrieval of archived data files and improves the efficiency of archived data query.

[0049] 5) Create a data verification thread for each archived object data file, and perform archived data object file verification to verify the authenticity of the archived data. Traverse the archived file archive status table, download the archived object data file that has been successfully uploaded, and then deserialize and parse the downloaded object data file to extract the data content of the archived object data file and the metadata information of the object data file; secondly, generate the object data file feature code for some field data that generates summary information according to the agreed algorithm (such as the SHA-1 algorithm), and compare it with the feature code in the metadata of the extracted object data file. After passing the consistency check, it indicates that the object data file has been successfully archived. Otherwise, it indicates that the data file has failed to be archived. At the same time, the verification result is written into the archive status table.

[0050] 6) The upload status and verification result of the archiving object data file corresponding to the archiving task are sent to the archiving task management process so that it can update the archiving execution status information of the archiving task queue.

[0051] The settlement data is archived in a power trading institution in a certain province, realizing the efficient archive storage of massive data, providing efficient data support for data archiving, efficient data processing, compliance auditing and operational analysis of the trading platform, and having the conditions to expand to other power trading centers across the country in the future. The method proposed in the present invention is different from the current big data file archiving. The present invention needs to solve the processing problem of massive data file archiving, including the efficiency of massive data file archiving to avoid long archiving time; the reliability of massive data archiving to avoid the loss and omission of archived data files. Therefore, the present invention proposes a multi-level decentralized management method of archive batches, archive tasks, single table data, and paging files in the archive management layer, and triggers the scheduling of data archive task execution by real-time monitoring of the data archiving process status, restarting the power trading business data archiving failure task or archive interruption task, avoiding the loss of archived data files, and improving the robustness of data archiving. In the data archiving execution layer, the single-table data of the power trading business is decentralized into several single-page data object files, and multi-threaded archive storage and archive data verification are performed, giving full play to the characteristics of object storage technology such as large storage capacity and high read and write efficiency.

[0052] Example 2 The present invention proposes a distributed archiving system based on power trading business data, such as Figure 4 As shown, it includes a task creation module, a task scheduling module and a task execution module; The task creation module is used to create a power trading business data archiving task; The creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

[0053] The task scheduling module is used to schedule the power trading business data archiving task according to the power trading business data archiving task; The scheduling of the power trading business data archiving task according to the power trading business data archiving task is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

[0054] While the archiving task is being executed, timely monitor the execution progress and status of the data archiving execution service on the archiving batch and the execution progress and status of each business data archiving task corresponding to the archiving batch according to the archiving task management service process; Among them, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to "1" to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

[0055] The task execution model is used to perform power trading business data archiving execution services based on power trading business data archiving task scheduling, thereby realizing decentralized archiving of business data.

[0056] The power trading business data archiving execution service based on the power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

[0057] The object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

[0058] Example 3 See also Figure 5As shown, the present invention also provides an electronic device 100 based on a distributed archiving method for power trading business data; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0059] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the distributed archiving method based on power trading business data described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0060] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0061] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a distributed archiving method based on power trading business data, and the processor 102 can execute the plurality of instructions to implement: Create a power trading business data archiving task; Scheduling power transaction business data archiving tasks according to power transaction business data archiving tasks; Based on the power trading business data archiving task scheduling, power trading business data archiving execution service is carried out to realize distributed archiving.

[0062] Example 4 If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0063] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0065] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0066] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A distributed archiving method based on power trading business data, characterized in that: include: Create a power trading business data archiving task; Scheduling power transaction business data archiving tasks according to power transaction business data archiving tasks; Based on the power trading business data archiving task scheduling, the power trading business data archiving execution service is carried out to realize the distributed archiving of business data.

2. The distributed archiving method based on power trading business data according to claim 1 is characterized in that: The creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

3. The distributed archiving method based on power trading business data according to claim 2 is characterized in that: The scheduling of the power trading business data archiving task according to the power trading business data archiving task is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

4. The distributed archiving method based on power trading business data according to claim 3 is characterized in that: While the archiving task is being executed, timely monitor the execution progress and status of the data archiving execution service on the archiving batch and the execution progress and status of each business data archiving task corresponding to the archiving batch according to the archiving task management service process; Among them, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to 1 to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

5. The distributed archiving method based on power trading business data according to claim 1 is characterized in that: The power trading business data archiving execution service based on the power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

6. The distributed archiving method based on power transaction business data according to claim 1 is characterized in that: The object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

7. A distributed archiving system based on power trading business data, characterized in that: include: A task creation module, wherein the task creation module is used to create a power transaction business data archiving task; A task scheduling module, wherein the task scheduling module is used to schedule the power trading business data archiving task according to the power trading business data archiving task; The task execution module is used to perform power trading business data archiving execution services based on power trading business data archiving task scheduling, thereby realizing decentralized archiving of business data.

8. The distributed archiving system based on power transaction business data according to claim 7 is characterized in that: The creation of the power trading business data archiving task includes the creation of a power trading business data archiving batch sequence, the configuration of an archiving task set, and the setting of archiving task execution parameters; Among them, the archiving task set corresponding to the archiving batch sequence is configured for the archiving batch sequence of the power trading business data.

9. The distributed archiving system based on power trading business data according to claim 8 is characterized in that: The scheduling of the power trading business data archiving task according to the power trading business data archiving task is specifically as follows: After the power trading business data archiving task is created, the archiving task management service process will store the batch sequence to be archived and the corresponding archiving task set into the archiving task queue according to the task priority, and the archiving task execution service process will actively poll the execution progress and status of the archiving batch tasks in the queue, find the batch sequence in the state of waiting to be archived, and apply for the archiving token issued by the archiving task management service to execute data archiving. At the same time, the archiving task management service process will update the execution status of the archiving task queue; when the archiving task is in a successful state, the archiving task management service will remove the archiving task from the task queue and add the newly added task to be archived to the task queue.

10. The distributed archiving system based on power transaction business data according to claim 9 is characterized in that: While the archiving task is being executed, timely monitor the execution progress and status of the data archiving execution service on the archiving batch and the execution progress and status of each business data archiving task corresponding to the archiving batch according to the archiving task management service process; Among them, the archiving task execution progress monitoring records the number of files and records that have been successfully archived, the number of files and records being processed, and the number of failed files and records for each archiving task; the archiving task execution status monitoring records the task progress status and task completion status of each archiving task; the archiving batch execution progress monitoring records the number of successfully completed archiving tasks, the number of archiving tasks being processed, and the number of archiving tasks that failed to be processed; the archiving batch task execution status monitoring records the task execution status and task completion status of the archiving batch; when all archiving tasks corresponding to the archiving batch number are completed successfully, the archiving status of the archiving batch sequence is set to 1 to indicate that all archiving tasks under the batch have been completed; when the archiving task corresponding to the archiving batch number is in the archiving failure status, a warning is issued.

11. The distributed archiving system based on power transaction business data according to claim 7, characterized in that: The power trading business data archiving execution service based on the power trading business data archiving task scheduling is specifically: Obtain the quantity information of the archived object data corresponding to the object data set, compare the quantity information of the archived object data with the object paging upload object data threshold, if it is greater than the object paging upload object paging threshold, then the archived object data is paging split according to the paging threshold, and each page of object data obtained after the split is converted into an archived object data file, and the file is numbered and named, and at the same time, an association relationship between the data archiving task and the archived object data file is established; if it is less than or equal to the object paging upload object paging threshold, the archived data object set is directly converted into an archived object data file; Create a data upload thread for each archive object data file, obtain the archive OSS storage space, storage file path, and archive object data file name, and serialize and upload to the OSS object server for persistent storage according to the archive configuration parameters; write the upload time and the upload success or failure information returned by the OSS object storage server into the archive file archive status table; the archive files in the OSS storage space are organized in layers according to the archiving scenario - archiving category - archiving object - archiving object group - archiving object file - archiving object attachment; Traverse the archive file archiving status table, download the archive object data file that has been uploaded successfully, deserialize and parse the downloaded object data file, extract the data content of the archive object data file and the metadata information of the object data file; secondly, generate an object data file feature code for the field data of some generated summary information, and compare it with the feature code in the metadata of the extracted object data file. After a consistency check, it indicates that the object data file has been successfully archived; otherwise, it indicates that the data file has failed to be archived, and the verification result is written into the archive status table, and the upload status of the archive object data file corresponding to the archiving task and the verification result are sent to the archiving task management process to complete the distributed archiving of power trading business data.

12. The distributed archiving system based on power transaction business data according to claim 7, characterized in that: The object data set is specifically: Get the data source configuration parameter of the time filter condition for the archived source data, and extract the archived data from the single table of the source business database corresponding to the archive task according to this parameter; Get the data source configuration parameters of the incremental / full filtering conditions. If it is a full configuration parameter, all the extracted data will be directly encapsulated into a full object set. If it is an incremental configuration parameter, the previous version of the archived data is downloaded from the archive OSS storage server and loaded into the archive execution server memory. After comparing it with the currently extracted archived data, after deleting duplicate data records, the newly added or changed data records are encapsulated into an object data set.

13. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the distributed archiving method based on power trading business data as described in any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the distributed archiving method based on power trading business data as described in any one of claims 1 to 6 are implemented.

Images