Data configurator design scheme for solving data updating problem of power transaction system

Abstract

The invention discloses a data configurator design scheme for solving a data updating problem of a power transaction system, belongs to the technical field of new energy station power generation, and aims to cope with changes of published data and constraints of business data by designing a data configuration layer as a central link of data processing and configuring a management layer. The data configuration layer depends on a preset configuration rule and a logic constraint system, and verification, conversion and standardized output of data are achieved. Secondly, a clear solution is given from the aspects of improving the data quality, monitoring the data quality and the like, fuzziness of a traditional solution is avoided, the data quality is improved while the problem is solved, data management and transaction efficiency of the electricity market are improved, and the method is suitable for popularization and application. And technical support is provided for informatization and intelligent transformation of the power industry.

Description

Technical Field

[0001] The present invention belongs to the technical field of new energy station power generation, and specifically is a data configurator design scheme for solving the data update problem of an electric power trading system. Background Art

[0002] At present, the power markets in various provinces are developing rapidly. The power trading decision-making support system needs to process a large amount of data published by trading centers in different provinces. These data are highly diverse and non-standard due to regional characteristics, which brings severe challenges to system integration, analysis and decision support. In this case, an efficient solution is urgently needed. The technical solution for data processing and management published by power trading systems in various provinces aims to solve the problem of data diversity and non-standardization of trading centers in multiple provinces through highly flexible and efficient configuration methods, so as to improve data quality, system stability and operational efficiency.

[0003] The existing technology is to customize the collection and analysis according to the content of the data published by the trading center, and then assemble the data into the format agreed upon by the server for transmission, and forward the request to the corresponding server through the reverse proxy Web server. After receiving the request, the server stores the data. The existing technology implementation scheme is such that all models in the full-link business process are built around the data published by the trading center. The above technical scheme has significant defects in the design and implementation stages, lacks the ability to respond quickly to changes in the data published by the trading center, and also lacks reusability. The high dependence on changes in the trading center data makes the system fault-tolerant. In addition, in the full-link interaction, when the upstream data changes, these schemes cannot issue early warning signals in time.

[0004] like Fig.11 , Fig.12 As shown, Fig.11 It shows the various links and problems in the data processing process of existing technologies. First of all, starting from the data provider to the crawler, back-end service and database, the entire process requires customized development and modeling. However, there are some challenges in this process, such as the slow update and iteration speed of the data provider, the crawler needs to be constantly adjusted to adapt to new requirements and changes in the data structure of the back-end service, and the back-end service faces the problems of data structure redundancy and insufficient expansion capabilities. In addition, the entire system lacks link monitoring capabilities, resulting in all nodes requiring R&D and business to invest a lot of energy and time to track and optimize.

[0005] Fig.12The impact of changes in data published by trading centers in the existing technology on the system is demonstrated. Taking the Shanxi Trading Center as an example, the data in the existing technical solution has structural changes and data redundancy problems, which will cause the crawler plug-in and back-end service code to be continuously upgraded and released to adapt to these changes. At the same time, the algorithm and BI system also need to adjust the strategy to clean these changing data to meet the needs of the data model. The data model is connected to multiple database tables, such as "Table 1", "Table 2" and "Table 3". In addition, the algorithm and BI system will feedback information to the data model to further optimize the data processing process.

[0006] In summary, the business model and process of the existing technical solution-assisted trading system are closely built around the trading center. However, once the data published by the trading center changes, the circular business relationship must respond quickly to deal with it, but the power trading market is extremely sensitive to the accuracy and timeliness of data, and any changes may have a great impact on customers' daily use, market analysis, trading decisions, etc. Summary of the invention

[0007] The purpose of the present invention is to address the current problems of data diversity and non-standardization as well as the lack of ability to quickly respond to changes in data published by trading centers.

[0008] In order to solve the above technical problems, the present invention provides a data configurator design solution for solving the data update problem of the power trading system, including:

[0009] Data access module, used to receive raw data from trading centers in multiple provinces;

[0010] A data backup module, used to back up the received original data to a database;

[0011] The data configuration analysis and processing module includes a configuration management layer, wherein the configuration management layer stores configuration data, wherein the configuration data includes key elements such as province mark, data type, interface identifier, verification logic and entity object mapping, and the data configuration analysis and processing module performs analysis, verification, conversion and standardization processing on the original data according to the configuration data;

[0012] The standardized output module is used to output the processed standardized data; wherein, the configuration management layer supports hot update of configuration, which can take effect immediately to meet the needs of rapid iteration and response to market changes.

[0013] Preferably, the configuration management layer includes a basic configuration layer, a business logic configuration layer, a monitoring and alarm configuration layer, a security and compliance configuration layer, and a data mapping and conversion configuration layer.

[0014] Preferably, the basic configuration layer includes a request layer, a configuration layer and a log layer. The request layer performs protocol comparison, creates a hard backup of data, and performs asynchronous task processing; the configuration layer marks the data by province and public or private; and the log layer records the operations of the request layer and the configuration layer in corresponding logs.

[0015] Preferably, the business logic configuration layer manages the logic of business processing, including business processing switches, logical constraints, data type labels and underlying data definitions; the data mapping and transformation configuration layer involves business models, class loaders, reflection and object generation, and is used to achieve data conversion and mapping; the security and compliance configuration layer ensures data security and compliance through validators and default settings; the monitoring and alarm configuration layer is used to issue an alarm notification when the key information of the original data and the configuration data is inconsistent or there is a difference in the constraints.

[0016] Preferably, the data configuration parsing and processing module records all request data through the MongoDB database; the configuration data is saved in the configuration center in JSON format, and each provincial transaction center has its own exclusive configuration set, each set contains multiple detailed configurations, and each configuration corresponds to different interfaces and business scenarios.

[0017] Preferably, verifying the original data includes the following steps:

[0018] S11. The adapter parser obtains data from the trading center, and the data dimension consists of province and data type;

[0019] S12, the adapter parser verifies the acquired data. The verification logic is to check whether the requested data province and interface match the protocol province and interface of the configuration management layer. The verification result is True or False.

[0020] S13. Classify the data according to the configuration dimension.

[0021] Preferably, the original data is compared with the necessary constraints of the corresponding provincial data in the configuration management layer. If there are differences, an alarm notification is issued through the monitoring and alarm configuration layer.

[0022] Preferably, the difference includes if a key is missing or a value type is changed during the verification process.

[0023] Preferably, if the monitoring and alarm configuration layer issues an alarm notification, the R&D or operation personnel modify the constraints of the configuration center according to the alarm information and re-trigger the data conversion scheduling.

[0024] Preferably, after verifying the original data, the standardized output module fills the data into the standard business object through the JAVA reflection mechanism according to the business object constraint relationship in the configuration, and then stores the data in the database.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. This solution emphasizes the design of flexible and rapid response from the perspective of "publishing data changes", and increases the ability to quickly adapt to changes in data formats and rules of various trading centers through the configuration layer, effectively reducing the impact of market rule adjustments on the system.

[0027] 2. This plan aims to improve data quality by targeting the changing nature of data release by various provinces. Through the data configuration mechanism, we can efficiently and accurately convert diverse non-standard data into a unified standard format, thereby significantly improving data quality.

[0028] 3. This solution uses a configuration-driven strategy from the perspective of "modularization promotes agile expansion", making business expansion in new provinces or the introduction of new rules simple and fast, greatly reducing the workload of repetitive development, reducing maintenance costs and the complexity of system expansion, and improving the maintainability and expansion flexibility of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0030] Figure 2 It is a schematic diagram comparing the processes of the prior art of the present invention and the present application;

[0031] Figure 3 A schematic diagram of the design background for configuring the management system of the present invention;

[0032] Figure 4 Schematic diagram of multi-province configuration management interface for the configuration management layer of the present invention Figure 1 ;

[0033] Figure 5 Schematic diagram of multi-province configuration management interface for the configuration management layer of the present invention Figure 2 ;

[0034] Figure 6 This is a schematic diagram for introducing a new configuration interface in the configuration management layer of the present invention;

[0035] Figure 7 A schematic diagram of the configuration management layer of the present invention;

[0036] Figure 8 A schematic diagram of the relationship between request data and configuration protocol of the present invention;

[0037] Fig. 9This is a schematic diagram of the alarm management interface of the present invention;

[0038] Fig.10 It is a schematic diagram of a complete solution of the present invention for data change and update scenarios;

[0039] Fig.11 It is a business process diagram of the existing technical solution;

[0040] Fig.12 A schematic diagram showing the impact of data changes published by existing trading centers on the system. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Electricity spot market: A market where operators that meet the entry requirements conduct day-ahead, intraday and real-time electricity transactions. The electricity spot market forms a market-clearing price that reflects the value of time and space through competition, and carries out auxiliary service transactions such as frequency regulation and standby.

[0043] Example 1: Figure 1 - Fig.10 As shown in the figure, the data configurator design scheme for solving the data update problem of the power trading system includes:

[0044] Data access module, used to receive raw data from trading centers in multiple provinces;

[0045] A data backup module, used to back up the received original data to a database;

[0046] The data configuration analysis and processing module includes a configuration management layer, wherein the configuration management layer stores configuration data, wherein the configuration data includes key elements such as province mark, data type, interface identifier, verification logic and entity object mapping, and the data configuration analysis and processing module performs analysis, verification, conversion and standardization processing on the original data according to the configuration data;

[0047] The standardized output module is used to output the processed standardized data; wherein, the configuration management layer supports hot update of configuration, which can take effect immediately to meet the needs of rapid iteration and response to market changes.

[0048] like Figure 1As shown, the core of this application is to design and implement an innovative data configuration layer as the central link of data processing, which plays the role of receiving, parsing and processing raw data from transaction centers in multiple provinces. The data configuration layer relies on pre-set configuration rules and logical constraint systems to realize data verification, conversion and standardized output. This design not only optimizes the data processing logic, but also demonstrates extremely high flexibility and adaptability, and can cleverly deal with the commonality and particularity of data models in different provinces. Through the advanced design strategies of the main table and related tables, the data storage structure is optimized to ensure data consistency and efficient access.

[0049] The data configuration layer includes four core stages: "data access", "data backup", "data configuration analysis and processing", and "standardized output". This design ensures the efficiency, accuracy and end-to-end controllability of data processing, and enhances the overall system performance and data security.

[0050] like Figure 2 As shown, the content in the figure mainly describes two different data processing links and their comparison. First of all, the figure mentions the point of "discovering data changes". In the prior art, when the data changes, the original processing link needs to process and publish codes for four nodes. These nodes are service center A, crawler, server and database (DB). In contrast, the link of the content of the present invention does not need to be published when the data changes. It only needs to modify the corresponding template to achieve immediate effect. This link also includes trading center A, crawler, database (DB) and algorithm, but by introducing the "DCL" template mechanism, the data processing process is significantly simplified, and the efficiency and flexibility are improved. The technical solution of the present invention provides reliable technical support for the robust operation of the power trading decision support system, and effectively copes with the complex and changing business environment.

[0051] Specifically, for the configuration management layer, the design background of the data configuration layer is introduced. In view of the uncertainty of the data published by the transaction center and the need to solve the problem of data change and update, the present invention introduces the configuration management layer as the core component, and uses the hot update capability of the configuration (a mechanism that can take effect without restarting the server program) to realize the immediate effectiveness of the configuration modification, meeting the needs of rapid iteration and response to market changes. The configuration content covers the configuration information required in the business processing process, such as: province marking, data type, interface identification, verification logic and entity object mapping and other key elements, providing flexible configuration and constraints for the data processing process.

[0052] like Figure 3As shown, the content described in the picture is about the functions and processes of the adapter parser. First, the adapter parser obtains data from trading center B, and its data dimensions consist of province and data type. Next, the adapter parser verifies the obtained data. The verification logic is to check whether the requested data province and interface match the protocol province and interface of the configuration management layer. The verification result is True or False. During the verification process, if it is found that the key is missing or the value type has changed, an early warning will be triggered. Finally, according to the configuration dimension (also composed of province and data type), the data is divided into three categories: configuration A, configuration B, and configuration C. Such a process design helps to ensure the consistency and accuracy of the data, and can also identify and respond to potential problems in a timely manner.

[0053] like Figure 4 , Figure 5 and Figure 6 As shown, a simple example of the newly added configuration interface in the configuration management layer and the multi-province configuration management interface in the configuration management layer are given. These configuration interfaces facilitate managers to perform early warning operations on the functions and processes of the adapter parser, and can clearly display specific template data and template styles.

[0054] It should be noted that the configuration data can be saved in the configuration center in JSON format. Each provincial transaction center has its own exclusive configuration collection. Each collection contains multiple detailed configurations, and each configuration corresponds to different interfaces and business scenarios.

[0055] Configuration data is saved in the configuration center in JSON format, and its immediate effect greatly simplifies the configuration adjustment process. Configuration details include but are not limited to provincial affiliation, data classification (public / private), interface naming, verification control switches, detailed constraints (such as data format, required requirements, null value processing and field alias matching) and entity object mapping paths. As mentioned above, each provincial trading center has its own exclusive configuration set, each of which contains multiple detailed configurations, each of which corresponds to different interfaces and business scenarios, building a close linkage between the configuration and the data publisher, and realizing "instant response" to data processing.

[0056] The core of this application is the hierarchical configuration system, which is the key to data automation and standardization. Through logical layering and component design, the system achieves modularity, flexibility and maintainability. Constraint configurations are labeled and classified to accurately map to request data. The data tags mainly include the following categories:

[0057] Province: This item is mainly used to map the configuration with the province corresponding to the corresponding trading center, usually represented by an enumeration value.

[0058] Type: Indicates whether it is public data or private data.

[0059] Interface name: Indicates the business to which the data belongs, such as unit, load, contract, etc.

[0060] Verification switch: indicates whether data verification is required. If verification is not performed, the necessary constraints will not take effect. It is usually stored as a Boolean value.

[0061] Necessary constraints: mainly describe the type of transaction center interface data, whether it is a required item, null value protection, alias mapping and other information.

[0062] Entity object: usually the path to the entity business object in the code. In the data conversion logic, the request data is converted into a business object and stored through reflection characteristics.

[0063] like Figure 7 As shown in the figure, this picture depicts a complex configuration management layer architecture, which consists of the basic configuration layer, business logic configuration layer, monitoring and alarm configuration layer, security and compliance configuration layer, and data mapping and transformation configuration layer. In the basic configuration layer, protocol comparison is first performed to ensure the security of communication, and then the data is marked as province and public and private for subsequent processing and analysis. Next, the system creates a hard backup of the data and performs asynchronous task processing. These operations will be recorded in the corresponding logs, such as request logs, marking logs, hard backup logs, and transformation logs. The business logic configuration layer is responsible for managing the logic of business processing, including data processing switches, logical constraints, data type labels, and bottom data definitions. The monitoring and alarm configuration layer implements the monitoring and notification functions for invalid warnings, comparison failures, and save failures. The security and compliance configuration layer ensures data security and compliance through validators and default settings. Its core part may include encryption, access control and other mechanisms. Finally, the data mapping and transformation configuration layer involves business models, class loaders, reflection, and object generation to achieve data conversion and mapping. Each layer of this system is closely connected to ensure the efficiency and accuracy of data processing.

[0064] like Figure 8 As shown, the relationship between request data and configuration protocol (request data on the left and configuration protocol on the right).

[0065] After obtaining the data from the transaction center, the request data will be written to the MongoDB database as a backup (as the original request log), and the scheduling will be triggered by asynchronous messages to convert the data from non-standardized to standardized. In this process, the system compares the original data with the necessary constraints of the corresponding provincial data in the configuration management layer. If the R&D personnel find that the key information of the transaction center data and the constraint data is inconsistent or there are differences in the constraints, the system will notify the R&D\operation personnel through the DingTalk alarm. The R&D\operation personnel will modify the constraints of the configuration center according to the missing fields or changed fields in the alarm, and then re-trigger the data conversion scheduling to ensure the consistency and accuracy of the data. The above is the comparison logic.

[0066] In addition, the original data is compared with the necessary constraints of the corresponding provincial data in the configuration management layer. If there are differences, an alarm notification is issued through the monitoring and alarm configuration layer. The difference includes if a key is missing or the value type changes during the verification process. If the monitoring and alarm configuration layer issues an alarm notification, the R&D or operation personnel will modify the constraints of the configuration center according to the alarm information and re-trigger the data conversion scheduling. Fig. 9 As shown in the alarm management interface, an alarm will be triggered when the data changes.

[0067] Data standardization output is the last step in the process. After the comparison logic is completed, the system will automatically fill in the standard business objects through the JAVA reflection mechanism according to the business object constraint relationship in the configuration (constructed through configuration-driven entity objects), and then store them in the database.

[0068] like Fig.10 As shown in the figure, a complete solution for data change and update scenarios is shown. It consists of multiple transaction centers and crawlers for crawling and collecting data. The data is first received and processed uniformly through Nginx, and then enters the original data backup stage. After that, the data is initially filtered and divided into real-time data and offline data. For real-time data, they first enter the asynchronous message processing link and then reach the data cleaning layer; for offline data, they directly enter the data cleaning layer and compare with the historical change data. In this process, any inconsistency will trigger an alarm. The cleaned data will be loaded into the data warehouse, including different types such as result tables, public tables, and atomic data. In addition, model maintenance templates for different provinces are provided to facilitate rapid response to market needs. Finally, the business end can obtain the required information by calling algorithms or BI functions.

[0069] In summary, the present invention successfully solves the problem of data diversity and non-standardization in the power trading system through innovative data configuration management layer and sophisticated data processing flow, improves data management and transaction efficiency in the power market, and provides technical support for the information and intelligent transformation of the power industry.

[0070] This application has significantly improved the technical architecture, data change processing speed, R&D efficiency, data quality, etc.:

[0071] The first is the technical architecture upgrade: from traditional end-to-end customized development to a unified data processing center, and use MongoDB to record all request data to facilitate historical scene backtracking or traffic playback, and use asynchronous messaging mechanisms to achieve business decoupling. When the request pressure is too high, the node capacity can be expanded on the consumer side to resolve it.

[0072] The second is the processing speed of data changes: through the data configuration layer, the associated constraints between the collection request and the server are established. When the transaction center announces data changes, only the corresponding constraint configuration needs to be modified, and downstream applications and services can respond without making any adjustments.

[0073] R&D efficiency has also been improved: the traditional end-to-end technical solution takes about one man-day to develop one interface on average. With the upgraded technical solution of this patent, one interface can be developed in just one hour, which greatly improves R&D efficiency.

[0074] Data quality is further improved: Through unified data cleaning, data processing, data modeling and other processes, the data finally stored in the database is standardized data that has been stored through a strict process, which greatly improves the data quality compared to the original solution;

[0075] Finally, this patent establishes a complete data monitoring and early warning solution for the power trading decision-making support system. By configuring management-level configuration constraints and comparing data during cleaning, when the trading center announces changes in data, R&D and operations personnel can quickly receive alerts and respond.

[0076] Embodiment 2: On the other hand, the present application also discloses a data configuration method for the data update problem of the power trading system, comprising the following steps:

[0077] S1. Design the data configuration layer, in which the received raw data is verified, converted and standardized according to the pre-set configuration rules and logical constraint system;

[0078] S2. After obtaining data from the trading center, the requested data is written into the database as a backup, and the data is converted from non-standardized to standardized through asynchronous message trigger scheduling;

[0079] S3. According to the business object constraint relationship in the configuration, fill it into the standard business object through the reflection mechanism and store it in the database.

[0080] This application provides solutions and ideas for the data scenario of the power spot trading decision-making support system, establishes a unified data processing center, and responds to changes in published data and constraints on business data through the configuration management layer. The configuration management layer is the core point of this application, and existing solutions lack perspectives in multiple aspects. Secondly, this application provides clear solutions from the aspects of improving data quality and monitoring data quality, avoiding the ambiguity of traditional solutions, and improving data quality while solving problems. At present, there are no applications for corresponding solutions or perspectives for improving data quality in power spot trading scenarios.

Claims

1. A data configurator design scheme for solving the data update problem of power trading system, characterized in that: include: Data access module, used to receive raw data from trading centers in multiple provinces; A data backup module, used to back up the received original data to a database; The data configuration analysis and processing module includes a configuration management layer, wherein the configuration management layer stores configuration data, wherein the configuration data includes key elements such as province mark, data type, interface identifier, verification logic and entity object mapping, and the data configuration analysis and processing module performs analysis, verification, conversion and standardization processing on the original data according to the configuration data; The standardized output module is used to output the processed standardized data; wherein, the configuration management layer supports hot update of configuration, which can take effect immediately to meet the needs of rapid iteration and response to market changes.

2. The data configurator design scheme for solving the data update problem of the power trading system according to claim 1 is characterized in that: The configuration management layer includes a basic configuration layer, a business logic configuration layer, a monitoring and alarm configuration layer, a security and compliance configuration layer, and a data mapping and conversion configuration layer.

3. The data configurator design scheme for solving the data update problem of the power trading system according to claim 2 is characterized in that: The basic configuration layer includes a request layer, a configuration layer and a log layer. The request layer performs protocol comparison, creates a hard backup of data, and performs asynchronous task processing; The configuration layer labels the data by province and public or private; The log layer records the operations of the request layer and the configuration layer in corresponding logs.

4. The data configurator design scheme for solving the data update problem of the power trading system according to claim 2 is characterized in that: The business logic configuration layer manages the logic of business processing, including business processing switches, logical constraints, data type labels and underlying data definitions; the data mapping and transformation configuration layer involves business models, class loaders, reflection and object generation, and is used to achieve data conversion and mapping; the security and compliance configuration layer ensures data security and compliance through validators and default settings; the monitoring and alarm configuration layer is used to issue an alarm notification when the key information of the original data and the configuration data is inconsistent or there is a difference in the constraints.

5. The data configurator design solution for solving the data update problem of the power trading system according to claim 1 is characterized in that: The data configuration parsing and processing module records all request data through the MongoDB database; the configuration data is saved in the configuration center in JSON format. Each provincial transaction center has its own exclusive configuration set, each set contains multiple detailed configurations, and each configuration corresponds to different interfaces and business scenarios.

6. The data configurator design solution for solving the data update problem of the power trading system according to claim 1 is characterized in that: Verification of the original data includes the following steps: S11. The adapter parser obtains data from the trading center, and the data dimension consists of province and data type; S12, the adapter parser verifies the acquired data. The verification logic is to check whether the requested data province and interface match the protocol province and interface of the configuration management layer. The verification result is True or False. S13. Classify the data according to the configuration dimension.

7. The data configurator design scheme for solving the data update problem of the power trading system according to claim 6 is characterized in that: The original data is compared with the necessary constraints of the corresponding provincial data in the configuration management layer. If there are any differences, an alarm notification is issued through the monitoring and alarm configuration layer.

8. The data configurator design solution for solving the data update problem of the power trading system according to claim 7 is characterized in that: The differences include if a key is missing or a value type is changed during the verification process.

9. The data configurator design scheme for solving the data update problem of the power trading system according to claim 7 is characterized in that: If the monitoring and alarm configuration layer issues an alarm notification, the R&D or operation personnel will modify the constraints of the configuration center based on the alarm information and re-trigger the data conversion scheduling.

10. The data configurator design solution for solving the data update problem of the power trading system according to claim 1, characterized in that: After the standard output module completes the verification of the original data, it fills the data into the standard business object through the JAVA reflection mechanism according to the business object constraint relationship in the configuration, and then stores it in the database.

Images