Streamed tdengine database data self-defined synchronization method for power system

Abstract

The application discloses a kind of flow TDengine database data self-defined synchronization method for power system, this method includes: database table and field synchronization method, data timing automatic synchronization method, database table is shown after synchronization.Database table and field synchronization method, by manual mode, add the database connection information of different levels, when manually triggering database or database table synchronization, connection path is automatically created in system, obtain database or table information and persist to local database, simultaneously it will be saved to Redis database.Data timing automatic synchronization method, in the scheduling task in system, create new scheduling task, and bind the data exchange connection line that has been created, that is, data can be extracted to receiver database at specified time point.Effectively improve data interaction efficiency, and power system data transmission efficiency, realize the purpose of data synchronization between different subject data in multi-layer architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system data processing, and particularly relates to a flow type TDengine database data self-defined synchronization method for a power system. BACKGROUND

[0002] Power digital transformation is an inevitable choice to respond to energy revolution and social development. On the one hand, with the rapid development of renewable energy, the power system needs to realize source-grid-load-storage collaboration to improve the consumption capacity of new energy and the flexibility of the power grid. On the other hand, with the diversification and individualization of user demand, the power system needs to realize multi-energy complementation to provide differentiated and diversified energy services. All of these need to rely on digital technology and intelligent technology to realize fine management and optimized scheduling of the power system.

[0003] In 2009, the State Electricity Regulatory Commission issued the Auxiliary Service Management Rules and the Power Plant Grid Connection Management Rules (the "two rules" for building plants), and various power grid assessment and compensation platforms were successively put into operation. However, over time, new energy power plants have also been included in the "two rules" assessment and compensation calculation range, and the number and data types of grid-connected power plants have rapidly increased, resulting in a huge increase in data volume, with a huge amount of data generated per second. Traditional relational databases cannot support massive data and have slow data interaction efficiency, which seriously affects the data transmission efficiency of the power system. When displaying the results, the existing platform uses the C / S mode, which is not only cumbersome to install but also has certain configuration requirements for the host used. With the upgrading of technology, the increase in power grid and power plant users, the platform has been unable to effectively adapt. Moreover, due to the limitations of the JAVA platform JVM, it cannot cache second-level data and cannot display detailed curves for users.

[0004] In summary, the existing database platform and synchronization method cannot meet the requirements of slow data interaction efficiency, low data transmission efficiency, and detailed result display requirements of users due to the increasing number of power plants, which has become a difficult problem for technical personnel in the field of power systems. SUMMARY

[0005] To solve the problems in the prior art, the present application solves the technical problem of providing a flow type TDengine database data self-defined synchronization method for a power system that can support massive data, effectively improve data interaction efficiency and power system data transmission efficiency, cache second-level data, display detailed curves for users, and realize data synchronization between different subject data in a multi-layer architecture.

[0006] To solve the above technical problems, the technical solution adopted by the present application is: a flow type TDengine database data self-defined synchronization method for a power system, including database table and field synchronization and data timing automatic synchronization, comprising the following steps:

[0007] (I) Database table and field synchronization

[0008] (1) Build a streaming TDengine database, a relational database mysql, and an in-memory database Redis in the application server;

[0009] (2) Deploy and run the system containing the synchronization function, perform "database registration", register and add the database connection information of the corresponding level, and use it for subsequent data synchronization;

[0010] (3) Perform "database table registration", select the corresponding level and select the database to be registered, automatically connect to the target database through the "synchronization" operation in the system, and pull all table information in the database, that is, the table of the target database is persisted to the local;

[0011] (4) Synchronize the database field, automatically create a connection and obtain the target table field information, and persist it to the local, and the user configures the table field to be synchronized by setting whether to synchronize, which is used to accurately control the database table field to be synchronized;

[0012] (5) Add "data exchange connection line", configure data flow, and use it for subsequent automatic data synchronization in the automatic task;

[0013] (II) Data timing automatic synchronization

[0014] (6) After the database table and field synchronization is completed, the user configures the scheduling task according to the business requirements and binds the data connection exchange line;

[0015] (7) The system running on the server automatically starts a thread at the time specified by the user, creates a connection according to the bound data connection exchange line;

[0016] (8) According to the user's settings of whether the database table is available / whether it is exchanged, and whether the table field is available, the data in the corresponding task period is extracted and persisted to the receiver database.

[0017] The above-mentioned method for customizing synchronization of streaming TDengine database data for a power system further includes database table synchronization after the data timing automatic synchronization is completed. The user configures a graph according to requirements, the system splices a sql statement from the database according to the defined parameters to pull out the required data, and displays the data in the graph according to the user's individual configuration.

[0018] The above-mentioned method for custom synchronization of streaming TDengine database data for power systems, in the process of synchronizing database tables and fields, the interface program calls the Mysql database through the adaptive driver library to obtain the corresponding information of the target database or database table.

[0019] The above-mentioned method for custom synchronization of streaming TDengine database data for power systems, in the process of synchronizing database tables and fields, the interface program calls the Mysql database through the adaptive driver library to obtain the corresponding information of the target database or database table.

[0020] table_name: database table name

[0021] table_comment: database table description

[0022] The above-mentioned method for custom synchronization of streaming TDengine database data for power systems, in the process of synchronizing database tables and fields, the interface program calls the Mysql database through the adaptive driver library to obtain the corresponding information of the target database or database table.

[0023] table_schema: database table name

[0024] column_name: field name

[0025] column_comment: field note

[0026] data_type: field type

[0027] character_maximum_length: length setting

[0028] column_key: database table field primary key identifier, default null, PRI primary key component

[0029] If the TDengine database field information is synchronized, the following corresponding field information will be obtained and persisted locally

[0030] Field: field name

[0031] Type: field type

[0032] Note: field note

[0033] Length: length setting.

[0034] The above-mentioned method for custom synchronization of streaming TDengine database data for power systems, in the process of data extraction and persistence to the receiving database in the corresponding task period in step (8), the tagged data in the streaming TDengine database adopts three-dimensional positioning:

[0035] tag_phy: main dimension

[0036] tag_app: Application dimension

[0037] data_time: Time dimension

[0038] value: value.

[0039] The aforementioned custom synchronization method for streaming TDengine database data in power systems allows for setting extraction time, extraction cycle, number of retries for failures, and configuring data cleaning logic after extraction according to business requirements during the timed automatic data synchronization process.

[0040] In the aforementioned custom synchronization method for streaming TDengine database data in power systems, during the three-dimensional positioning process, the super data table in the TDengine database has four fields: subject dimension, application dimension, time dimension, and value. All data can be located through the three-dimensional fields to facilitate data query and modification. The subject dimension records the subject identifier, the application dimension records the data category, and the time dimension records the data generation time.

[0041] The above-mentioned custom synchronization method for streaming TDengine database data in power systems involves the user writing data processing logic within the system according to actual business needs. In the scheduling task section, the user binds the custom processing logic block with a custom name, and the system will automatically start the corresponding data cleaning process at the user-configured time and persist the processed data as needed.

[0042] The aforementioned custom synchronization method for streaming TDengine database data in power systems includes a group layer, a power generation company layer, and a power plant layer.

[0043] The advantages of this invention for a custom synchronization method of streaming TDengine database data in power systems are as follows: The interface program is written in Java. During the synchronization of database tables and fields, a manual triggering mechanism is used. Based on the user-selected database or table, after manual triggering, the corresponding thread automatically creates a connection, parses the target database or table, synchronizes the parsed information to the local database, and destroys the thread after completion. During scheduled automatic synchronization, this part is combined with a scheduled task and bound to the configured database exchange line. The system will then synchronize the user-defined source database data to the target database at a specified time. During the post-synchronization display process, a browser kernel is used for data interaction in client mode. It supports configuring different display graphics according to the user's business needs, currently supporting various common data display charts on the market. Users bind data marked with tags within the system to the graphics, supporting personalized settings and refresh time settings, allowing users to intuitively see the extracted data through the graphics. This invention achieves customized and rapid data synchronization between different entities in a multi-layered architecture, including the group level, power generation company level, and power plant level, based on the same database. It can support massive amounts of data while effectively improving data interaction efficiency and power system data transmission efficiency. Furthermore, it can cache data at the second level, displaying detailed curves to users. This provides strong technical support for the rapidly developing power grid industry and is suitable for industrial applications. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0045] Figure 2 Flowchart of the database table and field synchronization process;

[0046] Figure 3 Flowchart of the automatic data synchronization process at regular intervals;

[0047] Figure 4 A flowchart showing the process after database table synchronization. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The application is based on a TDengine database, which is a big data platform designed and optimized for the Internet of Things, the Internet of Vehicles, the Industrial Internet, IT operation and maintenance, etc. In addition to the core 10 times faster time series database function, it also provides cache, data subscription, streaming computing and other functions, which can minimize the complexity of research and development and operation and maintenance, and the core code, including cluster functions, is all open source. It uses a SQL interface, which has low learning costs. In addition, it is simple to install, deploy and maintain. It is suitable for data with the following characteristics: time series, with timestamp; structured; data rarely have update operations; data source is unique; relative to Internet applications, read more and write less; users are interested in trends over time; data has a retention period; data queries are often based on time periods and certain groups of devices; in addition to storage and query, real-time analysis and calculation operations are often required; traffic is smooth and predictable; and data volume is huge. The method can greatly improve data interaction efficiency and power system data transmission efficiency for power grid system data synchronization and interaction.

[0050] As shown in Figure 1 A flow TDengine database data self-defined synchronization method for a power system, the method comprising: a database table and field synchronization method, a data timing automatic synchronization method, and a database table synchronization post-display. The database table and field synchronization method adds different levels of database connection information through a manual method. When the database or database table synchronization is manually triggered, a connection path is automatically created in the system, the database or table information is obtained and persisted to a local database, and is also saved to a Redis database. The data timing automatic synchronization method creates a new scheduling task in a scheduling task in the system, and binds the created data exchange connection line. Thus, the data can be automatically extracted to a receiver database at a specified time point. The database table synchronization post-display uses a browser kernel to display the results in a client manner.

[0051] Specifically, the self-defined synchronization method comprises database table and field synchronization and data timing automatic synchronization, and comprises the following steps:

[0052] (I) Database table and field synchronization

[0053] (1) A flow TDengine database, a relational database mysql, and an in-memory database Redis are built in an application server.

[0054] (2), deploy and run the system containing the synchronization function, carry out "database registration", add the registration of the corresponding hierarchical database connection information, for subsequent data synchronization; in this step, regarding the system applied by the application, if the method is applied in the system written in java language, jdk and other application middleware <such as rabbitMQ, etc.> need to be configured during deployment, different systems will be different during deployment due to different functions. That is, different program languages form a system, and the required files and functions are reasonably selected according to the situation of different systems. The technology is prior art, which is not described in detail here.

[0055] (3), "database table registration" is carried out, the corresponding level is selected and the database to be registered is selected, through the "synchronization" operation in the system, the system is automatically connected to the target database, and all table information in the database is pulled, that is, the table of the target database is persisted to the local;

[0056] (4), synchronize the database field, the system automatically creates a connection and obtains the target table field information, and persists to the local, the user configures the table field to be synchronized by setting whether to synchronize, which is used to accurately control the database table field to be synchronized;

[0057] (5), add "data exchange connection line", configure data flow, which is used for subsequent automatic data synchronization in automatic task; for example: configure the data source as xx power company, the database as databaseCompany, the data table as tableCompany, the data receiver as xx power group, the database as databaseGroup, and the data table as tableGroup.

[0058] (II), data timing automatic synchronization

[0059] (6), after the synchronization of the database table and the field is completed, the user configures the scheduling task according to the business requirement, and binds the data connection exchange line; for example: the user binds the data exchange line 1 <xx power company to xx power group> according to the business requirement, and clicks start task, at 1 o'clock in the morning every day, the data will be automatically transferred according to the information configured in the "data connection exchange line".

[0060] (7), the system running on the server automatically starts the thread at the time specified by the user, and creates a connection according to the bound data connection exchange line;

[0061] (8), according to whether the database table is available / whether the table field is available, the data in the corresponding task period is extracted and persisted to the receiver database.

[0062] As Figure 2 shown, in the database table and field synchronization process, the interface program calls the Mysql database through the adaptive driver library to obtain the corresponding information of the target database or database table. The synchronized database table information such as mysql obtains the following information and persists to the local at the same time:

[0063] table_name: database table name

[0064] table_comment: database table description

[0065] The synchronized table field information such as mysql obtains the following corresponding field information and persists to the local:

[0066] table_schema: database table name

[0067] column_name: field name

[0068] column_comment: field note

[0069] data_type: field type

[0070] character_maximum_length: length setting

[0071] column_key: database table field primary key identifier, default null, PRI primary key component

[0072] If the TDengine database field information is synchronized, the following corresponding field information will be obtained and persisted to the local

[0073] Field: field name

[0074] Type: field type

[0075] Note: field note

[0076] Length: length setting.

[0077] The hierarchy of the application includes a group layer, a power generation company layer and a power plant layer. Specifically, the group layer refers to the five major power generation groups in China; the power generation company layer refers to the next level of the power generation group; and the power plant layer refers to the next level of the power generation company, which is the last level.

[0078] As Figure 3As shown, in the data timing automatic synchronization process, the extraction time, extraction period, failure retry number can be set according to the demand, and the data cleaning logic after data extraction can be configured according to the business demand. The configuration process of the data cleaning logic is that the user writes the data processing logic according to the actual business needs in the system, binds the self-defined processing logic block through the self-defined name in the scheduling task part, and the system will automatically start the corresponding data cleaning processing at the time configured by the user and persist the processed data according to the needs. For example, the coal consumption data in power industry, this data is normally >0, if the data taken is <0, it can be discarded or alarmed through the self-defined data cleaning logic.

[0079] In the process of data extraction and persistence to the receiver database in the corresponding task period, the tagged data in the stream TDengine database is three-dimensionally positioned:

[0080] tag_phy: main dimension

[0081] tag_app: application dimension

[0082] data_time: time dimension

[0083] value: value.

[0084] Specifically, in the super data table of TDengine database, the table field has 4, including main dimension, application dimension, time dimension and value. All data can be positioned through three-dimensional field to facilitate data query and modification. The main dimension records the main identification, such as group code, unit code, etc. The application dimension records the data category, such as unit output data, coal price, weather data, etc. The time dimension records the data generation time.

[0085] Taking the group layer and power generation company layer as an example, the specific self-defined synchronization process is as follows:

[0086] (1) In the ''database registration'', configure the group layer, power generation company layer database connection information; (2) In ''data table registration'', select the group layer and synchronize the data table, select the power generation company layer and synchronize the data table; (3) In ''data field'', select the group layer and select the table that needs to be data transfer, synchronize the source table field to the local through synchronization, and synchronize the company layer table field to the local through the same operation; (4) Configure ''data exchange connection line'', bundle the group layer and the company layer, and configure data extraction sql according to business needs; (5) Enter the scheduling task interface, configure the scheduling task time according to business needs, and bind the configured connection line; (6) Start the task to realize automatic data synchronization. The different subject data mentioned in the application is the data of multi-layer architecture, and the synchronization method mainly performs data transfer operation on the specified database table and field, so that the purpose of customizing and quickly synchronizing different subject data of multi-layer architecture is achieved.

[0087] As shown in Figure 4 To realize the detailed curve display for users, the application further includes a database table synchronization display part after the data timing automatic synchronization is completed. The user configures the graph according to the needs, the system splices the sql statement from the database according to the defined parameters to pull out the required data, and displays the data in the graph according to the user's individual configuration. The system supports setting the refresh frequency, the time range of the value and the individual display.

[0088] Finally, it should be noted that: the above examples are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and not to limit it, the protection scope of the present application is not limited to this. Those skilled in the art should understand that: any person skilled in the art in the technical range disclosed by the present application can modify or easily think of changes to the technical solutions recorded in the foregoing examples, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and all should be covered in the protection scope of the present application.

Claims

1. A method for custom synchronization of streaming TDengine database data for power systems, characterized in that, It includes database table and field synchronization, data timing automatic synchronization, including the following steps: (1) Database table and field synchronization (1) Build a stream TDengine database, a relational database mysql and an in-memory database Redis in the application server; (2) "Database registration" is performed, and the database connection information of the corresponding level is registered and added, which is used for subsequent data synchronization; (3) "Database table registration" is performed, the corresponding level is selected, and the database to be registered is selected. Through the "synchronization" operation, the system automatically connects to the target database, and all table information in the database is pulled, that is, the table of the target database is persisted to the local; (4) Synchronize the database field. The system automatically creates a connection and obtains the target table field information, and persists it to the local. Users configure the table fields that need to be synchronized by setting whether to synchronize, which is used to accurately control the database table fields of the synchronized data; (5) Add "data exchange connection line" for subsequent control of data flow in automatic tasks; (2) Data timing automatic synchronization (6) After the database table and field synchronization is completed, the user configures the scheduling task according to the business requirements and binds the data exchange connection line; (7) The system running on the server automatically starts a thread at the time specified by the user, and creates a connection according to the bound data exchange connection line; (8) According to the user's settings of whether the database table is available / whether it is exchanged, and whether the table field is available, the data in the corresponding task period is extracted and persisted to the receiver database.

2. The method for stream TDengine database data self-defined synchronization for power system according to claim 1, characterized in that: After the data timing automatic synchronization is completed, the database table synchronization is displayed. The user configures the graph according to the requirements, the system splices the sql statement according to the defined parameters to pull the required data from the database, and displays the data in the graph according to the user's individual configuration.

3. The method for stream TDengine database data self-defined synchronization for power system according to claim 1, characterized in that: In the database table and field synchronization process, the interface program calls the Mysql database through the adaptive driver library to obtain the corresponding information of the target database or database table.

4. The method for stream TDengine database data self-defined synchronization for power system according to claim 3, characterized in that: Synchronizing database table information will get the following information and persist it to the local: table_name: database table name table_comment: database table description Synchronizing table field information will get the following corresponding field information and persist it to the local: table_schema: database table name column_name: field name column_comment: field note data_type: field type character_maximum_length: length setting column_key: database table field primary key identifier, default null, PRI primary key component If TDengine database field information is synchronized, the following corresponding field information will be obtained and persisted to the local Field: field name Type: field type Note: field note Length: length setting.

5. The method for stream TDengine database data self-defined synchronization for power system according to claim 1, characterized in that: In the step (8), the tagged data in the streaming TDengine database is three-dimensionally positioned during the data extraction and persistence to the receiver database within the corresponding task cycle: tag_phy: main dimension tag_app: application dimension data_time: time dimension.

6. The method for stream TDengine database data self-customized synchronization for power system according to claim 1, characterized in that: In the data timing automatic synchronization process, the extraction time, extraction cycle, failure retry number can be set according to the requirements, and the data cleaning logic after data extraction can be configured according to the business requirements.

7. The method for stream TDengine database data self-customized synchronization for power system according to claim 1, characterized in that: The hierarchy includes a group layer, a power generation company layer, and a power plant layer.

Images