Method and terminal for dynamically writing change into time sequence database based on feature information

Abstract

The invention discloses a method and a terminal for writing a change dynamic into a time sequence database. The method comprises the following steps: controlling the time sequence database to construct a super table of corresponding data; controlling a flow calculation engine to load table field information in the super table to a memory queue; and when the data uploaded to the time sequence database is subscribed, controlling a flow calculation engine to clean non-table field information of the corresponding data and writing residual effective data information into a corresponding super table in the time sequence database. A time sequence database is selected to replace a traditional database to accept a dynamic change message, the query and write-in performance of data is improved, and the disk space is reduced; according to the method, the super table corresponding to the data is constructed in the time sequence database, and the table field information of the super table is extracted by using the flow calculation engine to serve as a basis for cleaning unknown data, so that effective information is ensured to be written into the time sequence database, and the probability of occurrence of abnormity during writing is reduced.

Description

[0001] This case is a divisional application based on the invention patent filed on August 14, 2023, with application number 202311019221.9 and titled "A method and terminal for dynamically writing changes into a time-series database". Technical Field

[0002] This invention relates to the field of data storage technology. In particular, it relates to a method and terminal for dynamically writing changes into a time-series database. Background Technology

[0003] With the increasing number of new energy vehicles in China, more and more charging stations are being built and operated. The charging piles, batteries, meters and other equipment in the charging stations generate a large amount of time-series data. This time-series data has analytical value for the operation and maintenance of charging stations. Therefore, it is necessary for the charging station equipment to upload the data to the server via the network and store it in the database.

[0004] With the development of the charging station business, the number of charging stations and the equipment within them are constantly increasing. As a result, the number of data fields that the equipment needs to report is also constantly increasing. The maintenance of the database table structure used to store charging station equipment data is becoming increasingly demanding, and the platform is also increasingly busy parsing and processing equipment data. Therefore, a method is needed to dynamically write IoT data and changes to a time-series database for charging stations in order to reduce the workload.

[0005] Traditional databases on the market suffer from drawbacks such as low query and write performance, large data footprint, limited data analysis capabilities, and poor compatibility with time-series data in IoT scenarios. Furthermore, data transmission to the database is prone to anomalies, impacting write performance. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method and terminal for dynamically writing changes into a time-series database, thereby solving the problems of low performance and abnormal data writing in traditional databases.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for dynamically writing changes to a time-series database includes the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data is subscribed to and uploaded to the time series database, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A terminal that dynamically writes changes to a time-series database includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data is subscribed to and uploaded to the time series database, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database.

[0009] The beneficial effects of this invention are: it provides a method and terminal for dynamically writing changes into a time-series database, using a time-series database instead of a traditional database to receive dynamic change messages, thereby improving data query and writing performance and reducing disk space usage; Meanwhile, a super table corresponding to the data is built in the time series database to ensure the consistency of the data structure. The table field information of the super table is extracted by the stream computing engine as the basis for cleaning unknown data, thereby ensuring that valid information is written into the time series database and reducing the probability of anomalies during writing. In addition, by using a stream computing engine to save table field information to an in-memory queue, the number of interactions with the time-series database during cleaning is reduced, thus improving the efficiency of data cleaning. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a method for dynamically writing changes into a time-series database according to a certain embodiment of the present invention; Figure 2 This is a flowchart illustrating a method for dynamically writing changes into a time-series database according to a certain embodiment of the present invention. Figure 3 This is a comparison diagram of the effects of writing changes dynamically into a time-series database according to a certain embodiment of the present invention; Figure 4 This is a schematic diagram of a terminal that dynamically writes changes into a time-series database according to a certain embodiment of the present invention; Label Explanation: 1. A terminal that dynamically writes changes to a time-series database; 2. Storage; 3. Processor; Detailed Implementation To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] Please refer to Figures 1 to 3A method for dynamically writing changes to a time-series database, comprising the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data is subscribed to and uploaded to the time series database, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database.

[0012] As can be seen from the above description, the beneficial effects of the present invention are: it provides a method and terminal for dynamically writing changes into a time-series database, using a time-series database instead of a traditional database to receive dynamic change messages, thereby improving data query and writing performance and reducing disk space. Meanwhile, a super table corresponding to the data is built in the time series database to ensure the consistency of the data structure. The table field information of the super table is extracted by the stream computing engine as the basis for cleaning unknown data, thereby ensuring that valid information is written into the time series database and reducing the probability of anomalies during writing. In addition, by using a stream computing engine to save table field information to an in-memory queue, the number of interactions with the time-series database during cleaning is reduced, thus improving the efficiency of data cleaning.

[0013] Among them, the super table is a data structure in time-series databases that can contain a series of sub-tables and is designed to meet the needs of big data and time-series data.

[0014] Further, step S1 specifically includes: S11. When a new data entry or a field of a new data entry is detected, the backend management system is controlled to construct the corresponding DDL data and send it to the message middleware. S12, The control flow computing engine subscribes to data from the message middleware; S13. When the DDL data is detected to be a schema change statement, the control flow calculation engine constructs an SQL statement based on the DDL data content. S14. Control the time-series database to create or modify the super table corresponding to the data according to the SQL statement.

[0015] As can be seen from the above description, the steps described above provide a specific method for constructing a super table in a time-series database, and its main process is as follows: S11: The control backend management system first checks for any new data entries or fields, such as new device types or device data fields. Once such a change is detected, the system constructs corresponding DDL (Data Definition Language) data based on the new data structure. This data describes how to create or modify the data structure in the database. Then, this DDL data is sent to the message middleware so that other services can subscribe to and use it.

[0016] S12: Stream computing engines (such as Flink) subscribe to data from message middleware. The computing engine can process large amounts of real-time data. It can obtain data from message middleware in real time and process it.

[0017] S13: When the stream processing engine receives DDL data and identifies it as a schema change statement (a descriptive statement about the structure and attributes of the database and tables), the stream processing engine constructs a corresponding SQL statement (a standard programming language for manipulating and querying the database) based on the content of the DDL data. This SQL statement can be used to create a new table in the database or add new fields to an existing table.

[0018] S14: The time-series database creates new data tables or adds new fields to existing data tables based on the SQL statements generated by the stream computing engine. In one embodiment of the present invention, when applied to an Internet of Things (IoT) network, new device types and device data fields can be effectively stored and queried.

[0019] In other words, the entire process can dynamically adapt to new device types and device data fields without manual intervention, greatly improving the system's flexibility and adaptability. At the same time, the entire process transmits data through a message middleware, making it easy to add new consumer services and improving the system's scalability.

[0020] Further, step S2 specifically includes: S21. The control flow calculation engine periodically obtains the table field information of the super table; S22. The control flow calculation engine loads the table field information into the memory queue.

[0021] As can be seen from the above description, processes S21 and S22 mainly involve obtaining table field information from the super table and loading this information into the memory queue. When the super table is continuously added or entries are added to the existing super table in the time series database, the stream computing engine controls the stream computing engine to periodically obtain table field information and cache it into the memory queue in order to avoid the original table field information in the memory queue becoming outdated and causing the valid information to be mistakenly cleaned.

[0022] Further, step S3 specifically includes: S31. When dynamic data to be uploaded to the time-series database is detected, the back-end management system is controlled to construct a string with corresponding feature information tags and send it to the message middleware. S32. When a new string is added to the message middleware, the control flow calculation engine reads the new string, cleans the non-table field information according to the feature information label, and caches the remaining valid information to the memory queue. S33. When the preset time is reached or the message accumulation in the memory queue exceeds the preset threshold, the remaining valid information is written into the corresponding super table in the time-series database.

[0023] As described above, steps S31, S32, and S33 mainly concern the data uploading, data cleaning, and caching and storage of valid information for the time-series database. The specific steps are explained below: S31: When dynamic data that needs to be uploaded to the time series database is detected, the backend management system will first construct a string with characteristic information tags and send this string to the message middleware.

[0024] In one embodiment of the present invention, this method is applied in an Internet of Things (IoT) network. The process uses a JSON string containing various device information, such as device ID, timestamp, and specific business data (e.g., voltage, current). Device ID and charging station ID are selected as feature information. Based on this feature information, a sub-table is constructed from the existing super-table. Information about the corresponding device is stored in the sub-table according to the feature information. Later, when querying data, queries can be performed through the sub-table instead of from the full dataset, improving query efficiency. Simultaneously, data on devices of the same type uses the same super-table, ensuring structural consistency. Furthermore, when comparing the same type of devices across different charging stations, data analysis can be performed using the super-table.

[0025] S32: The stream processing engine subscribes to data in the message middleware. When it receives data that needs to be uploaded to the time-series database, the stream processing engine first performs data cleaning, i.e., removing non-table field information. This step is mainly to ensure that the data structure uploaded to the database matches the database table structure. The cleaned valid information (i.e., data containing only table field information) is cached in an in-memory queue.

[0026] S33: The stream processing engine triggers data write operations based on a preset time or the message backlog in the memory queue. When the preset time is reached or the message backlog exceeds a preset threshold, valid information cached in the memory queue is written to the corresponding super table in the time-series database. Specifically, the preset time is 1 second, and the preset threshold is 500 messages. Triggering data write operations by setting preset time and thresholds allows for flexible adjustment of the data write frequency and batch size according to the actual system load, improving system flexibility and scalability.

[0027] Furthermore, the time series database is the TDEngine time series database.

[0028] As described above, the selected time-series database is TDEngine. In one embodiment of this invention, in terms of business compatibility, TDEngine, being a time-series database, matches the time-series data of devices in the Internet of Things (IoT) network and provides downsampling, automatic filling of missing values, and common functions (average, maximum, minimum, latest value, etc.). Regarding disk space usage, TDEngine's data compression ratio is at least 5 times, resulting in less disk space usage compared to traditional databases. In terms of query performance, TDEngine automatically caches the latest data for each charging station and each device, greatly improving query performance. Regarding write performance, stress testing analysis shows that under the same conditions, TDEngine's write performance is superior to traditional databases and other common time-series databases. Figure 3 As shown.

[0029] A terminal 1 that dynamically writes changes to a time-series database includes a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor. The processor 3 executes the computer program to complete the above steps.

[0030] This invention provides a method and terminal for dynamically writing changes into a time-series database, mainly applied to dynamically writing IoT network data and schema changes into a time-series database. The following is a description with reference to specific embodiments: Please refer to Figures 1 to 3 Embodiment 1 of the present invention is: a method for dynamically writing changes into a time-series database, comprising the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data is subscribed to and uploaded to the time series database, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database.

[0031] As can be seen from the above description, the beneficial effects of the present invention are: it provides a method and terminal for dynamically writing changes into a time-series database, using a time-series database instead of a traditional database to receive dynamic change messages, thereby improving data query and writing performance and reducing disk space. Meanwhile, a super table corresponding to the data is built in the time series database to ensure the consistency of the data structure. The table field information of the super table is extracted by the stream computing engine as the basis for cleaning unknown data, thereby ensuring that valid information is written into the time series database and reducing the probability of anomalies during writing. In addition, by using a stream computing engine to save table field information to an in-memory queue, the number of interactions with the time-series database during cleaning is reduced, thus improving the efficiency of data cleaning.

[0032] Please refer to Figures 1 to 3 The second embodiment of the present invention is as follows: Based on the first embodiment, step S1 specifically includes: S11: The control backend management system first checks for any new data entries or fields, such as new device types or device data fields. Once such a change is detected, the system constructs corresponding DDL (Data Definition Language) data based on the new data structure. This data describes how to create or modify the data structure in the database. Then, this DDL data is sent to the message middleware so that other services can subscribe to and use it.

[0033] S12: Stream computing engines (such as Flink) subscribe to data from message middleware. The computing engine can process large amounts of real-time data. It can obtain data from message middleware in real time and process it.

[0034] S13: When the stream processing engine receives DDL data and identifies it as a schema change statement (a descriptive statement about the structure and attributes of the database and tables), the stream processing engine constructs a corresponding SQL statement (a standard programming language for manipulating and querying the database) based on the content of the DDL data. This SQL statement can be used to create a new table in the database or add new fields to an existing table.

[0035] S14: The time-series database creates new data tables or adds new fields to existing data tables based on the SQL statements generated by the stream computing engine. In one embodiment of the present invention, when applied to an Internet of Things (IoT) network, new device types and device data fields can be effectively stored and queried.

[0036] In other words, the entire process can dynamically adapt to new device types and device data fields without manual intervention, greatly improving the system's flexibility and adaptability. At the same time, the entire process transmits data through a message middleware, making it easy to add new consumer services and improving the system's scalability.

[0037] Step S2 specifically involves: S21. The control flow calculation engine periodically obtains the table field information of the super table; S22. The control flow calculation engine loads the table field information into the memory queue.

[0038] In other words, processes S21 and S22 mainly involve obtaining table field information from the super table and loading this information into the memory queue. When the super table is continuously added or entries are added to the existing super table in the time series database, the stream computing engine controls the stream computing engine to periodically obtain table field information and cache it into the memory queue in order to avoid the original table field information in the memory queue becoming outdated and causing the valid information to be mistakenly cleaned.

[0039] Specifically, examples are as follows: The data for adding a device and the data field for adding a device in the backend management system web interface is as follows: The statement `{"ddl":true,"source":{"table":"BMS","type":"CREATE"},"tableChange":{"num":"INT"}}`, where `ddl` is true, indicates that this is a schema change statement, requiring a database table structure change. `table` represents the specific device table name, `type` indicates whether this is a new device table or a newly added table field, and `tableChange` indicates the fields involved and their types.

[0040] Example of SQL statements used to create supertables in a time-series database: CREATE STABLE IOT.BMS(`ts` timestamp,`cur` INT,`vol` INT); Example of an SQL statement for adding a new field in a super table of a time-series database: ALTER STABLE IOT.BMS ADD COLUMN `cur` INT; Please refer to Figures 1 to 3 The third embodiment of the present invention is as follows: Based on the first embodiment, step S3 specifically includes: S31: When dynamic data that needs to be uploaded to the time series database is detected, the backend management system will first construct a string with characteristic information tags and send this string to the message middleware.

[0041] In one embodiment of the present invention, this method is applied in an Internet of Things (IoT) network. The process uses a JSON string containing various device information, such as device ID, timestamp, and specific business data (e.g., voltage, current). Device ID and charging station ID are selected as feature information. Based on this feature information, a sub-table is constructed from the existing super-table. Information about the corresponding device is stored in the sub-table according to the feature information. Later, when querying data, queries can be performed through the sub-table instead of from the full dataset, improving query efficiency. Simultaneously, data on devices of the same type uses the same super-table, ensuring structural consistency. Furthermore, when comparing the same type of devices across different charging stations, data analysis can be performed using the super-table.

[0042] Specifically, an example of a JSON string is as follows: {"did":"0000000001","mid":"BMS-1","ts":1685510005004,"vol":1,"cur":2}; The fields `did`, `mid`, and `ts` are feature information tags that must be included in every data entry. `did` represents the charging station ID, `mid` represents the device ID, and `ts` represents the data timestamp. Other fields are specific device business fields, such as voltage and current.

[0043] S32: The stream processing engine subscribes to data in the message middleware. When it receives data that needs to be uploaded to the time-series database, the stream processing engine first performs data cleaning, i.e., removing non-table field information. This step is mainly to ensure that the data structure uploaded to the database matches the database table structure. The cleaned valid information (i.e., data containing only table field information) is cached in an in-memory queue.

[0044] S33: The stream processing engine triggers data write operations based on a preset time or the message backlog in the memory queue. When the preset time is reached or the message backlog exceeds a preset threshold, valid information cached in the memory queue is written to the corresponding super table in the time-series database. Specifically, the preset time is 1 second, and the preset threshold is 500 messages. Triggering data write operations by setting preset time and thresholds allows for flexible adjustment of the data write frequency and batch size according to the actual system load, improving system flexibility and scalability.

[0045] Specifically, the process of writing to the time series database is as follows: Example data (i.e., newly added data): {"did":"0000000001","mid":"BMS-1","ts":1685510005004,"vol":1,"cur":2}; Example SQL statement constructed: INSERT INTO `0000000001_BMS-1` USING BMS (`mid`,`did`) TAGS ('BMS-1','0000000001') (`vol`,`cur`,`ts`) VALUES (1,2,1685510005004); Here, `0000000001_BMS-1` indicates that data is written to the device table with mid of BMS-1 ​​in the charging station with dot 0000000001; USING BMS indicates the use of a super table structure for devices such as BMS. The table structure has been automatically created in the time series database in step S1. (`mid`,`did`) TAGS ('BMS-1','0000000001') are the feature information labels for the current data in the time series database; (`vol`,`cur`,`ts`) VALUES (1,2,1685510005004) is the business information of the current data, and ts is the data timestamp.

[0046] Please refer to Figure 3 The fourth embodiment of the present invention is as follows: In any of the methods in embodiments one to three, the time series database is the TDEngine time series database.

[0047] Please refer to Figure 4 Embodiment 5 of the present invention is: a terminal for dynamically writing changes into a time-series database, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it completes any one of the methods for dynamically writing changes into a time-series database as described in Embodiments 1 to 4.

[0048] In summary, the present invention provides a method and terminal for dynamically writing changes to a time-series database, which uses a time-series database instead of a traditional database to receive dynamic change messages, thereby improving data query and writing performance and reducing disk space usage. Meanwhile, a super table corresponding to the data is built in the time series database to ensure the consistency of the data structure. The table field information of the super table is extracted by the stream computing engine as the basis for cleaning unknown data, thereby ensuring that valid information is written into the time series database and reducing the probability of anomalies during writing. In addition, by using a stream computing engine to save table field information to an in-memory queue, the number of interactions with the time-series database during cleaning is reduced, thus improving the efficiency of data cleaning.

[0049] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for dynamically writing changes into a time-series database based on feature information, characterized in that: Including the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data uploaded to the time series database is subscribed to, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database. Step S3 specifically involves: S31. When dynamic data to be uploaded to the time-series database is detected, the backend management system is controlled to construct a string with corresponding feature information tags and send it to the message middleware; the feature information includes device ID and charging station ID, and devices of the same type use the same super table; S32. When a new string is added to the message middleware, the control flow calculation engine reads the new string, cleans the non-table field information according to the feature information label, and caches the remaining valid data information to the memory queue. S33. When the preset time is reached or the message accumulation in the memory queue exceeds the preset threshold, the remaining valid data information is written into the corresponding super table in the time-series database.

2. The method for dynamically writing changes into a time-series database based on feature information according to claim 1, characterized in that: Step S1 specifically involves: S11. When a new data entry or a field of a new data entry is detected, the backend management system is controlled to construct the corresponding DDL data and send it to the message middleware. S12, The control flow computing engine subscribes to data from the message middleware; S13. When the DDL data is detected to be a schema change statement, the control flow calculation engine constructs an SQL statement based on the DDL data content. S14. Control the time-series database to create or modify the super table corresponding to the data according to the SQL statement.

3. The method for dynamically writing changes into a time-series database based on feature information according to claim 2, characterized in that: Step S2 specifically involves: S21. The control flow calculation engine periodically obtains the table field information of the super table; S22. The control flow calculation engine loads the table field information into the memory queue.

4. The method for dynamically writing changes into a time-series database based on feature information according to claim 1, characterized in that: The time series database is the TDEngine time series database.

5. A terminal that dynamically writes changes into a time-series database based on feature information, characterized in that: Includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps: S1. Control the construction of a super table for the corresponding data in the time series database; S2. The control flow computing engine loads the table field information of the super table into the memory queue; S3. When data uploaded to the time series database is subscribed to, the control flow computing engine cleans the non-table field information of the corresponding data and writes the remaining valid data information into the corresponding super table in the time series database. Step S3 specifically involves: S31. When dynamic data to be uploaded to the time-series database is detected, the backend management system is controlled to construct a string with corresponding feature information tags and send it to the message middleware; the feature information includes device ID and charging station ID, and devices of the same type use the same super table; S32. When a new string is added to the message middleware, the control flow calculation engine reads the new string, cleans the non-table field information according to the feature information label, and caches the remaining valid data information to the memory queue. S33. When the preset time is reached or the message accumulation in the memory queue exceeds the preset threshold, the remaining valid data information is written into the corresponding super table in the time-series database.

6. A terminal that dynamically writes changes into a time-series database based on feature information according to claim 5, characterized in that: Step S1 specifically involves: S11. When a new data entry or a field of a new data entry is detected, the backend management system is controlled to construct the corresponding DDL data and send it to the message middleware. S12, The control flow computing engine subscribes to data from the message middleware; S13. When the DDL data is detected to be a schema change statement, the control flow calculation engine constructs an SQL statement based on the DDL data content. S14. Control the time-series database to create or modify the super table corresponding to the data according to the SQL statement.

7. A terminal that dynamically writes changes into a time-series database based on feature information according to claim 6, characterized in that: Step S2 specifically involves: S21. The control flow calculation engine periodically obtains the table field information of the super table; S22. The control flow calculation engine loads the table field information into the memory queue.

8. A terminal that dynamically writes changes into a time-series database based on feature information according to claim 5, characterized in that: The time series database is the TDEngine time series database.

Images