A method and system for auditing disordered data in a time series database

Abstract

The present invention discloses a method and system for auditing disordered data in a time series database. The method uses the time primary key of the time series data as a basis. When the database performs a storage operation, the method determines whether disordered behavior is generated by monitoring the time sequence of the current data and the data being inserted. If disordered behavior is not generated, it is skipped. If disordered behavior is generated, the disordered data information is recorded and an audit file is formed. The user can quickly find the location of the disordered data according to the audit file information. The present invention can quickly locate the disordered data, and then analyze the cause of the disordered data, so as to better guarantee the database service.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method and system for auditing disordered data in a time series database. Background Art

[0002] In the Internet of Things (IoT) environment, sensor devices are widely used in various scenarios, such as intelligent transportation systems, environmental monitoring stations, and smart homes. These devices continuously generate a large amount of data, providing a rich data source for various applications. However, with the increasing number of sensor devices and the sharp increase in data volume, these data that should have been strictly arranged in chronological order often appear out of order due to sensor failures, network delays, and uncertainties in data transmission. The emergence of out-of-order data has brought many impacts. Ideally, data is written one by one in chronological order, but the existence of out-of-order data breaks this rule, causing the database to require additional logic to process these out-of-order data, thereby affecting the overall write performance. Out-of-order data also has a negative impact on the effect of data compression algorithms. The emergence of out-of-order data makes it difficult for compression algorithms to achieve optimal results because the algorithm needs to process more out-of-order data blocks, increasing the consumption of storage space.

[0003] At present, the only thing we can do is to find that the query speed is slow during the query and realize that there is disorder, then re-sort and export the data, rebuild the database to fill the data, or export the data through stream and calculation solutions, sort the whole data and then import it. However, it is basically a manual perception of the query, and then do the overall backup operation, which is inefficient and slow. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for auditing disordered data in a time series database, aiming to solve the technical problem of how to quickly locate disordered data, and then analyze the causes of disordered data, so as to better protect database services.

[0005] The objective of the present invention is to adopt the following technical scheme to achieve: a method for auditing disordered data in a time series database, which uses the time primary key of the time series data as a basis. When performing storage operations in the database, it determines whether disordered behavior occurs by monitoring the time sequence of the current data and the data being inserted. If disordered behavior does not occur, it is skipped. If disordered behavior occurs, the disordered data information is recorded and an audit file is formed. The user can quickly find the location of the disordered data based on the audit file information.

[0006] Furthermore, the method for auditing disordered data in a time series database includes the following steps:

[0007] S1: Add configuration items and determine whether to enable out-of-order auditing;

[0008] S2: Determine whether disordered behavior occurs, record data information, and write audit files in batches;

[0009] S3: Check the audit file and perform follow-up processing.

[0010] Furthermore, step S1 is specifically as follows: adding a configuration item DisordAudit to the CFG file of the database, and determining the value of the configuration item DisordAudit. If the configuration item DisordAudit is 1, the out-of-order audit function is enabled; if the configuration item DisordAudit is 0, the out-of-order audit function is disabled.

[0011] Furthermore, step S2 includes the following sub-steps:

[0012] S21: Compare the current time of the inserted data with the time of the previous data. If the time of the current data is less than the time of the previous data, it indicates that disorder occurs.

[0013] S22: Record the information of current data and put it into cache;

[0014] S23: When the data in the cache reaches a preset value, the data information is written into the audit file in batches.

[0015] Furthermore, the data information includes one or more of timestamp information of current data, table information, user information and current operating system time information.

[0016] Furthermore, step S2 also includes the following sub-steps:

[0017] S24: The audit file decides to segment the disordered data in the database according to the data file configuration parameter DURATION of the current database.

[0018] Furthermore, step S3 includes the following sub-steps:

[0019] S31: Query the database based on the content recorded in the audit file;

[0020] S32: Based on the query result, further analyze the cause of the disorder so as to perform optimization.

[0021] Furthermore, step S3 also includes the following sub-steps:

[0022] S33: Determine the interval of the disordered data through the record content in the audit file, generate the corresponding query SQL statement, query the disordered data in the database, and then export the disordered data involved in sequence to a file for query.

[0023] Furthermore, step S3 also includes the following sub-steps:

[0024] S34: Delete the data in the disordered data interval in the current database, and import the data in the exported data file in the order of filling historical data to avoid query performance loss caused by cross-order data insertion.

[0025] A disordered data auditing system in a time series database is used to implement the above-mentioned disordered data auditing method in a time series database, including a monitoring module, a disordered data information recording module and a query module, wherein:

[0026] The monitoring module uses the time primary key of the time series data as the basis. When performing storage operations in the database, it monitors the time sequence of the current data and the data being inserted to determine whether disorder occurs.

[0027] The disordered data information recording module records the data that generates disordered behavior and forms an audit file;

[0028] The query module quickly finds the location of the disordered data based on the audit file information.

[0029] The beneficial effects of the present invention are:

[0030] Efficiency: Once disorder occurs, traditional methods cannot locate the disorder interval (due to the built-in sorting algorithm of the time series database system, the disordered data will be displayed in order when it is queried, but the query time will be very long). By adding this audit method, the audit file can be queried at any time to locate the disordered data interval and device number, and the problem can be quickly discovered.

[0031] Import volume: This method can be used to specify the disordered interval, and only the disordered data can be exported sequentially, deleted, and then imported sequentially. There is no need to export, sort, delete, and import all the data, which reduces the amount of imported data and improves efficiency.

[0032] Certainty: Users only need to check the audit file to find out whether disordered behavior has occurred. There is no need to manually rely on query time to fuzzily perceive whether disorder has occurred, which improves the user experience.

[0033] Data integrity: further ensures data integrity.

[0034] Applicability: It is extremely applicable in time series scenarios where there is a large amount of sensor data and a large amount of disordered data is generated due to network failures, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0036] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0038] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0039] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0040] See also Figure 1 A method for auditing disordered data in a time series database mainly uses the time primary key of the time series data as the basis. When the database is entered, it monitors the time sequence of the current data and the data being inserted to determine whether disordered behavior occurs. If not, it will be skipped. Once it occurs, a disordered data information record will be generated. The record will contain: current time, disordered data time, user name, subtable name, super table name and ordinary table name, etc., and the above data information is combined into an audit file. Users can quickly find the location of disordered data based on the audit file information, and make special processing on the interval of disordered data, or analyze the cause of disordered data.

[0041] Specifically, the method for auditing disordered data in the time series database includes the following steps:

[0042] Step 1: Add configuration items. Since the development of the audit function will cause certain performance loss, this function may not be used in some scenarios. Therefore, a configuration item needs to be added to the database CFG file: DisordAudit. When DisordAudit is 1, the data audit function is turned on; when DisordAudit is 0, the data audit function is turned off. In this embodiment, DisordAudit defaults to 0, that is, the data audit function is turned off by default.

[0043] Step 2: Perform out-of-order data audit. Check whether the out-of-order audit function is enabled according to the database configuration. If enabled, open the audit behavior code in the database insertion path. The logic of this part of the code mainly compares the current time of the inserted data with the time of the previous data. If the time of the current data is less than the previous data, it means that out-of-order behavior has occurred.

[0044] Furthermore, if disorder occurs, the timestamp information, table information (super table, sub-table, ordinary table), user information, current operating system time and other information of the current data are recorded and put into the cache. When the data in the cache reaches a certain amount (it can be set according to the actual situation, and the present invention is not limited in this regard), it is written into the audit file in batches. The audit file is determined by the data file configuration parameter DURATION (DURATION: the time span of data file storage data) of the current database, and is segmented according to the time set by the configuration parameter. Segmentation is very helpful for the subsequent analysis of the causes of disorder and disordered data processing, avoiding the concentration of data in one file and the difficulty in processing large data.

[0045] Step 3: Check the audit file results and take corresponding actions.

[0046] Specifically, by checking whether the content in the audit file is correct and querying the content of the audit record in the database, the user can use the audit file to find the corresponding device (car, sensor, etc.) for the table name that produces the disorder, and then further analyze the cause of the disorder for optimization.

[0047] Alternatively, you can determine the interval of disordered data through the records in the audit file, generate the corresponding query SQL statement, query the disordered data from the database, and then export the disordered data involved in a sequential manner. After confirming that the export is correct, delete the data in the disordered data interval in the current database, and import the data in the exported data file in the order of filling historical data. This can avoid the query performance loss caused by the insertion of cross-ordered data and further improve data query performance.

[0048] The present invention also provides a disordered data auditing system in a time series database, which is used to implement the above-mentioned disordered data auditing method in a time series database. The system includes a monitoring module, a disordered data information recording module and a query module, wherein:

[0049] The monitoring module uses the time primary key of the time series data as the basis. When performing storage operations in the database, it monitors the time sequence of the current data and the data being inserted to determine whether disorder occurs.

[0050] The disordered data information recording module records the data that generates disordered behavior and forms an audit file;

[0051] The query module quickly finds the location of the disordered data based on the audit file information.

[0052] Based on the above embodiments, the present invention has at least the following technical effects:

[0053] Once the traditional method generates disorder, it is impossible to locate the disorder interval (due to the built-in sorting algorithm of the time series database system, the disordered data will be displayed in order when querying, but the query time will be very long). By adding this audit method, you can query the audit file at any time, locate the disordered data interval and device number, and quickly find the problem. This method can specify the disorder interval, only export the disordered data in sequence, delete it, and then import it in sequence. There is no need to export all the data, sort it, delete all of it, and then import all of it, which reduces the amount of imported data and improves efficiency. Users only need to check the audit file to find out whether disordered behavior has occurred. There is no need to manually rely on the query time to fuzzily perceive whether disorder has occurred, which improves the user experience. It further guarantees the integrity of the data and is extremely applicable to time series scenarios where there is a lot of sensor data and a large amount of disordered data is generated due to network failures.

[0054] It should be noted that, for the above embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification belong to preferred embodiments, and the actions involved are not necessarily required by the present application.

[0055] The above embodiments describe the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the changes and modifications made by those skilled in the art shall be within the scope of protection of the appended claims of the present invention without departing from the spirit and scope of the present invention.

Claims

1. A method for auditing disordered data in a time series database, characterized in that: Based on the time primary key of time series data, when performing storage operations in the database, the time sequence of the current data and the data being inserted is monitored to determine whether disordered behavior occurs. If disordered behavior does not occur, it is skipped. If disordered behavior occurs, disordered data information records will be generated and an audit file will be formed. Users can quickly find the location of disordered data based on the audit file information; The steps include: S1: Add a configuration item and determine whether to enable out-of-order auditing; Step S1 is specifically: add a configuration item DisordAudit in the CFG file of the database, and determine the value of the configuration item DisordAudit. If the configuration item DisordAudit is 1, the out-of-order auditing function is enabled; if the configuration item DisordAudit is 0, the out-of-order auditing function is disabled; S2: Determine whether disordered behavior occurs, record data information, and write batches into audit files; Step S2 includes the following sub-steps: S21: Compare the current time of the inserted data with the time of the previous data. If the time of the current data is less than the time of the previous data, it indicates that disorder occurs. S22: Record the information of current data and put it into cache; S23: When the data in the cache reaches a preset value, the data information is written into the audit file in batches; S24: The audit file decides to segment the disordered data in the database according to the data file configuration parameter DURATION of the current database; S3: Check the audit file and do subsequent processing; Step S3 includes the following sub-steps: S31: Query the database based on the content recorded in the audit file; S32: further analyzing the cause of disorder according to the query result so as to perform optimization; S33: Determine the interval of the disordered data through the record content in the audit file, generate the corresponding query SQL statement, query the disordered data in the database, and then export the disordered data involved in sequence to a file for query; S34: Delete the data in the disordered data interval in the current database, and import the data in the exported data file in the order of filling historical data to avoid query performance loss caused by cross-order data insertion.

2. The method for auditing disordered data in a time series database according to claim 1, characterized in that: The data information includes one or more of timestamp information of current data, table information, user information and current operating system time information.

3. A disordered data auditing system in a time series database, used to implement a disordered data auditing method in a time series database as described in any one of claims 1 to 2, characterized in that: It includes a monitoring module, a disordered data information recording module and a query module, among which: The monitoring module uses the time primary key of the time series data as the basis. When performing storage operations in the database, it monitors the time sequence of the current data and the data being inserted to determine whether disorder occurs. The disordered data information recording module records the data that generates disordered behavior and forms an audit file; The query module quickly finds the location of the disordered data based on the audit file information.

Images