Time series database information storage and management system

Abstract

The present invention discloses a time series database information storage and management system, comprising: a data receiving module for receiving time series data from different data sources and converting it into a unified format; a data storage module for storing the converted time series data in a database and sorting it according to time series; a data query module for retrieving the corresponding time series data from the database according to the user's query request and presenting it to the user in the form of a chart or table; a data management module for performing addition, deletion, modification, and query operations on the time series data in the database; and a data analysis and processing module for performing pre-processing operations such as cleaning, deduplication, and missing value processing on the received time series data. Through its modular design and advanced data processing technology, the present invention provides enterprises with a powerful, flexible, and efficient tool for managing, analyzing, and predicting time series data, thereby supporting better business decision-making and operational optimization.

Description

Technical Field

[0001] The present invention belongs to the field of information storage, and more specifically, relates to a time series database information storage and management system. Background Art

[0002] Time series data is data recorded in chronological order, typically containing timestamps, metrics, and associated tags or metadata. This data is generated in large quantities in areas such as IoT devices, financial markets, and environmental monitoring. Time series data has several notable characteristics: each data point is timestamped, which is crucial for data analysis and computation; this data is typically structured, with predefined data types or fixed lengths;

[0003] Existing technologies often experience performance bottlenecks when faced with large-scale data applications, such as cloud computing, big data, and artificial intelligence, making it difficult to effectively scale to meet growing data demands. Many enterprises are forced to frequently replace storage devices due to system non-scalability, increasing costs and operational complexity.

[0004] At the same time, internet-connected storage devices are vulnerable to hacker attacks, leading to data leaks. Common storage methods such as mobile phones, personal computers, servers, and cloud storage all carry security risks. The information security industry primarily focuses on protecting information transmission and attack defense, while neglecting storage security. For example, issues such as the authenticity, confidentiality, and non-repudiation of stored information are rarely addressed, leading to significant security risks in the information storage sector.

[0005] In addition, due to the different information needs of different entities, various types of data are sealed in different systems, forming data islands, which is not conducive to data sharing and openness. Different units choose different storage formats and database designs, making data management and integration extremely complicated and increasing management costs. Therefore, we propose a time series database information storage and management system. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technology. The proposed time series database information storage and management system, through its modular design and advanced data processing technology, provides enterprises with a powerful, flexible and efficient tool for managing, analyzing and predicting time series data, thereby supporting better business decision-making and operational optimization.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] Time series database information storage and management system, including:

[0009] The data receiving module is used to receive time series data from different data sources and convert it into a unified format;

[0010] The data storage module is used to store the converted time series data in the database and sort them according to the time series;

[0011] The data query module is used to retrieve the corresponding time series data from the database according to the user's query request and present it to the user in the form of charts or tables;

[0012] The data management module is used to perform operations such as adding, deleting, modifying, and querying time series data in the database, as well as backup and recovery operations;

[0013] The data analysis and processing module is used to perform preprocessing operations such as cleaning, deduplication and missing value processing on the received time series data, as well as analytical operations such as statistical analysis, trend prediction and anomaly detection on the stored time series data.

[0014] Preferably, the system further comprises:

[0015] User authority management module, used to control and manage user operation permissions;

[0016] The system monitoring module is used to monitor and alarm the system's operating status in real time.

[0017] Preferably, the data receiving module includes:

[0018] Multi-protocol support submodule, which receives data through multiple communication protocols and can parse and unify these data streams from different protocols;

[0019] Data format parser, used to load different parsing plug-ins to parse the received data;

[0020] A data transformation and mapping engine that maps raw data into a database schema, including handling times and values ​​for time series data, as well as mapping any metadata tags;

[0021] The data quality monitoring unit performs real-time quality checks on data entering the system. The checks include but are not limited to range checks, integrity checks, and accuracy tests to ensure that only high-quality data is accepted and stored.

[0022] Preferably, the data storage module adopts a distributed storage architecture, stores data in multiple nodes through sharding technology to achieve horizontal expansion, and adopts an optimized write strategy to achieve higher write performance through memory writing and subsequent disk sequential writing, avoiding the overhead of random writing;

[0023] The data storage module should have a built-in data compression and downsampling unit, utilize the regularity of time series data, adopt differential compression, dictionary compression and other algorithms to reduce the data volume, and automatically downsample to merge high-precision data into low-precision data to further save storage space.

[0024] Preferably, the specific workflow of the data query module is:

[0025] 1) Users construct queries through the query interface, specifying time range, dimensions, and aggregation function conditions;

[0026] 2) The query module parses the query request submitted by the user and passes it to the query optimizer to generate an efficient execution plan;

[0027] 3) Based on the optimized execution plan, the query engine accesses relevant data indexes and storage to execute the query operation;

[0028] 4) Return the query result set to the user, supporting paging and on-demand loading to improve the performance of large data processing;

[0029] 5) Present the data in charts or other formats.

[0030] Preferably, the data analysis and processing module establishes a prediction model to analyze and process the data, and the specific process is as follows:

[0031] 1) Identify the business problems or analysis objectives that need to be solved and develop a data analysis plan to address the problems;

[0032] 2) Determine the scope and type of data required based on analysis requirements and obtain data from designated sources;

[0033] 3) Build an applicable prediction model based on the analysis purpose and train the model;

[0034] 4) Present the analysis results through reports, charts, and visual interfaces, adjust the analysis models and processes based on business feedback, continuously monitor the analysis results, and perform iterative optimization.

[0035] Preferably, the prediction model is selected based on the importance and relevance of features to reduce model complexity and avoid overfitting, and the partial derivative of the loss function with respect to the weight is used to guide the update of the weight, which is expressed as:

[0036]

[0037] Where η is the learning rate, is the partial derivative of the loss function with respect to the current weight, E is the loss function, and w is any weight in the network;

[0038] The prediction model is trained using the back propagation algorithm. First, the error of the output layer is calculated, and then it is back propagated layer by layer to the input layer. For each layer, the algorithm calculates the contribution of each neuron to the total error and adjusts the weight accordingly. If the input of neuron j is net j , the output is o j , the error is δ j , with weight w ji , the expression is:

[0039] Δw ji =η·δ j o i ;

[0040] Where, δ j It is calculated based on the type of neuron and the activation function.

[0041] Preferably, the prediction model adopts a nonlinear function during training so that the model can capture complex data features, and the expression is:

[0042]

[0043] Where σ(x) represents the output of the Sigmoid function, which is a real number with a range of 0 to 1, x is the real value input to the Sigmoid function, e is the base of the natural logarithm, -x represents the opposite of the input value, and 1+e -x It is the denominator that ensures that the result of the fraction is mapped between 0 and 1.

[0044] Preferably, in the prediction model, a loss function is used to measure the difference between the predicted value and the actual value of the neural network, and the expression is:

[0045]

[0046] Where M is the number of categories, y c is the true label, is the predicted output, log is the logarithmic function with the natural constant e as the base;

[0047] The calculation process of the above formula is:

[0048] For each category c, calculate the true label yc and the predicted probability The product of

[0049] Take the logarithm of this product

[0050] Adding up all these values ​​gives us the final loss E.

[0051] The time series database information storage and management system provided by the present invention has the following advantages over the prior art:

[0052] Efficient data reception and conversion: By supporting a multi-protocol data reception module, the system can process time series data from different data sources and convert it into a unified format. This improves data availability and consistency, providing a solid foundation for subsequent data analysis and management.

[0053] Optimized data storage: Using a distributed storage architecture and sharding technology, the system can achieve horizontal scalability and improve data writing performance. Built-in data compression and downsampling units further save storage space while ensuring data integrity and accuracy.

[0054] Flexible data query and display: The data query module allows users to retrieve data based on different time ranges, dimensions, and aggregation function conditions. Query results can be displayed in charts or tables, supporting paging and on-demand loading, improving the performance of large data processing.

[0055] Comprehensive data management: The data management module not only supports the addition, deletion, modification, and query of time series data, but also includes backup and recovery functions to ensure data security and reliability.

[0056] In-depth data analysis and processing: The data analysis and processing module can perform pre-processing operations such as cleaning, deduplication, and missing value processing on time series data, and perform analytical operations such as statistical analysis, trend prediction, and anomaly detection. This helps to extract valuable information from the data and support decision-making.

[0057] User rights management and system monitoring: The user rights management module ensures the security of the system and prevents unauthorized access and operation; the system monitoring module monitors the operating status of the system in real time, promptly discovers and alerts potential problems;

[0058] Intelligent prediction model: By building and training prediction models, the system can conduct in-depth analysis of time series data, provide business insights, and predict future trends. Model training uses backpropagation algorithms and nonlinear functions to capture complex data features and improve prediction accuracy.

[0059] Application of loss function: The prediction model uses the loss function to measure the difference between the predicted value of the neural network and the actual value. By optimizing the loss function, the model weight is adjusted to improve the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is the architecture diagram of the time series database information storage and management system of the present invention;

[0061] Figure 2This is a specific workflow diagram of the data query module in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] The present invention provides a time series database information storage and management system. Through its modular design and advanced data processing technology, it provides enterprises with a powerful, flexible and efficient tool for managing, analyzing and predicting time series data, thereby supporting better business decision-making and operational optimization.

[0064] System includes:

[0065] The data receiving module is used to receive time series data from different data sources and convert it into a unified format; the data receiving module includes:

[0066] Multi-protocol support submodule, which receives data through multiple communication protocols and can parse and unify these data streams from different protocols;

[0067] Data format parser, used to load different parsing plug-ins to parse the received data;

[0068] A data transformation and mapping engine that maps raw data into a database schema, including handling times and values ​​for time series data, as well as mapping any metadata tags;

[0069] The data quality monitoring unit performs real-time quality checks on data entering the system, including but not limited to range checks, completeness checks, and accuracy tests, to ensure that only high-quality data is accepted and stored;

[0070] The data storage module is used to store the converted time series data in the database and sort it according to the time series. The data storage module adopts a distributed storage architecture and stores data on multiple nodes through sharding technology to achieve horizontal expansion. It also adopts an optimized write strategy to achieve higher write performance through memory write and subsequent disk sequential write, avoiding the overhead of random write.

[0071] The data storage module should have a built-in data compression and downsampling unit, utilize the regularity of time series data, adopt differential compression, dictionary compression and other algorithms to reduce the data volume, and automatically downsample to merge high-precision data into low-precision data to further save storage space.

[0072] The data query module is used to retrieve the corresponding time series data from the database according to the user's query request and present it to the user in the form of charts or tables; Figure 2 , the specific workflow of the data query module is as follows:

[0073] 1) Users construct queries through the query interface, specifying time range, dimensions, and aggregation function conditions;

[0074] 2) The query module parses the query request submitted by the user and passes it to the query optimizer to generate an efficient execution plan;

[0075] 3) Based on the optimized execution plan, the query engine accesses relevant data indexes and storage to execute the query operation;

[0076] 4) Return the query result set to the user, supporting paging and on-demand loading to improve the performance of large data processing;

[0077] 5) Present the data in charts or other formats.

[0078] The data management module is used to perform operations such as adding, deleting, modifying, and querying time series data in the database, as well as backup and recovery operations;

[0079] Specifically, the functions of the data management module are as follows:

[0080] Data addition, deletion, modification and query operations:

[0081] Adding data allows users to add new time series data to the database; this usually involves validating and formatting the data to ensure that the structure and integrity of the data meet system requirements;

[0082] Delete data. Users can delete time series data that is no longer needed from the database. The deletion operation needs to be handled with caution to avoid accidentally deleting important data.

[0083] To modify data, users can update existing time series data in the database. The modification operation must ensure data consistency and verify the validity of the new data.

[0084] To query data, users can query time series data based on specific conditions, such as time range, specific indicators, etc.; query operations need to be optimized to support fast data retrieval.

[0085] Data backup:

[0086] Automatic backup: the system can be configured to automatically back up data regularly to prevent data loss;

[0087] Manual backup: users can also manually trigger the data backup process as needed;

[0088] Incremental backup and full backup, support incremental backup and full backup;

[0089] Data Recovery

[0090] Disaster recovery: in the event of data corruption or loss, backup data can be used for recovery;

[0091] Restore to a specific point in time, which supports restoring data to a specific point in time. This is very useful for recovering data lost due to operational errors.

[0092] Data security and rights management

[0093] Access control ensures that only authorized users can perform data addition, deletion, modification, and query operations;

[0094] Data encryption: encrypt sensitive data to protect the security of data during storage and transmission.

[0095] Through the above functions and technologies, the data management module can effectively protect and manage time series data, ensure data availability and security, and provide flexible data operation capabilities.

[0096] The data analysis and processing module is used to perform preprocessing operations such as cleaning, deduplication, and missing value processing on the received time series data, as well as statistical analysis, trend prediction, and anomaly detection on the stored time series data. The data analysis and processing module establishes a prediction model to analyze and process the data. The specific process is as follows:

[0097] 1) Identify the business problems or analysis objectives that need to be solved and develop a data analysis plan to address the problems;

[0098] 2) Determine the scope and type of data required based on analysis requirements and obtain data from designated sources;

[0099] 3) Build an applicable prediction model based on the analysis purpose and train the model;

[0100] 4) Present analysis results through reports, charts, and visual interfaces, adjust analysis models and processes based on business feedback, continuously monitor analysis results, and perform iterative optimization;

[0101] The prediction model is selected based on the importance and relevance of features to reduce model complexity and avoid overfitting. The partial derivative of the loss function with respect to the weight is used to guide the update of the weight. The expression is:

[0102]

[0103] Where η is the learning rate, is the partial derivative of the loss function with respect to the current weight, E is the loss function, and w is any weight in the network;

[0104] The prediction model is trained using the back propagation algorithm. First, the error of the output layer is calculated, and then it is back propagated layer by layer to the input layer. For each layer, the algorithm calculates the contribution of each neuron to the total error and adjusts the weight accordingly. If the input of neuron j is net j , the output is o j , the error is δ j , with weight w ji , the expression is:

[0105] Δw ji =η·δ j o i ;

[0106] Where, δ j It is calculated based on the type of neuron and the activation function;

[0107] During training, the prediction model uses a nonlinear function to enable the model to capture complex data features. The expression is:

[0108]

[0109] Where σ(x) represents the output of the Sigmoid function, which is a real number with a range of 0 to 1, x is the real value input to the Sigmoid function, e is the base of the natural logarithm, -x represents the opposite of the input value, and 1+e -x It is the denominator part, which ensures that the result of the fraction will be mapped between 0 and 1;

[0110] In the prediction model, the loss function is used to measure the difference between the predicted value of the neural network and the actual value, and the expression is:

[0111]

[0112] Where M is the number of categories, y c is the true label, is the predicted output, log is the logarithmic function with the natural constant e as the base;

[0113] The calculation process of the above formula is:

[0114] For each category c, calculate the true label y c and predicted probability The product of

[0115] Take the logarithm of this product

[0116] Adding up all these values ​​gives us the final loss E.

[0117] User authority management module, used to control and manage user operation permissions;

[0118] Specifically, the functions of the user rights management module are as follows:

[0119] User Authentication:

[0120] Login verification, which verifies the credentials provided by the user (such as username and password) to confirm their identity;

[0121] Multi-factor authentication: support for multi-factor authentication (MFA) to add an extra layer of security, such as combining a password and a mobile phone verification code;

[0122] Role-based access control:

[0123] Role definition, defining different user roles, each with a specific set of permissions;

[0124] Role assignment: assign roles to specific users or user groups to simplify permission management;

[0125] 3. Permission management:

[0126] Fine-grained permission control can precisely control the data and functions that users can access and operate;

[0127] Data-level permissions restrict users to access only specific data sets and prevent them from accessing other users' data;

[0128] Audit and Compliance:

[0129] Operation logs record all user operations, including login, data query, modification, etc., for auditing and backtracking;

[0130] Compliance reporting: Automatically generate compliance reports that show detailed records of user activities to meet regulatory requirements.

[0131] The user rights management module provides a secure and controllable access control environment for the time series database information storage and management system, ensuring that only authorized users can access and operate data in the system. It also provides the necessary audit and compliance support to meet the enterprise's security and regulatory requirements.

[0132] System monitoring module, used for real-time monitoring and alarm of the system's operating status;

[0133] Specifically, the system monitoring module functions are as follows:

[0134] Performance Monitoring:

[0135] Response time monitoring, tracking the response time of the system processing requests to ensure that the service responds quickly;

[0136] Throughput monitoring measures the number of requests processed by the system per unit time and evaluates system capabilities;

[0137] Error Monitoring:

[0138] Error rate monitoring, tracking error rates in the system, such as database query failure rate or API error rate;

[0139] Exception capture: capture exceptions in the system and record detailed information for subsequent analysis;

[0140] Resource usage monitoring:

[0141] CPU usage, monitor the system's CPU usage to ensure the processor is not overused;

[0142] Memory usage, track memory usage and prevent memory leaks or shortages;

[0143] Disk space monitoring: monitor disk usage to ensure there is enough storage space;

[0144] Network Monitoring:

[0145] Network traffic monitoring: monitor the traffic of the network interface to ensure stable network connection and sufficient bandwidth;

[0146] Connection monitoring, monitoring the connection status between the database and the application;

[0147] Alarm mechanism:

[0148] Threshold setting: set thresholds for various monitoring indicators, and trigger alarms once the thresholds are exceeded;

[0149] Notification method: sending alarm notifications via email, SMS or instant messaging;

[0150] The system monitoring module can ensure the efficient operation of the time series database information storage and management system, promptly detect and resolve potential problems, and thus ensure the stability and availability of the system.

[0151] In summary, compared with the prior art, the present invention has the following advantages:

[0152] Efficient data reception and conversion: By supporting a multi-protocol data reception module, the system can process time series data from different data sources and convert it into a unified format. This improves data availability and consistency, providing a solid foundation for subsequent data analysis and management.

[0153] Optimized data storage: Using a distributed storage architecture and sharding technology, the system can achieve horizontal scalability and improve data writing performance. Built-in data compression and downsampling units further save storage space while ensuring data integrity and accuracy.

[0154] Flexible data query and display: The data query module allows users to retrieve data based on different time ranges, dimensions, and aggregation function conditions. Query results can be displayed in charts or tables, supporting paging and on-demand loading, improving the performance of large data processing.

[0155] Comprehensive data management: The data management module not only supports the addition, deletion, modification, and query of time series data, but also includes backup and recovery functions to ensure data security and reliability.

[0156] In-depth data analysis and processing: The data analysis and processing module can perform pre-processing operations such as cleaning, deduplication, and missing value processing on time series data, and perform analytical operations such as statistical analysis, trend prediction, and anomaly detection. This helps to extract valuable information from the data and support decision-making.

[0157] User rights management and system monitoring: The user rights management module ensures the security of the system and prevents unauthorized access and operation; the system monitoring module monitors the operating status of the system in real time, promptly discovers and alerts potential problems;

[0158] Intelligent prediction model: By building and training prediction models, the system can conduct in-depth analysis of time series data, provide business insights, and predict future trends. Model training uses backpropagation algorithms and nonlinear functions to capture complex data features and improve prediction accuracy.

[0159] Application of loss function: The prediction model uses the loss function to measure the difference between the predicted value of the neural network and the actual value. By optimizing the loss function, the model weight is adjusted to improve the performance of the model.

[0160] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Time series database information storage and management system, characterized by: include: Data receiving module, data storage module, data query module, data management module; The data analysis and processing module is used to perform preprocessing operations such as cleaning, deduplication, and missing value processing on the received time series data, as well as statistical analysis, trend prediction, and anomaly detection on the stored time series data. The data analysis and processing module establishes a prediction model to analyze and process the data. The specific process is as follows: 1) Identify the business problems or analysis objectives that need to be solved and develop a data analysis plan to address them; 2) Determine the scope and type of data required based on analysis requirements and obtain data from designated sources; 3) Construct an applicable prediction model based on the analysis purpose and train the model. The prediction model is selected based on the importance and relevance of features to reduce model complexity and avoid overfitting. The partial derivative of the loss function with respect to the weight is used to guide the update of the weight. The expression is: ； Where η is the learning rate, is the partial derivative of the loss function with respect to the current weight, E is the loss function, and w is any weight in the network; The prediction model is trained using the back propagation algorithm. First, the error of the output layer is calculated, and then it is back propagated layer by layer to the input layer. For each layer, the algorithm calculates the contribution of each neuron to the total error and adjusts the weight accordingly. If the input of neuron j is net j , the output is o j , the error is δ j , with weight w ji , the expression is: ； Where, δ j It is calculated based on the type of neuron and the activation function; 4) Present analysis results through reports, charts, and visual interfaces, adjust analysis models and processes based on business feedback, continuously monitor analysis results, and perform iterative optimization; During training, the prediction model uses a nonlinear function to enable the model to capture complex data features. The expression is: ； Where σ(x) represents the output of the Sigmoid function, which is a real number with a range of 0 to 1, x is the real value input to the Sigmoid function, e is the base of the natural logarithm, −x represents the opposite of the input value, and 1+e −x It is the denominator part, which ensures that the result of the fraction will be mapped between 0 and 1; In the prediction model, the loss function is used to measure the difference between the predicted value of the neural network and the actual value, and the expression is: ； Where M is the number of categories, y c is the true label, Is the predicted output, log is the logarithmic function with the natural constant e as the base; The calculation process of the above formula is: For each category c, calculate the true label y c and predicted probability The product of Take the logarithm of this product ; Adding up all these values ​​gives us the final loss E.

2. The time series database information storage and management system according to claim 1, characterized in that: The system also includes: User authority management module, used to control and manage user operation permissions; The system monitoring module is used to monitor and alarm the system's operating status in real time.

3. The time series database information storage and management system according to claim 1, characterized in that: The data receiving module includes: Multi-protocol support submodule, which receives data through multiple communication protocols and can parse and unify these data streams from different protocols; Data format parser, used to load different parsing plug-ins to parse the received data; A data transformation and mapping engine that maps raw data into a database schema, including handling times and values ​​for time series data, as well as mapping any metadata tags; The data quality monitoring unit performs real-time quality checks on data entering the system. The checks include but are not limited to range checks, integrity checks, and accuracy tests to ensure that only high-quality data is accepted and stored.

4. The time series database information storage and management system according to claim 1, characterized in that: The data storage module adopts a distributed storage architecture, which stores data in multiple nodes through sharding technology to achieve horizontal expansion, and adopts an optimized write strategy to achieve higher write performance through memory writing and subsequent disk sequential writing, avoiding the overhead of random writes; The data storage module should have a built-in data compression and downsampling unit, utilize the regularity of time series data, adopt differential compression, dictionary compression and other algorithms to reduce the data volume, and automatically downsample to merge high-precision data into low-precision data to further save storage space.

5. The time series database information storage and management system according to claim 1, characterized in that: The specific workflow of the data query module is as follows: 1) Users construct queries through the query interface, specifying time range, dimensions, and aggregation function conditions; 2) The query module parses the query request submitted by the user and passes it to the query optimizer to generate an efficient execution plan; 3) Based on the optimized execution plan, the query engine accesses relevant data indexes and storage to execute the query operation; 4) Return the query result set to the user, supporting paging and on-demand loading to improve the performance of large data processing; 5) Present the data in charts or other formats.

Images