Archive file storage method, system and storage medium for real-time database

Abstract

A method, system, and storage medium for archiving files for a real-time database. The storage method includes the following steps: receiving written snapshot data through data middleware and creating an archive file for a corresponding time range; storing the snapshot data in a data page of a set size, and archiving the data after the set number of data pages are filled; storing the data to be archived in units of data pages into an archive file within the current corresponding time range; determining whether the time range of the data page to be archived exceeds the time range of the current archive file; if not, stopping data archiving; if so, stopping the current archive file storage and switching to a new archive file for the next time range. The present invention also provides an archive file storage system and computer-readable storage medium for a real-time database. The present invention can ensure that snapshot data will not be lost during archiving, ensuring data integrity, while also ensuring data coherence and consistency.

Description

Technical Field

[0001] The present invention relates to the technical field of data archiving of real-time databases, and in particular to an archive file storage method, system and storage medium oriented to real-time databases. Background Art

[0002] With the rapid advancement of information technology, the popularization of information technology has also promoted the development of various industries. As a branch of the development of database systems, real-time databases effectively combine real-time processing with database technology. They can directly store a large amount of industrial enterprise production data in real time and convert it into intuitive and effective information for real-time supervision and data analysis operations.

[0003] A key characteristic of real-time databases is real-time performance. Achieving this requires not only high-quality data collection but also crucial data storage. For modern industrial enterprises, production process information is a crucial resource. The ability to access this information at any time significantly contributes to decision-making. Therefore, maintaining the integrity and validity of this information is crucial. This requires real-time databases to store all necessary information, prevent loss, and efficiently access data. Existing real-time database technologies include those that utilize disk sector read / write methods, allowing for direct data read and write without the need for indexing. However, this approach has significant limitations and is generally used in embedded systems. Other approaches leverage the characteristics of file systems, archiving real-time data in chronologically ordered files and establishing appropriate index structures such as linked lists, hash tables, or trees. These approaches are more adaptable and better suited to the characteristics of operating systems and storage devices. In the existing technology, the index is generally stored in the header of the archive file. Although this method increases the indexing speed, it is not convenient to dynamically build and modify the index. Moreover, when the scale of tag points is large, the index part will take up a lot of space, resulting in disk waste and data redundancy. It will also consume more disk I / O and CPU processing time when querying data, affecting the performance of the real-time database. Summary of the Invention

[0004] The purpose of the present invention is to address the problem in the above-mentioned prior art that the access performance of data archiving is limited by the operating system or the scale of tag points when the real-time database is archiving data, and to provide an archive file storage method, system and storage medium for real-time databases, which improves the efficiency of data access while ensuring the integrity of data archiving.

[0005] In order to achieve the above object, the present invention has the following technical solutions:

[0006] In a first aspect, an embodiment provides an archive file storage method for a real-time database, comprising the following steps:

[0007] Receive the written snapshot data through the data middleware and create archive files for the corresponding time range;

[0008] Snapshot data is stored in data pages of a set size, and data is archived after the set number of data pages is filled.

[0009] The data to be archived is stored in the archive file within the current corresponding time range in units of data pages;

[0010] Determine whether the time range of the data page to be archived exceeds the time range of the current archive file. If not, data archiving is stopped. If so, stop the current archive file storage and switch to a new archive file in the next time range.

[0011] As a preferred solution of the archive file storage method for real-time database of the present invention, the archive file is automatically created, and the time range of automatic creation and the initial size of the file are custom configured. If the size of the archived data within the time range exceeds the archive file size, the archive file will grow adaptively, and the growth amount is custom configured.

[0012] As a preferred solution of the archive file storage method for real-time database of the present invention, the archive file is created manually, and the manually created archive file is used for archiving and storing historical data.

[0013] As a preferred solution of the archive file storage method for real-time database of the present invention, only the data of one tag point is stored in one data page, and the content stored in one data page includes the data page ID, the TagID corresponding to the tag point name, the number of data items and each data item.

[0014] As a preferred solution of the archive file storage method for real-time database of the present invention, the archive file format separates the data file and the index file, and is divided into the following four files: .rdf data file, .sum satellite data file, .idx index file, and .ref index reference file; specifically:

[0015] .rdf data files are data page archive storage files, which store data values ​​and interpolation values ​​in units of data pages;

[0016] The .sum satellite data file stores the statistical values ​​calculated for each data page, including the maximum value, minimum value, cumulative value, weighted average value, and arithmetic mean value;

[0017] The .idx index file is a second-level index file that stores n red-black trees, corresponding to n label points, in the form of tree nodes. Each tree node corresponds to a data page, and the node stores the red-black tree node information and index key value. The index key value stores the start and end time of the data page and the data page ID. By comparing the tree node with the query time, the data page is found to locate the queried data.

[0018] The .ref index reference file is the first-level index file, which stores a persistent hash table, including a bucket area and a linked list area. The bucket area stores bucket nodes, and each node stores the linked list head node. The linked list area stores n linked list nodes, corresponding to n red-black trees. Each node stores the ID of the label point, the root node of the red-black tree, and the next linked list node. The second-level index is located by searching the linked list node to find the root node of the red-black tree in the .idx index file.

[0019] As a preferred solution of the archive file storage method for a real-time database of the present invention, the method stores the data to be archived in the archive file within the current corresponding time range in units of data pages, and sequentially stores the .rdf data file, the .sum satellite data file, the .idx index file, and the .ref index reference file, specifically comprising the following steps:

[0020] The data to be stored is stored in batches in .rdf data files in units of data pages;

[0021] Calculate the .sum satellite data of each data page. If the data page tag type is a basic data type, the calculated statistical value is stored in the satellite data file; if the type is other types, no calculation is done;

[0022] Create and update a red-black tree. First, store the data page start and end times and the data page ID in each newly added tree node. Then, determine the red-black tree to be inserted by the tag point ID. Finally, update the parent node and child node information of the relevant node. The tree node is stored in the .idx index file.

[0023] To create an updated index reference, first, store the label point ID and the red-black tree root node in each newly added linked list node. Then, using the hash remainder algorithm, use the label point ID to find the remainder of the number of buckets and calculate the bucket into which the label falls, thus obtaining the linked list into which the linked list node is to be inserted. Finally, point the tail node of the linked list to this node. The linked list node is stored in the .ref index reference file.

[0024] As a preferred solution of the archive file storage method for real-time database of the present invention, the basic data types include int32, int64, float32, and float64.

[0025] As a preferred solution of the archive file storage method for real-time database of the present invention, when the archive file is switched, the time of the new archive file is immediately after the previous archive file, and the time between the archive files does not overlap.

[0026] In a second aspect, an embodiment of the present invention further provides an archive file storage system for a real-time database, comprising:

[0027] The archive file creation module is used to receive the written snapshot data through the data middleware and create archive files of the corresponding time range;

[0028] A data archiving module is used to store snapshot data in data pages of a set size and archive the data after the set number of data pages is filled;

[0029] The storage module is used to store the data to be archived in the archive file within the current corresponding time range in units of data pages;

[0030] The archive file switching module is used to determine whether the time range of the data page to be archived exceeds the time range of the current archive file. If not, data archiving is stopped. If so, the current archive file storage is stopped and switched to a new archive file in the next time range.

[0031] In a third aspect, an embodiment of the present invention further proposes a computer-readable storage medium storing a computer program, which implements the steps of the archive file storage method for a real-time database when executed by a processor.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] When the present invention accepts snapshot data, an archive file is automatically created to ensure that the snapshot data will not be lost when archiving, thereby ensuring the integrity of the data; the time ranges between the archive files are connected to ensure the coherence of the data; the time between the archive files does not overlap, thereby ensuring the consistency of the data.

[0034] Furthermore, the present invention separates the archive file index from the data file. This facilitates dynamic data index construction and eliminates the need to modify the data file when tag point authorization changes, saving system resources. Furthermore, archive file repair is more efficient when the system crashes or restarts. The index file uses a red-black tree index, which takes up little space and offers fast insertion and retrieval speeds, improving data access performance.

[0035] Furthermore, the present invention refines the archive file system, storing data separately in data files and satellite data files. Frequently accessed data storage values ​​and interpolations are stored in the data files and loaded into memory when accessed; while infrequently accessed data statistics are stored in the satellite data files and loaded into memory when needed. This effectively increases memory utilization and further improves database performance.

[0036] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 Flowchart of the archive file storage method for a real-time database according to an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of an archive file system according to an embodiment of the present invention;

[0040] Figure 3 Flowchart of the archive file system storage method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer and more specific, the present invention is further described in detail below in conjunction with specific embodiments and accompanying drawings. The present invention can also be implemented or applied through different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0042] See also Figure 1 An embodiment of the archive file storage method for a real-time database of the present invention includes the following steps:

[0043] Step 1: Data archiving begins. First, the real-time database server receives the written snapshot data through the data middleware and automatically creates an archive file for the corresponding time range.

[0044] The time range and initial file size for automatic archive file creation are customizable. The default time range is 1 day, and the default initial file size is 1GB. These settings take effect the next time an archive file is automatically created. If the archived data size exceeds the archive file size within the time range, the archive file will automatically grow, and the growth rate can also be customized. Archive files can also be created manually for archiving historical data.

[0045] Step 2: Before archiving, the snapshot data is first stored in a data page of a certain size in the snapshot buffer. After a certain amount of data pages are filled, the data is archived.

[0046] Specifically, a data page stores only the data of one tag point. The page size is generally 4KB. Each page stores the data page ID, the tag ID corresponding to the tag point name, the number of data items, and each data item. In other words, the data of a tag point is stored in multiple data pages and archived in the form of data pages. When it is accessed, the corresponding data page ID is found through the index file.

[0047] See also Figure 2 In data archiving, the archive file system separates data files from index files and is divided into four files: .rdf data file, .sum satellite data file, .idx index file, and .ref index reference file, among which:

[0048] .rdf data files are data page archive storage files, which store data values ​​and interpolation values ​​in units of data pages;

[0049] The .sum satellite data file stores statistical values ​​such as the maximum value (MAX), minimum value (MIN), cumulative value (CUSUM), weighted average value, and arithmetic mean value (AVG) calculated for each data page;

[0050] The .idx index file is a second-level index file that primarily stores n red-black trees, corresponding to n tag points, stored as tree nodes. Each tree node corresponds to a data page, and the node stores the red-black tree node information and index key value. The index key value stores the start and end time of the data page and the data page ID. The data page to be queried is located by comparing the tree node with the query time.

[0051] The .ref index reference file is the first-level index file, storing a persistent hash table consisting of a bucket area and a linked list area. The bucket area stores bucket nodes, each containing the linked list header. The linked list area stores n linked list nodes, corresponding to n red-black trees. Each node contains the ID of the tag point, the root node of the red-black tree, and the next linked list node. The second-level index is located by searching the linked list nodes to find the root node of the red-black tree in the index file.

[0052] Step 3: The data to be archived is stored in the archive file within the current corresponding time range in units of data pages, and the data file, satellite data file, index file and index reference file are stored in sequence, such as Figure 3 As shown, the specific steps include:

[0053] S31: The data to be stored is stored in batches in a data file in units of data pages;

[0054] S32: Calculate the satellite data of each data page. If the data page tag type is a basic data type (such as int32, int64, float32, float64, etc.), the calculated statistical value is stored in the satellite data file; if the type is other types, no calculation is performed;

[0055] S33: Create and update the red-black tree. First, store the data page start and end time and data page ID in each newly added tree node. Then, determine the red-black tree to be inserted by the tag point ID. Finally, update the parent node and child node information of the relevant node. The tree nodes are stored in the index file.

[0056] S34: Create and update the index reference. First, store the tag point ID and the red-black tree root node in each newly added linked list node. Then, use the hash remainder algorithm to calculate the bucket number using the tag point ID. This calculates the bucket the tag falls into, thus obtaining the linked list node to be inserted into. Finally, point the tail node of the linked list to this node. The linked list node is stored in the index reference file.

[0057] Step 4: Repeat steps 2 and 3 until the time range of the data page to be archived exceeds the time range of the current archive file, and then switch to the new archive file for storage.

[0058] The present invention is applied in the data archiving module of the Ruiteng real-time database system. When data is archived and stored, by storing data values, statistical values ​​and indexes in data files, satellite data files, indexes and index reference files respectively, it is possible to conveniently dynamically build data indexes, reduce data file modifications, save system resources, and at the same time improve memory utilization during data access, further improving database performance.

[0059] In one specific application instance, when querying the historical storage value of a single tag point, the specific steps are as follows:

[0060] 1) In the index reference file, use the tag point ID to calculate the remainder of the bucket size and obtain the bucket node;

[0061] 2) Get the linked list head node in the bucket node, and get the linked list node by traversing and comparing the tag point ID with the TagID in the linked list node;

[0062] 3) Get the index red-black tree root node of the label point in the linked list node;

[0063] 4) In the index file, traverse from the root node of the red-black tree according to the query time, compare the query time with the red-black tree node time, and obtain the query time data if all are hit; continue traversing if some are hit, return the corresponding time data, and then traverse to return the remaining data.

[0064] Another embodiment of the present invention further provides an archive file storage system for a real-time database, comprising:

[0065] The archive file creation module is used to receive the written snapshot data through the data middleware and create archive files of the corresponding time range;

[0066] A data archiving module is used to store snapshot data in data pages of a set size and archive the data after the set number of data pages is filled;

[0067] The storage module is used to store the data to be archived in the archive file within the current corresponding time range in units of data pages;

[0068] The archive file switching module is used to determine whether the time range of the data page to be archived exceeds the time range of the current archive file. If not, data archiving is stopped. If so, the current archive file storage is stopped and switched to a new archive file in the next time range.

[0069] Another embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the steps of the archive file storage method for a real-time database when executed by a processor.

[0070] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the steps of the real-time database-oriented archive file storage method described in this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the server.

[0071] The server can be a computing device such as a smartphone, laptop, PDA, or cloud server. The server may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the server may include more or fewer components, or a combination of certain components, or different components. For example, the server may also include input and output devices, network access devices, and buses.

[0072] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0073] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk equipped on the server, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is about to be output.

[0074] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0077] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0078] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for storing archive files in a real-time database, characterized in that: The following steps are involved: Receive the written snapshot data through the data middleware and create archive files for the corresponding time range; Snapshot data is stored in data pages of a set size, and data is archived after the set number of data pages is filled. The data to be archived is stored in the archive file within the current corresponding time range in units of data pages; Determine whether the time range of the data page to be archived exceeds the time range of the current archive file. If not, data archiving stops. If so, stop storing the current archive file and switch to a new archive file with the next time range. The archive file supports both automatic and manual creation. In the case of automatic creation, the time range and initial file size can be customized. If the size of the archived data exceeds the archive file size within the time range, the archive file will grow adaptively, and the growth amount can be customized. A data page stores data of only one tag point. The contents stored in a data page include the data page ID, the TagID corresponding to the tag point name, the number of data items, and each data item. The archive file format separates the data file from the index file, and is divided into the following four files: .rdf data file, .sum satellite data file, .idx index file, and .ref index reference file; specifically: .rdf data files are archive storage files for data pages; The .sum satellite data file stores the statistical values ​​calculated for each data page; The .idx index file stores n red-black trees as the second-level index; The .ref index reference file stores a persistent hash table as the first level index.

2. The archive file storage method for real-time database according to claim 1, characterized in that: The archive files are created manually, and the manually created archive files are used for archiving and storing historical data.

3. The archive file storage method for real-time database according to claim 1, characterized in that: .rdf data files store data values ​​and interpolation values ​​in units of data pages; .sum satellite data files store maximum, minimum, cumulative, weighted average, and arithmetic mean values; The .idx index file stores n red-black trees, corresponding to n label points, in the form of tree nodes. Each tree node corresponds to a data page, and the node stores the red-black tree node information and index key value. The index key value stores the start and end time of the data page and the data page ID. By comparing the tree node with the query time, the data page is found to locate the queried data. The hash table of the .ref index reference file includes a bucket area and a linked list area; the bucket area stores bucket nodes, and each node stores the linked list head node; the linked list area stores n linked list nodes, corresponding to n red-black trees, and each node stores the ID of the label point, the root node of the red-black tree, and the next linked list node. The second-level index is located by searching the linked list node to find the root node of the red-black tree in the .idx index file.

4. The archive file storage method for real-time database according to claim 3, characterized in that: The method of storing the data to be archived in an archive file within the current corresponding time range in units of data pages, and sequentially storing the .rdf data file, the .sum satellite data file, the .idx index file, and the .ref index reference file, specifically includes the following steps: The data to be stored is stored in batches in .rdf data files in units of data pages; Calculate the .sum satellite data of each data page. If the data page tag type is a basic data type, the calculated statistical value is stored in the satellite data file; if the type is other types, no calculation is done; Create and update a red-black tree. First, store the data page start and end times and the data page ID in each newly added tree node. Then, determine the red-black tree to be inserted by the tag point ID. Finally, update the parent node and child node information of the relevant node. The tree node is stored in the .idx index file. To create an updated index reference, first, store the label point ID and the red-black tree root node in each newly added linked list node. Then, using the hash remainder algorithm, use the label point ID to find the remainder of the number of buckets and calculate the bucket into which the label falls, thus obtaining the linked list into which the linked list node is to be inserted. Finally, point the tail node of the linked list to this node. The linked list node is stored in the .ref index reference file.

5. The archive file storage method for real-time database according to claim 4, characterized in that: The basic data types include int32, int64, float32, and float64.

6. The archive file storage method for real-time database according to claim 1, characterized in that: When the archive files are switched, the time of the new archive file is immediately after the time of the previous archive file, and there is no overlap between the times of the archive files.

7. An archive file storage system for real-time database, characterized in that: include: The archive file creation module is used to receive the written snapshot data through the data middleware and create archive files of the corresponding time range; A data archiving module is used to store snapshot data in data pages of a set size and archive the data after the set number of data pages is filled; The storage module is used to store the data to be archived in the archive file within the current corresponding time range in units of data pages; The archive file switching module is used to determine whether the time range of the data page to be archived exceeds the time range of the current archive file. If not, data archiving is stopped. If so, the current archive file storage is stopped and switched to a new archive file with the next time range; The archive file supports both automatic and manual creation. In the case of automatic creation, the time range and initial file size can be customized. If the size of the archived data exceeds the archive file size within the time range, the archive file will grow adaptively, and the growth amount can be customized. A data page stores data of only one tag point. The contents stored in a data page include the data page ID, the TagID corresponding to the tag point name, the number of data items, and each data item. The archive file format separates the data file from the index file, and is divided into the following four files: .rdf data file, .sum satellite data file, .idx index file, and .ref index reference file; specifically: .rdf data files are archive storage files for data pages; The .sum satellite data file stores the statistical values ​​calculated for each data page; The .idx index file stores n red-black trees as the second-level index; The .ref index reference file stores a persistent hash table as the first level index.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the archive file storage method for a real-time database as claimed in any one of claims 1 to 6 are implemented.

Images