Cold and hot data adaptive indexing method, system, storage medium and server

Abstract

A method, system, storage medium and server for adaptive indexing of hot and cold data, the method comprising the following steps: writing hot data into a memory; constructing an index with a red-black tree structure for the hot data written into the memory; judging whether there is a condition that satisfies conversion to cold data in the current hot data, and converting the hot data that satisfies the condition into cold data and storing it in a disk, releasing the corresponding cache space; converting the index converted into cold data from a red-black tree structure to a skip linked list structure. At the same time, a system, storage medium and server for adaptive indexing of hot and cold data are also disclosed. When receiving real-time data writing, a large amount of data stored in the memory is marked as hot data, and by constructing an index structure of a red-black tree for the hot data file, the red-black tree structure has good insertion performance, which can improve data archiving performance, and constructing an index structure of a skip linked list for the cold data, the skip linked list structure has a fast search speed, which can improve data query performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of data indexing, and in particular relates to a method, system, storage medium and server for adaptively indexing hot and cold data. Background Art

[0002] The data of each measurement point in the real-time database archive data is generally stored discretely in different disk sectors. Most real-time databases use indexing to locate data. There are also methods to directly read and write disk sectors, but most of them are used in embedded systems and are not suitable for general use. Therefore, the data reading and writing performance is closely related to the quality of the data indexing method. In the existing real-time database indexing technology, the establishment of indexes also has various structures, such as tables or tree structures.

[0003] Traditional methods use arrays, balanced binary trees or B-trees as indexes. Among them, arrays as indexes take up less space and have faster search speeds in an ordered manner, but have very low insertion efficiency; balanced binary trees have a smaller operation time complexity and higher access performance, but the effective utilization rate of each node is very low, resulting in low memory utilization; B-tree nodes contain more keywords and can fully utilize the principle of disk blocks, but have low efficiency in sequential search or deletion operations. Therefore, traditional methods are not conducive to improving real-time database performance.

[0004] Among the improved methods, B+ tree is used as index. Compared with balanced binary tree or B tree, it has fewer levels, more keyword data per node, faster query speed, and all keyword data addresses are on leaf nodes, so the query speed is more stable. Although this improvement has greatly improved the performance of the database, there is still much room for improvement in terms of real-time data characteristics.

[0005] When real-time data is first written, it is stored in the cache in large quantities. This part of data is written and accessed frequently, and real-time data can also be called hot data. After a period of time, hot data is transferred to disk storage, especially archived data, which is unlikely to be supplemented with historical data. It can be called non-hot data or cold data. The performance bottleneck of hot data lies mainly in the speed of building indexes and writing to disk, because the index performance in the cache is usually relatively high, and different index structures have little effect on data read and write performance. Cold data generally has only some read operations and rarely write operations. At this time, the index structure plays a key role in performance. In summary, the existing technology is not enough to fully utilize the characteristics of real-time data, and further optimization can be performed based on data characteristics. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system and storage medium for adaptive indexing of cold and hot data to address the problem of insufficient read and write performance of real-time databases in the above-mentioned prior art, construct appropriate indexes for cold and hot data respectively, and adaptively switch indexing methods to comprehensively improve the read and write performance of real-time databases.

[0007] In order to achieve the above object, the present invention has the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a method for adaptively indexing hot and cold data, comprising the following steps:

[0009] Writing hot data to storage;

[0010] Construct an index with a red-black tree structure for the hot data written into the memory;

[0011] Determine whether the current hot data meets the conditions for conversion to cold data, and convert the hot data that meets the conditions into cold data and store it on disk to release the corresponding cache space;

[0012] The index converted to cold data is converted from a red-black tree structure to a skip linked list structure.

[0013] As a preferred solution of the hot and cold data adaptive indexing method of the present invention, the hot data is recent historical archived data, and the user accesses it for writing data operations and reading data operations.

[0014] As a preferred solution of the hot and cold data adaptive indexing method of the present invention, the cold data is long-term historical archived data, and the user accesses it to read the data.

[0015] As a preferred solution of the hot and cold data adaptive indexing method of the present invention, the conditions for conversion to cold data include spatial or temporal conditions, wherein the spatial condition is that the space occupied by the hot data reaches a threshold, and when the spatial condition is met, the long-term hot data is converted to cold data, and the temporal condition is that the hot data that reaches the conversion time is converted to cold data.

[0016] As a preferred solution of the hot and cold data adaptive indexing method of the present invention, the long-term hot data is the earliest hot data written; the conversion time is custom configured according to the frequency and size of data writing.

[0017] As a preferred solution of the hot and cold data adaptive indexing method of the present invention, the index is adaptively converted from a red-black tree structure to a skip linked list structure.

[0018] In a second aspect, an embodiment of the present invention provides a cold and hot data adaptive indexing system, including:

[0019] A hot data writing module, used for writing hot data into a memory;

[0020] An index building module, used to build an index having a red-black tree structure for hot data written into the storage;

[0021] The data conversion module is used to determine whether the current hot data meets the conditions for conversion to cold data, and convert the hot data group that meets the conditions into cold data and store it in the disk to release the corresponding cache space;

[0022] The index conversion module is used to convert the index converted into cold data from a red-black tree structure to a skip linked list structure.

[0023] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the cold and hot data adaptive indexing method when executed by a processor.

[0024] In a fourth aspect, an embodiment of the present invention provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the hot and cold data adaptive indexing method when executing the computer program.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects:

[0026] When receiving real-time data writes, a large amount of data stored in the memory is marked as hot data. The hot data file is indexed as a red-black tree, which has good insertion performance and can improve data archiving performance. The cold data is indexed as a skip list, which has a fast search speed and can improve data query performance. It is determined whether the current hot data meets the conditions for conversion to cold data, and the index converted to cold data is converted from a red-black tree structure to a skip list structure. Different index structures are used according to the characteristics of real-time data to comprehensively improve the read and write performance of the real-time database.

[0027] Furthermore, whether hot data of the present invention can be converted into cold data includes two conditions: space and time. On the one hand, space restrictions ensure the space of the memory, prevent storage bottlenecks from affecting data access performance, and rationally use system resources; on the other hand, time restrictions ensure that long-term data is marked as cold data in a timely manner, thereby timely converting the index structure and improving the query performance of the overall data.

[0028] Furthermore, the process of converting the index of the present invention from a red-black tree structure to a jump linked list structure is performed adaptively, and the corresponding index structure is adaptively established according to the different operation focuses of hot data and cold data. The real-time database will automatically execute the index conversion task according to predetermined conditions. The process does not require human intervention, thereby enhancing the intelligence and flexibility of the real-time database.

[0029] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 Flow chart of the hot and cold data adaptive indexing method according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of the cold and hot data and index conversion process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below in conjunction with specific embodiments and drawings. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0034] See also Figure 1 The present invention proposes a method for adaptively indexing hot and cold data, comprising the following steps:

[0035] Step 1: First, receive the real-time data write and store it in the cache as hot data in large quantities. The hot data is written into files and stored in high-speed storage.

[0036] Hot data is recent historical archived data. When real-time data is written, the latest data is snapshot data, and all subsequent data can be called historical data. The newly written historical data can be called recent data. Recent data files are often accessed by users, and there may be a large number of data write operations.

[0037] Step 2: See Figure 2The index building module M1 in builds an index for the hot data file, and the index structure is a red-black tree.

[0038] The red-black tree does not need to be adjusted when inserting data, so it can still maintain good index creation performance under a large number of data write operations. The red-black tree takes up little space and has a great speed advantage after the index is loaded into memory. In addition, the red-black tree traversal is an ordered sequence, and the data retrieval performance is also very good.

[0039] Step 3: Determine whether there is any hot data that meets the conditions for conversion to cold data in the current hot data. If so, store the hot data that meets the conditions as cold data files on the disk to release the corresponding cache space.

[0040] Cold data is generally long-term data. After a period of time, there is rarely any new data written to the historical data file, so the data will be archived. This part of historical data can be called long-term data. Generally, users mostly read data when accessing it.

[0041] See also Figure 2 In the data conversion module M2, the cold and hot data conversion conditions are divided into two conditions: space and time. The conversion is performed when one of the two conditions is met, where:

[0042] The space occupied by hot data is detected in the space conversion condition. When a threshold is reached, and new hot data is written, the earliest historical data written in the current hot data can be regarded as long-term data, and the long-term hot data is converted to cold data.

[0043] The time conversion condition will detect the time from when the hot data is written to the present. When the hot data reaches the conversion time, this part of historical data will be regarded as long-term data and converted into cold data. The conversion time can be customized according to the frequency and size of data writing.

[0044] Step 4: Convert the index structure of the files converted to cold data, and convert the red-black tree structure into a skip linked list structure.

[0045] See also Figure 2 The index conversion module M3 in the system converts the index into an adaptive conversion. When a data file is detected, if it is a hot data file, no index conversion task is performed. If it is converted into a cold data file, the real-time database will automatically perform the index conversion task according to the predetermined conditions, delete the red-black tree index, and construct a skip list index.

[0046] The skip linked list is slower than the red-black tree structure when writing data, but the indexing performance will be improved by about 80% when reading data. It will show good indexing performance when the read operations far outnumber the write operations, thus maximizing the performance of historical data queries.

[0047] The hot and cold data adaptive indexing method proposed in the present invention has been verified and applied in the data indexing module of the Ruiteng real-time database system. When indexing data files, red-black tree indexes are constructed for hot data files by distinguishing between hot and cold data, and jump linked list indexes are adaptively converted for files converted to cold data. The present invention constructs indexes that are more suitable for write or read operations according to different data characteristics, comprehensively improving the database indexing performance; at the same time, it realizes automatic data index conversion, improves the flexibility of the database, and saves human operation resources.

[0048] Another embodiment of the present invention further provides a cold and hot data adaptive indexing system, including:

[0049] A hot data writing module, used for writing hot data into a memory;

[0050] An index building module, used to build an index having a red-black tree structure for hot data written into the storage;

[0051] The data conversion module is used to determine whether the current hot data meets the conditions for conversion to cold data, and convert the hot data group that meets the conditions into cold data and store it in the disk to release the corresponding cache space;

[0052] The index conversion module is used to convert the index converted into cold data from a red-black tree structure to a skip linked list structure.

[0053] Another embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the steps of the cold and hot data adaptive indexing method when executed by a processor.

[0054] Another embodiment of the present invention further provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the hot and cold data adaptive indexing method when executing the computer program.

[0055] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the steps of the hot and cold data adaptive indexing method described in the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the server.

[0056] The server may be a computing device such as a notebook, a desktop computer, or a cloud server. The server may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the server may include more or fewer components, or a combination of certain components, or different components. For example, the server may also include input and output devices, network access devices, buses, etc.

[0057] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0058] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0059] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0060] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0061] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0062] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0063] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for adaptive indexing of hot and cold data, characterized in that: The following steps are involved: Writing hot data to storage; Construct an index with a red-black tree structure for the hot data written into the memory; Determine whether the current hot data meets the conditions for conversion to cold data, and convert the hot data that meets the conditions into cold data and store it on disk to release the corresponding cache space; The index converted to cold data is converted from a red-black tree structure to a skip linked list structure.

2. The cold and hot data adaptive indexing method according to claim 1 is characterized in that: The hot data is recent historical archived data, and when users access it, they perform write data operations and read data operations.

3. The cold and hot data adaptive indexing method according to claim 1, characterized in that: The cold data is long-term historical archived data, and when users access it, it is a data read operation.

4. The cold and hot data adaptive indexing method according to claim 1, characterized in that: The conditions for converting to cold data include space and time conditions, wherein the space condition is that the space occupied by hot data reaches a threshold, and when the space condition is met, the long-term hot data is converted to cold data, and the time condition is that the hot data that reaches the conversion time is converted to cold data.

5. The cold and hot data adaptive indexing method according to claim 4 is characterized in that: The long-term hot data is the hot data written earliest; the conversion time is custom configured according to the frequency and size of data writing.

6. The cold and hot data adaptive indexing method according to claim 1, characterized in that: The index is converted from a red-black tree structure to a skip list structure in an adaptive manner.

7. A cold and hot data adaptive indexing system, characterized in that: include: A hot data writing module, used for writing hot data into a memory; An index building module, used to build an index having a red-black tree structure for hot data written into the storage; The data conversion module is used to determine whether the current hot data meets the conditions for conversion to cold data, and convert the hot data group that meets the conditions into cold data and store it in the disk to release the corresponding cache space; The index conversion module is used to convert the index converted into cold data from a red-black tree structure to a skip linked list structure.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the cold and hot data adaptive indexing method as described in any one of claims 1 to 6 are implemented.

9. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the cold and hot data adaptive indexing method as described in any one of claims 1 to 6 are implemented.

Images