A database management method and system based on hybrid index and data layering

Abstract

The application discloses a database management method and system based on mixed index and data layering, and belongs to the technical field of database storage. The method is as follows: according to the use frequency and access attribute of stored data, setting several types of data tables, setting corresponding index modes for different types of data tables; judging the type of the data table to be accessed through a received transaction request, generating an access index; if the access index is a primary index, performing data processing in the node memory through the primary index; if the access index is a mixed index, performing data processing in the node memory according to the data storage position provided by the mixed index. Therefore, by implementing the application, the problem that the existing memory database cannot process massive data when processing high-concurrency tasks can be solved.

Description

Technical Field

[0001] This application belongs to the field of database storage technology, specifically relating to a database management method and system based on hybrid indexing and data layering. Background Technology

[0002] An in-memory database is a database system that distributes data across multiple computer nodes. All data is typically stored in the memory of each node, and these nodes achieve global access and management of the data through a network communication subsystem. This ensures logical data consistency while physically distributing the data across multiple geographical locations. Therefore, in-memory databases are suitable for handling large-scale data and high-concurrency requirements, but they require complex sharding design and data distribution management.

[0003] While in-memory databases can handle real-time transactions in high-concurrency environments, their memory expansion is difficult, making it hard to cope with the storage needs of massive amounts of data. Although a hybrid index approach can be used to store all data using a cold and hot data cache, facilitating the storage of massive amounts of data, this model cannot provide high real-time responsiveness and concurrent operations. Therefore, how to find an effective data storage method to enable in-memory databases to simultaneously achieve high-concurrency real-time processing and storage of massive amounts of data is an urgent problem to be solved. Summary of the Invention

[0004] This application proposes a database management method and system based on hybrid indexing and data layering, which can solve the problem that existing in-memory databases cannot handle massive amounts of data when dealing with high-concurrency tasks.

[0005] The first aspect of this application provides a database management method based on hybrid indexing and data layering, the method comprising: Based on the frequency of use and access attributes of the stored data, several types of data tables are set, and corresponding index modes are set for different types of data tables; wherein, the index mode includes a first-level index and a hybrid index, the first-level index points to the node memory of the in-memory database, and the hybrid index points to the local hard disk; The type of data table to be accessed is determined by the received transaction request, and an access index is generated. If the access index is a first-level index, then data processing is performed in the node's memory through the first-level index; If the access index is a hybrid index, then data processing is performed in the node's memory according to the data storage location provided by the hybrid index.

[0006] The above scheme categorizes data tables into multiple types based on the frequency of data usage and access attributes. Frequently accessed data is stored in memory, while infrequently accessed data is stored on the local hard drive, achieving hierarchical data storage. Furthermore, for data in node memory, a first-level index is used for retrieval to reduce search resource usage, enabling rapid location of the data to be processed and facilitating rapid parallel processing of multiple data processing tasks. Through hybrid indexing, the database is ensured to store large amounts of data regardless of node memory limitations, and storage costs are significantly reduced when storing data on the local hard drive compared to node memory. Therefore, the type of table containing the data to be accessed can be determined based on transaction requests, thereby determining the corresponding access index and quickly and accurately locating the data to be processed. This achieves the goal of expanding data storage space while maintaining the database's high concurrency characteristics, allowing the database to store massive amounts of data when handling high-concurrency tasks.

[0007] In one possible implementation of the first aspect, several data table types are set according to the frequency of use and access attributes of the stored data, and corresponding index patterns are set for different types of data tables, specifically as follows: The usage frequency includes high frequency and low frequency. If the most recent data access time of the stored data exceeds the first time, the access attribute is a hot attribute; if the most recent data access time of the stored data does not exceed the first time, the access attribute is a cold attribute. Define a replication table and a partition table. The replication table is used to store the high-frequency data, and the partition table is used to store the low-frequency data. Set the index mode for the replicated table, and set the index mode for the partitioned table according to the access attributes.

[0008] The above scheme stores frequently accessed data in a replicated table to ensure that this data can be retrieved quickly. Infrequently accessed, large-scale data is stored in a partitioned table, achieving hierarchical data storage.

[0009] In one possible implementation of the first aspect, a copy of the replication table is stored in the memory of each of the nodes; the data in the partition table is bound to a designated node memory according to the region division.

[0010] In one possible implementation of the first aspect, the index mode of the replication table is set, and the index mode of the partition table is set according to the access attribute, specifically as follows: Set the index mode of the replicated table to a first-level index; Based on the access attributes, the partition table is divided into a memory partition table and a hybrid partition table; the memory partition table stores data with hot attributes, and the hybrid partition table stores data with cold attributes. The index mode of the memory partition table is set to a first-level index, and the index mode of the hybrid partition table is set to a hybrid index. The indexes of both the memory partition table and the hybrid partition table are stored in the node's memory.

[0011] The above scheme employs different indexing patterns for different data tables to achieve fast retrieval. First-level retrieval allows for quick location of data storage; the hybrid index ensures all data is persistently stored on the local hard drive, providing stable storage space that can be continuously expanded as data volume grows. Furthermore, storing only recently used data and indexes in node memory improves data access speed.

[0012] In one possible implementation of the first aspect, the hybrid index engine is set in the memory of each node; The hybrid index includes a primary index and a secondary index; The primary index is used to map the partition keys of the data table to data attributes, and a hash index is used as the index storage; wherein, the data attributes include file number, offset and file version number; The secondary index is used to map index field values ​​to the primary key list. It uses a B+ tree as the index storage and loads the data to be processed in units of data blocks.

[0013] In one possible implementation of the first aspect, if the access index is a hybrid index, then data processing is performed in the node memory according to the data storage location provided by the hybrid index, specifically as follows: If the data storage location is the node memory, then data processing is performed in the node memory; If the data storage location is a local hard disk, the data page to be processed is read from the local hard disk into the node memory through the pointer of the hybrid index, and then the data page to be processed is processed.

[0014] The above solution does not require knowledge of the actual storage location of the data; it only needs to process the data based on the information provided by the access index.

[0015] In one possible implementation of the first aspect, the in-memory database includes several nodes, each node contains multiple sites, and each site contains a CPU core, memory, and hard disk partitions. The stations do not share memory or hard disk partitions; the stations use a lock-free data processing method.

[0016] The above scheme forces frequently accessed and recently accessed data to be stored in the node's memory and implements a lock-free design, which enables data processing to enjoy memory-level speed and reduces lock overhead, thus supporting high-concurrency processing.

[0017] The second aspect of this application provides a database management system based on hybrid indexing and data hierarchical structure, the system comprising: a data table classification module, an index generation module, a direct storage data access module, and a hybrid storage data access module; The data table classification module is used to set several types of data tables based on the frequency of use and access attributes of the stored data, and to set corresponding index modes for different types of data tables. The index modes include a first-level index and a hybrid index. The first-level index points to the node memory of the in-memory database, and the hybrid index points to the local hard disk. The index generation module is used to determine the type of data table to be accessed based on the received transaction requests, and to generate an access index. The direct storage data access module is used to perform data processing in the node memory through the first-level index if the access index is a first-level index; The hybrid storage data access module is used to load data from the local hard disk into the memory cache through the hybrid index if the access index is a hybrid index, and then perform data processing in the memory cache.

[0018] A third aspect of this application provides a terminal device, the device comprising: a terminal device including a processor and a memory, the memory storing a computer program, wherein the processor executes the computer program to implement the steps of a database management method based on hybrid indexing and data hierarchical as described in any embodiment of this application.

[0019] A fourth aspect of this application provides a storage medium that stores computer-readable program code, which, when executed, implements the steps of a database management method based on hybrid indexing and data hierarchical structure as described in any one of the embodiments of this application. Attached Figure Description

[0020] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating a specific process of a database management method based on hybrid indexing and data layering, provided in an embodiment of this application. Figure 2This is a structural diagram of a database management system based on hybrid indexing and data layering, provided in one embodiment of this application; Figure 3 This is a structural diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0024] First Embodiment In in-memory databases, when multiple nodes process transactions simultaneously, issues such as dirty reads and non-repeatable reads can easily occur in high-concurrency environments due to network latency and excessive data partitioning, leading to a significant degrade in database performance. Furthermore, when querying or updating multiple data tables, throughput limitations prevent the parallel processing of a large number of requests, resulting in low data processing efficiency and increased data table query costs. Therefore, the main research objective of this application is to enable databases to achieve high real-time response and concurrent operations while meeting the storage needs of massive amounts of data, achieving an optimal balance between cost, capacity, and performance.

[0025] like Figure 1 As shown, to address the problem in existing technologies of difficulty in achieving high-concurrency data processing when a large amount of data is stored in the database, the first embodiment of this application provides a detailed flowchart of a database management method based on hybrid indexing and data layering. This embodiment's database management method based on hybrid indexing and data layering includes steps S1 to S4, detailed below: Step S1: Based on the frequency of use and access attributes of the stored data, set several types of data tables and set corresponding index modes for different types of data tables.

[0026] Based on the frequency of use and access attributes of stored data, this application innovatively plans three different types of data tables to improve data access efficiency and expand storage space according to actual needs, so as to achieve dynamic differentiated storage based on the table level.

[0027] First, the stored data is divided into high-frequency and low-frequency categories based on usage frequency. A replicated table stores the high-frequency data, while a partitioned table stores the low-frequency data. The difference between a replicated table and a partitioned table is that a replicated table maintains a complete copy in the memory of each node in the in-memory database, ensuring strong data consistency and preventing conflicts caused by data modifications. A partitioned table stores large amounts of low-frequency data; the data within the table is horizontally partitioned according to a partition key, and each partition is bound to a designated node in memory.

[0028] Then, based on the local characteristics of the stored data, such as access attributes, the storage mode of the partition table is set.

[0029] Specifically, this application provides two storage modes: MEMORY mode and HYBRID mode. In MEMORY mode, the entire partition table's data is typically stored in the node's memory; in HYBRID mode, all data of the partition table is stored on the local hard disk, while the node's memory only stores the partition table's complete data index and active data page cache. Furthermore, the local hard disk also stores a complete data operation log.

[0030] The access attribute refers to whether the data has been accessed recently. Therefore, data whose most recent access time exceeds a set first time is considered to have a "hot" access attribute; conversely, data whose most recent access time is less than a set first time is considered to have a "cold" access attribute. Thus, data with a "hot" attribute indicates that it has been accessed recently. To ensure faster location on subsequent accesses, the "memory" mode is used to store hot attributes.

[0031] Therefore, the hot or cold attribute of data is determined by whether it has been accessed recently.

[0032] For partitioned tables, they are further divided into memory partitioned tables and hybrid partitioned tables based on the hot / cold attributes of the stored data. Memory partitioned tables use the MEMORY schema to store hot-attribute data; hybrid partitioned tables use the HYBRID schema to store cold-attribute data.

[0033] The indexes of both the memory partition table and the hybrid partition table are stored in the node's memory. The index mode of the memory partition table is set to a first-level index, and the index mode of the hybrid partition table is set to a hybrid index.

[0034] Since both the replicated table and the memory partitioned table store data in the node's memory, a first-level index is used for retrieval, enabling fast data location. The hybrid partitioned table uses a hybrid index that points to the local hard drive.

[0035] Because memory is a scarce resource, the data stored in memory uses the simplest possible indexes, and is only used to store the most frequently accessed indexes and data, making the best use of the resources.

[0036] The in-memory database in this embodiment comprises multiple nodes. Each node, as the physical deployment unit of the database, can contain multiple sites (i.e., the node memory mentioned above). A share-nothing architecture is adopted between sites, meaning resources are isolated between sites, and data is processed through serialized, lock-free transactions, thereby supporting high-concurrency task execution. However, sites can communicate with each other via high-speed network within the nodes.

[0037] Because the data in a hybrid partitioned table using the HYBRID model is stored on the local hard drive, the amount of data stored is not limited by the node's memory and can grow linearly with the local hard drive's storage space. At the same time, since there is no memory sharing between nodes, the number of nodes can also be easily scaled horizontally and linearly.

[0038] Furthermore, the site-specific memory (node ​​memory) is an independent computing and storage unit, containing a dedicated CPU core, memory, and hard disk partitions. The memory includes modified data copies, local partition indexes, and hot attribute data caches; the hard disk partitions are used to store complete data and data operation logs for the local partitions.

[0039] Because sites do not share memory or hard drives, and only synchronize modified data through a replica synchronization protocol, transactions are executed strictly in the order of modification, enjoying memory-level speed while reducing lock overhead.

[0040] Step S2: Determine the type of the data table to be accessed based on the received transaction request, and generate an access index.

[0041] Upon receiving a transaction request, the SQL language of the request is parsed to extract information such as the transaction type and the type of data table to be accessed. Replicated tables, memory-partitioned tables, and hybrid partitioned tables all have corresponding indexing methods; therefore, a corresponding access index can be generated based on the table type.

[0042] The retrieval engine provided in this application embodiment is key to supporting table-level hierarchical storage. It is embedded in the memory of each node and establishes different index modes for different types of data tables, including first-level indexes and hybrid indexes.

[0043] The first-level index points directly to the node's memory. Therefore, if the transaction request requires access to data in the replicated table or the memory partitioned table, the generated access index directly points to the data copy in the node's memory corresponding to the replicated table or the memory partitioned table.

[0044] Optionally, the first-level index in this embodiment of the application uses a red-black tree index to achieve fast data reading to support high-concurrency operations and reduce the additional memory overhead required by the index.

[0045] The hybrid index includes a primary index and a secondary index. The primary index is used to map the partition keys of the data table (in this embodiment, it refers to a hybrid partitioned table) to data attributes, and uses a hash index for index storage. The data attributes include file number, offset, and file version number. The secondary index is used to map the index field values ​​to the primary key list, and uses a B+ tree for index storage, loading the data to be processed in units of data blocks.

[0046] If the transaction request requires access to data in a hybrid partitioned table, the generated access index first points to the data header of the corresponding replica of the data partition. If the data to be accessed is stored in node memory, then it points to node memory; if the data to be accessed is stored on the local hard drive, then it points to the local hard drive.

[0047] Therefore, the indexing mechanism provided in this application embodiment enables accurate data retrieval without needing to know the actual storage location of the data, allowing the in-memory database to handle real-time transactions in a high-concurrency environment.

[0048] Step S3: If the access index is a first-level index, then data processing is performed in the node memory through the first-level index.

[0049] For node memory and local hard disk, this application provides two storage managers. The cache manager uses an optimization algorithm to manage data pages in node memory. The stored index entries will contain an "IN_MEM" bit to mark whether the corresponding data page is currently in node memory. The local hard disk manager organizes data in the form of appending write log files and is responsible for writing data pages and transaction records moved out of node memory to the local hard disk.

[0050] Therefore, if the access index is a first-level index, the corresponding node memory can be accessed directly through the first-level index, and the stored data copy can be processed in the node memory to modify the target access data in the data copy.

[0051] For example, by coordinating the gateway to parse the transaction request and determining that the target partition of the data T1 to be accessed is in node memory A, then node memory A is accessed directly, and the data record in data copy T2 in node memory A is modified.

[0052] After the modifications are complete, the data operation records generated during this data processing are written to the local hard drive's data operation log. Then, the modified data copy is updated to the data copies in the memory of all other nodes, and the data page containing data T1 is asynchronously flushed back to the local hard drive by a background thread.

[0053] The above steps clarify that transaction atomicity is guaranteed by synchronously writing data to the local hard drive, while data modification, refreshing, and cross-node replication are performed asynchronously in the background, which ensures low latency for front-end transactions.

[0054] Step S4: If the access index is a hybrid index, then data processing is performed in the node memory according to the data storage location provided by the hybrid index.

[0055] If the access index is a hybrid index, the system first determines whether the data to be accessed is stored in the node's memory or on the local hard drive. If it is stored in the node's memory, the corresponding node memory is accessed directly to modify the data, and the modification method is the same as that of the first-level index. If it is stored on the local hard drive, the data page to be processed is first read from the local hard drive and then into the node's memory using the pointer of the hybrid index, and then the data page to be processed is processed.

[0056] The operations after data processing are the same as in step S3, so I won't go into details here.

[0057] Therefore, accessing the hybrid partition table pointed to by the hybrid index requires going through the critical process of "in-memory index -> cache judgment -> local hard disk loading" to ensure that massive amounts of data can be stored on the local hard disk. This expands the database storage space while ensuring low data processing latency, enabling the in-memory database to store massive amounts of data under high concurrency.

[0058] Implementing the embodiments of this application has the following beneficial effects: This application embodiment categorizes data tables into multiple types based on the frequency of data usage and access attributes. Frequently accessed data is stored in memory, while infrequently accessed data is stored on the local hard drive, achieving hierarchical data storage. Furthermore, for data in node memory, a first-level index is used for retrieval to reduce search resource usage, enabling rapid location of the data to be processed and facilitating rapid parallel processing of multiple data processing tasks. Through hybrid indexing, the database is ensured to store large amounts of data regardless of node memory limitations, and storage costs are significantly reduced compared to storing data on the local hard drive. Therefore, the type of table containing the data to be accessed can be determined based on transaction requests, thereby identifying the corresponding access index and quickly and accurately locating the data to be processed. This achieves the goal of expanding data storage space while maintaining the database's high concurrency characteristics, allowing the database to store massive amounts of data when handling high-concurrency tasks.

[0059] Second Embodiment Furthermore, in order to implement the database management system based on hybrid indexing and data layering corresponding to the above method embodiments, and to achieve the corresponding functions and technical effects, Figure 2A structural diagram of a database management system based on hybrid indexing and data layering is provided. For ease of explanation, only the parts relevant to this embodiment are shown. The database management system based on hybrid indexing and data layering provided in this application embodiment includes: The data table classification module 201 is used to set several types of data tables according to the usage frequency and access attributes of the stored data, and to set corresponding index modes for different types of data tables; wherein, the index mode includes a first-level index and a hybrid index, the first-level index points to the node memory of the memory database, and the hybrid index points to the local hard disk.

[0060] In this application embodiment, based on the usage frequency and access attributes of the stored data, this application embodiment innovatively plans three different types of data tables to improve data access efficiency and expand storage space according to actual needs, so as to achieve dynamic differentiated storage based on the table level.

[0061] First, the stored data is divided into high-frequency and low-frequency categories based on usage frequency. A replicated table stores the high-frequency data, while a partitioned table stores the low-frequency data. The difference between a replicated table and a partitioned table is that a replicated table maintains a complete copy in the memory of each node in the in-memory database, ensuring strong data consistency and preventing conflicts caused by data modifications. A partitioned table stores large amounts of low-frequency data; the data within the table is horizontally partitioned according to a partition key, and each partition is bound to a designated node in memory.

[0062] Then, based on the local characteristics of the stored data, such as access attributes, the storage mode of the partition table is set.

[0063] Specifically, this application provides two storage modes: MEMORY mode and HYBRID mode. In MEMORY mode, the entire partition table's data is typically stored in the node's memory; in HYBRID mode, all data of the partition table is stored on the local hard disk, while the node's memory only stores the partition table's complete data index and active data page cache. Furthermore, the local hard disk also stores a complete data operation log.

[0064] The access attribute refers to whether the data has been accessed recently. Therefore, data whose most recent access time exceeds a set first time is considered to have a "hot" access attribute; conversely, data whose most recent access time is less than a set first time is considered to have a "cold" access attribute. Thus, data with a "hot" attribute indicates that it has been accessed recently. To ensure faster location on subsequent accesses, the "memory" mode is used to store hot attributes.

[0065] Therefore, the hot or cold attribute of data is determined by whether it has been accessed recently.

[0066] For partitioned tables, they are further divided into memory partitioned tables and hybrid partitioned tables based on the hot / cold attributes of the stored data. Memory partitioned tables use the MEMORY schema to store hot-attribute data; hybrid partitioned tables use the HYBRID schema to store cold-attribute data.

[0067] The indexes of both the memory partition table and the hybrid partition table are stored in the node's memory. The index mode of the memory partition table is set to a first-level index, and the index mode of the hybrid partition table is set to a hybrid index.

[0068] Since both the replicated table and the memory partitioned table store data in the node's memory, a first-level index is used for retrieval, enabling fast data location. The hybrid partitioned table uses a hybrid index that points to the local hard drive.

[0069] Because memory is a scarce resource, the data stored in memory uses the simplest possible indexes, and is only used to store the most frequently accessed indexes and data, making the best use of the resources.

[0070] The in-memory database in this embodiment comprises multiple nodes. Each node, as the physical deployment unit of the database, can contain multiple sites (i.e., the node memory mentioned above). A share-nothing architecture is adopted between sites, meaning resources are isolated between sites, and data is processed through serialized, lock-free transactions, thereby supporting high-concurrency task execution. However, sites can communicate with each other via high-speed network within the nodes.

[0071] Because the data in a hybrid partitioned table using the HYBRID model is stored on the local hard drive, the amount of data stored is not limited by the node's memory and can grow linearly with the local hard drive's storage space. At the same time, since there is no memory sharing between nodes, the number of nodes can also be easily scaled horizontally and linearly.

[0072] Furthermore, the site-specific memory (node ​​memory) is an independent computing and storage unit, containing a dedicated CPU core, memory, and hard disk partitions. The memory includes modified data copies, local partition indexes, and hot attribute data caches; the hard disk partitions are used to store complete data and data operation logs for the local partitions.

[0073] Because sites do not share memory or hard drives, and only synchronize modified data through a replica synchronization protocol, transactions are executed strictly in the order of modification, enjoying memory-level speed while reducing lock overhead.

[0074] The index generation module 202 is used to determine the type of the data table to be accessed based on the received transaction request and generate an access index.

[0075] In this embodiment, upon receiving a transaction request, the SQL language of the transaction request is parsed to extract information such as the transaction type and the type of data table to be accessed. Replicated tables, memory partitioned tables, and hybrid partitioned tables all have corresponding indexing methods; therefore, a corresponding access index can be generated based on the table type.

[0076] The retrieval engine provided in this application embodiment is key to supporting table-level hierarchical storage. It is embedded in the memory of each node and establishes different index modes for different types of data tables, including first-level indexes and hybrid indexes.

[0077] The first-level index points directly to the node's memory. Therefore, if the transaction request requires access to data in the replicated table or the memory partitioned table, the generated access index directly points to the data copy in the node's memory corresponding to the replicated table or the memory partitioned table.

[0078] Optionally, the first-level index in this embodiment of the application uses a red-black tree index to achieve fast data reading to support high-concurrency operations and reduce the additional memory overhead required by the index.

[0079] The hybrid index includes a primary index and a secondary index. The primary index is used to map the partition keys of the data table (in this embodiment, it refers to a hybrid partitioned table) to data attributes, and uses a hash index for index storage. The data attributes include file number, offset, and file version number. The secondary index is used to map the index field values ​​to the primary key list, and uses a B+ tree for index storage, loading the data to be processed in units of data blocks.

[0080] If the transaction request requires access to data in a hybrid partitioned table, the generated access index first points to the data header of the corresponding replica of the data partition. If the data to be accessed is stored in node memory, then it points to node memory; if the data to be accessed is stored on the local hard drive, then it points to the local hard drive.

[0081] Therefore, the indexing mechanism provided in this application embodiment enables accurate data retrieval without needing to know the actual storage location of the data.

[0082] The direct storage data access module 203 is used to perform data processing in the node memory through the first-level index if the access index is a first-level index.

[0083] In this embodiment of the application, two storage managers are provided for node memory and local hard disk. The cache manager uses an optimization algorithm to manage data pages in node memory, and the stored index entries will contain an "IN_MEM" bit to mark whether the corresponding data page is currently in node memory. The local hard disk manager organizes data in the form of appending write log files and is responsible for writing data pages and transaction records moved out of node memory to the local hard disk.

[0084] Therefore, if the access index is a first-level index, the corresponding node memory can be accessed directly through the first-level index, and the stored data copy can be processed in the node memory to modify the target access data in the data copy.

[0085] For example, by coordinating the gateway to parse the transaction request and determining that the target partition of the data T1 to be accessed is in node memory A, then node memory A is accessed directly, and the data record in data copy T2 in node memory A is modified.

[0086] After the modifications are complete, the data operation records generated during this data processing are written to the local hard drive's data operation log. Then, the modified data copy is updated to the data copies in the memory of all other nodes, and the data page containing data T1 is asynchronously flushed back to the local hard drive by a background thread.

[0087] The hybrid storage data access module 204 is used to load data from the local hard disk into the memory cache through the hybrid index if the access index is a hybrid index, and then perform data processing in the memory cache.

[0088] If the access index is a hybrid index, the system first determines whether the data to be accessed is stored in the node's memory or on the local hard drive. If it is stored in the node's memory, the corresponding node memory is accessed directly to modify the data, and the modification method is the same as that of the first-level index. If it is stored on the local hard drive, the data page to be processed is first read from the local hard drive and then into the node's memory using the pointer of the hybrid index, and then the data page to be processed is processed.

[0089] The operations after data processing are the same as in step S3, so I won't go into details here.

[0090] Therefore, accessing the hybrid partition table pointed to by the hybrid index requires going through the critical process of "in-memory index -> cache judgment -> local hard disk loading" to ensure that massive amounts of data can be stored on the local hard disk. This expands the database storage space while ensuring low data processing latency, enabling the in-memory database to store massive amounts of data under high concurrency.

[0091] Implementing the embodiments of this application has the following beneficial effects: This application embodiment categorizes data tables into multiple types based on the frequency of data usage and access attributes. Frequently accessed data is stored in memory, while infrequently accessed data is stored on the local hard drive, achieving hierarchical data storage. Furthermore, for data in node memory, a first-level index is used for retrieval to reduce search resource usage, enabling rapid location of the data to be processed and facilitating rapid parallel processing of multiple data processing tasks. Through hybrid indexing, the database is ensured to store large amounts of data regardless of node memory limitations, and storage costs are significantly reduced compared to storing data on the local hard drive. Therefore, the type of table containing the data to be accessed can be determined based on transaction requests, thereby identifying the corresponding access index and quickly and accurately locating the data to be processed. This achieves the goal of expanding data storage space while maintaining the database's high concurrency characteristics, allowing the database to store massive amounts of data when handling high-concurrency tasks.

[0092] Furthermore, Figure 3 This is a structural diagram of a terminal device provided in one embodiment of this application. Figure 3 As shown, the terminal device 3 of this embodiment includes: at least one processor 30 (in... Figure 3 (Only one is shown in the image) and a memory 31 and a computer program 32 stored in the memory 31 and executable on the at least one processor, wherein when the processor 30 executes the computer program 32, it can implement the steps of a database management method based on hybrid indexing and data hierarchical as described in any one of the embodiments of this application.

[0093] The terminal device 3 may be a computing device such as a desktop computer, a cloud server, or a laptop computer, and the computing device may include, but is not limited to, a processor 30 and a memory 31. Figure 3 This is merely an example of terminal device 3 and does not constitute a limitation on terminal device 3. It may include more or fewer components than those shown in the figure.

[0094] This application provides a storage medium that stores computer-readable program code, which, when executed, implements the steps of the database management method based on hybrid indexing and data hierarchical structure described above.

[0095] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, or improvements made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A database management method based on hybrid indexing and data layering, characterized in that, include: Based on the frequency of use and access attributes of the stored data, several types of data tables are set, and corresponding index modes are set for different types of data tables; wherein, the index mode includes a first-level index and a hybrid index, the first-level index points to the node memory of the in-memory database, and the hybrid index points to the local hard disk; The type of data table to be accessed is determined by the received transaction request, and an access index is generated. If the access index is a first-level index, then data processing is performed in the node's memory through the first-level index; If the access index is a hybrid index, then data processing is performed in the node's memory according to the data storage location provided by the hybrid index.

2. The database management method based on hybrid indexing and data layering according to claim 1, characterized in that, Based on the frequency of use and access attributes of the stored data, several types of data tables are set, and corresponding index patterns are set for different types of data tables, specifically: The usage frequency includes high frequency and low frequency. If the most recent data access time of the stored data exceeds the first time, the access attribute is a hot attribute; if the most recent data access time of the stored data does not exceed the first time, the access attribute is a cold attribute. Define a replication table and a partition table. The replication table is used to store the high-frequency data, and the partition table is used to store the low-frequency data. Set the index mode for the replicated table, and set the index mode for the partitioned table according to the access attributes.

3. The database management method based on hybrid indexing and data layering according to claim 2, characterized in that, The replication table stores a copy in the memory of each node; the data in the partition table is bound to a designated node memory according to the region division.

4. The database management method based on hybrid indexing and data layering according to claim 2, characterized in that, The specific steps for setting the index mode of the replication table and setting the index mode of the partition table according to the access attributes are as follows: Set the index mode of the replicated table to a first-level index; Based on the access attributes, the partition table is divided into a memory partition table and a hybrid partition table; the memory partition table stores data with hot attributes, and the hybrid partition table stores data with cold attributes. The index mode of the memory partition table is set to a first-level index, and the index mode of the hybrid partition table is set to a hybrid index. The indexes of both the memory partition table and the hybrid partition table are stored in the node's memory.

5. The database management method based on hybrid indexing and data layering according to claim 1, characterized in that, The engine for the hybrid index is set in the memory of each node; The hybrid index includes a primary index and a secondary index; The primary index is used to map the partition keys of the data table to data attributes, and a hash index is used as the index storage; wherein, the data attributes include file number, offset and file version number; The secondary index is used to map index field values ​​to the primary key list. It uses a B+ tree as the index storage and loads the data to be processed in units of data blocks.

6. The database management method based on hybrid indexing and data layering according to claim 1, characterized in that, If the access index is a hybrid index, then data processing is performed in the node memory according to the data storage location provided by the hybrid index, specifically as follows: If the data storage location is the node memory, then data processing is performed in the node memory; If the data storage location is a local hard disk, the data page to be processed is read from the local hard disk into the node memory through the pointer of the hybrid index, and then the data page to be processed is processed.

7. The database management method based on hybrid indexing and data layering according to claim 1, characterized in that, The in-memory database comprises several nodes, each node contains multiple sites, and each site contains a CPU core, memory, and hard disk partitions; the sites do not share memory and hard disk partitions; the sites employ a lock-free data processing method.

8. A database management system based on hybrid indexing and data layering, characterized in that, include: Data table classification module, index generation module, direct storage data access module, and hybrid storage data access module; The data table classification module is used to set several types of data tables based on the frequency of use and access attributes of the stored data, and to set corresponding index modes for different types of data tables. The index modes include a first-level index and a hybrid index. The first-level index points to the node memory of the in-memory database, and the hybrid index points to the local hard disk. The index generation module is used to determine the type of data table to be accessed based on the received transaction requests, and to generate an access index. The direct storage data access module is used to perform data processing in the node memory through the first-level index if the access index is a first-level index; The hybrid storage data access module is used to load data from the local hard disk into the memory cache through the hybrid index if the access index is a hybrid index, and then perform data processing in the memory cache.

9. A terminal device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the steps of a database management method based on hybrid indexing and data hierarchical as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores computer-readable program code, which, when executed, implements the steps of a database management method based on hybrid indexing and data hierarchical structure as described in any one of claims 1 to 7.

Images