Artificial intelligence-based data tiering method, apparatus, device and medium

Abstract

The application provides an artificial intelligence-based data hierarchical storage method and device, electronic equipment and storage medium. The artificial intelligence-based data hierarchical storage method comprises the following steps: obtaining business data, calculating the heat of the business data to obtain heat data; dividing a preset business database into multiple levels to store the heat data; screening all levels according to the storage capacity to obtain a boundary level; obtaining a scheduling level based on the boundary level to store the heat data in different storage media, wherein the storage media comprise a first storage medium and a second storage medium; and calling the business data in the second storage medium based on the first storage medium. The application divides the business database into multiple levels, calculates the boundary level and the scheduling level, realizes the dynamic hierarchical storage of the business data, and effectively improves the storage efficiency of the business database.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data hierarchical storage method, apparatus, electronic device and storage medium based on artificial intelligence. Background Technology

[0002] LSM-tree (Log-Structured Merge-Tree) is a key-value storage structure commonly used in the underlying storage engines of NoSQL database systems. As is well known, hard disks, as non-volatile storage media, carry the main persistent data portion of a database system and constitute a major component of its cost. Due to its write amplification and read amplification principles, in enterprise-level applications, to meet performance requirements, the typical recommended storage medium for LSM-trees is NVMe hard drives, which offer high read / write throughput and high random read / write performance. However, their unit storage cost is several times higher than that of traditional hard disk drives (HDDs).

[0003] LSM-trees require data merging; otherwise, they will lead to unlimited data expansion and continuous performance degradation. Currently, the common merging method is to store data on disk in a hierarchical manner. However, how to fully utilize both NVMe and HDD space while strictly ensuring that NVMe space is not exhausted, thereby improving database storage efficiency, is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of the above, it is necessary to propose a data hierarchical storage method, device, electronic device and storage medium based on artificial intelligence to solve the technical problem of how to improve the storage efficiency of databases.

[0005] This application provides a data hierarchical storage method based on artificial intelligence, the method comprising:

[0006] Acquire business data and calculate the popularity of the business data to obtain popularity data;

[0007] The preset business database is divided into multiple levels to store the popularity data, and each level includes a storage capacity.

[0008] Based on the storage capacity, all levels are filtered to obtain the boundary level;

[0009] Based on the boundary level, the scheduling level is obtained to store the heat data in different storage media, the storage media including a first storage media and a second storage media;

[0010] The service data in the second storage medium is accessed based on the first storage medium.

[0011] In some embodiments, obtaining business data and calculating the popularity of the business data to obtain popularity data includes:

[0012] Retrieve the business data to be stored and count the frequency of each business data being accessed;

[0013] The frequency with which each business data is accessed is used as the corresponding business data popularity, and the business data with popularity is used as popularity data.

[0014] In some embodiments, dividing the preset business database into multiple levels to store the popularity data includes:

[0015] The preset business database is divided into multiple levels according to different storage capacities, and the storage capacity and the level correspond one-to-one.

[0016] All levels are sorted in ascending order of storage capacity, with the earlier the level is sorted, the smaller the corresponding storage capacity.

[0017] The popularity data is stored sequentially in descending order of popularity into the sorted levels, with the more popular the data appearing earlier in the sorted levels.

[0018] In some embodiments, the step of filtering all levels based on the storage capacity to obtain the boundary level includes:

[0019] Based on the sorting results of the hierarchy, the storage capacity corresponding to each hierarchy is accumulated from front to back to obtain the accumulated capacity;

[0020] The layers whose accumulated capacity meets the preset capacity threshold range are designated as boundary layers.

[0021] In some embodiments, obtaining the scheduling level based on the boundary level to store the heat data in different storage media includes:

[0022] Calculate the boundary capacity of the storage capacity corresponding to the boundary level, whereby the boundary capacity is used to characterize the maximum capacity that the storage capacity corresponding to the boundary level can be elastically expanded.

[0023] The scheduling level is obtained by comparing the boundary capacity with the capacity of the first storage medium;

[0024] Based on the scheduling hierarchy, the database containing the popularity data is stored in different storage media, including a first storage media and a second storage media.

[0025] In some embodiments, obtaining the scheduling level by comparing the boundary capacity and the capacity of the first storage medium includes:

[0026] If the boundary capacity is greater than the capacity of the first storage medium, then the boundary level is used as the scheduling level.

[0027] If the boundary capacity is not greater than the capacity of the first storage medium, then the next level in the sorting result of the boundary level is taken as the scheduling level.

[0028] In some embodiments, the step of accessing business data in the second storage medium based on the first storage medium includes:

[0029] Allocate cache space in the first storage medium;

[0030] The cache space stores the metadata of the second storage medium to access the business data in the second storage medium.

[0031] This application embodiment also provides a data hierarchical storage device based on artificial intelligence, the device comprising:

[0032] A calculation unit is used to acquire business data and calculate the popularity of the business data to obtain popularity data;

[0033] A partitioning unit is used to divide a preset business database into multiple levels to store the popularity data, and each level includes a storage capacity.

[0034] A filtering unit is used to filter all levels based on the storage capacity to obtain the boundary level;

[0035] The acquisition unit is used to acquire the scheduling level based on the boundary level in order to store the heat data in different storage media, the storage media including a first storage medium and a second storage medium;

[0036] The invocation unit is used to invoke business data in the second storage medium based on the first storage medium.

[0037] This application embodiment also provides an electronic device, the electronic device comprising:

[0038] Memory, storing at least one instruction;

[0039] The processor executes the instructions stored in the memory to implement the AI-based hierarchical data storage method.

[0040] This application also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the AI-based hierarchical data storage method.

[0041] This application effectively improves the storage efficiency of the business database by dividing the pre-defined business database into multiple levels and storing business data hierarchically by defining boundary levels and scheduling levels. At the same time, it can optimize the data retrieval method of the second storage medium using the first storage medium. Attached Figure Description

[0042] Figure 1 This is a flowchart of a preferred embodiment of the AI-based hierarchical data storage method involved in this application.

[0043] Figure 2 This is a functional block diagram of a preferred embodiment of the AI-based hierarchical data storage device involved in this application.

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the data hierarchical storage method based on artificial intelligence involved in this application.

[0045] Figure 4 This is a structural diagram of the SST distribution pattern of each level of an LSM-tree under a hierarchical strategy.

[0046] Figure 5 This is a schematic diagram of the storage structure of data blocks and metadata in the SST file within the database. Detailed Implementation

[0047] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0048] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] This application provides a data hierarchical storage method based on artificial intelligence, which can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0051] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0052] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0053] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0054] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the data hierarchical storage method based on artificial intelligence according to this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different needs.

[0055] S10, acquire business data and calculate the popularity of the business data to obtain popularity data.

[0056] In an optional embodiment, obtaining business data and calculating the popularity of the business data to obtain popularity data includes:

[0057] S101: Obtain the business data to be stored and count the frequency of each business data being accessed;

[0058] S102, the frequency of each business data being called is taken as the popularity of the corresponding business data, and the business data with popularity is taken as the popularity data.

[0059] In this optional embodiment, large amounts of business data can be acquired through both active and passive acquisition. Active acquisition can be achieved by proactively reading business data from various business channels using preset scripts and configuration files. Passive acquisition can be performed using proprietary data transmission tools, commercial transmission tools, or open-source transmission tools.

[0060] In this optional embodiment, the frequency with which each business data is accessed within a preset period can be counted. The preset period can be a day, an hour, or other periods; no specific limitation is made in this solution. The data volume of each identical business data within the preset period can be used as the frequency with which that business data is accessed.

[0061] In this optional embodiment, the frequency with which each business data is accessed can be used as the popularity of the corresponding business data. In this solution, all business data with popularity is used as popularity data.

[0062] In this way, by calculating the popularity of business data, it is easier to store business data in layers according to popularity in subsequent processes, which facilitates the storage and retrieval of business data.

[0063] S11, the preset business database is divided into multiple levels to store the popularity data, and each level includes a storage capacity.

[0064] In an optional embodiment, dividing the preset business database into multiple levels to store the popularity data includes:

[0065] S111 divides the preset business database into multiple levels according to different storage capacities;

[0066] S112, Sort all levels according to the storage capacity from small to large, with the storage capacity corresponding to the earlier level being sorted being smaller;

[0067] S113, The popularity data is stored in the sorted hierarchy from front to back in descending order of popularity, with the higher the popularity data, the earlier the corresponding hierarchy is sorted.

[0068] In this optional embodiment, the preset database can be an LSM-tree based database. LSM-tree (Log-Structured Merge-Tree) is a key-value storage structure commonly used in the underlying storage engines of NoSQL database systems. When the data in memory of an LSM-tree accumulates to a certain size, the memory data is flushed to persistent storage as an SST (Sorted String Table), which is an ordered set of key-value pairs. SSTs are stored on the hard drive as files and are the primary persistent data form for LSM-trees.

[0069] In this optional embodiment, the preset business database is divided into multiple levels according to different storage capacities, and the storage capacity and the level correspond one-to-one. For example... Figure 4 As shown, this is the SST distribution pattern of each level of LSM-tree under the hierarchical strategy. The SST of level 0 is directly written from memory, and its size is mainly limited by the configuration related to the memory data size and the real-time load.

[0070] In this optional embodiment, all levels can be sorted in ascending order of storage capacity, with the earlier levels corresponding to smaller storage capacities. For example... Figure 4 As shown, starting from level 1, each level has a limited storage capacity, and the storage capacity of each level increases by a fixed multiple in ascending order of level. In LSM-tree, this fixed multiple is used as the fan-out multiple. In this scheme, the fan-out multiple of the storage capacity between adjacent levels is 10.

[0071] In this optional embodiment, the popularity data is stored sequentially from highest to lowest popularity into sorted levels. The higher the popularity of the popularity data, the higher the level of the sorted data. This is because in an LSM-tree, the lower the level of popularity data, the higher the priority of the data being accessed.

[0072] In this way, by storing popularity data in a pre-defined hierarchical business database, business data can be quickly accessed based on popularity, thereby improving the efficiency of business data retrieval.

[0073] S12, based on the storage capacity, all levels are filtered to obtain the boundary level.

[0074] In an optional embodiment, the step of filtering all levels based on the storage capacity to obtain the boundary level includes:

[0075] S121, according to the sorting result of the hierarchy, the storage capacity corresponding to each hierarchy is accumulated from front to back to obtain the accumulated capacity;

[0076] S122. Take the level whose cumulative capacity meets the preset capacity threshold range as the boundary level.

[0077] In this optional embodiment, the storage capacities corresponding to each level are accumulated in order from front to back according to the sorting result of the levels. Denote the total storage capacity between level 0 and level n as g(n). In this solution, the total storage capacity g(n) is used as the cumulative capacity. Taking Figure 4 as an example, then g(3) = 56 GB, g(4) = 556 GB, g(5) = 5556 GB.

[0078] In this optional embodiment, NVMe hard disks with large read / write throughput and high random read / write performance and relatively traditional mechanical hard disks (HDD) can be used to jointly store the hot data in the service database. Among them, since the cost of NVMe hard disks is higher, in order to maximize the storage efficiency of hot data, in this solution, NVMe hard disks are preferentially used to store hot data, and when the NVMe hard disks are fully loaded, the mechanical hard disks (HDD) are continued to be used to store the hot data.

[0079] In this optional embodiment, denote the standard storage capacity of the NVMe hard disk as s. In this solution, when g(n) <= s < g(n + 1), level n is taken as the boundary level. The reason is that the standard storage capacity s of the NVMe hard disk is difficult to meet the cumulative capacity corresponding to the next level of the boundary level n. At this time, the hot data included in levels 0 to n needs to be stored in the NVMe hard disk, and the hot data included in all levels after level n needs to be stored in the mechanical hard disk (HDD).

[0080] In this way, by screening the boundary level, the storage medium of the hot data can be adjusted in time to ensure that the service database always maintains a high storage efficiency.

[0081] S13. Obtain the scheduling level based on the boundary level to store the hot data in different storage media.

[0082] In an optional embodiment, the obtaining the scheduling level based on the boundary level to store the hot data in different storage media includes:

[0083] S131. Calculate the boundary capacity of the storage capacity corresponding to the boundary level, and the boundary capacity is used to represent the maximum capacity that the storage capacity corresponding to the boundary level can be elastically expanded;

[0084] S132. Compare the boundary capacity with the capacity of the first storage medium to obtain the scheduling level;

[0085] S133. Store the database where the heat data is located in different storage media based on the scheduling level. The storage media include a first storage medium and a second storage medium.

[0086] In this optional embodiment, the storage capacity corresponding to each level is only a reference value in the LSM-tree, that is, there is a certain flexible space for the storage capacity corresponding to each level. In an actual high-load scenario, the cumulative capacity g(n) corresponding to the nth level of the LSM-tree can even expand to three times the storage capacity of level n. Therefore, if the storage space of NVMe is exhausted, the operation of the LSM-tree for adding new SSTs to NVMe will receive an error message of "no available space" from the file system, which will trigger a series of emergency mechanisms such as "Write Stall" in the database program, resulting in a significant decrease in the performance of the hard disk.

[0087] In this optional embodiment, if the storage method of the heat data is directly judged according to the relationship g(n)<=s<g(n + 1), it may cause waste of the storage space of the storage medium. Exemplarily, denote the storage space of NVMe as 400GB and the storage space of HDD as 2TB. Since g(3)<400GB<g(4), all heat data from level 0 to level 3 will be stored as SSTs on NVMe, and all heat data included in level 4 and subsequent levels will be stored as SSTs on HDD. At this time, there is a theoretical waste of 400GB - 56GB = 344GB in the expensive NVMe storage space.

[0088] In this optional embodiment, to solve the adverse situations that the storage space of NVMe may be wasted and exhausted, in this solution, the boundary capacity of the storage capacity corresponding to the boundary level is calculated, and the scheduling level is obtained by comparing the boundary capacity and the capacity of the first storage medium. The scheduling level can dynamically adjust the SST between NVMe and HDD according to the real-time available storage space of NVMe.

[0089] In this optional embodiment, three times the cumulative capacity g(n) of the boundary level n is used as the boundary capacity of the boundary level, denoted as G(n). In this solution, the storage medium NVMe is used as the first storage medium, and the storage medium HDD is used as the second storage medium. Among them, if the boundary capacity is greater than the capacity s of the first storage medium, the boundary level is used as the scheduling level; if the boundary capacity is not greater than the capacity s of the first storage medium, the next level in the sorting result of the boundary level is used as the scheduling level, and the boundary level n still fixedly selects NVMe to store the heat data.

[0090] In this optional embodiment, the popularity data can be stored in a first storage medium and a second storage medium based on the scheduling level. Specifically, this solution only dynamically adjusts the popularity data within the scheduling level; for other levels below the scheduling level, the first storage medium (NVMe) is always used to store the corresponding popularity data; for other levels above the scheduling level, the second storage medium (NVMe) is always used to store the corresponding popularity data.

[0091] In this way, the storage space of the first storage medium NVMe can be effectively utilized while preventing the storage space of the first storage medium NVMe from being exhausted, thereby further improving the storage efficiency of the business database.

[0092] S14, the business data in the second storage medium is retrieved based on the first storage medium.

[0093] In an optional embodiment, the step of accessing business data in the second storage medium based on the first storage medium includes:

[0094] S141, allocate cache space in the first storage medium;

[0095] S142, Based on the cache space, store the metadata of the second storage medium to call the business data in the second storage medium.

[0096] In this optional embodiment, the LSM-tree principle dictates that it may be necessary to traverse and read the metadata of multiple SSTs before accurately reading the data or determining its existence. However, the second storage medium HDD has relatively low efficiency in reading random hot data, which results in a long retrieval time for hot data stored on the second storage medium HDD when it is called in the business database. The first storage medium NVMe does not have this problem.

[0097] In this optional embodiment, to address the issue of low read efficiency of the second storage medium (HDD) for random hot data, this solution allocates a specific cache space in the first storage medium to store the metadata of the second storage medium. This is because the metadata portion only occupies approximately 1% of the total storage space for the hot data, and caching the metadata portion of all SSTs on the HDD requires only a small amount of NVMe space.

[0098] like Figure 5As shown in this optional embodiment, the metadata portion includes a footer block, an index block, a metadata index block, and a metadata block. When retrieving popular data from the business database, it is necessary to first locate the index block based on the footer block, then locate the metadata index block based on the index block, then locate the metadata block corresponding to the popular data being retrieved from the metadata index block, and finally obtain the popular data being retrieved from the metadata block. Therefore, when writing SST to the second storage medium HDD, it is necessary to simultaneously update the metadata portion to the NVMe cache space. In this way, when reading SST metadata, it is possible to first select to retrieve it from the NVMe cache space, and if it is not found, then select to further retrieve it from the second storage medium HDD, thereby realizing and completing the hierarchical storage and fast retrieval of business data.

[0099] This effectively solves the problem of low access efficiency when reading hot data using the second storage medium HDD, and improves the data access efficiency of the business database.

[0100] Please see Figure 2 , Figure 2 This is a functional block diagram of a preferred embodiment of the AI-based hierarchical data storage device of this application. The AI-based hierarchical data storage device 11 includes a calculation unit 110, a partitioning unit 111, a filtering unit 112, an acquisition unit 113, and a retrieval unit 114. The module / unit referred to in this application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and perform a fixed function, and are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0101] In an optional embodiment, the calculation unit 110 is used to acquire business data and calculate the popularity of the business data to obtain popularity data.

[0102] In an optional embodiment, obtaining business data and calculating the popularity of the business data to obtain popularity data includes:

[0103] Retrieve the business data to be stored and count the frequency of each business data being accessed;

[0104] The frequency with which each business data is accessed is used as the corresponding business data popularity, and the business data with popularity is used as popularity data.

[0105] In this optional embodiment, large amounts of business data can be acquired through both active and passive acquisition. Active acquisition can be achieved by proactively reading business data from various business channels using preset scripts and configuration files. Passive acquisition can be performed using proprietary data transmission tools, commercial transmission tools, or open-source transmission tools.

[0106] In this optional embodiment, the frequency with which each business data is accessed within a preset period can be counted. The preset period can be a day, an hour, or other periods; no specific limitation is made in this solution. The data volume of each identical business data within the preset period can be used as the frequency with which that business data is accessed.

[0107] In this optional embodiment, the frequency with which each business data is accessed can be used as the popularity of the corresponding business data. In this solution, all business data with popularity is used as popularity data.

[0108] In an optional embodiment, the partitioning unit 111 is used to divide a preset business database into multiple levels to store the popularity data, and each level includes a storage capacity.

[0109] In an optional embodiment, dividing the preset business database into multiple levels to store the popularity data includes:

[0110] The pre-defined business database is divided into multiple levels based on different storage capacities;

[0111] All levels are sorted in ascending order of storage capacity, with the earlier the level is sorted, the smaller the corresponding storage capacity.

[0112] The popularity data is stored sequentially in descending order of popularity into the sorted levels, with the more popular the data appearing earlier in the sorted levels.

[0113] In this optional embodiment, the preset database can be an LSM-tree based database. LSM-tree (Log-Structured Merge-Tree) is a key-value storage structure commonly used in the underlying storage engines of NoSQL database systems. When the data in memory of an LSM-tree accumulates to a certain size, the memory data is flushed to persistent storage as an SST (Sorted String Table), which is an ordered set of key-value pairs. SSTs are stored on the hard drive as files and are the primary persistent data form for LSM-trees.

[0114] In this optional embodiment, the preset business database is divided into multiple levels according to different storage capacities, and the storage capacity and the level correspond one-to-one. For example... Figure 4 As shown, this is the SST distribution pattern of each level of LSM-tree under the hierarchical strategy. The SST of level 0 is directly written from memory, and its size is mainly limited by the configuration related to the memory data size and the real-time load.

[0115] In this optional embodiment, all levels can be sorted in ascending order of storage capacity, with the earlier levels corresponding to smaller storage capacities. For example... Figure 4 As shown, starting from level 1, each level has a limited storage capacity, and the storage capacity of each level increases by a fixed multiple in ascending order of level. In LSM-tree, this fixed multiple is used as the fan-out multiple. In this scheme, the fan-out multiple of the storage capacity between adjacent levels is 10.

[0116] In this optional embodiment, the popularity data is stored sequentially from highest to lowest popularity into sorted levels. The higher the popularity of the popularity data, the higher the level of the sorted data. This is because in an LSM-tree, the lower the level of popularity data, the higher the priority of the data being accessed.

[0117] In an optional embodiment, the filtering unit 112 is used to filter all levels based on the storage capacity to obtain the boundary level.

[0118] In an optional embodiment, the step of filtering all levels based on the storage capacity to obtain the boundary level includes:

[0119] Based on the sorting results of the hierarchy, the storage capacity corresponding to each hierarchy is accumulated from front to back to obtain the accumulated capacity;

[0120] The layers whose accumulated capacity meets the preset capacity threshold range are designated as boundary layers.

[0121] In this optional embodiment, the storage capacity corresponding to each level is accumulated sequentially from front to back according to the sorting result of the levels, and the total storage capacity from level 0 to level n is denoted as g(n). In this scheme, the total storage capacity g(n) is used as the accumulated capacity. Figure 4 For example, g(3) = 56GB, g(4) = 556GB, g(5) = 5556GB.

[0122] In this optional embodiment, NVMe hard drives with high read / write throughput and high random read / write performance and relatively traditional hard disk drives (HDDs) can be used to jointly store the hot data in the business database. Since NVMe hard drives are more expensive, this solution prioritizes using NVMe hard drives to store hot data in order to maximize storage efficiency, and continues to use HDDs to store hot data when the NVMe hard drives are fully loaded.

[0123] In this optional embodiment, the standard storage capacity of the NVMe hard disk is denoted as s. In this solution, when s satisfies g(n) <= s < g(n + 1), level n is used as the boundary level. The reason is that it is difficult for the standard storage capacity s of the NVMe hard disk to meet the cumulative capacity corresponding to the next level of the boundary level n. At this time, the hot data included in levels 0 to level n needs to be stored in the NVMe hard disk, and the hot data included in all levels after level n needs to be stored in a mechanical hard disk (HDD).

[0124] In an optional embodiment, the obtaining unit 113 is configured to obtain a scheduling level based on the boundary level to store the hot data in different storage media.

[0125] In an optional embodiment, the obtaining a scheduling level based on the boundary level to store the hot data in different storage media includes:

[0126] Calculating a boundary capacity of the storage capacity corresponding to the boundary level, where the boundary capacity is used to represent the maximum capacity that the storage capacity corresponding to the boundary level can be elastically expanded;

[0127] Comparing the boundary capacity with the capacity of the first storage medium to obtain a scheduling level;

[0128] Based on the scheduling level, storing the database where the hot data is located in different storage media, where the storage media includes a first storage medium and a second storage medium.

[0129] In this optional embodiment, the storage capacity corresponding to each level is only a reference value in the LSM-tree, that is, there is a certain elastic space for the storage capacity corresponding to each level. In an actual high-load scenario, the cumulative capacity g(n) corresponding to the nth level of the LSM-tree can even expand to three times the storage capacity of level n. Therefore, if the storage space of the NVMe is exhausted, the LSM-tree will receive an error message of "no available space" from the file system for subsequent new SST operations on the NVMe, thereby triggering a series of emergency mechanisms such as "Write Stall" in the database program, resulting in a significant decrease in the hard disk performance.

[0130] In this optional embodiment, if the storage method of the heat data is directly judged according to the relationship g(n) <= s < g(n + 1), it may cause waste of the storage space of the storage medium. Exemplarily, the storage space of the NVMe is recorded as 400GB, and the storage space of the HDD is 2TB. Since g(3) < 400GB < g(4), all heat data from level 0 to level 3 will be stored as SST on the NVMe, and all heat data included in level 4 and subsequent levels will be stored as SST on the HDD. At this time, there is a theoretical waste of 400GB - 56GB = 344GB in the expensive NVMe storage space.

[0131] In this optional embodiment, to solve the adverse situations that the storage space of the NVMe may be wasted and exhausted, in this solution, the boundary capacity corresponding to the boundary level is calculated, and the scheduling level is obtained by comparing the boundary capacity with the capacity of the first storage medium. The scheduling level can dynamically adjust the SST between the NVMe and the HDD according to the real-time available storage space of the NVMe.

[0132] In this optional embodiment, three times the cumulative capacity g(n) corresponding to the boundary level n is used as the boundary capacity of the boundary level, denoted as G(n). In this solution, the storage medium NVMe is used as the first storage medium, and the storage medium HDD is used as the second storage medium. Among them, if the boundary capacity is greater than the capacity s of the first storage medium, the boundary level is used as the scheduling level; if the boundary capacity is not greater than the capacity s of the first storage medium, the next level of the boundary level in the sorting result is used as the scheduling level, and the boundary level n still fixedly selects the NVMe to store the heat data.

[0133] In this optional embodiment, the heat data can be stored in the first storage medium and the second storage medium respectively based on the scheduling level. Among them, in this solution, only the heat data in the scheduling level is dynamically adjusted. For the remaining levels lower than the scheduling level, the first storage medium NVMe is fixedly selected to store the corresponding heat data; for the remaining levels higher than the scheduling level, the second storage medium HDD is fixedly selected to store the corresponding heat data.

[0134] In an optional embodiment, the calling unit 114 is used to call the service data in the second storage medium based on the first storage medium.

[0135] In an optional embodiment, the calling of the service data in the second storage medium based on the first storage medium includes:

[0136] Create a cache space in the first storage medium;

[0137] The cache space stores the metadata of the second storage medium to access the business data in the second storage medium.

[0138] In this optional embodiment, the LSM-tree principle dictates that it may be necessary to traverse and read the metadata of multiple SSTs before accurately reading the data or determining its existence. However, the second storage medium HDD has relatively low efficiency in reading random hot data, which results in a long retrieval time for hot data stored on the second storage medium HDD when it is called in the business database. The first storage medium NVMe does not have this problem.

[0139] In this optional embodiment, to address the issue of low read efficiency of the second storage medium (HDD) for random hot data, this solution allocates a specific cache space in the first storage medium to store the metadata of the second storage medium. This is because the metadata portion only occupies approximately 1% of the total storage space for the hot data, and caching the metadata portion of all SSTs on the HDD requires only a small amount of NVMe space.

[0140] like Figure 5 As shown in this optional embodiment, the metadata portion includes a footer block, an index block, a metadata index block, and a metadata block. When retrieving popular data from the business database, it is necessary to first locate the index block based on the footer block, then locate the metadata index block based on the index block, then locate the metadata block corresponding to the popular data being retrieved from the metadata index block, and finally obtain the popular data being retrieved from the metadata block. Therefore, when writing SST to the second storage medium HDD, it is necessary to simultaneously update the metadata portion to the NVMe cache space. In this way, when reading SST metadata, it is possible to first select to retrieve it from the NVMe cache space, and if it is not found, then select to further retrieve it from the second storage medium HDD, thereby realizing and completing the hierarchical storage and fast retrieval of business data.

[0141] As can be seen from the above technical solutions, this application can effectively improve the storage efficiency of the business database by dividing the preset business database into multiple levels and storing business data hierarchically by defining boundary levels and scheduling levels. At the same time, it can optimize the data retrieval method of the second storage medium using the first storage medium.

[0142] Please see Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement the data hierarchical storage method based on artificial intelligence described in any of the above embodiments.

[0143] In an alternative embodiment, the electronic device 1 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as an AI-based data tiered storage program.

[0144] Figure 3 Only electronic device 1 with memory 12 and processor 13 is shown. It will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0145] Combination Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer-readable instructions to implement an artificial intelligence-based hierarchical data storage method, and the processor 13 can execute the plurality of instructions to achieve:

[0146] Acquire business data and calculate the popularity of the business data to obtain popularity data;

[0147] The preset business database is divided into multiple levels to store the popularity data, and each level includes a storage capacity.

[0148] Based on the storage capacity, all levels are filtered to obtain the boundary level;

[0149] Based on the boundary level, the scheduling level is obtained to store the heat data in different storage media, the storage media including a first storage media and a second storage media;

[0150] The service data in the second storage medium is accessed based on the first storage medium.

[0151] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0152] Those skilled in the art will understand that the schematic diagram is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. Electronic device 1 can be a bus-type structure or a star-type structure. Electronic device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, electronic device 1 may also include input / output devices, network access devices, etc.

[0153] It should be noted that electronic device 1 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0154] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a data hierarchical storage program based on artificial intelligence, but also to temporarily store data that has been output or will be output.

[0155] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a data hierarchy storage program based on artificial intelligence), and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0156] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the AI-based hierarchical data storage method described above, for example... Figure 1 The steps are shown.

[0157] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a calculation unit 110, a partitioning unit 111, a filtering unit 112, an acquisition unit 113, and a calling unit 114.

[0158] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the AI-based hierarchical data storage method described in the various embodiments of this application.

[0159] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0160] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0161] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0162] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0163] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0164] This application also provides a computer-readable storage medium (not shown), which stores computer-readable instructions. These computer-readable instructions are executed by a processor in an electronic device to implement the AI-based hierarchical data storage method described in any of the above embodiments.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0166] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0168] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A data hierarchical storage method based on artificial intelligence, characterized in that, The method includes: Acquire business data and calculate the popularity of the business data to obtain popularity data; The preset business database is divided into multiple levels to store the popularity data, and each level includes a storage capacity. The process of filtering all levels based on the storage capacity to obtain boundary levels includes: accumulating the storage capacity of each level from front to back according to the sorting result of the levels to obtain the accumulated capacity; and taking the level whose accumulated capacity meets the preset capacity threshold range as the boundary level. Based on the boundary level, a scheduling level is obtained to store the popularity data in different storage media. The storage media include a first storage media and a second storage media. The process of obtaining the scheduling level based on the boundary level to store the popularity data in different storage media includes: calculating the boundary capacity of the storage capacity corresponding to the boundary level, whereby the boundary capacity characterizes the maximum capacity that the storage capacity corresponding to the boundary level can elastically expand; comparing the boundary capacity with the capacity of the first storage media to obtain the scheduling level; and storing the database containing the popularity data in different storage media based on the scheduling level. The service data in the second storage medium is accessed based on the first storage medium.

2. The data hierarchical storage method based on artificial intelligence as described in claim 1, characterized in that, The process of acquiring business data and calculating the popularity of that business data to obtain popularity data includes: Retrieve the business data to be stored and count the frequency of each business data being accessed; The frequency with which each business data is accessed is used as the corresponding business data popularity, and the business data with popularity is used as popularity data.

3. The data hierarchical storage method based on artificial intelligence as described in claim 1, characterized in that, The step of dividing the preset business database into multiple layers to store the popularity data includes: The preset business database is divided into multiple levels according to different storage capacities, and the storage capacity and the level correspond one-to-one. All levels are sorted in ascending order of storage capacity, with the earlier the level is sorted, the smaller the corresponding storage capacity. The popularity data is stored sequentially in descending order of popularity into the sorted levels, with the more popular the data appearing earlier in the sorted levels.

4. The data hierarchical storage method based on artificial intelligence as described in claim 1, characterized in that, The step of comparing the boundary capacity with the capacity of the first storage medium to obtain the scheduling level includes: If the boundary capacity is greater than the capacity of the first storage medium, then the boundary level is used as the scheduling level. If the boundary capacity is not greater than the capacity of the first storage medium, then the next level in the sorting result of the boundary level is taken as the scheduling level.

5. The data hierarchical storage method based on artificial intelligence as described in claim 1, characterized in that, The step of accessing business data in the second storage medium based on the first storage medium includes: Allocate cache space in the first storage medium; The cache space stores the metadata of the second storage medium to access the business data in the second storage medium.

6. A data hierarchical storage device based on artificial intelligence, characterized in that, The apparatus is used to implement the data hierarchical storage method based on artificial intelligence as described in any one of claims 1 to 5, the apparatus comprising: A calculation unit is used to acquire business data and calculate the popularity of the business data to obtain popularity data; A partitioning unit is used to divide a preset business database into multiple levels to store the popularity data, and each level includes a storage capacity. A filtering unit is used to filter all levels based on the storage capacity to obtain the boundary level; The acquisition unit is used to acquire the scheduling level based on the boundary level in order to store the heat data in different storage media, the storage media including a first storage medium and a second storage medium; The invocation unit is used to invoke business data in the second storage medium based on the first storage medium.

7. An electronic device, characterized in that, The electronic device includes: Memory, which stores computer-readable instructions; and The processor executes computer-readable instructions stored in the memory to implement the AI-based hierarchical data storage method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the data hierarchical storage method based on artificial intelligence as described in any one of claims 1 to 5.

Images