Data access method and computing device

Abstract

Embodiments of the present application provide a data access method and a computing device. The method comprises: in the case that all read requests access first version index data, detecting the index data version update, and performing the following flow switching operation; wherein the first version is the current latest version; following the index data version update, controlling the read requests to be gradually switched from few to many to access the latest version index data, and controlling the remaining read requests to access the historical version index data; and in the case that all read requests are switched to access the latest version index data, stopping the execution of the flow switching operation. The technical solution provided by the embodiments of the present application reduces the influence on the data reading efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data access method and a computing device. BACKGROUND

[0002] The storage system adopting the LSM-Tree (The Log-Structured Merge-Tree) architecture writes data in an append manner without updating the original data. The write operation is first written to the memory, and when the memory data reaches a corresponding threshold, it is frozen as a layer and then written to the persistent storage medium and merged with the data in the persistent storage medium. The data on the persistent storage medium can also be stored in layers according to the hierarchical organization, and the data in each layer is merged with the data in the next layer when it reaches a corresponding threshold. All the written data, whether in the memory or in the persistent storage medium, is sorted by the key and then stored.

[0003] An existing storage system adopting the LSM-Tree architecture stores data in the form of data blocks in the persistent storage medium, and establishes index information for each data block to quickly locate the data block. The index information of the data block and the memory data are organized to form index data, and it can be known that the data write operation and the data merge operation update the index data. In addition, in order to improve the data reading efficiency, the hot data block in the persistent storage medium is cached to the memory. When a read request for target data corresponding to a target key is received, the memory data recorded in the index data is first accessed, and if the target data does not exist in the memory data, the target data block where the target data is located is determined based on the index information of the data block. Then, the cached data is first read, and if the target data block exists in the cached data, the target data can be read therefrom, and if not, the persistent storage medium is read layer by layer until the target data block is found to read the target data.

[0004] Since the merge operation reorders the data of adjacent layers to generate a new data block and updates the index data, and in the case that the cached data is not updated, if a read request is received, based on the updated index data, the target data is in the new data block, and the cached data does not cache the new data block, the persistent storage medium is read, which may cause a large number of read requests to access the persistent storage medium, resulting in read performance jitter, read delay, and affecting data reading efficiency. SUMMARY

[0005] Embodiments of the present application provide a data access method and a computing device to solve the technical problem of affecting data reading efficiency in the prior art.

[0006] In a first aspect, the embodiments of the present application provide a data access method, comprising:

[0007] In the case that all read requests access the first version index data, detecting the index data version update, and performing the following stream switching operation; wherein the first version is the current latest version;

[0008] Following the index data version update, gradually switching the read requests from few to many to access the latest version index data, and controlling the remaining read requests to access the historical version index data;

[0009] In the case that all read requests are switched to access the latest version index data, stopping the execution of the stream switching operation.

[0010] Optionally, after stopping the stream switching operation, the method further comprises:

[0011] Deleting the historical version index data.

[0012] Optionally, further comprising:

[0013] Releasing the data blocks in the persistent storage medium that are not referenced by any version index data.

[0014] Optionally, the controlling the remaining read requests to access the historical version index data comprises:

[0015] Controlling the remaining read requests to access the first version index data.

[0016] Optionally, further comprising:

[0017] After stopping accessing any historical version index data, deleting the historical version index data.

[0018] Optionally, the stream switching operation further comprises:

[0019] Determining whether to write new data;

[0020] If yes, controlling all read requests to switch to access the latest version index data;

[0021] If no, following the index data version update, gradually switching the read requests from few to many to access the latest version index data, and controlling the remaining read requests to access the historical version index data;

[0022] Optionally, the stream switching operation further comprises:

[0023] Following the index data version update, determining the stream switching ratio corresponding to each version update;

[0024] According to the split ratio, a corresponding number of read requests are controlled to switch to access the latest version of the index data;

[0025] The rest of the read requests are controlled to access the historical version of the index data.

[0026] Optionally, determining the split ratio corresponding to each version update of the index data according to the number of version updates of the index data comprises:

[0027] According to the equal proportion increasing principle, the split ratio corresponding to each version update of the index data is determined based on an incremental ratio starting from zero.

[0028] Optionally, the method further comprises:

[0029] Adjusting the incremental ratio in combination with the current flush operation frequency and / or compression operation frequency.

[0030] In a second aspect, an embodiment of the present application provides a computing device, comprising a storage component and a processing component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component; and the processing component is used to implement the data access method according to the first aspect.

[0031] In a third aspect, an embodiment of the present application provides a storage engine, comprising:

[0032] A detection module is configured to detect the version update of the index data in the case that all read requests access the first version of the index data, and trigger the execution of the split operation;

[0033] A split module is configured to execute the split operation to control the read requests to gradually switch to access the latest version of the index data from few to many, and control the rest of the read requests to access the historical version of the index data, and stop the execution of the split operation in the case that all read requests switch to access the latest version of the index data.

[0034] In a fourth aspect, an embodiment of the present application provides a storage system based on the LSM-tree architecture, comprising a persistent storage medium and the storage engine according to the third aspect.

[0035] In the embodiments of the present application, in the case that all read requests access the first version index data, wherein the first version is the current latest version, the index data version update is detected, and the following cut-flow operation is performed; following the version update of the index data, the read requests are gradually switched from few to many to access the latest version index data, and the remaining read requests access the historical version index data; if it is detected that all read requests are switched to access the latest version index data, the cut-flow operation is stopped. According to the principle of from few to many, with the version update, the read traffic is gradually migrated to access the latest version index data, part of the read requests still access the historical version index data, and all traffic is directly migrated to the latest version index data, and in the gradual migration process, the cache data is updated accordingly, achieving the effect of gradually warming up the cache data, which can avoid the read delay caused by the read performance jitter of directly accessing the latest version index data, and improves the data read efficiency.

[0036] These aspects or other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 A flow chart of one embodiment of a data access method provided by the present application is shown;

[0039] Figure 2 A cut-flow schematic diagram of an actual application of the embodiments of the present application is shown;

[0040] Figure 3 A flow chart of another embodiment of a data access method provided by the present application is shown;

[0041] Figure 4 A structural schematic diagram of one embodiment of a storage engine provided by the present application is shown;

[0042] Figure 5 A structural schematic diagram of one embodiment of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0043] In order to make those skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0044] In some of the flowcharts described in the specification and claims of the present application and in the above-described figures, a plurality of operations are included which occur in a particular order, but it should be clearly understood that the operations can be performed in the order in which they appear herein or in parallel, and the serial numbers of the operations, such as 101, 102, etc., are merely used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, the flowcharts can include more or fewer operations, and the operations can be performed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. herein are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do "first" and "second" represent different types.

[0045] The technical solutions of the embodiments of the present application are mainly applied to a storage system adopting an LSM-Tree (The Log-Structured Merge-Tree) architecture.

[0046] In order to facilitate understanding of the technical solutions of the present application, the technical terms that can appear in the embodiments of the present application are first explained as follows:

[0047] Key-Value (key-value) storage system: a storage system that stores data in the form of key-value pairs.

[0048] LSM-Tree: a data storage structure suitable for a Key-Value storage system, which writes data in an append manner, and the write operation is first inserted into memory, and after the memory data reaches the corresponding threshold, it is written to a persistent storage medium. The persistent storage medium stores data in layers, and after the data of each layer reaches the corresponding threshold, it is written to the next layer. Therefore, data is written in an append manner, and there can be multiple versions of data corresponding to the same key, with the new version of data located at a higher level and the old version of data possibly located at a lower level. The read operation starts from the highest level and sequentially searches for the latest version of data from lower levels, and terminates when the latest version of data is found.

[0049] LSM-tree storage system: a storage system adopting an LSM-tree architecture.

[0050] compaction (compression): LSM-tree storage system, due to the use of append manner to write data, there can be data redundancy, so that data can be merged (merge) through compaction operation, Compaction operation will constantly merge the data of adjacent levels and write to a lower level. Specifically, the data of adjacent two or more layers to be merged can be read out, sorted by key, and if there are multiple versions of the same key, only the new version of data is retained and the old version of data is deleted, and then written to a lower layer.

[0051] Active-Memtable: In LSM-tree storage system, active memory table in memory, write operation will first be written into the active memory table in memory.

[0052] Immutable-Memtable: In LSM-tree storage system, frozen memory table in memory. After the active memory table is full, the active memory table will be switched to the frozen memory table. Then, the frozen memory table will be flushed to the persistent storage medium.

[0053] Persistent storage medium: Storage device for persistent storage of data, such as disk, optical disc, etc. The data on the persistent storage medium in the LSM-tree storage system is stored in layers according to the hierarchical organization, assuming that it includes L0, L1, L2 three layers, the flush operation will split the data in the frozen memory table into data blocks and flush them into the L0 layer, and can be merged with the L0 layer data through the Compaction operation. L0, L1, L2 will also be merged through the Compaction operation.

[0054] Data block: Each layer of the persistent storage medium stores data in the form of data blocks. In order to quickly locate the data block, each data block will set the corresponding index information to index, which can be specifically the metadata of the data block, which can include data block size, data block identifier, key value range, etc. It can be seen that flush operation and Compaction operation may generate new data blocks, but will not introduce new data, only switch operation will introduce new data.

[0055] Index data: In order to facilitate data management, the current LSM-tree storage system is composed of memory data (such as Active-Memtable and Immutable-Memtable) and index information of each data block. The read operation will first access the index data, first find the Active-Memtable, then find the Immutable-Memtable, then locate the data block based on the index information, and then find each layer in the persistent storage medium. It can be seen that switch operation, flush operation and Compaction operation will update the index data. Taking the persistent storage medium including L0, L1, L2 three layers as an example, before and after the index data is updated, Active-Memtable and Immutable-Memtable may be different, Immutable-Memtable and L0 may be different, L0 may be different, L0 and L1 may be different, L1 and L2 may be different, and L2 may be different.

[0056] Cache data: In order to improve the data read efficiency, the data block in the persistent storage medium that meets certain conditions such as high access frequency or latest writing is cached as a hot data block in the memory, which is called cache data. After the read operation is located to a certain data block based on the index information, the cache data is first searched. If the cache data hits the data block, it can be directly read, otherwise the each layer in the persistent storage medium is searched again.

[0057] Reference: The index information of the data block is recorded in the index data, which means that the data block is referenced. Since the compaction operation generates new data blocks and old data blocks, for example, it is assumed that data block 1 of L0 and data block 2 of L1 layer are merged to generate data block 3 and data block 4, and data block 3 and data block 4 are written to L1 layer. Among them, data block 3 and data block 4 are new data blocks, data block 1 of L0 layer and data block 2 of L1 layer are old data blocks, and the old data blocks will not be released immediately. Since the index data update will generate different versions of index data, a data block may be referenced by multiple versions of index data, so the number of index data referencing the data block can be indicated by the reference count. If the reference count of a certain data block is 0, it means that it is not referenced by any version of index data, and at this time the data block will be released.

[0058] Cache invalidation: If the data block in the cache data is not referenced by a certain index data, the data read through accessing the index data will fail, and the cache data will be invalidated with respect to the index data.

[0059] As can be known from the foregoing description, since the compaction operation will reorder the data of adjacent layers, generate new data blocks, and update the index data. However, the cache data will not be updated immediately. If a read request is received, based on the updated index data, if the target data requested for reading is located in the new data block, and the cache data does not cache the new data block, it will be read from the persistent storage medium. If there are a large number of such read requests accessing the persistent storage medium, it will cause the read performance to fluctuate, produce read delay, and affect the data read efficiency.

[0060] In order to improve the data reading efficiency, the inventors find that in the LSM-tree storage system, the switch operation, the flush operation and the compaction operation all update the index data, but only the switch operation introduces new data into the storage system, and the flush operation and the compaction operation do not introduce new data, but only change the data organization structure, for example, a certain data is located in data block A, and after the compaction operation, the data is located in data block B, and the storage system simultaneously exists data block A and data block B, and the data can be obtained from data block A or data block B. Accordingly, the inventors think that after the flush operation or the compaction operation, whether the historical version of the index data can be temporarily retained, and in the case of generating new version index data, the historical version of the index data is not deleted, so that the old data block in the cache data will still be referenced, and will not be immediately released from the persistent storage medium, the old data block is referenced by the historical version of the index data and is not referenced by the new version of the index data, and the new data is referenced by the new version of the index data, and in the case that the target data to be accessed exists in the old data and the new data, then for the same read operation, the target data can be successfully read whether the new version of the index data or the historical version of the index data is accessed, and the historical version of the index data can be directly obtained from the cache data, which can ensure the data reading efficiency, and the cache invalidation of the new version data requires reading the persistent storage medium. However, the inventors also think that the cache data will also be updated, and after the flush operation or the compaction operation, the historical version of the index data is still accessed, and with the passage of time, the cache data is updated, and the reading from the cache data will fail, and the cache data is invalidated relative to the historical version of the index data, and the persistent storage medium still needs to be read.

[0061] According to the above application process, the inventor has proposed the technical scheme of the present application through a series of researches. In the embodiment of the present application, in the case that all read requests access first version index data, wherein the first version is the current latest version, the index data version update is detected, and the following flow switching operation is performed; following the version update of the index data, the read requests are gradually switched from few to many to access the latest version of the index data, and the remaining read requests access the historical version of the index data; if it is detected that all read requests are switched to access the latest version of the index data, the flow switching operation is stopped. According to the principle of gradually migrating from few to many, the read traffic is gradually migrated to access the latest version of the index data, and part of the read requests still access the historical version of the index data, and all the traffic is directly migrated to the latest version of the index data. During the gradual migration process, the cache data is updated accordingly, achieving the effect of gradually warming up the cache data. The read delay caused by the read performance jitter of directly accessing the latest version of the index data can be avoided, the data read efficiency is improved, and the cache hit rate is improved.

[0062] The technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0063] Figure 1 A flowchart of one embodiment of the data access method provided in the embodiments of the present application, the technical scheme of the present embodiment can be executed by a storage engine, and the method can include the following steps:

[0064] 101: In the case that all read requests access first version index data, detect index data version update, and perform step 102 flow switching operation.

[0065] The first version index data refers to the current latest version index data, which is named as the first version index data for convenience of description.

[0066] All read requests can refer to all read requests received at one time.

[0067] The index data version update can be achieved by detecting whether a new version of index data is generated or detecting whether a merge operation is performed. Specifically, it can be detected whether a flush operation or a compaction operation is performed, and if a flush operation or a compaction operation is performed, it can be considered that the index data version is updated.

[0068] 102: Following the version update of the index data, gradually switching from few to more read requests to access the latest version of the index data, and controlling the remaining read requests to access the historical version of the index data.

[0069] The remaining read requests can refer to the read requests received except for accessing the latest version of the index data.

[0070] Following the version update of the index data, for a read request received at a time, gradually switching from few to more read requests to access the latest version of the index data, and controlling the remaining read requests to access the historical version of the index data. Starting from the first version, as the number of version updates increases, more and more read requests access the latest version of the index data until all access the latest version of the index data, and fewer and fewer read requests access the historical version of the index data until the access amount is zero.

[0071] Optionally, after each version update of the index data, for a read request received at a time, a first number of read requests can be controlled to access the latest version of the index data, and a second number of read requests remaining outside the first number of read requests can be controlled to access the historical version of the index data. Among them, the first number starts from 0 and increases with the increase of the number of version updates, and the second number decreases with the increase of the number of version updates until it is 0. The number of first number increased each time can be the same or different, and the number of second number decreased each time can be the same or different.

[0072] 103: Detecting the case where all read requests are switched to access the latest version of the index data, and stopping the execution of the flow splitting operation.

[0073] That is, when the flow splitting operation causes the second number to be 0 and all read traffic accesses the latest version of the index data, the flow splitting operation can be stopped. At this time, the latest version of the index data will be used as the first version of the index data, and the step 101 will continue to be executed.

[0074] Among them, the data reading operation performed after the read request accesses the index data is the same as the traditional way, for example, the read request can carry a target key corresponding to the target data to be read. First, read the memory data in the index data, if the target data corresponding to the target key exists, it is ended, if not, based on the target key and the key value range in the index information, the target data block corresponding to the target key can be determined, and then it can be first searched in the cache data to see if the target data block exists, if yes, it is directly read, otherwise, it is read from the persistent storage medium layer by layer, the specific reading process is not repeated here.

[0075] In this embodiment, the read traffic is gradually migrated to the latest version index data according to the principle of from less to more, part of the read requests still access the historical version index data, and all the traffic is directly migrated to the latest version index data. During the gradual migration process, the cache data is updated accordingly, achieving the effect of gradually warming up the cache data. This can avoid the read delay caused by the read performance jitter of directly accessing the latest version index data, improve the data read efficiency, and improve the cache hit rate.

[0076] For example, if the target data to be read is located in the cache data, it is located in data block 1 before the merging operation, and data block 1 is also stored in the cache data. After the merging operation, it is located in data block 2, and the index data after the merging operation no longer references data 1. The target data can be obtained from data block 1 of the cache data using the index data before the merging operation without reading the persistent storage medium, and the target data can only be obtained from data block 2 of the persistent storage medium using the index data after the merging operation. According to the technical solution of the present application, most of the read traffic accesses the historical version index data initially, which can ensure the data read efficiency. As the version is updated, the cache data is also updated gradually, and data block 2 will be saved to the cache data as the access heat increases. Most of the read traffic accesses the latest version index data, which can obtain the target data directly from the cache data with a higher probability, improve the cache hit rate, and ensure the data read efficiency. Therefore, gradually warming up the cache data according to the technical solution of the present application can reduce the read performance jitter and improve the data read efficiency.

[0077] In some embodiments, after stopping the flow splitting operation, the method can further include:

[0078] Deleting the index data of the historical version.

[0079] In order to avoid data redundancy and improve storage performance, in addition, the data block not referenced by any version index data can be released.

[0080] After deleting the index data of the historical version, the data block referenced by each version is no longer referenced, and the reference count can be reduced accordingly. When the reference count is 0, it can indicate that the data block is not referenced by any version index data, and therefore the data block with a reference count of 0 can be released, that is, the data block with a reference count of 0 is deleted from the storage system.

[0081] In the control of the rest of the read request access history version index data, as an optional way, the storage system includes a plurality of history version index data, which can be divided into different groups of read requests accessing different history version index data, that is, the read traffic is shared by a plurality of history version index data. The rest of the read request can be equally divided according to the number of history version index data, and the different history version index data is equally divided by the read request. Of course, it can also be unevenly divided, and the read request from small to large is sequentially borne according to the time sequence of the history version index data.

[0082] As another optional way, the rest of the read request can be controlled to access the same history version index data.

[0083] Optionally, for different times of index data version update, the rest of the read request can be controlled to access the same history version index data.

[0084] In addition, as another optional way, the rest of the read request can be controlled to access the first version index data.

[0085] That is, after each index data version update, the rest of the read request can be controlled to access the first version index data.

[0086] In some embodiments, the method can further include:

[0087] After stopping accessing any history version index data, the history version index data is deleted.

[0088] If the rest of the read request accesses the first version index data, the history version index data between the first version index data and the latest version index data can be deleted, reducing the occupation of the storage space. Due to the deletion of the index data, the reference of the data block will be changed, and the data block not referenced by any version index data can be deleted, further reducing the occupation of the storage space.

[0089] In some embodiments, when performing the cut stream operation, it can be determined first whether new data is written;

[0090] If yes, control all read requests to switch to access the latest version index data;

[0091] If no, follow the version update of the index data, control the read request from small to large to gradually switch to access the latest version of the index data, and control the rest of the read request to access the history version index data;

[0092] As an optional way, whether new data is written can be determined by comparing the index data of the adjacent two versions.

[0093] Since the active memory table is included in the index data, by comparing the active memory tables in the index data of two adjacent versions, if they are different, it can be determined that new data is written, otherwise new data is not written.

[0094] As another optional way, it can also be determined whether the switch operation exists, if yes, it can be determined that data is written, otherwise it is considered that new data is not written.

[0095] Among them, following the version update of the index data, it can be determined that after each index data version update, the first quantity corresponding to the latest version of the index data and the second quantity corresponding to the historical version data, the corresponding access quantity can be used to control the corresponding number of read requests to access different versions of index data. The first quantity and the second quantity corresponding to each index data version update can be pre-set, or can be realized by other means.

[0096] As an optional implementation, the flow switching operation can include:

[0097] Following the version update of the index data, determine the flow switching ratio corresponding to each version update;

[0098] According to the flow switching ratio, control the corresponding number of read requests to switch to access the latest version of the index data;

[0099] Control the remaining read requests to access the historical version of the index data.

[0100] Among them, with the increase of the number of version updates, the flow switching ratio increases, and the increase ratio can be the same or different each time.

[0101] Suppose that after a version update, 1 million read requests are received, and the flow switching ratio is 30%, then the access quantity corresponding to the flow switching ratio is 300,000 read requests. Among them, 300,000 read requests will be switched to access the latest version of the index data, and the remaining 700,000 read requests will still access the historical version of the index data.

[0102] Optionally, following the version update of the index data, the flow switching ratio corresponding to each version update can be determined based on the increasing ratio starting from zero according to the equal ratio increase principle.

[0103] Through equal ratio increase, the read delay rate after each index data version update can be balanced, and the data read efficiency after each index data version update can be ensured to be similar, so as to ensure the user reading experience.

[0104] The increment ratio can be set in combination with actual conditions, or can be dynamically adjusted. The increment ratio can be determined in combination with the current flush frequency and / or compaction frequency, and therefore, in some embodiments, the method can further include adjusting the increment ratio in combination with the current flush frequency and / or compaction frequency.

[0105] In addition, the flush data amount and the merged data amount can also be determined in combination. The flush data amount can refer to the amount of data written into the persistent storage medium by the flush operation, and the merged data amount can refer to the amount of data merged by the compaction operation.

[0106] The increment ratio corresponding to different flush operation frequencies, different flush data amounts, different compaction operation frequencies, and different merged data amounts can be preset.

[0107] Optionally, the total switch time can also be determined in combination with the switch operation frequency, and then the increment ratio can be determined based on the total switch time and the current flush operation frequency or compaction operation frequency.

[0108] For example, based on the flush operation frequency and the total switch time, the version update prediction times within the total switch time can be determined. Similarly, based on the compaction operation frequency and the total switch time, the version update prediction times within the total switch time can also be determined. Then, the increment ratio can be determined according to the version update prediction times. For example, if the version update times are 4 times, the increment can be set to be greater than or equal to 25%, for example, 30%.

[0109] Of course, the input operation frequency and the compaction operation frequency can also be averaged or minimized as the predetermined frequency, and the version update prediction times can be determined based on the predetermined frequency and the total switch time. For example, if the predetermined frequency is to update once every 20 seconds, and the total switch time is 60 seconds, the version update prediction times are 3 times, and so on.

[0110] It should be noted that the above is only an example of a possible determination method of the increment ratio, and the present application does not specifically limit this.

[0111] For ease of understanding, it is assumed that the increment ratio is 30%, the index data of the first version is SV0, the index data SV1 is obtained after version update based on SV0, the index data SV2 is obtained after version update based on SV1, the index data SV3 is obtained after version update based on SV2, and so on, until the index data SVm is obtained. m where m is a positive integer. In combination with Figure 2 The switch ratio diagram is shown in the following figure:

[0112] When all read requests access SV0, the flush or compaction operation is detected to generate SV1, at which time the stream switching operation can be started to switch to SV1, and the stream switching ratio corresponding to SV1 is 0+30%=30%, that is, after SV1 is generated, 30% of the read requests received at one time access SV1, and 70% of the read requests still access SV0.

[0113] After that, the flush or compaction operation is continued to generate SV2, and the stream switching ratio corresponding to SV2 is 30%+30%=60%, at which time the switching to SV1 can be stopped and switching to SV2 is turned on, that is, after SV1 is generated, 60% of the read requests received at one time access SV1, and 40% of the read requests still access SV0.

[0114] After that, the flush or compaction operation is continued to generate SV3, and the stream switching ratio corresponding to SV3 is 60%+30%=90%, at which time the switching to SV2 can be stopped and switching to SV3 is turned on, that is, after SV3 is generated, 90% of the read requests received at one time access SV3, and 10% of the read requests still access SV0.

[0115] After that, the flush or compaction operation is continued to generate SV4, and the stream switching ratio corresponding to SV4 is 100%, at which time the switching to SV3 can be stopped and switching to SV4 is turned on, that is, after SV4 is generated, all read requests received at one time access SV4, and the access amount of SV0 is 0. The stream switching operation is completed, at which time the stream switching operation can be stopped, and the current SV4 can continue to perform the above operation as SV0.

[0116] Among them, it is assumed that after SV2 is obtained, the switch operation is performed to generate SV5, at which time, due to the introduction of new data by the switch operation, all read requests received at one time will be switched to access SV5, and the access amount of SV0 is 0, so as to ensure normal data access.

[0117] In an actual application, the data access process can be performed according to the flowchart shown in Figure 3 The data access method shown in Figure 3 may include the following steps:

[0118] 301: When all read requests access the first version of index data, detect index data version update and perform stream switching operation. The stream switching operation can include the following steps 302-305.

[0119] 302: Determine whether to write new data, if yes, execute step 303, if no, execute step 304.

[0120] 303: Determine the proportion of the cut stream corresponding to each version update.

[0121] 304: According to the proportion of the cut stream, control the corresponding number of read requests to switch to access the latest version of index data.

[0122] 305: Control the remaining read requests to access the first version of index data.

[0123] 306: Control all read requests to switch to access the latest version of index data.

[0124] 307: Detect that all read requests switch to access the latest version of index data, stop executing the cut stream operation.

[0125] 308: After stopping accessing any historical version of index data, delete the historical version of index data.

[0126] 309: Release the data block in the persistent storage medium which is not referenced by any version of index data.

[0127] The specific operations of each step can be found in the corresponding embodiments described above, which will not be repeated here.

[0128] The embodiments of the present application also provide a storage engine, as shown in Figure 4 The storage engine can include:

[0129] The detection module 401 is configured to detect the version update of the index data when all read requests access the first version of index data, and trigger the execution of the cut stream operation.

[0130] The cut stream module 402 is configured to execute the cut stream operation to follow the version update of the index data, control the read requests to gradually switch from few to many to access the latest version of index data, and control the remaining read requests to access the historical version of index data; and detect that all read requests switch to access the latest version of index data, and stop executing the cut stream operation.

[0131] Figure 4 The storage engine can execute Figure 1 or Figure 3 The implementation principle and technical effects of the data access method described in the embodiments will not be repeated. For the specific manner of operation of each module and unit of the storage engine in the above embodiments, it has been described in detail in the embodiments related to the method, which will not be described in detail here.

[0132] In one possible design,Figure 4 The storage engine of the illustrated embodiment can be configured in a computing device, thus as Figure 5 illustrated, the embodiments of the present application also provide a computing device, which can include a storage component 501 and a processing component 502; the storage component 501 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 502, and can implement the data access method of the embodiments of the present application as Figure 1 or Figure 3 illustrated.

[0133] The processing component 502 can include one or more processors to execute the computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic elements, for executing the above method.

[0134] The storage component 501 is configured to store various types of data to support operations on the computing device. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0135] Of course, the computing device can also include other components, such as an input / output interface, a communication component, etc.

[0136] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0137] The communication component is configured to facilitate wired or wireless communication between the computing device and other devices, etc.

[0138] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform, and the computing device can be a cloud server, and the processing component, the storage component, etc. can be a basic server resource rented or purchased from the cloud computing platform.

[0139] The embodiments of the present application also provide a computer readable storage medium storing a computer program, and the computer program is executed by a computer to implement the data access method of the embodiments of the present application as Figure 1 or Figure 3 illustrated.

[0140] In addition, the embodiment of the present application further provides a storage system based on the LSM-tree architecture, which can persist the storage medium and the storage engine as described above. Figure 4

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0142] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0143] Through the foregoing description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and the necessary general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the foregoing technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0144] Finally, it should be noted that: the foregoing embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the foregoing embodiments of the present application have been described in detail, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.​

Claims

1. A data access method, characterized in that, include: If all read requests access the first version of the index data, detect if the index data is updated and perform the following stream switching operation; wherein, the first version is the latest version. Following the version update of the index data, determine the corresponding flow switching ratio for each version update; according to the flow switching ratio, control a corresponding number of read requests to switch to access the latest version of the index data, and control the remaining read requests to access the historical version of the index data; If all read requests are switched to accessing the latest version of the index data, the flow switching operation is stopped.

2. The method according to claim 1, characterized in that, After stopping the flow cutting operation, the method further includes: Delete the index data of historical versions.

3. The method according to claim 1, characterized in that, Also includes: Release data blocks in persistent storage that are not referenced by any version of the indexed data.

4. The method according to claim 1, characterized in that, The control of other read requests accessing historical version index data includes: Control the access of other read requests to the first version of the index data.

5. The method according to claim 4, characterized in that, Also includes: After stopping access to any historical version of the index data, delete that historical version of the index data.

6. The method according to claim 1, characterized in that, The flow cutting operation further includes: Determine whether to write new data; If so, control all read requests to switch to accessing the latest version of the index data; If not, following the version update of the index data, control the read requests from few to many to gradually switch to accessing the latest version of the index data, and control the remaining read requests to access the historical version of the index data.

7. The method according to claim 1, characterized in that, The process of following the version updates of the index data and determining the traffic switching ratio corresponding to each version update includes: Following the version updates of the index data, the flow switching ratio corresponding to each version update is determined from zero based on the incremental ratio, following the principle of proportional increase.

8. The method according to claim 7, characterized in that, Also includes: The increment ratio is adjusted based on the current brushing operation frequency and / or compression operation frequency.

9. A computing device, characterized in that, It includes a storage component and a processing component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component; the processing component is used to implement the data access method as described in any one of claims 1 to 8.

Images