Method, apparatus, and program product for managing indexes of a stream data store system

Abstract

The present disclosure relates to a method, device and program product for managing an index of a streaming data storage system. An active tier of the index comprises a first set of shards, and shards in the first set of shards are used to store indexes of a portion of data objects in the storage system. In the method, in response to determining that a state of the first set of shards satisfies a predetermined expansion condition, a second set of shards is created in the index, shards in the second set of shards are used to store indexes of data objects that are to be entered into the storage system. A tier in which the second set of shards is located is marked as an active tier. Other tiers outside the tier in which the second set of shards is located are marked as frozen tiers. The storage system is managed based on shards in the active tier and the frozen tiers in the index. Further, a corresponding electronic device and computer program product are provided. The number of shards in the storage system can be dynamically set to process streaming data at a higher speed, and is suitable for processing streaming data that is constantly entered into the storage system.

Description

TECHNICAL FIELD

[0001] Implementations of the present disclosure relate to management of a storage system, and more particularly, to a method, device and computer program product for managing indexes in a streaming data storage system. BACKGROUND

[0002] Streaming data is a continuous and endless stream of data generated by various data sources. Generally, streaming data is generated at a high speed and in a large amount. Common examples of streaming data include log data of various application systems, measurement data of various sensors in the Internet of Things, and various real-time data collected by monitoring systems, etc. Streaming data can include a large amount of information and is usually stored in a time series manner.

[0003] In order to facilitate data search, a technical solution for establishing indexes for streaming data has been proposed. However, the existing technical solution has poor scalability. As time goes by, when the amount of streaming data continues to grow, the existing index technology cannot effectively manage more and more streaming data, which will cause the data access performance to decline and further reduce the efficiency of later data analysis. At this time, how to manage the indexes in the streaming data storage system in a more convenient and effective way has become a research hotspot. SUMMARY

[0004] Therefore, it is desirable to develop and implement a technical solution for managing the indexes of a streaming data storage system in a more effective way. It is desirable that the technical solution can be compatible with existing storage systems, and by modifying various configurations of the existing storage systems, the indexes of the storage system can be managed in a more effective way.

[0005] According to a first aspect of the present disclosure, a method for managing indexes of a streaming data storage system is provided. An active layer of the indexes includes a first set of shards, and a shard in the first set of shards is used to store indexes of a part of data objects in the storage system. In the method, in response to determining that a state of the first set of shards satisfies a predetermined expansion condition, a second set of shards is created in the indexes, and a shard in the second set of shards is used to store indexes of data objects to be entered into the storage system. The layer in which the second set of shards is located is marked as an active layer. Layers other than the layer in which the second set of shards is located are marked as frozen layers. The storage system is managed based on the shards in the active layer and the frozen layers in the indexes.

[0006] According to a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; a volatile memory; and a memory coupled with the at least one processor, the memory having stored therein instructions that, when executed by the at least one processor, cause the device to perform the method according to the first aspect of the present disclosure.

[0007] According to a third aspect of the present disclosure, there is provided a computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions for performing the method according to the first aspect of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0008] The features, advantages, and other aspects of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which several implementations of the present disclosure are illustrated, by way of example, not limitation. In the drawings:

[0009] Figure 1 A schematic diagram illustrating the working process of a storage system in which the method of the present disclosure can be implemented is shown;

[0010] Figure 2 A block diagram illustrating an index of a storage system according to one technical solution is shown;

[0011] Figure 3 A flow chart illustrating a method for managing an index of a storage system according to one implementation of the present disclosure is shown;

[0012] Figure 4 A block diagram illustrating a method for managing an index of a storage system according to one implementation of the present disclosure is shown;

[0013] Figure 5 A block diagram illustrating an extended index according to one implementation of the present disclosure is shown;

[0014] Figure 6 A block diagram illustrating an extended index according to one implementation of the present disclosure is shown;

[0015] Figure 7 A block diagram illustrating an index after intra-layer merging according to one implementation of the present disclosure is shown;

[0016] Figure 8 A block diagram illustrating an index after inter-layer merging according to one implementation of the present disclosure is shown; and

[0017] Figure 9 A block diagram illustrating an apparatus for managing an index of a storage system according to an exemplary implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Preferred implementations of the present disclosure will be described herein below with reference to the accompanying drawings. While preferred implementations of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the implementations set forth herein. Rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0019] The term "includes" and its variants are used inclusively herein and mean "includes but is not limited to." The term "or" means "and / or" unless otherwise specifically indicated. The term "based on" means "based, at least in part, on." The terms "one implementation" and "an implementation" mean "at least one implementation." The term "another implementation" means "at least one additional implementation." The terms "first," "second," and the like can refer to different or same objects. Other explicit or implicit definitions can also be included below.

[0020] A variety of stream data storage systems have been developed. For the convenience of description, only stream data storage systems for storing log data of application systems will be described as examples in the context of the present disclosure. Log data can be continuously generated in an application system, for example, log data can be generated every minute (or other time interval). At this time, the generated log data will be stored in the storage system in a time sequence. In order to facilitate later search and processing of log data, an index can be established for the stored data.

[0021] Full-text search-based indexes (e.g., Lucene indexes, etc.) have been proposed and have been widely used in various application systems. Full-text indexes can dynamically manage data objects (e.g., log data) stored in a storage system and can provide an inverted index of words in data objects to data objects. In the process of data analysis, the inverted index helps to score search results, and if more search words are mapped to the same data object, it is considered that the data object is more relevant to the search words and can have a higher score.

[0022] The number of data objects that can be accommodated by existing full-text indexes is limited. When faced with endless stream data, it is necessary to first partition the stream data entering the storage system. Resilient search solutions have been proposed and can be partitioned according to the range of keywords of data objects or hash values, and then each newly arrived data object can be stored in the corresponding partition. At this time, the index of each data object can be stored in the partition. In the context of the present disclosure, the collection of all indexes can be referred to as the index of the storage system, and the partition can be referred to as a "shard".

[0023] In the following, reference is made to Figure 1A summary describing shard-based indexing. Figure 1 A block diagram schematically illustrates the working process 100 of a storage system in which the method of the present disclosure may be implemented. Figure 1 As shown, the storage system 110 includes an index 120, where the index 120 may include multiple shards. Assuming that a predetermined hash function can map the keyword of a data object to a range of 0-255, the above range can be divided into multiple shards (for example, 3): the first shard may include hash values ​​of 0-84, the second shard may include hash values ​​of 85-169, and the third shard may include hash values ​​of 170-255. When a data object 130 enters the storage system 110, a hash value 132 of the data object 130 may be obtained. Assuming that the hash value 132 of the data object 130 is "20", the data object 130 may be divided into shard 122, and the index of the data object 130 may be stored in the shard 122. Here, the index of the data object 130 may point to the storage space in the storage system 110 where the data object 130 is stored.

[0024] However, the number of shards is fixed when the index is initially created. Furthermore, because inverse indexes store a mapping of terms to data objects, it's not possible to directly separate data objects. A common approach is to delete a portion of a data object belonging to an existing shard and then insert it into a newly created shard. This approach is similar to reindexing the data object and requires significant time and computational resources.

[0025] On the other hand, a fixed number of shards has limited data processing capacity and cannot cope with the ever-changing volume of streaming data. If the incoming streaming data exceeds the processing capacity of the shards, the speed of index generation will be slower than the speed of streaming data entering the storage system, resulting in a data backlog. However, if too many shards are set in the initial stage, queries will need to be executed across all shards, resulting in low query efficiency. In this regard, it is desirable to manage the index of the streaming data storage system in a more convenient and efficient manner.

[0026] According to an exemplary implementation of the present disclosure, a technical solution for dynamically managing the index of a storage system is proposed. Specifically, an initial number of shards can be set for the index, and streaming data entering the storage system can be processed according to the initial number of shards. Various types of requests executed against the storage system can be continuously monitored, and the number of shards in the storage system can be dynamically adjusted based on the load of the shards in the index. Using the exemplary implementation of the present disclosure, streaming data can be processed at an appropriate speed. In this way, streaming data that continuously enters the storage system can be processed.

[0027] According to an example implementation of the present disclosure, an index can include one or more tiers and each tier can include a set of shards. Specifically, initially an index can only include an active tier with a fixed number of shards. When the shard load in this tier is too high, a new active tier can be recreated and the original active tier can be set as a frozen tier. Figure 2 A block diagram of an index 200 of a storage system according to one technical solution is schematically shown. As shown, assume that the shard load in the initial tier 220 is too high, a tier 230 with more number of shards can be generated. At this time, the tier 220 can be set as a frozen tier while the tier 230 is set as an active tier. Figure 2

[0028] In the following, reference will be made to Figure 3 More details about managing an index will be described. Figure 3 A flow chart of a method 300 for managing an index of a storage system according to one implementation of the present disclosure is schematically shown. According to an example implementation of the present disclosure, initially the active tier of an index includes a first set of shards and the shards in the first set of shards are used to store an index of a portion of data objects in the storage system. The first set of shards herein has a predetermined fixed number (e.g. 3). At this time, each shard stores an index of a plurality of data objects and the hash values of the keys of the plurality of data objects belong to the hash value range of the shard.

[0029] In the context of the present disclosure, an index of a data object can represent a pointer to the specific address of the data object in the storage system. An index of a storage system can represent the overall collection of indexes of all data objects in the storage system. For example, an index of data objects can include indexes of data objects in each shard in the active tier and indexes of data objects in each shard in one or more frozen tiers.

[0030] In the following, reference will be made to Figure 4 More details about the initial first set of shards will be described. Figure 4 A block diagram of an index 400 for managing a storage system according to one implementation of the present disclosure is schematically shown. Figure 4 The index 400 in FIG. 4 can be an initially created index with a fixed number of shards. In Figure 4 In FIG. 4, the index 400 can be implemented based on a hash ring and include 3 shards, where the shard 410 is used to store indexes of data objects with hash values of 0-84, the shard 420 is used to store indexes of data objects with hash values of 85-169, and the shard 430 is used to store indexes of data objects with hash values of 170-255.

[0031] ​According to example implementations of the present disclosure, the index of the data objects can include a forward index (i.e., a mapping of keywords to terms in the data objects) and a reverse index (i.e., a mapping of terms in the data objects to the data objects). With example implementations of the present disclosure, a full-text search based index can be established, which in turn facilitates post-processing and analysis of the data objects in the storage system.

[0032] It will be appreciated that, Figure 4 The case where the hash value ranges from 0 to 255 and the active tier includes 3 shards is shown only schematically. According to example implementations of the present disclosure, the hash value can have a different range, and the active tier can include other number of shards. For example, the hash value can range from 0 to 63, and the active tier can include 4 shards. In this case, the hash value ranges involved by the respective shards are 0-15, 16-31, 32-47, and 48-63, respectively.

[0033] During the operation of the storage system, the status of the first set of shards can be periodically collected. At block 310, in response to determining that the status of the first set of shards satisfies a predetermined expansion condition, a second set of shards is created in the index. Here, the shards in the second set of shards are used to store the index of the data objects that are to be entered into the storage system. According to example implementations of the present disclosure, the second number of shards of the second set of shards can be an integer multiple of the first number of shards of the first set of shards. For example, the initial first set of shards includes 3 shards, and the created second set of shards can include 6 shards. Each shard has a predetermined resource, and thus the workload of each shard has an upper limit. By creating more shards, the respective shards can manage and store the index of the data objects entered into the respective shards in parallel.

[0034] According to example implementations of the present disclosure, the predetermined expansion condition can include that the number of indexes stored by the first set of shards is higher than a threshold number. It will be appreciated that as more data objects are stored into the storage system, the number of indexes stored in each shard will increase, at which time the workload of the shard will rise and the performance will decrease. Thus, when the number of indexes stored by the respective shards in the active tier reaches the predetermined threshold number, the delay of various requests performed for the storage system will increase, and the performance of the storage system will decrease. Further, since the processing capacity of the resource allocated to the shard has an upper limit, if the number of indexes continues to increase, the resource usage will reach the upper limit, and thus a situation such as a hang occurs. At this time, the expansion process of the index can be performed in order to generate more shards, thereby alleviating the pressure of the respective shards in the current active tier and preventing a situation of responding too slowly or even a hang.

[0035] According to an example implementation of the present disclosure, the predetermined expansion condition can include that the speed of storing indexes of the plurality of data objects entering the storage system based on the first set of shards is lower than the speed of the plurality of data objects entering the storage system. When a new data object enters the storage system, the computing resource of a shard will create an index for the new data object and store the created index in the shard. Since there is a certain time overhead for creating and storing the index, given a large amount of data objects entering the storage system, there is not enough time to process each of the entering data objects. This will result in the processing capacity provided by the shard being unable to meet the speed of the data objects entering the storage system. Thus, the speed of generating and storing the index becomes a bottleneck of the storage system and causes the performance of the storage system to degrade. At this time, an expansion process can be performed to generate more shards so as to improve the overall processing capacity of the plurality of shards.

[0036] Figure 5 A block diagram of the expanded index 500 according to one implementation of the present disclosure is schematically shown. As Figure 5 shown, the 3 shards 410, 420 and 430 in the initial active tier are expanded to 6 shards: namely, shards 512, 514, 522, 524, 532 and 534. Specifically, the hash value range 0-84 of the shard 410 can be divided into two, at which time the key range of the shard 512 can be 0-42 and the key range of the shard 514 can be 43-84. Similarly, the key ranges of 522, 524, 532 and 534 can be determined respectively.

[0037] At block 320, the tier in which the second set of shards is located is marked as an active tier (as shown by the solid line of the outer circle). At block 330, the tiers other than the tier in which the second set of shards is located are marked as frozen tiers (as shown by the dashed line of the inner circle). It will be appreciated that at any time during the operation of the storage system, there is only one active tier in the index. At this time, the second set of shards can replace the first set of shards and become the "active" state. The active tier and the frozen tier can have different functions, some requests performed on the storage system only involve the active tier, some requests only involve the frozen tier, and some requests will involve both the active tier and the frozen tier. Figure 5 Figure 5 According to an example implementation of the present disclosure, the state of the current active tier can be constantly monitored and the method 300 can be periodically performed. If the state of the current active tier including 6 shards meets the predetermined expansion condition, more shards (e.g., 12 shards) can be created again. At this time, the state of the index will be as

[0038] A block diagram of the expanded index 600 according to one implementation of the present disclosure is schematically shown. As Figure 6 Figure 6 A block diagram of the expanded index 600 according to one implementation of the present disclosure is schematically shown. As Figure 6 ​​As shown, the outermost hash ring (shown in solid line) illustrates the current active tier 630 including 12 shards, the middle hash ring (shown in dashed line) illustrates the frozen tier 620 including 6 shards, and the innermost hash ring (shown in dashed line) illustrates the frozen tier 610 including 3 shards.

[0039] At this point, the expansion procedure is successfully performed and the storage system can be managed with the expanded index. In the following, return to Figure 3 More details are described regarding managing the storage system with the active tier and one or more frozen tiers in the index. At block 340, the storage system is managed based on the shards in the active tier and the frozen tiers in the index. According to an example implementation of the present disclosure, the storage system can receive multiple types of requests: an insert request, an update request, an upsert request (a combination of update and insert), a delete request, and a search request. In the following, how to manage various types of requests performed on the storage system based on the active tier and the shards in the frozen tiers in the index will be described respectively.

[0040] According to an example implementation of the present disclosure, if an insert request is received for inserting a data object into the storage system, an index of the data object can be generated. Further, the index of the data object can be stored to the shard corresponding to the data object in the active tier. Specifically, which shard to store the index of the data object can be determined based on the key of the data object to be inserted. In the following, return to Figure 5 how to perform the insert operation is described. Assume that the hash value of the key of the data object to be inserted is 70, based on the hash value ranges of the shards described above, it is known that the hash value range of the shard 514 is 43-84, at this point the index of the data object should be stored to the shard 514. For another example, assume that the hash value of the key of the data object to be inserted is 30, at this point the index of the data object should be stored to the shard 512 (hash value range is 0-42).

[0041] It will be appreciated that the above only refers to Figure 5 The case where the active tier includes 6 shards is described. If the index is Figure 6 As shown, the case where the active tier includes 12 shards. The hash value ranges of the 12 shards can be determined respectively based on similar manners, and the shard matching the hash value of the key of the data object to be inserted is found according to the ranges.

[0042] According to an example implementation of the present disclosure, if an update request for updating a data object in the storage system is received, the data object can be first searched in the active tier, i.e., it can be determined whether the index of the data object is stored in the active tier. If it is determined that the index of the data object is stored in a shard in the active tier, the address of the data object to be updated in the storage system can be found directly based on the index in the shard. The data object can be read from the address and updated based on the update request, and then the index of the updated data object can be stored to the shard. Since the index of the data object to be updated is in the active tier at this time, the index operation related to the update request can be processed directly in the active tier.

[0043] In the following, the case that the index to be updated is stored to the active tier is described with reference to Figure 5 Assume that the hash value of the key of the data object to be updated is 70, and the index of the data object is found in the shard 514 in the active tier. The address of the data object can be found based on the index (e.g., the index points to the storage space 0x00...00-0x0f...ff in the storage system in a pointer manner). At this time, the data in the storage space can be updated based on the update request, and the index of the updated data object is stored to the shard 514.

[0044] According to an example implementation of the present disclosure, if it is determined that the index of the data object is not stored in the active tier, the update request can be forwarded to each frozen tier in the index, and similar operations can be performed in each frozen tier. If it is determined that the index of the data object to be updated is stored in a shard in the frozen tier, the address of the data object to be updated in the storage system can be found based on the index in the shard. The data object can be read from the address and updated based on the update request. Since the index of the data object to be updated is in the frozen tier (the index in the frozen tier cannot be modified) at this time, a new index can be created for the data object that has been updated, and the new index is stored to a shard in the active tier. Specifically, based on the hash value of the key of the data object, a shard in the active tier can be found whose hash value range matches the hash value. Further, the new index can be stored to the found shard. With the example implementation of the present disclosure, it can be ensured that the latest index of the updated data object is stored in the active tier.

[0045] In the following, the case that the index to be updated is stored to the active tier is described with reference to Figure 5The case where the index of the data object to be updated is stored to the frozen layer will be described. It is assumed that the hash value of the key of the data object to be updated is 60, and the index of the data object is found in the shard 410 in the frozen layer. The address of the data object (e.g., storage space 0x10...00-0x1f...ff) can be found based on the index. At this time, the data in the storage space can be updated based on the update request, and the index of the updated data object is stored in the shard in the active layer. In the active layer, the hash value range of the shard 514 is 43-84 and matches the hash value 60. Thus, the index of the updated data object can be stored to the shard 514 in the active layer.

[0046] It will be appreciated that since the data object has been updated at this time, the old index in the shard in the frozen layer is no longer valid, and the old index of the data object can be deleted from the shard in the frozen layer. In the example above, the old index can be deleted from the shard 410 in the frozen layer. With the exemplary implementation of the present disclosure, it can be ensured that the old index that is no longer valid is removed from the index. Thereafter, when an operation needs to be performed on the updated data object, the corresponding latest index can be found from the shard in the active layer.

[0047] In the storage system, an update insert request for a data object can be performed. The update insert request indicates that if the specified data object does not exist in the storage system, the data object is inserted into the storage system; if the data object exists in the storage system, the data object in the storage system is updated. If the update insert request for updating and inserting the data object in the storage system is received, it can be determined whether the index of the data object is stored in the active layer. If it is determined that the index of the data object is stored in the shard in the active layer, the data object is updated and the index of the updated data object is stored to the shard. At this time, since the index of the data object already exists in the active layer, it indicates that the data object has been stored in the storage system. Thus, the update insert request at this time performs an update action, and the process is similar to the case where the index of the data object to be updated is in the active layer described above, and thus will not be described again.

[0048] According to the exemplary implementation of the present disclosure, if it is determined that the index of the data object is not stored in the active layer, the update insert request is forwarded to each frozen layer. At this time, it can be found in each frozen layer whether the index of the data object exists. If it is determined that the index of the data object is not stored in any one of the frozen layers, it indicates that the data object is not stored in the storage system. Thus, the update insert request at this time performs an insert action, and the index of the data object can be stored to the shard in the active layer. The process at this time is similar to the case where the insert request is performed described above, and thus will not be described again.

[0049] According to example implementations of the present disclosure, if it is determined that the index of the data object is stored in the shard in the frozen tier, it indicates that the data object has already been stored in the storage system. Thus, the update insertion request herein performs an update action. Specifically, the data object can be updated based on the update insertion request, and the index of the updated data object is stored to the shard in the active tier. Here, the process is similar to the case that the index of the data object to be updated is in the frozen tier as described above, thus not repeated. According to example implementations of the present disclosure, since the data object is updated at this time, the old index of the data object in the frozen tier is no longer valid, and the old index of the data object can be deleted from the shard in the frozen tier.

[0050] It will be appreciated that the plurality of shards in the active tier and one or more frozen tiers in the index store the indices of all data objects in the storage system, thus when performing a search request, the search needs to be performed in the active tier and the one or more frozen tiers respectively. According to example implementations of the present disclosure, if a search request for searching data objects in the storage system is received, the search request can be forwarded to the active tier and the frozen tier respectively. The search request can be performed in each tier respectively, and the search results from the active tier and the frozen tier are provided.

[0051] For example, assume that the index herein is a reverse index, and a full-text search is desired to be performed in the log storage system to find logs including the keyword "fault". At this time, the search request can be forwarded to the active tier and all frozen tiers in the index. Then, the search can be performed in the reverse index in each shard in the active tier and all frozen tiers to find all logs including the keyword "fault". With example implementations of the present disclosure, it can be ensured that the full-text search is performed in a more accurate and efficient manner.

[0052] As the storage system operates, there can be data objects that include errors or are no longer valid in the storage system, at this time, the specified data object can be deleted from the storage system. According to example implementations of the present disclosure, if a deletion request for deleting a data object from the storage system is received, it can be first determined whether the index of the data object is stored in the active tier. If it is determined that the index of the data object is stored in the shard in the active tier, the index of the data object can be directly deleted from the shard in the active tier.

[0053] Specifically, based on the hash value of the key of the data object to be deleted, the corresponding shard in the active tier can be found, and the storage space where the data object to be deleted is located can be found at the address specified by the index in the shard. The data in the storage space can be cleared and released, and since the data object itself has been deleted at this time, the index of the data object is no longer valid, and the index can be deleted from the shard in the active tier. In this way, it can be ensured that the data objects and the indexes in the storage system have a one-to-one correspondence.

[0054] According to an exemplary implementation of the present disclosure, a deletion queue can be set for the frozen tier. If it is determined that the index of the data object is not stored in the active tier, the deletion request can be added to the deletion queue in order to notify the deletion request to each shard in the frozen tier. Specifically, similar operations can be performed in each frozen tier. In a certain frozen tier, if it is determined that the index of the data object is stored in the shard in the frozen tier, the storage space where the data object to be deleted is located can be found from the shard. The data in the storage space can be cleared and released. Since the data object itself has been deleted at this time, the index of the data object is no longer valid, and the index can be deleted from the shard in the frozen tier. In this way, it can be ensured that the data objects and the indexes in the storage system have a one-to-one correspondence.

[0055] According to an exemplary implementation of the present disclosure, the update history of the deletion queue can be saved in the cache file. For example, if a certain deletion request is added to the deletion queue, the deletion request can be recorded in the cache file, and the state of the deletion request is set to “not executed”. If the deletion request has been successfully executed, the state of the deletion request can be marked as “executed”.

[0056] According to an exemplary implementation of the present disclosure, when receiving an access request for the storage system, the access request can be filtered based on the above-mentioned cache file. Here, the access request includes at least any one of the update request, the deletion request and the search request. If the access request involves a data object that has been deleted recorded in the cache file, a notification that the data object has been deleted can be directly provided. Specifically, if an access request for a data object in the storage system is received, a deletion request associated with the data object can be searched in the cache file. If the deletion request is found, it indicates that the data object to be accessed has been deleted, and a notification can be directly provided without having to search in each tier in the index respectively.

[0057] Having described the process for expanding shards in an index above, according to an example implementation of the present disclosure, the status of shards in an active tier can be monitored periodically and expansion operations can be performed when needed. It will be appreciated that workloads in a storage system can vary and excessive tiers and / or shards can increase the complexity of management. At this time, a shrink process can be performed for shards in an index. The shrink process can include intra-tier shrinking and inter-tier shrinking, which will be described below with reference to Figure 7 and Figure 8 Further details are provided.

[0058] According to an example implementation of the present disclosure, if it is determined that the number of indexes stored by a tier in an index is below a threshold number, shards in the tier can be merged. It will be appreciated that as delete requests, update requests, or update insert requests are performed in a storage system, the number of indexes stored in a tier can decrease. At this time, excessive shards in a tier can be merged in order to reduce the computational resource overhead and management complexity of the excessive shards.

[0059] Assuming that the number of indexes in a frozen tier 620 in an index 600 as shown in Figure 6 decreases below a threshold number, merging can be performed in the frozen tier 620. Specifically, shards 512 and 514 can be merged, shards 522 and 524 can be merged, and shards 532 and 534 can be merged. Figure 7 A block diagram of an index 700 after intra-tier merging is performed according to one implementation of the present disclosure is schematically shown. In Figure 7 , shard 710 is from shards 512 and 514 before merging, shard 720 is from shards 522 and 524 before merging, and shard 730 is from shards 532 and 534 before merging.

[0060] According to an example implementation of the present disclosure, the number of shards in a tier before merging is an integer multiple of the number of shards in the tier after merging. For example, the number of shards "6" in the frozen tier 620 before merging is twice the number of shards "3" in the frozen tier 740 after merging. With an example implementation of the present disclosure, expansion and shrink processes are performed based on integer multiples. In this way, by merging between shards with the same hash value space, the complexity of managing an index can be simplified and the likelihood of anomalies in expansion and shrink processes can be reduced.

[0061] According to an example implementation of the present disclosure, if the speed of storing the indexes of the plurality of data objects entering the storage system based on the shards in a tier is higher than the speed of the plurality of data objects entering the storage system, the shards in the tier can be merged. It will be appreciated that it is required herein to ensure that the processing speed of the merged shards in the tier is higher than the speed of the plurality of data objects entering the storage system. Assuming that the speed of writing data objects to the storage system is greatly reduced, and at this time the processing speed of the 12 shards in the active tier 630 is already much higher than the speed of the data objects entering the storage system, at this time the 12 shards can be merged into 6 shards.

[0062] It will be appreciated that if the number of tiers in the index is too large, the management complexity will also increase, at which time inter-tier merging can be performed. According to an example implementation of the present disclosure, if it is determined that the number of indexes stored by a plurality of tiers in the index is lower than a threshold number, the shards in the plurality of tiers can be merged. In Figure 7 In the example shown in FIG. 7, the shards in the frozen tier 740 and the frozen tier 610 can be merged. Specifically, the shards 410 and 710 can be merged, the shards 420 and 720 can be merged, and the shards 430 and 730 can be merged. Figure 8 An example block diagram of the index 800 after performing inter-tier merging according to an implementation of the present disclosure is schematically shown in FIG. 8. In Figure 8 In the example shown in FIG. 8, the frozen tier 840 is from the Figure 7 In the example shown in FIG. 8, the frozen tier 840 is from the

[0063] It will be appreciated that in order to avoid the adverse effects of failures occurring in the merging on the index, each relevant tier before merging can be backed up in advance, and if an operation exception or other failure occurs during the merging process, each relevant tier can be rolled back to the version backed up in advance. If the merging process is successfully performed, the backed up version can be deleted. According to an example implementation of the present disclosure, it can be ensured that when a failure occurs, the index of the storage system can be restored to the correct version before, thereby improving the reliability of index management.

[0064] In the foregoing, reference has been made to Figures 2 to 8Detailed description of the example of the method 300 according to the present disclosure will be given, it will be appreciated that the method described above can be executed on a single device, or can be executed in a distributed manner on multiple devices. The implementation of the corresponding apparatus will be described hereinafter. According to an example implementation of the present disclosure, an apparatus for managing the index of a storage system of streaming data is provided. Here, the active tier of the index comprises a first set of shards, and the shards in the first set of shards are used to store the index of a portion of data objects in the storage system. The apparatus comprises: a creation module configured to, in response to determining that the state of the first set of shards satisfies a predetermined expansion condition, create a second set of shards in the index, the shards in the second set of shards being used to store the index of data objects to be entered into the storage system; an active tier marking module configured to mark the tier in which the second set of shards is located as an active tier; a frozen tier marking module configured to mark other tiers except the tier in which the second set of shards is located as frozen tiers; and a management module configured to manage the storage system based on the shards in the active tier and the frozen tiers in the index. According to an example implementation of the present disclosure, the apparatus can comprise modules for performing other steps in the method 300 described above.

[0065] Figure 9 A block diagram of an apparatus 900 for managing the index of a storage system according to an example implementation of the present disclosure is schematically shown. As shown, the apparatus 900 comprises a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the apparatus 900 can also be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0066] A plurality of components in the apparatus 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the apparatus 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0067] The various processes and processes described above, such as the method 300, can be performed by the processing unit 901. For example, in some implementations, the method 300 can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some implementations, portions or all of the computer program can be loaded onto and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the CPU 901, one or more steps of the method 300 described above can be performed. Alternatively, in other implementations, the CPU 901 can also be configured in any other suitable manner to implement the above-described processes / methods.

[0068] According to an example implementation of the present disclosure, there is provided an electronic device comprising: at least one processor; a volatile memory; and a memory coupled with the at least one processor, the memory having stored therein instructions which, when executed by the at least one processor, cause the device to perform a method for managing an index of a storage system for streaming data. An active tier of the index comprises a first set of shards, and a shard of the first set of shards is used to store an index of a portion of data objects in the storage system. The method comprises: in response to determining that a state of the first set of shards satisfies a predetermined expansion condition, creating a second set of shards in the index, a shard of the second set of shards being used to store an index of a data object to be entered into the storage system; marking a tier in which the second set of shards is located as the active tier; marking other tiers outside the tier in which the second set of shards is located as frozen tiers; and managing the storage system based on the shards in the active tier and the frozen tiers in the index.

[0069] According to an example implementation of the present disclosure, the predetermined expansion condition comprises at least one of: a number of indexes stored by the first set of shards being higher than a threshold number; and a speed of storing indexes of a plurality of data objects entering the storage system based on the first set of shards being lower than a speed of the plurality of data objects entering the storage system.

[0070] According to an example implementation of the present disclosure, managing the storage system comprises: in response to receiving an insertion request for inserting a data object into the storage system, generating an index of the data object; and storing the index of the data object to a shard corresponding to the data object in the active tier.

[0071] According to an example implementation of the present disclosure, managing the storage system comprises: in response to receiving an update request for updating a data object in the storage system, determining whether an index of the data object is stored in the active tier; and in response to determining that the index of the data object is stored in a shard in the active tier, updating the data object and storing an index of the updated data object to the shard.

[0072] According to example implementations of the present disclosure, the method further includes forwarding the update request to the frozen tier in response to determining that the index of the data object is not stored in the active tier, and updating the data object and storing the index of the updated data object to a shard in the active tier in response to determining that the index of the data object is stored in a shard in the frozen tier.

[0073] According to example implementations of the present disclosure, the method further includes deleting the index of the data object from the shard in the frozen tier.

[0074] According to example implementations of the present disclosure, managing the storage system includes, in response to receiving an update-insert request for updating an inserted data object in the storage system, determining whether an index of the data object is stored in an active tier, and in response to determining that the index of the data object is stored in a shard in the active tier, updating the data object and storing the index of the updated data object to the shard.

[0075] According to example implementations of the present disclosure, the method further includes forwarding the update-insert request to the frozen tier in response to determining that the index of the data object is not stored in the active tier, and storing the index of the data object to a shard in the active tier in response to determining that the index of the data object is not stored in the frozen tier.

[0076] According to example implementations of the present disclosure, the method further includes updating the data object based on the update-insert request in response to determining that the index of the data object is stored in a shard in the frozen tier, and storing the index of the updated data object to a shard in the active tier.

[0077] According to example implementations of the present disclosure, the method further includes deleting the index of the data object from the shard in the frozen tier.

[0078] According to example implementations of the present disclosure, the method managing the storage system includes, in response to receiving a search request for searching a data object in the storage system, forwarding the search request to an active tier and a frozen tier, respectively, and providing search results from the active tier and the frozen tier.

[0079] According to example implementations of the present disclosure, the method managing the storage system includes, in response to receiving a delete request to delete a data object from the storage system, determining whether an index of the data object is stored in an active tier, and in response to determining that the index of the data object is stored in a shard in the active tier, deleting the index of the data object from the shard in the active tier.

[0080] According to example implementations of the present disclosure, the method further includes adding the delete request to a delete queue of the frozen tier to notify a shard in the frozen tier of the delete request in response to determining that the index of the data object is not stored in the active tier.

[0081] According to an example implementation of the present disclosure, the method further includes deleting the index of the data object from the shard in the frozen tier in response to determining that the index of the data object is stored in the shard in the frozen tier.

[0082] According to an example implementation of the present disclosure, the method further includes searching for a delete request associated with the data object based on an update history of the delete queue in response to receiving an access request for the data object in the storage system, wherein the access request comprises at least any one of an update request, a delete request, and a search request; and providing a notification that the data object has been deleted in response to finding the delete request.

[0083] According to an example implementation of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transient computer readable medium and comprises machine executable instructions for performing a method according to the present disclosure.

[0084] According to an example implementation of the present disclosure, a computer readable medium is provided. The computer readable medium has stored thereon machine executable instructions, which, when executed by at least one processor, cause the at least one processor to implement a method according to the present disclosure.

[0085] The present disclosure can be a method, apparatus, system, and / or computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present disclosure.

[0086] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0087] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0088] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some implementations, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, which may be executed by the computer-readable program instructions to implement various aspects of the present disclosure.

[0089] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0091] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0093] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for managing an index of a storage system for streaming data, wherein an active layer of the index includes a first set of shards, and shards in the first set of shards are used to store indexes of a portion of data objects in the storage system, comprising: In response to determining that the status of the first group of shards meets a predetermined expansion condition, creating a second group of shards in the index, wherein the shards in the second group of shards are used to store indexes of data objects that are about to enter the storage system, wherein the predetermined expansion condition includes at least any one of the following: The number of indexes stored in the first group of shards is greater than a threshold number; as well as storing an index of the plurality of data objects entering the storage system based on the first set of shards at a rate lower than a rate at which the plurality of data objects enter the storage system; Mark the layer where the second group of slices resides as the active layer; Marking other layers except the layer where the second group of slices are located as frozen layers; as well as The storage system is managed based on shards in the active tier and the frozen tier in the index.

2. The method of claim 1 , wherein managing the storage system comprises: In response to receiving an insert request for inserting a data object into the storage system, generating an index of the data object; as well as The index of the data object is stored in a shard corresponding to the data object in the active layer.

3. The method of claim 1 , wherein managing the storage system comprises: In response to receiving an update request for updating a data object in the storage system, determining whether an index of the data object is stored in the active layer; as well as In response to determining that the index of the data object is stored in a shard in the active tier, the data object is updated and the updated index of the data object is stored in the shard.

4. The method according to claim 3, further comprising: In response to determining that the index of the data object is not stored in the active tier, forwarding the update request to the frozen tier; as well as In response to determining that the index of the data object is stored in a shard in the frozen tier, the data object is updated and the updated index of the data object is stored in a shard in the active tier.

5. The method according to claim 4, further comprising: The index of the data object is deleted from the shard in the frozen tier.

6. The method of claim 1 , wherein managing the storage system comprises: In response to receiving an upsert request for upsert a data object in the storage system, determining whether an index of the data object is stored in the active layer; as well as In response to determining that the index of the data object is stored in a shard in the active tier, the data object is updated and the updated index of the data object is stored in the shard.

7. The method according to claim 6, further comprising: In response to determining that the index of the data object is not stored in the active tier, forwarding the upsert request to the frozen tier; as well as In response to determining that the index of the data object is not stored in the frozen tier, the index of the data object is stored to a shard in the active tier.

8. The method according to claim 7, further comprising: in response to determining that the index of the data object is stored in a shard in the frozen tier, updating the data object based on the upsert request; as well as The updated index of the data object is stored in the shard in the active layer.

9. The method according to claim 8, further comprising: The index of the data object is deleted from the shard in the frozen tier.

10. The method of claim 1 , wherein managing the storage system comprises: In response to receiving a search request for searching for a data object in the storage system, forwarding the search request to the active layer and the frozen layer respectively; as well as Search results from the active layer and the frozen layer are provided.

11. The method of claim 1 , wherein managing the storage system comprises: In response to receiving a delete request to delete a data object from the storage system, determining whether an index of the data object is stored in the active layer; as well as In response to determining that the index of the data object is stored in a shard in the active tier, the index of the data object is deleted from the shard in the active tier.

12. The method according to claim 11, further comprising: In response to determining that the index of the data object is not stored in the active layer, The deletion request is added to a deletion queue of the frozen layer to notify the shards in the frozen layer of the deletion request.

13. The method according to claim 12, further comprising: In response to determining that the index of the data object is stored in a shard in the frozen tier, the index of the data object is deleted from the shard in the frozen tier.

14. The method according to claim 12, further comprising: In response to receiving an access request for a data object in the storage system, searching for a delete request associated with the data object based on an update history of the delete queue, wherein the access request includes at least any one of an update request, a delete request, and a search request; and In response to finding the deletion request, providing notification that the data object has been deleted.

15. The method of claim 1 , wherein managing the storage system comprises: merging shards in a tier in the index in response to at least any one of: a number of indexes stored in the tier of the index being below a threshold number; and a rate at which indexes for a plurality of data objects entering the storage system are stored based on the shards in the tier being higher than a rate at which the plurality of data objects enter the storage system, The number of shards of the layer before merging is an integer multiple of the number of shards of the layer after merging.

16. The method of claim 15, wherein the index includes the active tier and a plurality of frozen tiers, and managing the storage system comprises: In response to determining that the number of indexes stored by the plurality of layers in the index is less than the threshold number, merging the shards in the plurality of layers.

17. The method according to claim 1, wherein the number of second shards in the second group of shards is an integer multiple of the number of first shards in the first group of shards, the shards in the first group of shards are used to store the index of a portion of data objects in the storage system based on a hash ring, and the index of a portion of data objects includes a reverse index.

18. An electronic device comprising: at least one processor; Volatile memory; as well as A memory coupled to the at least one processor, the memory having instructions stored therein, the instructions, when executed by the at least one processor, causing the apparatus to perform the method according to any one of claims 1 to 17.

19. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions for performing the method according to any one of claims 1-17.

Images