Methods, devices, and programs for managing indexes in storage systems.

Abstract

This disclosure relates to methods, apparatus, and program products for managing indexes in a storage system. In one method, a purge period is received for clearing expired events in a storage system, the storage system comprising multiple events in a data stream. Based on the purge period, time shards for managing the indexes in the storage system are determined. In response to determining that the occurrence time of a first event entering the storage system from the data stream is within the time shard, a sharded index node is created for the time shard, the sharded index node including the index node of the first event in the storage system. The sharded index node is added to the indexes of the storage system. Corresponding apparatus and computer program products are provided. Using exemplary implementations of this disclosure, indexes of a large number of events entering the storage system can be managed according to the occurrence time of the events. Furthermore, the storage system can be queried and updated accurately and efficiently.

Description

Technical Field

[0001] The various implementations of this disclosure relate to storage systems, and more specifically, to methods, apparatus, and computer program products for managing indexes of storage systems. Background Technology

[0002] With the development of data storage technology, various data storage devices are now able to provide users with increasingly higher data storage capabilities, and data access speeds have also greatly improved. For streaming data storage systems, a large number of events in the data stream are continuously stored in the storage system. Currently, technical solutions for creating indexes for events have been proposed. However, the performance of existing technical solutions is not satisfactory. Therefore, how to create and manage indexes for the storage system with higher performance, thereby improving the overall performance of the storage system, has become a technical challenge. Summary of the Invention

[0003] Therefore, it is desirable to develop and implement a technical solution for managing storage system indexes in a more efficient manner. This solution is expected to manage storage system indexes more conveniently and effectively. Furthermore, it is hoped that the created indexes can be used to improve the access and update efficiency of the storage system.

[0004] According to a first aspect of this disclosure, a method for managing an index of a storage system is provided. In this method, a purge period for clearing expired events in the storage system is received, the storage system including multiple events in a data stream. Based on the purge period, time shards for managing the index of the storage system are determined. In response to determining that the occurrence time of a first event entering the storage system from the data stream is within the time shard, a sharded index node is created for the time shard, the sharded index node including the index node of the first event in the storage system. The sharded index node is added to the index of the storage system.

[0005] According to a second aspect of this disclosure, an electronic device is provided, comprising: at least one processor; volatile memory; and a memory coupled to the at least one processor, the memory having instructions stored therein, the instructions causing the device to perform a method according to the first aspect of this disclosure when executed by the at least one processor.

[0006] According to a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a non-transient computer-readable medium and includes machine-executable instructions for performing the method according to a first aspect of this disclosure. Attached Figure Description

[0007] The features, advantages, and other aspects of various implementations of this disclosure will become more apparent from the accompanying drawings and the following detailed description, which illustrate, by way of example and not limitation, several implementations of this disclosure. In the drawings:

[0008] Figure 1 A block diagram of a storage system according to a technical solution is shown schematically;

[0009] Figure 2 A block diagram illustrating a process for managing an index of a storage system according to an exemplary implementation of this disclosure is shown schematically.

[0010] Figure 3 A flowchart illustrating an exemplary implementation of this disclosure for managing an index of a storage system is shown.

[0011] Figure 4 A block diagram illustrating a sharded index node according to an exemplary implementation of this disclosure is shown.

[0012] Figure 5 A block diagram illustrating the process of creating multiple sharded index nodes based on multiple time shards according to an exemplary implementation of this disclosure is shown.

[0013] Figure 6 A block diagram illustrating the state of each shard index node in an index according to an exemplary implementation of this disclosure is shown.

[0014] Figure 7 A block diagram illustrating a process for removing expired events according to an exemplary implementation of this disclosure is shown; and

[0015] Figure 8 A block diagram of a device for managing indexes of a storage system, according to an exemplary implementation of the present disclosure, is shown schematically. Detailed Implementation

[0016] Preferred implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While preferred implementations of this disclosure are shown in the drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the implementations set forth herein. Rather, these implementations are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0017] The term “comprising” and its variations, as used herein, signify open inclusion, i.e., “including but not limited to.” Unless otherwise stated, the term “or” means “and / or.” The term “based on” means “at least partially based on.” The terms “one example implementation” and “one implementation” mean “at least one example implementation.” The term “another implementation” means “at least one additional implementation.” The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] In the context of this disclosure, events in a data stream can involve various types, and these events can occur at different times. For example, in an application environment that monitors the operation of a data center, the data stream may include log data from various devices within the data center. In this case, events in the data stream may be log data generated at various points in time (e.g., every minute, or other time intervals). As another example, in an application environment that monitors road traffic conditions, the data stream may include images, videos, and / or other data acquired by multiple acquisition devices at multiple locations.

[0019] It will become clear that data streams are continuous and can involve massive amounts of data. For ease of storage, large amounts of events can be divided into multiple shards based on hash algorithms, and these shards can be stored on different storage devices. For simplicity, please refer to [link to previous text]. Figure 1 Describe an overview of the storage system. Figure 1 A block diagram of a storage system 100 according to one technical solution is shown schematically. Figure 1 As shown, storage system 100 may include data shards 122, ..., 132, etc. A threshold capacity for each data shard can be pre-specified; when the amount of data in a data shard reaches this threshold capacity, a new data shard is created. A technical solution for creating indexes for individual data shards has been proposed. For example, index 120 can be created for data shard 122, ..., and index 130 can be created for data shard 132. In this case, the indexes of storage system 100 will include index group 110, and index group 110 includes indexes 120, ..., 130 for each data shard. Furthermore, storage system 100 can be managed based on index group 110.

[0020] However, users of storage system 100 need to develop specialized applications to manage index group 110. When a query request is received, a hash algorithm can be used to find one or more indexes corresponding to the query request in index group 110, and then one or more query results can be returned. Furthermore, the returned one or more query results can be filtered to serve as the final query result.

[0021] It will be understood that during the management of index group 110, faults and / or misoperations may occur, leading to errors in the index group. Therefore, existing index management schemes cannot effectively manage the indexes of the storage system. Furthermore, events in the storage system have expiration dates; when some events expire, they must be removed from the storage system. This requires traversing each index in index group 110 to find the expired events. Therefore, index group 110 based on existing technical solutions cannot provide satisfactory performance.

[0022] To address the aforementioned shortcomings, this disclosure provides a technical solution for managing indexes in a storage system. Specifically, indexes can be created based on a cleanup cycle for clearing expired events in the storage system and according to the occurrence times of multiple events in the data stream. Assuming a cleanup cycle of one day, indexes can be created using one day as the time unit. That is, separate index nodes are created for events occurring within each day.

[0023] Using the exemplary implementation of this disclosure, indexes can be created within the storage system according to a cleanup cycle and time sequence for a large number of events. This allows for index management within the storage system, eliminating the need for users to develop their own index management applications. Furthermore, the index nodes associated with expired events can be directly located from each index node, resulting in more efficient updates to the storage system's content.

[0024] See below. Figure 2 A general description of an exemplary implementation of this disclosure is provided. Figure 2 A block diagram schematically illustrates a process 200 for managing an index of a storage system according to an exemplary implementation of this disclosure. Arrows 210 indicate the sequence of events, and data flow 250 may include a large number of events 251, ..., 252, 253, ..., 254, ..., 255, ..., 256, etc. Data flow 210 is continuous and can continuously generate new events over time. Figure 2 As shown, events in the storage system have an expiration period of 260 and a cleanup period of 270. For example, the expiration period of 260 can be specified as 7 days (or another time length), and the cleanup period of 270 can be specified as 1 day (or another time length). In this case, events stored in the storage system within the last 7 days are valid, and expired events older than 7 days can be cleared from the storage system at 00:00 (or another time point) every day.

[0025] According to an exemplary implementation of this disclosure, time shards can be determined according to a cleanup period 270, and separate shard index nodes can be created for events occurring within those time shards. For example, if events 251, ..., 252 occur on day X, then time shard 240 includes 24 hours of day X. Separate shard index nodes 230 can be created for events 251, ..., 252. Similarly, if time shard 242 includes 24 hours of day X+1, separate shard index nodes 232 can be created for events 253, ..., 254. And so on, if time shard 244 includes 24 hours of day X+7, separate shard index nodes 234 can be created for events 255, ..., 254. The created shard index nodes can be added to the root node 220 of the index, allowing traversal of events in the storage system based on the child nodes of the root node.

[0026] Each shard index node relates to events occurring within one day; therefore, when clearing expired events, the clearing operation can be performed on a shard index node basis. Specifically, at 0:00 on day X+7, events occurring within day X become expired. At this time, the shard index node 230 pointing to the expired events can be found via the root node 220, and all expired events can be deleted from the storage system. Using the exemplary implementation of this disclosure, it is not necessary to traverse every index node in the index to find expired events; instead, it is possible to directly determine whether the multiple events pointed to by each shard index node have expired based on the time shard corresponding to that shard index node. If the determination result is "yes," all events pointed to by that shard index node can be directly deleted from the storage system. In this way, the indexes of the storage system can be managed more efficiently, thereby simplifying the storage system update process.

[0027] See below. Figure 3 Further details describe an exemplary implementation according to this disclosure. Figure 3 A flowchart illustrating a method 300 for managing indexes in a storage system according to an exemplary implementation of this disclosure is shown schematically. Method 300 can be executed at computing resources within the storage system 100. In this case, users of the storage system 100 do not need to develop their own index management applications, but can directly use the index management services provided within the storage system 100. Figure 3 At box 310, a cleanup cycle for clearing expired events in storage system 100, which includes multiple events in a data stream, is received.

[0028] As time passes, a large number of events continuously enter storage system 100. Typically, in applications such as monitoring, only recently occurring events (e.g., events that occurred within the last week) are of interest, while events that occurred much earlier are not. Therefore, storage system 100 needs to be updated periodically. An expiration date can be set for the events in storage system 100, and expired events can be periodically removed from storage system 100 according to a clearing cycle.

[0029] It will be understood that events can involve occurrence time and reception time. Occurrence time refers to the time when an event occurs, such as the time a log entry is generated. Reception time refers to the time when the event is received by storage system 100, that is, the time the event enters storage system 100. Due to potential delays in data transmission, reception time will be later than occurrence time, and in situations such as network failures, there may be significant delays. Expired events are events whose occurrence time does not conform to the validity period of storage system 100. For example, in the case of a 7-day validity period, an event that occurred more than 7 days ago is an expired event.

[0030] At box 320, the time shards for the indexes used to manage the storage system are determined based on the purge cycle. Here, a time shard refers to the length of time corresponding to the created sharded index node. For example, if the purge cycle is 1 day, the time shard can be set to a length of 1 day. In this case, the time shard can include the period from 0:00 to 24:00 of a day.

[0031] At box 330, if the first event of the data stream entering storage system 100 is determined (e.g., Figure 2 If the occurrence time of event 251 (shown) falls within time shard 240, then a shard index node 230 is created for time shard 240. This shard index node 230 includes the index node for the first event in the storage system. According to an exemplary implementation of this disclosure, the index node for the first event can be created based on various methods currently known and / or to be developed in the future. Assuming the current time is 0:00 on day X, a shard index node can be created for the time shard containing day X. If it is determined that the first event occurs between 0:00 and 24:00 on day X, an index node can be created for the first event (e.g., a leaf node pointing to the storage address of the event in the storage system), and the created index node is added to the shard index node 230.

[0032] According to one exemplary implementation of this disclosure, an index can be created for subsequent events occurring within time shard 240. Specifically, if it is determined that the occurrence time of a second event entering the storage system 100 in the data stream is within time shard 240, an index node can be created for the second event, and the created index node can be added to the shard index node 230.

[0033] According to one exemplary implementation of this disclosure, the sharded index node 230 may include multiple levels. See below for further details. Figure 4 More details about segmented index node 230. Figure 4 A block diagram 400 schematically illustrates a sharded index node 230 according to an exemplary implementation of this disclosure. (See diagram 400 for details.) Figure 4 As shown, time slice 240 can be divided into multiple time windows according to a predetermined time window length. If the length of each time window is specified as 1 hour, time slice 240 can be divided into 24 time windows (e.g., Figure 4 (Time windows 410, ..., 420 in the time frame). A corresponding window index node can be created for each time window.

[0034] According to an exemplary implementation of this disclosure, if the occurrence time of the first event is determined to be within time window 410 of multiple time windows, a window index node 430 can be created for time window 410 based on the first event. An index node can be created for each event occurring within time window 410, and the created index nodes can be added to window index node 430. In this case, window index node 430 may include an index node pointing to the address of each event in the storage system. Window index node 430 can be added to the shard index node 230 of the time shard 240 to which time window 410 belongs.

[0035] According to one exemplary implementation of this disclosure, each time window can be divided into finer granularities. For example, time window 410 can be divided into time segments 412, ..., 414 according to a predetermined time length, and time window 420 can be divided into time segments 422, ..., 424. For example, a time window 410 with a length of 1 hour can be divided into 60 time segments, where each time segment is 1 minute long. A corresponding segment index node can be created for each time segment. For example, segment index nodes 440, ... can be created for time segment 412, and segment index node 442 can be created for time segment 414.

[0036] Fragmented index node 230 can have a multi-level structure. For example... Figure 4 As shown, the slice index node 230 may include window index nodes 430, ..., 432. As indicated by arrow 450, window index node 430 may correspond to time window 410. Further, window index node 430 may include segment index nodes 440, ..., 442. As indicated by arrow 452, segment index node 440 may correspond to time segment 412, and segment index node 442 may correspond to time segment 414. Although... Figure 4As not shown in the diagram, the segmented index node 440 may further include index nodes for multiple events occurring within the time segment 412, with each leaf node pointing to the address of an event in the storage system.

[0037] According to an exemplary implementation of this disclosure, each index node can have a corresponding time range. For example, the time range of sharded index node 230 may include 0:00 to 24:00 on day X, the time range of window index node 430 may include 0:00 to 1:00 on day X, the time range of window index node 432 may include 23:00 to 24:00 on day X, the time range of segmented index node 440 may include 0:00 to 0:01 on day X, and the time range of segmented index node 442 may include 0:59 to 1:00 on day X. In this way, events occurring at different times can be managed with fine time granularity.

[0038] According to one exemplary implementation of this disclosure, an index node can be created when the current time enters the time range corresponding to the index node, and the created index node can be added to the parent index node. According to another exemplary implementation of this disclosure, an index node can be set to read-only mode when the current time has exceeded the time range corresponding to the index node. It will be understood that the time an event enters the storage system will be later than the event's occurrence time; therefore, the index node can be set to read-only mode when the current time has exceeded the aforementioned time range for a predetermined duration. In this way, it can be ensured that the index of a late event is added to the correct index node corresponding to its occurrence time.

[0039] The process of creating sharded inodes has already been described; in the following section, we will return to... Figure 3 Describe more details about creating an index. Figure 3 At box 340, add sharded index node 230 to the index of storage system 100. In this way, expired events in storage system 100 can be updated on a time-shard basis.

[0040] As time progresses, newly created sharded inodes can be continuously added to the indexes of the storage system. According to an exemplary implementation of this disclosure, if it is determined that the current time exceeds time shard 240, another time shard 242 can be determined based on the current time and the purge cycle for managing the indexes of storage system 100. See below for further details. Figure 5 Describe the operations related to multiple time slices. Figure 5 A block diagram (500) schematically illustrates a process for creating multiple sharded inodes based on multiple time shards according to an exemplary implementation of this disclosure. (See diagram for example.) Figure 5As shown, assuming the current time has exceeded time slice 240 (that is, it has passed 24:00 on day X), time slice 242 can be determined (that is, from 0:00 to 24:00 on day X+1). At this time, time slice 242 is the subsequent time slice immediately following time slice 240.

[0041] Furthermore, sharded index nodes 232 can be created within time shard 242 as described above. At this point, the storage system's index will include multiple sharded index nodes, and each sharded index node corresponds to an index of an event occurring within a time shard. Similar to the process described above, corresponding index nodes can be created for events occurring within time shard 242. Over time, sharded index nodes 232 can include a multi-level index structure.

[0042] It will be understood that there is a certain difference between the time the event is received in storage system 100 and the time it occurs. For example... Figure 5 As shown, event 252, indicated by shading, occurs within time shard 240. Due to network failures and / or latency, event 252 may enter storage system 100 within time shard 240 or in the next time shard 242. According to an exemplary implementation of this disclosure, regardless of which time shard the event enters storage system 100, the inode of the event should be added to the shard inode of the time shard to which the event belongs. Specifically, if it is determined that event 252 entering storage system 100 in data stream 250 occurs within time shard 240, the index of event 252 in storage system 100 can be added to shard inode 230.

[0043] According to an exemplary implementation of this disclosure, when an event occurring within a new time shard enters the storage system 100, a shard index node can be created for that new time shard. Specifically, if the occurrence time of event 253 entering the storage system 100 in the data stream is within time shard 242, a shard index node 232 can be created for time shard 242. Furthermore, the shard index node 232 can be added to the index, and the index node for event 253 can be added to the shard index node 232. Using this exemplary implementation, the index can be managed according to the occurrence time of events, thereby facilitating the management of the storage system 100 according to the time range corresponding to each index node when querying and updating the storage system 100.

[0044] According to an exemplary implementation of this disclosure, if the current time has reached or exceeded the end time of time slice 240, the shard inode 230 can be marked as read-only. According to an exemplary implementation of this disclosure, after entering a new time slice 242, a period of waiting can be continued to ensure that the inodes for late events are added to the correct shard inode. According to an exemplary implementation of this disclosure, before marking the state as read-only, it can be detected whether each event occurring within time slice 240 has already been stored in storage system 100. If the result is yes, the shard inode 230 is marked as read-only. If the result is no, a period of waiting can continue.

[0045] It will be understood that although the foregoing only schematically illustrates the execution of method 300 at one computing resource in storage system 100, according to an exemplary implementation of this disclosure, method 300 can be executed in parallel at multiple computing resources. Specifically, multiple threads of one or more computing devices in storage system 100 can create inodes for a large number of events entering storage system 100. It will be understood that each time slice corresponds to only one slice inode, and inodes created separately by multiple threads can be added to the corresponding slice inodes.

[0046] Using the exemplary implementation of this disclosure, multiple computing resources can process a large number of events entering storage system 100 in parallel. The multiple computing resources can operate independently; for example, they can independently create inodes for events occurring within time slice 240. Furthermore, the inodes created by each computing resource can be added to segmented inode 230.

[0047] As time goes by, new sharded index nodes can be continuously added to the root node 220 of the index. See below for more details. Figure 6 More details describing the state of nodes in the index. Figure 6 A block diagram 600 schematically illustrates the state of each shard index node in an index according to an exemplary implementation of this disclosure. The root node 220 includes shard index node 230 (pointing to an event occurring on day X), shard index node 232 (pointing to an event occurring on day X+1), ..., and shard index node 234 (pointing to an event occurring on day X+7). Assuming the current time is day X+7, shard index node 230 becomes an expired index node 540, while shard index nodes 232, ..., 234 are valid index nodes 530. At this time, shard index node 234 is in a read-write state, meaning it can be indexed to for events occurring on day X+7 and can also be read to provide query services. Shard index nodes 232, etc., are in a read-only state 510 and can only be read, not written to.

[0048] According to an exemplary implementation of this disclosure, a query operation can be performed based on the valid index node 530 in the index. Specifically, if a query request is received to find an event in the storage system 100 that meets a specified time condition, the sharded index node corresponding to the specified time condition is searched in the index. Assuming the specified time condition is 00:20 on day X+1, the sharded index node 232 can be found in the valid index node 530, and the event that meets the specified time condition can be obtained from the sharded index node 232. If the sharded index node 232 includes a multi-level index structure, the index node matching the specified time condition can also be found based on the time range of each level of index node, thereby finding the corresponding event in the storage system 100.

[0049] According to an exemplary implementation of this disclosure, updates to the storage system 100 can be managed based on individual shard inodes in an index. Shard inodes in the index that have not reached their expiration date can be identified, and events associated with the found shard inodes can be removed from the storage system 100. See below for further details. Figure 7 More details are described regarding the update of storage system 100. Figure 7 A block diagram illustrating a process 700 for removing expired events according to an exemplary implementation of this disclosure is shown. Assuming it is determined that a shard inode 230 has expired, all events associated with that shard inode 230 can be retrieved in the storage system 100, and these events can be deleted.

[0050] like Figure 7 As shown, sharded index nodes 230 include a multi-level index structure, and window index nodes 430, ..., 432 respectively include indexes of events occurring in each hour of day X. Each window index node can be traversed sequentially; specifically, window index node 430 includes segment index nodes 710, ..., and 712. Based on segment index node 710, index nodes of events occurring within that time segment can be found, i.e., leaf nodes 720, ..., 722. Based on these leaf nodes, events 730, ..., 732 stored in the storage system can be found, and the found events can be deleted. Furthermore, sharded index nodes 230 and their child nodes (as shown in the shaded area) can be removed from the index. Using the exemplary implementation of this disclosure, events that meet the expiration criteria can be quickly found based on the index of storage system 100. In this way, expired data in storage system 100 can be accurately and effectively removed, thereby ensuring that storage system 100 can include sufficient available storage space.

[0051] As mentioned above Figures 2 to 7Examples of methods according to this disclosure have been described in detail, and implementations of corresponding apparatus will be described below. According to an exemplary implementation of this disclosure, an apparatus for managing an index of a storage system is provided. The apparatus includes: a receiving module configured to receive a purge cycle for clearing expired events in the storage system, the storage system including a plurality of events in a data stream; a determining module configured to determine a time shard for managing the index of the storage system based on the purge cycle; a creating module configured to create a sharded index node for the time shard in response to determining that the occurrence time of a first event entering the storage system from the data stream is within the time shard, the sharded index node including an index node for the first event in the storage system; and an adding module configured to add the sharded index node to an index of the storage system. According to an exemplary implementation of this disclosure, the apparatus further includes modules for performing other steps in the method 300 described above.

[0052] Figure 8 A block diagram of a device 800 for managing an index of a storage system, according to an exemplary implementation of the present disclosure, is shown schematically. As shown, the device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. Various programs and data required for the operation of the device 800 may also be stored in RAM 803. The CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0053] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0054] The various processes and procedures described above, such as method 300, can be executed by processing unit 801. For example, in some implementations, method 300 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some implementations, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more steps of method 300 described above can be performed. Alternatively, in other implementations, CPU 801 can also be configured in any other suitable manner to implement the above-described processes / methods.

[0055] According to an exemplary implementation of this disclosure, an electronic device is provided, including: at least one processor; volatile memory; and a memory coupled to the at least one processor, the memory having instructions stored therein, the instructions causing the device to perform a method for managing an index of a storage system when executed by the at least one processor. The method includes: receiving a purge period for clearing expired events in the storage system, the storage system including a plurality of events in a data stream; determining time shards for managing the index of the storage system based on the purge period; creating sharded index nodes for the time shard in response to determining that the occurrence time of a first event entering the storage system in the data stream is within the time shard, the sharded index nodes including an index node for the first event in the storage system; and adding the sharded index nodes to the index of the storage system.

[0056] According to an exemplary implementation of this disclosure, creating a sharded index node includes: dividing a time shard into multiple time windows according to a predetermined length of the time window; in response to determining that the occurrence time of a first event is within a time window of the multiple time windows, creating a window index node for the time window, the window index node including the index node of the first event in the storage system; and adding the window index node to the sharded index node.

[0057] According to an exemplary implementation of this disclosure, the method further includes: in response to determining that the occurrence time of a second event entering the storage system in the data stream is within a time shard, adding the index node of the second event in the storage system to the shard index node.

[0058] According to an exemplary implementation of this disclosure, the method further includes: in response to determining that the current time exceeds a time shard, determining another time shard for managing the storage system based on the current time and a cleanup period, the other time shard being adjacent to the time shard.

[0059] According to an exemplary implementation of this disclosure, the method further includes: in response to determining that the occurrence time of a third event entering the storage system in the data stream is within another time shard, creating another shard index node for the other time shard, the other shard index node including an index of the third event in the storage system; and adding the other shard index node to the index.

[0060] According to an exemplary implementation of this disclosure, the method further includes: marking the sharded index node as read-only.

[0061] According to an exemplary implementation of this disclosure, the method further includes: in response to receiving a query request to query for an event that satisfies a specified time condition in the storage system, searching in the index for a sharded index node corresponding to the specified time condition; and obtaining the event that satisfies the specified time condition based on the sharded index node.

[0062] According to an exemplary implementation of this disclosure, the method further includes: receiving the validity period of an event in the storage system; in response to determining that a time shard of a sharded index node does not meet its validity period, removing the event associated with the sharded index node from the storage system; and removing the sharded index node from the index.

[0063] According to an exemplary implementation of this disclosure, adding an index node of the second event in the storage system to a shard index node includes: creating an index node of the second event in the storage system using computing resources among multiple computing resources in the storage system; and adding the index node to the shard index node.

[0064] According to an exemplary implementation of this disclosure, a computer program product is provided, which is tangibly stored on a non-transient computer-readable medium and includes machine-executable instructions for performing a method according to this disclosure.

[0065] According to an exemplary implementation of this disclosure, a computer-readable medium is provided. The computer-readable medium stores machine-executable instructions that, when executed by at least one processor, cause the at least one processor to implement the method according to this disclosure.

[0066] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0067] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0068] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0069] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some implementations, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is customized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0070] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0072] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0074] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for managing indexes in a storage system, comprising: A system including a processor receives a cleanup cycle for clearing expired events in a storage system, wherein the storage system includes multiple events in a data stream, and wherein the cleanup cycle includes a first amount of time elapsed between iterations of performing the cleanup of the expired events in the storage system; Based on the first time quantity, a time slice including a second time quantity is determined, wherein the slice inode is configured to store an indication of events occurring during the second time quantity within the index of the storage system; In response to determining that the first occurrence time of a first event in the events entering the storage system in the data stream corresponds to the second time quantity, a shard index node is created for the time shard, wherein the shard index node includes the first index node of the first event in the storage system; Add the sharded index node to the index of the storage system; as well as The indication of the events that occurred during the time period corresponding to the second time quantity is stored within the sharded index node.

2. The method according to claim 1, wherein creating the sharded index node comprises: The time slice is divided into multiple time windows according to the predetermined length of the time window; In response to determining that the first occurrence time of the first event is within one of the plurality of time windows, a window index node is created for the time window, wherein the window index node includes the index node of the first event in the storage system; as well as Add the window index node to the shard index node.

3. The method according to claim 1, further comprising: In response to determining that the second occurrence time of the second event entering the storage system in the data stream is within the time shard, the second index node of the second event in the storage system is added to the shard index node.

4. The method of claim 3, wherein adding the second event to the second index node in the storage system as the shard index node comprises: The second event is created in the storage system by using computing resources among multiple computing resources in the storage system; as well as Add the second index node to the shard index node.

5. The method of claim 1, further comprising: In response to determining that the current time exceeds the time shard, another time shard for managing the index of the storage system is determined based on the current time and the purge cycle, wherein the other time shard is adjacent to the time shard.

6. The method of claim 5, further comprising: In response to determining that the third occurrence time of a third event entering the storage system in the data stream is within the other time slice, another shard index node is created for the other time slice, wherein the other shard index node includes a third index of the third event in the storage system; and Add the other shard index node to the index.

7. The method of claim 5, further comprising: The sharded index node is marked as read-only.

8. The method of claim 1, further comprising: In response to receiving a query request in the storage system that queries for an event that satisfies a specified time condition, the shard index node corresponding to the specified time condition is searched in the index; as well as Based on the sharded index node, obtain the event that satisfies the specified time condition.

9. The method of claim 1, further comprising: Receive the validity period of events in the storage system; In response to determining that the time shard of the sharded index node does not meet the validity period, the event associated with the sharded index node is removed from the storage system; as well as Remove the shard index node from the index.

10. An electronic device, comprising: At least one processor; Volatile memory; as well as A memory coupled to the at least one processor, the memory having instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the device to perform an operation, the operation including: Receive a cleanup cycle for clearing expired events in a storage system, wherein the storage system includes multiple events in a data stream; Based on the clearing cycle, a time shard including a time amount is determined, wherein the shard inode is configured to store indications of events occurring during the time period indicated by the time shard within the index of the storage system; In response to determining that the first occurrence time of a first event in the events entering the storage system in the data stream is within the time slice, a shard index node is created for the time slice, wherein the shard index node includes the index node of the first event in the storage system; Add the sharded index node to the index of the storage system; and The indication of the event that occurred during the time period indicated by the time segment is stored within the segment index node.

11. The device of claim 10, wherein creating the shard index node comprises: The time slice is divided into multiple time windows according to the predetermined length of the time window; In response to determining that the first occurrence time of the first event is within one of the plurality of time windows, a window index node is created for the time window, wherein the window index node includes the index node of the first event in the storage system; as well as Add the window index node to the shard index node.

12. The device of claim 10, wherein the operation further comprises: In response to determining that the second occurrence time of the second event entering the storage system in the data stream is within the time shard, the second index node of the second event in the storage system is added to the shard index node.

13. The apparatus of claim 12, wherein adding the second event to the second index node in the storage system as a shard index node comprises: The second event is created in the storage system by using computing resources among multiple computing resources in the storage system; as well as Add the second index node to the shard index node.

14. The apparatus of claim 10, wherein the operation further comprises: In response to determining that the current time exceeds the time shard, another time shard for managing the index of the storage system is determined based on the current time and the purge cycle, wherein the other time shard is adjacent to the time shard.

15. The device of claim 14, wherein the operation further comprises: In response to determining that the third occurrence time of a third event entering the storage system in the data stream is within the other time slice, another shard index node is created for the other time slice, wherein the other shard index node includes a third index of the third event in the storage system; and Add the other shard index node to the index.

16. The apparatus of claim 14, wherein the operation further comprises: The sharded index node is marked as read-only.

17. The apparatus of claim 10, wherein the operation further comprises: In response to receiving a query request in the storage system that queries for an event that satisfies a specified time condition, the shard index node corresponding to the specified time condition is searched in the index; as well as Based on the sharded index node, obtain the event that satisfies the specified time condition.

18. The apparatus of claim 10, wherein the operation further comprises: Receive the validity period of events in the storage system; In response to determining that the time shard of the sharded index node does not meet the validity period, the event associated with the sharded index node is removed from the storage system; as well as Remove the shard index node from the index.

19. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions for performing the method according to any one of claims 1 to 9.

Images