Method, apparatus, and program product for managing indexes of a storage system

Abstract

The present disclosure relates to methods, devices and program products for managing indexes of a storage system. In one method, a plurality of events included in a data stream to be stored to the storage system are received. At least one event of the plurality of events occurring within a time window is determined according to a predetermined length of the time window and occurrence times of the plurality of events. A window index node is created based on the at least one event, the window index node including an index of the at least one event. The window index node is added to an index in response to determining that a current time point satisfies a threshold time point corresponding to the time window, the threshold time point indicating that a number of events in the data stream occurring within the time window that have been received reaches a threshold number. Corresponding devices and computer program products are provided. With exemplary implementations of the present disclosure, indexes can be created in time for a large number of events entering the storage system. Further, the storage system can be queried and updated accurately and efficiently.

Description

TECHNICAL FIELD

[0001] Implementations of the present disclosure relate to a storage system, and more particularly, to a method, device and computer program product for managing indexes of a storage system. BACKGROUND

[0002] With the development of data storage technology, various data storage devices have been able to provide users with increasingly higher data storage capabilities, and the data access speed has also been greatly improved. For a storage system of streaming data, a large number of events in the data stream are continuously stored to the storage system. At present, a technical solution for creating indexes for events has been proposed. However, the performance of the existing technical solution is not satisfactory. At this time, how to create and manage the indexes of the storage system with higher performance, and further improve the performance of the entire storage system, has become a technical problem. SUMMARY

[0003] Therefore, it is desirable to develop and implement a technical solution for managing the indexes of the storage system in a more effective manner. It is desirable that the technical solution can manage the indexes of the storage system in a more convenient and effective manner. Further, it is desirable that the created indexes can be used to improve the access and update efficiency of the storage system.

[0004] According to a first aspect of the present disclosure, a method for managing indexes of a storage system is provided. In the method, a plurality of events included in a data stream to be stored to the storage system are received. At least one event of the plurality of events occurring within a time window is determined according to a predetermined length of the time window and occurrence times of the plurality of events. A window index node is created based on the at least one event, the window index node including an index of the at least one event. In response to determining that a current time point satisfies a threshold time point corresponding to the time window, the window index node is added to the indexes, the threshold time point indicating that a number of events in the data stream occurring within the time window and having been received reaches a threshold number.

[0005] According to a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; a volatile memory; and a memory coupled with the at least one processor, the memory having stored therein instructions that, when executed by the at least one processor, cause the device to perform the method according to the first aspect of the present disclosure.

[0006] According to a third aspect of the present disclosure, a computer program product is provided, the computer program product being tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions for performing the method according to the first aspect of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0007] The features, advantages, and other aspects of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which several implementations of the present disclosure are illustrated. The detailed description and drawings are merely illustrative of the present disclosure, wherein:

[0008] Figure 1 A block diagram of a storage system is schematically shown in accordance with one aspect;

[0009] Figure 2 A block diagram of a process for managing an index of a storage system is schematically shown in accordance with an example implementation of the present disclosure;

[0010] Figure 3 A flow diagram of a method for managing an index of a storage system is schematically shown in accordance with an example implementation of the present disclosure;

[0011] Figure 4 A block diagram of a process for receiving a plurality of events by a plurality of computing resources is schematically shown in accordance with an example implementation of the present disclosure;

[0012] Figure 5 A block diagram of a process for managing an index of a storage system is schematically shown in accordance with an example implementation of the present disclosure;

[0013] Figure 6 A block diagram of a process for adding a segment index node to an index is schematically shown in accordance with an example implementation of the present disclosure;

[0014] Figure 7 A block diagram of a process for updating an index is schematically shown in accordance with an example implementation of the present disclosure;

[0015] Figure 8 A block diagram of a process for updating an index is schematically shown in accordance with an example implementation of the present disclosure; and

[0016] Figure 9 A block diagram of an apparatus for managing an index of a storage system is schematically shown in accordance with an example implementation of the present disclosure. DETAILED DESCRIPTION

[0017] Preferred implementations of the present disclosure will be described herein below with reference to the accompanying drawings. While the preferred implementations of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the implementations set forth herein. Rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0018] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example implementation" and "an implementation" mean "at least one example implementation." The term "another implementation" means "at least one additional implementation." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] In the context of the present disclosure, the events in the data stream may involve multiple types, and each event may occur at a different time. For example, in an application environment that monitors the operating status of a data center, the data stream may include log data of each device in the data center. In this case, the events in the data stream may be log data generated at various time points (e.g., every minute, or other time intervals). For another example, in an application environment that monitors road traffic conditions, the data stream may include images, videos, and / or other data collected by multiple collection devices at multiple locations.

[0020] It will be understood that the data flow is continuous and may involve massive amounts of data. For ease of storage, a large number of events may be divided into multiple shards based on a hash algorithm, etc., and the multiple shards may be stored in different storage devices. For ease of description, first refer to Figure 1 Describes an overview of the storage system. Figure 1 The block diagram of a storage system according to a technical solution is schematically shown. Figure 1 As shown, storage system 100 may include data shards 122, ..., 132, etc. Currently, technical solutions have been proposed for creating indexes for individual data shards. For example, indexes 120, ..., can be created for data shard 122, and index 130 can be created for data shard 132. In this case, the index of storage system 100 will include index group 110, and index group 110 includes indexes 120, ..., 130 for each data shard. Furthermore, storage system 100 can be managed based on index group 110.

[0021] However, users of storage system 100 need to develop specialized applications to manage index group 110. When a query request is received, one or more indexes corresponding to the query request can be found in index group 110 based on a hash algorithm, and one or more query results can be returned. Furthermore, the returned one or more query results can be filtered to serve as the final query result. It will be appreciated that during the index management process, failures and / or misoperations may occur, leading to errors in the index. Therefore, existing index management solutions are not effective in managing storage system indexes.

[0022] To address the above-mentioned drawbacks, the implementation method of the present disclosure provides a technical solution for managing the index of a storage system. This technical solution proposes creating an index based on the occurrence time of multiple events in a data stream. Specifically, a time window and a threshold time point of the time window are proposed. If the current time exceeds the threshold time point of a given time window, it means that among the multiple events that have occurred within the given time window, the number of events that have entered the storage system has reached the threshold number. At this time, the index of the event that has entered the storage system can be added to the total index of the storage system.

[0023] Using the exemplary implementations of the present disclosure, indexes can be created within the storage system for events occurring within each time window, based on the predetermined length of the time window. This allows for index management within the storage system, eliminating the need for users to develop their own index management applications. Furthermore, using the exemplary implementations of the present disclosure, as time passes, an index can be created for events that have entered the storage system each time the current time point reaches a threshold time point for the corresponding time window. This allows for timely and efficient index creation, avoiding delays in the index creation process.

[0024] In the following, see Figure 2 A process according to an exemplary implementation of the present disclosure is generally described. Figure 2 A block diagram of a process 200 for managing an index of a storage system according to an exemplary implementation of the present disclosure is schematically shown. A data stream 210 may include a large number of events. Here, the data stream 210 is continuous and may generate new events over time. Figure 2 As shown, the length of time window 230 can be pre-specified (e.g., 1 minute or other values). For example, events occurring within time window 230 may include events 212, 214, ..., 216, and 218. In an application environment for monitoring the operating status of a data center, events may include logs generated by various computing devices at different time points.

[0025] It will be understood that events herein involve occurrence time and reception time. Occurrence time refers to the time when an event occurs, for example, when a log is generated; reception time refers to the time when the event is received by the storage system 100, that is, the time when the event enters the storage system 100. Due to potential delays in data transmission, the reception time will be later than the occurrence time, and in situations such as network failures, there may be a significant delay.

[0026] It will be understood that events that occur in chronological order are not necessarily received by the storage system 100 in the same order. Figure 2As indicated by arrow 222, event 212 is received at time point T1; as indicated by arrow 224, event 214 is received at time point T3; as indicated by arrow 226, event 216 is received at time point T2; and as indicated by arrow 228, event 218 is received at time point T4.

[0027] According to one exemplary implementation of the present disclosure, a threshold time point 240 can be set for the time window 230. As shown, time points T1, T2 and T3 are before the threshold time point 240, and time point T4 is after the threshold time point 240. That is, events 212, 214 and 216 are received before the threshold time point 240, and event 218 is received after the threshold time point 240. After events 212, 214 and 216 have entered the storage system 100, index nodes can be created for these events. When the current time exceeds the threshold time point 240, the created index nodes can be added to the index of the storage system 100. In this way, it can be ensured that the index is created with high efficiency. Figure 2

[0028] In the following, reference will be made to Figure 3 to describe more details according to one exemplary implementation of the present disclosure. Figure 3 A flow chart of a method 300 for managing an index of a storage system according to an exemplary implementation of the present disclosure is schematically shown. The method 300 can be executed at a computing resource in the storage system 100. In this way, a user of the storage system 100 does not have to create an application for managing the index by himself, but can directly use the index management service provided inside the storage system 100. In Figure 3 At block 310, a plurality of events included in a data stream 210 to be stored to the storage system 100 are received. Here, the receiving order of the plurality of events can be different from the occurrence order, that is, the receiving order of the plurality of events can be out-of-order.

[0029] At block 320, at least one event of the plurality of events occurring within a time window 230 is determined according to the predetermined length of the time window 230 and the occurrence time of the plurality of events. Each event can have a timestamp for representing the occurrence time. The length of the time window 230 can be set to 1 minute (or other values), and the time window can be constantly sliding forward as time elapses. Assuming that the time window 230 includes 0thto 1stminute, at least one event occurring in the 0thto 1stminute can be determined according to the timestamps of the respective events.

[0030] ​At block 330, a window index node is created based on the at least one event, the window index node comprising an index of the at least one event. According to one example implementation of the present disclosure, the window index node for the at least one event occurring within the time window 230 can be created based on indexing techniques that have been developed so far and / or will be developed in the future. For example, an index node can be first created for each event occurring within the time window 230, and each created index node can be appended to the window index node. According to one example implementation of the present disclosure, the index nodes can be created based on full-text search techniques.

[0031] It will be appreciated that although the method 300 is only schematically shown as being performed at one computing resource in the storage system 100. The storage system can comprise a plurality of computing resources. The plurality of events can be received by a single computing resource, alternatively and / or additionally, the plurality of events can be received in parallel by a plurality of computing resources. At this point, the plurality of computing resources can operate in parallel in order to improve the efficiency of receiving the events and creating the corresponding index nodes based on the received events. When the events are received by a plurality of computing resources, the events entering the storage system can be randomly assigned to any of the computing resources. Alternatively and / or additionally, the events can be assigned to the computing resources in accordance with the workloads of the respective computing resources.

[0032] In particular, the method 300 can be performed by a plurality of threads of one or more computing devices in the storage system 100. With the example implementation of the present disclosure, the plurality of computing resources can handle a large number of events entering the storage system 100 in parallel. The plurality of computing resources can operate independently, for example, the segmented index nodes for the events occurring within the time window 230 can be created independently. Further, the segmented index nodes created by the respective computing resources can be appended to the window index node 520.

[0033] In the following, reference will be made to Figure 4 Further details regarding the operation of the plurality of computing resources will be described. Figure 4 A block diagram of a process 400 of receiving a plurality of events by a plurality of computing resources according to an example implementation of the present disclosure is schematically shown. As Figure 4 As shown, the plurality of computing resources 410,..., and 412 can receive respective events in parallel. For example, events 212 and 216 are received by the computing resource 410,..., events 214 and 218 are received by the computing resource 412. At this point, each computing resource can create a respective index node for the respective events received. For example, the computing resource 410 can create the segmented index node 420 for events 212 and 216, and the computing resource 412 can create the segmented index node 422 for events 214 and 218. At this point, the time range of each segmented index node 420,..., 422 is the same, i.e. the time range of the time window 230.

[0034] According to one example implementation of the present disclosure, a window index node can include multiple levels. For example, multiple segment index nodes can be created based on multiple computing resources, respectively, and each segment index node can be added to the window index node. In the following, reference will be made to Figure 5 More details about creating the index are described. Figure 5 A block diagram of a process 500 of managing an index of a storage system according to an example implementation of the present disclosure is schematically shown. As Figure 5 shown, the index can include a root node 510, which can include window index nodes 520,..., 522 established based on events within respective time windows. As shown by arrow 566, the window index node 520 can correspond to the time window 230, and the window index node 522 can correspond to another time window (not shown).

[0035] Further, the window index node 520 can include segment index nodes 420,..., 422 created based on multiple computing resources 410,..., 412, respectively. Here, each of the multiple segment index nodes corresponds to events received by one of the multiple computing resources. As shown by arrow 562, the segment index node 420 corresponds to the computing resource 410, and as shown by arrow 564, the segment index node 422 corresponds to the computing resource 412. The segment index node 420 can include leaf nodes 540,..., 542 corresponding to events occurring within the time window 230. Specifically, each leaf node can point to an event stored in the storage system 100. For example, the leaf node 540 can correspond to a stored event 550, and the leaf node 542 can correspond to a stored event 552. In this way, the multiple computing resources can create index nodes for a large number of events in a parallel manner, thereby improving the overall performance of the storage system.

[0036] Having described the process of creating the window index node, the segment index nodes, and the leaf nodes, in the following, reference will be made to Figure 3 how the various nodes created are added to the index of the storage system 100. In Figure 3 at block 340, in response to determining that the current time point satisfies the threshold time point 240 corresponding to the time window 230, the window index node 520 is added to the index. Here, the threshold time point 240 indicates that the number of events in the data stream 210 that have been received among the multiple events occurring within the time window 230 reaches a threshold number.

[0037] According to one example implementation of the present disclosure, the threshold number can be expressed in percentage, e.g., 95% (or other numerical value). At this time, the threshold time point 240 represents the time point at which 95% of the events occurring within the time window 230 have been transmitted to the storage system 100. According to one example implementation of the present disclosure, the threshold number can be expressed in absolute value. Assuming that 1000 events have occurred within the time window 230, the time point at which 950 events enter the storage system 100 can be taken as the threshold time point 240.

[0038] According to one example implementation of the present disclosure, the threshold number can be a fuzzy value. Assuming that historical experience indicates that at the time point 10 seconds (or other length of time) after the end time of the time window 230, most of the events occurring within the time window 230 have entered the storage system 100, the threshold time point 240 can be directly set to the time point 10 seconds after the time window 230.

[0039] With the example implementation of the present disclosure, it is not necessary to wait until all the events occurring within the time window 230 have entered the storage system 100 to create an index for all the events, but at the threshold time point 240, the window index node 520 created based on most of the events that have entered the storage system 100 is added to the root node 510 of the index of the storage system 100. In this way, the index for a large number of events that enter the storage system 100 can be created in a timely and efficient manner.

[0040] According to one example implementation of the present disclosure, after the window node 520 has been added to the root node 510 of the index, each segment index node 530,..., and 532 in the window index node 520 can be set to a read-only state. At this time, each segment index node 530,..., and 532 cannot be modified, but can only be queried to serve access requests from the outside world. It will be understood that due to network delay and other reasons, some events occurring within the time window 230 can enter the storage system 100 after the window index node 520 has been added to the index. At this time, another segment index node can be created for these late events.

[0041] Specifically, if it is determined that the event in the data stream 210 is received after the threshold time point 240, another new segment index node can be created based on the event. In the following, the process 600 for adding a segment index node to the index according to the example implementation of the present disclosure will be described with reference to FIG. 6. Figure 6 More details about creating a new segment index node will be described. Figure 6 A block diagram of the process 600 for adding a segment index node to the index according to the example implementation of the present disclosure is schematically shown. As shown in FIG. 6, the process 600 includes the following steps. Figure 6As shown, assuming that events enter the storage system 100 after the threshold time point 240, a segment index node 610 can be created for the event and the segment index node 610 can be added to the window index node 520 corresponding to the time of occurrence of the event.

[0042] The leaf node 620 of the segment index node 610 can point to the event 630 stored in the storage system 100. When a subsequent event occurring within the time window 230 enters the storage system 100, a new leaf node can be created for the subsequent event in a similar manner and the new leaf node can be connected to the segment index node 610. According to one example implementation of the present disclosure, the index of all late events occurring within the time window 230 can be appended to the segment index node 610. According to one example implementation of the present disclosure, if the number of late events is large (e.g., above a specified threshold), multiple segment index nodes can be created and each segment index node can involve a number of events below the specified threshold. Further, the start time and the end time of the segment index node 610 can be set based on the occurrence times of the respective late events.

[0043] According to one example implementation of the present disclosure, merging can be performed for segment index nodes involving fewer events. In this way, the indexes of the storage system 100 can be managed with higher efficiency. According to one example implementation of the present disclosure, the indexes can be further optimized after all events occurring within the time window 230 have been indexed. For example, two or more segment index nodes within the same time window can be merged. Specifically, if two (or more) segment index nodes within a certain time window involve a small number of events (e.g., below a predetermined threshold), the segment index nodes can be merged. According to one example implementation of the present disclosure, two or more segment index nodes within different time windows can be merged. For example, a threshold length of a time segment can be set. Assuming that one segment index node of a time window and another segment index node of a subsequent time window involve only a small number of events, the two segment index nodes can be merged.

[0044] According to one example implementation of the present disclosure, additional conditions can be set for the merging operation. For example, if it is determined that the total length of two consecutive time windows corresponding to two consecutive window index nodes satisfies a predetermined length condition, the respective two segment index nodes of the two consecutive window index nodes can be merged. As another example, a threshold can be set for the total number of events of the merged segment index nodes and the merging operation can only be performed if the total number of events is below the threshold. With the example implementations of the present disclosure, the structure of the indexes can be optimized and the indexes can be stored and used with higher efficiency.

[0045] With the example implementation of the present disclosure, the created index can include a multi-level index structure, and each non-leaf node in the index can include a time range of events related to the node. For example, the time range of the root node 510 involves a portion between the occurrence time of the earliest event and the occurrence time of the latest event in the storage system 100. For another example, the time range of the window index node 520 involves a portion between the start time and the end time of the time window 230. For yet another example, the time range of the segment index nodes 420,..., 422 is the same as the time window 230. After the index has been created, the time ranges in the index can be used to manage time-related operations in the storage system 100.

[0046] In the following, how to use the index to perform a query in the storage system 100 will be described. According to one example implementation of the present disclosure, if a query request is received to query events in the storage system 100 that satisfy a specified time condition, a window index node corresponding to the specified time condition can be first looked up in the index. Assuming that the query is expected to occur between 0 to 1 minute, a child node with a time range between 0 to 1 minute can be looked up in each child node of the root node 510 of the index. At this time, the window index node 520 can be found, and then events satisfying the specified time condition can be obtained based on the found window index node 520.

[0047] Specifically, events satisfying the specified time condition can be looked up in the child nodes of the window index node 520 (i.e., the plurality of segment index nodes 420,..., 422 and 610), and corresponding leaf nodes can be further found in the segment index nodes, and then the events desired to be queried can be determined in the addresses of the storage system 100. With the example implementation of the present disclosure, events corresponding to the specified time condition can be quickly and accurately found by using the time ranges of the index nodes in the multi-level index.

[0048] According to one example implementation of the present disclosure, the established index can be used to manage updates of the storage system 100. It will be appreciated that as time goes by, a large number of events continuously enter the storage system 100. Generally speaking, in a monitoring or the like application environment, only events occurring recently (e.g., events occurring in the last week) are concerned, and events occurring much earlier are not concerned. Thus, the storage system 100 needs to be updated periodically, and the created index can be used to manage the update process of the storage system 100.

[0049] According to an exemplary implementation of the present disclosure, if a removal request is received to remove an event that meets an expiration condition from storage system 100, a window index node corresponding to the expiration condition can be searched in the index. Subsequently, the event associated with the window index node can be removed from storage system 100. According to an exemplary implementation of the present disclosure, the expiration time point specified by the expiration condition may fall between two time windows or within a time window. This will be described below in different scenarios.

[0050] If the expiration time falls between two time windows, the expiration condition does not span the time windows, that is, the occurrence time of all events in the time window before the expiration time meets the expiration condition. In this case, all events corresponding to the time window can be deleted. Figure 7 Schematically shows a block diagram of a process 700 for updating an index according to an exemplary implementation of the present disclosure. Figure 7 As shown in FIG. 7 , the expiration time point 710 specified by the expiration condition is at the end time of the time window 230, so the end time of the window index node 520 satisfies the expiration condition. At this time, all events within the time window 230 should be removed. Therefore, all events associated with the window index node 520 can be removed from the storage system based on the window index node 520. Specifically, the subtree ( Figure 7 The shaded portion is shown in the middle), and all expired events in the storage system 100 are found.

[0051] According to an exemplary implementation of the present disclosure, the expiration time point may fall within the time window, and the expiration condition spans the time window. That is, some events within the time window meet the expiration condition, while other events do not meet the expiration condition. Figure 8 Describes more details about the situations in which some events meet the expiration criteria. Figure 8 Schematically shows a block diagram of a process 800 of updating an index according to an exemplary implementation of the present disclosure. Figure 8 As shown, the expiration time point 810 specified by the expiration condition is located within the time window 230, so the end time of the time window 230 does not meet the expiration condition, and the start time of the time window 230 meets the expiration condition.

[0052] At this time, based on the multiple segment index nodes and expiration conditions in the time window 230, the event that meets the expiration condition can be searched within the time window 230. Figure 8As shown, the leaf node 540 of the event that meets the expiration condition can be found in the segment index node 420, and the leaf node 540 and the stored event 550 shown as the shadow can be deleted. Similar processing can be performed on each segment index node separately to find all events that meet the expiration condition.

[0053] By using the exemplary implementation of the present disclosure, events that meet the expiration condition can be quickly found based on the index of the storage system 100. In this way, expired data in the storage system 100 can be accurately and effectively removed, thereby ensuring that the storage system 100 can include sufficient available storage space.

[0054] According to an exemplary implementation of the present disclosure, after the expired data has been removed from the storage system 100, the index may be updated based on the events that have been removed from the storage system 100. Figure 7 and Figure 8 In the index shown, the shaded portion may be deleted to ensure that each node in the index corresponds to the latest data in the storage system 100 .

[0055] See above for Figures 2 to 8 An example of the method according to the present disclosure is described in detail, and the implementation of the corresponding device will be described below. According to the exemplary implementation of the present disclosure, a device for managing the index of a storage system is provided. The device includes: a receiving module, configured to receive a plurality of events included in a data stream to be stored in the storage system; a determining module, configured to determine at least one event that occurs within a time window among the plurality of events according to a predetermined length of the time window and the occurrence time of the plurality of events; a creating module, configured to create a window index node based on at least one event, the window index node including an index of at least one event; and an adding module, configured to add the window index node to the index in response to determining that the current time point meets a threshold time point corresponding to the time window, the threshold time point indicating that the number of received events that occur within the time window in the data stream reaches a threshold number. According to the exemplary implementation of the present disclosure, the device further includes a module for executing the other steps in the method 300 described above.

[0056] Figure 9A block diagram of a device 900 for managing indexes of a storage system according to an example implementation of the present disclosure is shown schematically. As shown, the device 900 includes a central processing unit (CPU) 901 that can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage unit 908. Various programs and data required by the device 900 to operate can also be stored in the RAM 903. The CPU 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0057] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; the storage unit 908, such as a magnetic disk, a magneto-optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0058] The various processes and procedures described above, such as the method 300, can be performed by the processing unit 901. For example, in some implementations, the method 300 can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 908. In some implementations, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the CPU 901, one or more steps of the method 300 described above can be performed. Alternatively, in other implementations, the CPU 901 can also be configured in any other appropriate manner to implement the processes / methods described above.

[0059] According to an example implementation of the present disclosure, there is provided an electronic device comprising: at least one processor; a volatile memory; and a memory coupled with the at least one processor, the memory having stored therein instructions that, when executed by the at least one processor, cause the device to perform a method for managing an index of a storage system. The method comprises: receiving a plurality of events included in a data stream to be stored to the storage system; determining, according to a predetermined length of a time window and occurrence times of the plurality of events, at least one event of the plurality of events that occurs within the time window; creating a window index node based on the at least one event, the window index node comprising an index of the at least one event; and in response to determining that a current time point satisfies a threshold time point corresponding to the time window, the threshold time point representing that a number of events in the data stream that occur within the time window and have been received reaches a threshold number, adding the window index node to the index.

[0060] According to an example implementation of the present disclosure, creating the window index node based on the at least one event comprises: creating a plurality of segment index nodes for the time window respectively using a plurality of computing resources in the storage system, a segment index node of the plurality of segment index nodes corresponding to an event of the at least one event received by a computing resource of the plurality of computing resources; and adding the segment index nodes to the window index node.

[0061] According to an example implementation of the present disclosure, the method further comprises: setting the segment index nodes in the window index node to a read-only state.

[0062] According to an example implementation of the present disclosure, the method further comprises: in response to determining that an event in the data stream is received after the threshold time point, creating another segment index node based on the event; and adding the another segment index node to the window index node.

[0063] According to an example implementation of the present disclosure, the method further comprises: in response to receiving a query request querying events satisfying a specified time condition in the storage system, finding a window index node corresponding to the specified time condition in the index; and obtaining the events satisfying the specified time condition based on the window index node.

[0064] According to an example implementation of the present disclosure, the method further comprises at least one of: merging at least two segment index nodes of the plurality of segment index nodes; and in response to determining that a total length of two consecutive time windows corresponding to two consecutive window index nodes in the index satisfies a predetermined length condition, merging at least two segment index nodes of the two consecutive window index nodes.

[0065] According to one example implementation of the present disclosure, the threshold time point is determined based on a number of events that occur within a time window, and the method is executed in parallel at a plurality of computing resources in the storage system.

[0066] According to one example implementation of the present disclosure, the method further includes, in response to receiving a removal request for removing events that satisfy the expiration condition from the storage system, looking up a window index node corresponding to the expiration condition in the index, and removing events associated with the window index node from the storage system.

[0067] According to one example implementation of the present disclosure, removing events associated with the window index node from the storage system includes at least one of, in response to determining that an end time of the window index node satisfies the expiration condition, removing all events associated with the window index node from the storage system, and in response to the end time not satisfying the expiration condition and a start time of the window node satisfying the expiration condition, removing events associated with at least a portion of the plurality of segment index nodes from the storage system based on the plurality of segment index nodes and the expiration condition.

[0068] According to one example implementation of the present disclosure, the method further includes updating the index based on events that have been removed from the storage system.

[0069] According to an example implementation of the present disclosure, a computer program product is provided, the computer program product being tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions for performing a method according to the present disclosure.

[0070] According to an example implementation of the present disclosure, a computer readable medium is provided. The computer readable medium has stored thereon machine executable instructions that, when executed by at least one processor, cause the at least one processor to implement a method according to the present disclosure.

[0071] The present disclosure can be a method, apparatus, system, and / or computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present disclosure.

[0072] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0073] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0074] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some implementations, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0075] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0076] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0077] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0078] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0079] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being shown and described in terms of certain implementations and overall functions, the implementations are not intended to exclude other implementations or technologies. Modifications and changes can be made in arrangement, operation, and details of the methods and apparatus described. Many modifications and variations of the described implementations are possible and will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. None, individually or taken in any combination, is to be treated as limiting the described implementations unless otherwise indicated.

Claims

1. A method for managing an index of a storage system, comprising: receiving a plurality of events included in a data stream to be stored into the storage system; determining, according to a predetermined length of a time window and occurrence times of the plurality of events, at least one event of the plurality of events occurring within the time window; creating a window index node based on the at least one event, the window index node including an index of the at least one event; and adding the window index node to the index in response to determining that a current time point satisfies a threshold time point corresponding to the time window, the threshold time point indicating that a number of events in the data stream occurring within the time window that have been received reaches a threshold number. 2.The method of claim 1, wherein creating the window index node based on the at least one event comprises: creating a plurality of segment index nodes for the time window respectively using a plurality of computing resources in the storage system, a segment index node of the plurality of segment index nodes corresponding to an event of the at least one event received by a computing resource of the plurality of computing resources; and adding the segment index nodes to the window index node. setting the segment index nodes in the window index node to a read-only state. 4.The method of claim 2, further comprising:

3. The method of claim 2, further comprising: in response to determining that an event in the data stream is received after the threshold time point, creating another segment index node based on the event; and adding the another segment index node to the window index node. 5.The method of claim 1, further comprising: in response to receiving a query request in the storage system querying events satisfying a specified time condition, finding a window index node corresponding to the specified time condition in the index; and obtaining events satisfying the specified time condition based on the window index node. 6.The method of claim 2, further comprising at least one of: merging at least two segment index nodes of the plurality of segment index nodes; and in response to determining that a total length of two consecutive time windows corresponding to two consecutive window index nodes in the index satisfies a predetermined length condition, merging at least two segment index nodes of the two consecutive window index nodes. 7.The method of claim 1, wherein the threshold time point is determined based on a number of events occurring within the time window, and the method is executed in parallel at a plurality of computing resources in the storage system. 8.The method of claim 2, further comprising: in response to receiving a removal request for removing events satisfying an expiration condition from the storage system, finding a window index node corresponding to the expiration condition in the index; and removing events associated with the window index node from the storage system. 9.The method of claim 8, wherein removing the events associated with the window index node from the storage system comprises at least one of: ​ ​ ​ ​ ​ remove, from the storage system, all events associated with the window index node in response to determining that the end time of the window index node satisfies the expiration condition; and remove, from the storage system, events associated with at least a portion of the plurality of segment index nodes based on the plurality of segment index nodes and the expiration condition in response to the end time not satisfying the expiration condition and the start time of the window node satisfying the expiration condition.

10. The method of claim 9, further comprising: update the index based on events that have been removed from the storage system.

11. An electronic device, comprising: at least one processor; a volatile memory; and a memory coupled with the at least one processor, the memory having instructions stored therein that, when executed by the at least one processor, cause the device to perform a method for managing an index of a storage system, the method comprising: receiving a plurality of events included in a data stream to be stored to the storage system; determining, in accordance with a predetermined length of a time window and occurrence times of the plurality of events, at least one event of the plurality of events that occurs within the time window; creating a window index node based on the at least one event, the window index node including an index of the at least one event; and adding the window index node to the index in response to determining that a current time point satisfies a threshold time point corresponding to the time window, the threshold time point representing that a number of events in the data stream that occur within the time window and have been received reaches a threshold number.

12. The device of claim 11, wherein creating the window index node based on the at least one event comprises: creating a plurality of segment index nodes for the time window using a plurality of computing resources in the storage system respectively, a segment index node of the plurality of segment index nodes corresponding to an event of the at least one event received by a computing resource of the plurality of computing resources; and adding the segment index nodes to the window index node. setting the segment index nodes in the window index node to a read-only state.

13. The apparatus of claim 12, wherein the method further comprises:

14. The device of claim 12, wherein the method further comprises: creating another segment index node based on an event of the data stream received after the threshold time point in response to determining that the event is received after the threshold time point; and adding the another segment index node to the window index node.

15. The device of claim 11, wherein the method further comprises: finding, in the index, a window index node corresponding to a specified time condition in response to receiving a query request to query events in the storage system that satisfy the specified time condition; and obtaining events that satisfy the specified time condition based on the window index node.

16. The device of claim 12, wherein the method further comprises at least either of: merging at least two segment index nodes of the plurality of segment index nodes; and ​ ​ ​ In response to determining that a total length of two consecutive time windows corresponding to two consecutive window index nodes in the index satisfies a predetermined length condition, merging at least two segment index nodes in the two consecutive window index nodes.

17. The device of claim 11, wherein the threshold time point is determined based on a number of events that occur within the time window, and the method is executed in parallel at a plurality of computing resources in the storage system.

18. The device of claim 12, wherein the method further comprises: In response to receiving a removal request for removing events from the storage system that satisfy an expiration condition, finding a window index node in the index that corresponds to the expiration condition; and removing events associated with the window index node from the storage system.

19. The device of claim 18, wherein removing the events associated with the window index node from the storage system comprises at least any of: In response to determining that an end time of the window index node satisfies the expiration condition, removing all events associated with the window index node from the storage system; In response to the end time not satisfying the expiration condition and a start time of the window node satisfying the expiration condition, removing events associated with at least a portion of the plurality of segment index nodes from the storage system based on the plurality of segment index nodes and the expiration condition; and updating the index based on events that have been removed from the storage system.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions for executing a method according to any one of claims 1 to 10.

Images