An index management method, apparatus, device, and medium

Abstract

The application discloses an index management method and device, equipment and medium, and relates to the technical field of enterprise networks. The method comprises the following steps: querying the current state information of each index every preset interval time, and judging whether the preset state update condition is met; updating the index relationship state of the corresponding index according to the judgment result, generating an index relationship table based on each index and the corresponding index relationship state; if a new index needs to be created at present, generating the index name and time range value of the new index according to the current creation time, and adding the new index to the index relationship table; when a target operation of document insertion or document retrieval needs to be performed, determining the target time range value based on the target time corresponding to the target operation, and screening the target index corresponding to the target time range value from the index relationship table to perform the target operation. The index relationship state can be updated in time, and the created index can be associated with the time range value to filter the appropriate index according to the time range.

Description

Technical Field

[0001] This invention relates to enterprise network technology, and in particular to an index management method, apparatus, device, and medium. Background Technology

[0002] In a security monitoring system, millions of security logs are collected and stored from security devices every day. These logs are then stored in Elasticsearch (a search engine) after undergoing ETL (Electronic Data Transmission) technology. The data retention period is usually required to be more than one year. The UI of the security monitoring system needs to provide various statistical charts based on the log data, such as the top 5 attack destinations in 24 hours and the distribution trend of attack events in 7 days. At the same time, there are many background tasks within the system that need to retrieve data from Elasticsearch regularly to perform various evaluations and calculations.

[0003] Elasticsearch can meet various data storage and retrieval needs, but when the data volume is very large, some problems can arise if Elasticsearch is not managed at the business level. For example, if the index cannot be filtered by time range during retrieval, and wildcards (such as event-*) are used to match the index, a large number of indexes and massive documents may be matched, which will slow down the query response speed and may even cause Elasticsearch to hang and be unable to respond to new requests. It is necessary to control the number of documents written to a single index; otherwise, no new documents can be written after the number of documents in the index reaches a certain number. It is also necessary to avoid generating too many indexes with few documents, as these index fragments will consume system file handles, CPU, and memory resources.

[0004] Currently, Elasticsearch version 6.7 and above supports index lifecycle management. Index lifecycle management can automatically manage the number of documents in an index and the lifespan of the index, which solves the problems of oversized indexes and index fragmentation to some extent. However, queries still match the index using wildcards, which is not flexible enough.

[0005] In summary, improving the flexibility of index matching is a problem that needs to be solved. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide an index management method, apparatus, device, and medium that can improve the flexibility of index matching. The specific solution is as follows:

[0007] Firstly, this application discloses an index management method, including:

[0008] At preset intervals, the target interface is called to query the current status information of each index, and it is determined whether the current status information meets the preset status update conditions.

[0009] Update the index relationship status of the corresponding index based on the judgment result, and generate an index relationship table based on each index and the index relationship status corresponding to each index;

[0010] If the determination result indicates that a new index needs to be created, then the index name and time range value of the new index are generated based on the current creation time, and the new index is added to the index relationship table;

[0011] When a target operation needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation, and a target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation.

[0012] Optionally, determining whether the current state information meets the preset state update conditions includes:

[0013] Determine the maximum number of index documents, index closing period, and index expiration period configured in the preset status update conditions, and determine the index relationship status corresponding to each index;

[0014] If the index relationship status of the index is a first preset status or a second preset status, then determine whether the number of index documents in the current status information exceeds the maximum number of index documents; the first preset status is used to characterize the index as a state where data is written preferentially, and the second preset status is used to characterize the index as a state where data can be written.

[0015] If the index relationship status of the index is the second preset status or the third preset status, then it is determined whether the index creation time in the current status information exceeds the index closing period; the third preset status is used to indicate that the index is in a read-only state.

[0016] If the index relationship status of the index is the second preset status, the third preset status, or the fourth preset status, then it is determined whether the index creation time in the current status information exceeds the index expiration period; the fourth preset status is used to indicate that the index is in a closed state.

[0017] Optionally, updating the index relationship status of the corresponding index based on the judgment result includes:

[0018] If the index relationship status of the index is a first preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to a second preset status, and a new index is created, and the index relationship status of the new index is set to the first preset status.

[0019] If the index relationship status of the index is the second preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to the third preset status.

[0020] If the index relationship status of the index is in the second preset status or the third preset status, and the index creation time corresponding to the index exceeds the index closing period, then the index relationship status of the index will be updated to the fourth preset status.

[0021] If the index relationship status of the index is a second preset status, a third preset status, or a fourth preset status, and the index creation time corresponding to the index exceeds the index expiration period, then the index relationship status of the index will be updated to a fifth preset status; the fifth status is used to indicate that the index is in an expired state.

[0022] Optionally, the index management method further includes:

[0023] Traverse the index relationship table to obtain the status of each index and its corresponding index relationship;

[0024] If all index relationship states corresponding to the index are in the fourth preset state, then the target interface is called to close the index;

[0025] If all index relationship states corresponding to the index are in the fifth preset state, then the target interface is called to delete the index.

[0026] Optionally, generating the index name and time range value of the new index based on the current creation time includes:

[0027] The index name of the new index is generated based on the current creation time, index category, and index sequence number;

[0028] The time range value of the new index is generated based on the current creation time and using a preset time partitioning strategy; the preset time partitioning strategy is to divide each month into several time regions and assign a value to each time region to obtain the corresponding time range value.

[0029] Optionally, when a target operation needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation, and a target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index, including:

[0030] When a document insertion operation needs to be performed on document data, the generation time of the document to be inserted is determined, and the target time range value corresponding to the generation time is determined using the preset time partitioning strategy.

[0031] The target index corresponding to the target time range value is selected from the index relationship table so that the document to be inserted can be saved to the target index;

[0032] When it is necessary to perform a document retrieval operation on the document data, the target retrieval time is determined, and the target time range value corresponding to the target retrieval time value is determined by the preset time partitioning strategy.

[0033] Target indexes corresponding to the target time range value are selected from the index relationship table so that the corresponding index name can be obtained based on the target index.

[0034] Optionally, the step of filtering the target index corresponding to the target time range value from the index relationship table includes:

[0035] Determine whether the target time range value has expired. If it has expired, the corresponding target index is an empty index. If it has not expired, determine whether there is an index in the index relationship table that corresponds to the target time range value.

[0036] If it does not exist, create a new index and use the new index as the target index;

[0037] If they exist, all indexes corresponding to the target time range and their index relationship status are obtained from the index relationship table, and the target indexes whose index relationship status meets the preset conditions are selected.

[0038] Secondly, this application discloses an index management device, comprising:

[0039] The update judgment module is used to call the target interface at preset intervals to query the current status information of each index and determine whether the current status information meets the preset status update conditions.

[0040] The relation table generation module is used to update the index relation status of the corresponding index according to the judgment result, and generate an index relation table based on each index and the index relation status corresponding to each index.

[0041] The new index creation module is used to generate the index name and time range value of the new index based on the current creation time if the judgment result determines that a new index needs to be created, and to add the new index to the index relationship table.

[0042] The target operation execution module is used to determine a target time range value based on the target time corresponding to the target operation when a target operation needs to be performed on document data, and to filter out the target index corresponding to the target time range value from the index relationship table so as to execute the target operation based on the target index; the target operation includes document insertion operation and document retrieval operation.

[0043] Thirdly, this application discloses an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] A processor for executing the computer program to implement the steps of the aforementioned disclosed index management method.

[0046] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed index management method.

[0047] As can be seen, this application calls the target interface at preset intervals to query the current status information of each index and determines whether the current status information meets the preset status update conditions; updates the index relationship status of the corresponding index based on the judgment result, and generates an index relationship table based on each index and the index relationship status corresponding to each index; if the judgment result determines that a new index needs to be created, the index name and time range value of the new index are generated based on the current creation time, and the new index is added to the index relationship table; when a target operation needs to be performed on document data, the target time range value is determined based on the target time corresponding to the target operation, and the target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation. Therefore, this application periodically calls the target interface to query the current status information of each index, updates the index relationship status of the corresponding index when the current status information meets the preset status update conditions, and generates an index relationship table based on each index and the corresponding index relationship status, which facilitates the lifecycle management of the index. When a new index needs to be created, the index name and time range value are generated based on the current creation time. This means that the application can associate the created index with a time range value. When document insertion and retrieval operations are performed on document data, the target time corresponding to the operation is determined, and then the target time range value is determined. The target time range value is then used to filter matching target indexes from the index relationship table before proceeding with subsequent operations. In other words, this application implements index filtering based on time ranges to match suitable indexes, improving the flexibility of index matching. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a flowchart of an index management method disclosed in this application;

[0050] Figure 2 This application discloses a specific index relationship transformation diagram;

[0051] Figure 3 This application discloses a flowchart of an index relationship status update process.

[0052] Figure 4 This application discloses a flowchart of an index relationship table synchronization process.

[0053] Figure 5 This application discloses a flowchart of an index lifecycle management process.

[0054] Figure 6 This application discloses a specific index management method flowchart;

[0055] Figure 7 This application discloses a flowchart for obtaining the index name during a document insertion operation.

[0056] Figure 8 This application discloses a flowchart for obtaining index names during a document retrieval process.

[0057] Figure 9 This is a schematic diagram of the structure of an index management device disclosed in this application;

[0058] Figure 10 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0060] Currently, Elasticsearch version 6.7 and later support index lifecycle management. Index lifecycle management can automatically manage the number of documents in an index and the index's lifespan, solving the problems of excessively large indexes and index fragmentation to some extent. However, queries still match indexes using wildcards, which is not flexible enough. Therefore, this application discloses an index management method, apparatus, device, and medium that can improve the flexibility of index matching.

[0061] See Figure 1 As shown in the figure, this application discloses an index management method, which includes:

[0062] Step S11: Call the target interface at preset intervals to query the current status information of each index, and determine whether the current status information meets the preset status update conditions.

[0063] In this embodiment, the Elasticsearch index management middleware calls the target interface to query the current status information of each index at preset intervals. The target interface is specifically the es interface, and the current status information includes the current number of documents stored in the index, the index creation time, etc. Furthermore, the above-mentioned determination of whether the current status information meets the preset status update conditions includes: determining the maximum number of index documents, the index closing period, and the index expiration period configured in the preset status update conditions, and determining the index relationship status corresponding to each index; if the index relationship status of the index is a first preset status or a second preset status, then determining whether the number of index documents in the current status information exceeds the maximum number of index documents; the first preset status is used to characterize the index as a state where data is preferentially written, and the second preset status is used to characterize the index as a state where data can be written; if the index relationship status of the index is the second preset status or a third preset status, then determining whether the index creation time in the current status information exceeds the index closing period; the third preset status is used to characterize the index as a read-only status; if the index relationship status of the index is the second preset status, the third preset status, or a fourth preset status, then determining whether the index creation time in the current status information exceeds the index expiration period; the fourth preset status is used to characterize the index as a closed status. It is understood that this embodiment will periodically obtain the number of indexed documents and the index creation time, and compare them with the maximum number of indexed documents, the index closing period and the index expiration period configured in the preset status update conditions, so as to update the index relationship status corresponding to the index based on the comparison results.

[0064] It should be noted that the index relationship status is used to identify the current state of the index relationship, that is, whether the index can be used to write data, whether it can be used to retrieve data, whether it has expired, etc. There are five index relationship statuses: active, writable, readonly, close, and expire. In this embodiment, the first preset status corresponds to active, indicating that the index corresponding to this index relationship is newly created, that is, the number of documents has not reached the maximum value, and data can be written first; the second preset status corresponds to writable, indicating that the index corresponding to this index relationship was created previously, that is, the number of documents has reached the maximum value, but has not exceeded 5% of the maximum value, and data can still be written; the third preset status corresponds to readonly, indicating that the index corresponding to this index relationship was created previously, and currently can only be read but not written, and the number of documents has exceeded 5% of the maximum value; the fourth preset status corresponds to close, indicating that data can no longer be read or written through this index relationship, but documents matching this index relationship should still be saved; the fifth preset status corresponds to expire, indicating that data can no longer be read or written through this index relationship, and documents matching this index relationship will no longer be saved. In addition, there are transition relationships between the five index relationship states. The state machine is used to describe and restrict these state transitions. The index relationship states have the following restrictions: among multiple index relationships of the same index category, only one can be in the active state; among multiple index relationships corresponding to the same time range key, multiple index relationships can simultaneously exist in the active, writable, and readonly states, but there can be at most one active state; among multiple index relationships corresponding to the same time range key, if one is in the closed state, then the others must also be in the closed state; among multiple index relationships corresponding to the same time range key, if one is in the expired state, then the others must also be in the expired state. Figure 2 This application discloses a specific index relationship transformation diagram, from Figure 2 As can be seen from this: active can be converted to writable; writable can be converted to readonly, close, and expire states; readonly can be converted to close and expire states; close can be converted to expire state; expire state cannot be converted further.

[0065] Step S12: Update the index relationship status of the corresponding index according to the judgment result, and generate an index relationship table based on each index and the index relationship status corresponding to each index.

[0066] In this embodiment, updating the index relationship status of the corresponding index based on the judgment result specifically includes: if the index relationship status of the index is a first preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to a second preset status, and a new index is created, with the index relationship status of the new index set to the first preset status; if the index relationship status of the index is a second preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to a third preset status; if the index relationship status of the index is a second or third preset status, and the index creation time corresponding to the index exceeds the index closing period, then the index relationship status of the index is updated to a fourth preset status; if the index relationship status of the index is a second, third, or fourth preset status, and the index creation time corresponding to the index exceeds the index expiration period, then the index relationship status of the index is updated to a fifth preset status; the fifth status is used to indicate that the index is in an expired state. See also Figure 3 As shown, Figure 3 This application discloses a flowchart of an index relationship status update process. As can be seen from the foregoing, the Elasticsearch index management middleware periodically calls the ES interface to query the status information of all current indexes, including the index creation time and the number of index documents, and determines whether the status information meets the preset status update conditions. Then, based on the judgment result, the index relationship status of the corresponding index is updated. If the index relationship status is in the first preset state (active), then it is checked whether the number of indexed documents exceeds the maximum number of indexed documents. If it does, the index relationship status is updated to the second preset state (writeable), a new index is created, and the index relationship status of the new index is set to the first preset state (active). If the index relationship status is in the second preset state (active), then it is checked whether the number of indexed documents exceeds the maximum number of indexed documents. If it does, the index relationship status is updated to the third preset state (readonly). If the index relationship status is in the second preset state (writeable) or the third preset state (readonly), then it is checked whether the index creation time exceeds the index closing period. If it does, the index relationship status is updated to the fourth preset state (close). If the index relationship status is in the second preset state (writeable), the third preset state (readonly), or the fourth preset state (close), then it is checked whether the index creation time exceeds the index expiration period. If it does, the index relationship status is updated to the fifth preset state (expire). This method can update the index relationship status of the corresponding index in a timely manner and can also automatically generate a new index after the limit is exceeded by monitoring the number of indexed documents.

[0067] Additionally, this embodiment can perform statistics on currently existing indexes to generate an index relationship table based on each index and the index relationship status corresponding to each index. It's understandable that when using the index management middleware on an Elasticsearch instance that has been running for some time, or when modifying the time partitioning strategy during the use of the index management middleware, the index relationship table must be synchronized first. The index relationship table contains four attributes: index category, index name, document time range key, and index relationship status. (See details...) Figure 4 As shown, Figure 4 This application discloses a flowchart for synchronizing an index relationship table. After calling the Elasticsearch interface to query the status information of all current indexes, including index creation time and the number of index documents, the Elasticsearch interface can be called to aggregate document time for each index at a daily granularity. This yields the time distribution of documents in the index. Then, using the index document time distribution from the previous step and the configured time differentiation granularity, the time range key contained in each index is calculated. Based on the index information and the configured strategy, the index relationship status of each time range key and the corresponding index is calculated. Finally, the index relationships of all indexes are combined to complete the synchronization of the index relationship table.

[0068] Furthermore, the above method also includes: traversing the index relationship table to obtain each index and its corresponding index relationship status; if all index relationship statuses corresponding to an index are in a fourth preset state, then calling the target interface to close the index; if all index relationship statuses corresponding to an index are in a fifth preset state, then calling the target interface to delete the index. That is, participants... Figure 5 As shown, Figure 5 This application discloses a flowchart for index lifecycle management. This embodiment traverses the index relationship table to obtain each index and its corresponding index relationship status. Based on the index relationship status, it automatically closes or deletes the index. When all index relationship statuses of an index are in the fourth preset state (close), the Elasticsearch interface is called to close the index. When all index relationship statuses of an index are in the fifth preset state (expire), the Elasticsearch interface is called to delete the index. Timely updates to the index relationship status facilitate index lifecycle management, enabling the automatic closure or deletion of expired indexes.

[0069] Step S13: If the judgment result determines that a new index needs to be created, then generate the index name and time range value of the new index according to the current creation time, and add the new index to the index relationship table.

[0070] In this embodiment, if it is determined that a new index needs to be created, the index name and time range value (i.e., time range key) of the new index are generated according to the current creation time. Then, an index relationship with an active status is generated according to the index name and time range value, and the generated new index is added to the index relationship table.

[0071] Step S14: When a target operation needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation, and a target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation.

[0072] In this embodiment, when a target operation such as document insertion or document retrieval needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation. Then, the target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index. That is, when inserting or retrieving document data, the Elasticsearch index management middleware first finds a suitable index through the index relationship table before performing subsequent operations. Furthermore, this process achieves the filtering of suitable indexes from the index relationship based on the time range value, improving the flexibility of index matching.

[0073] As can be seen, this application calls the target interface at preset intervals to query the current status information of each index and determines whether the current status information meets the preset status update conditions; updates the index relationship status of the corresponding index based on the judgment result, and generates an index relationship table based on each index and the index relationship status corresponding to each index; if the judgment result determines that a new index needs to be created, the index name and time range value of the new index are generated based on the current creation time, and the new index is added to the index relationship table; when a target operation needs to be performed on document data, the target time range value is determined based on the target time corresponding to the target operation, and the target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation. Therefore, this application periodically calls the target interface to query the current status information of each index, updates the index relationship status of the corresponding index when the current status information meets the preset status update conditions, and generates an index relationship table based on each index and the corresponding index relationship status, which facilitates the lifecycle management of the index. When a new index needs to be created, the index name and time range value are generated based on the current creation time. This means that the application can associate the created index with a time range value. When document insertion and retrieval operations are performed on document data, the target time corresponding to the operation is determined, and then the target time range value is determined. The target time range value is then used to filter matching target indexes from the index relationship table before proceeding with subsequent operations. In other words, this application implements index filtering based on time ranges to match suitable indexes, improving the flexibility of index matching.

[0074] See Figure 6 As shown, this application discloses a specific index management method. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0075] Step S21: Call the target interface at preset intervals to query the current status information of each index, and determine whether the current status information meets the preset status update conditions.

[0076] Step S22: Update the index relationship status of the corresponding index according to the judgment result, and generate an index relationship table based on each index and the index relationship status corresponding to each index.

[0077] Step S23: If the judgment result determines that a new index needs to be created, then generate the index name of the new index based on the current creation time, index category, and index sequence number.

[0078] In this embodiment, if the index relationship status of an index is in the first preset state "active" and the number of indexed documents exceeds the maximum number of indexed documents, then it is determined that a new index needs to be created. The index name of the new index is generated based on the current creation time, index category, and index sequence number. It should be noted that the Elasticsearch index management middleware does not directly associate the index name. The index name can be generated using any strategy, such as index category + random number, index category + number, or index category + timestamp. In this embodiment, in order to intuitively show the index creation time from the index name, the method of index category - current creation time (i.e., index creation date) - index sequence number is used. For example, event_common-2022.02.27-000001. If a second index is to be generated on February 27, 2022, then the date remains the same, the sequence number is incremented by 1, and the index name is event_common-2022.02.27-000002.

[0079] Step S24: Generate the time range value of the new index based on the current creation time and using a preset time partitioning strategy, and add the new index to the index relationship table; the preset time partitioning strategy is to divide each month into several time regions and assign a value to each time region to obtain the corresponding time range value.

[0080] In this embodiment, it is also necessary to generate the time range value of the new index based on the current creation time and using a preset time partitioning strategy. Specifically, the preset time partitioning strategy divides each month into several time zones and assigns a value to each time zone to obtain the corresponding time range value. Specifically, the time partitioning strategy can be daily, weekly, monthly, or other strategies. For example, if a month is partitioned into the first, middle, and last ten days of the month, then the time range key generated for January 2022 would have three values, as follows:

[0081] 2022.01-1 represents the time range from January 1, 2022 to January 10, 2022;

[0082] 2022.01-2 represents the time range from January 11, 2022 to January 20, 2022;

[0083] The time range represented by 2022.01-3 is from 2022.01.21 to 2022.01.31.

[0084] Step S25: When a document insertion operation needs to be performed on the document data, the generation time of the document to be inserted is determined, and the target time range value corresponding to the generation time is determined using the preset time partitioning strategy; the target index corresponding to the target time range value is selected from the index relationship table so that the document to be inserted can be saved to the target index.

[0085] In this embodiment, when a document insertion operation needs to be performed on document data, the generation time of the document to be inserted is first determined, and then the target time range value corresponding to the generation time is determined using a preset time partitioning strategy. For example, if the generation time of the document to be inserted is 2021.12.15, the corresponding target time range value is 2021.12-2. Then, the indexes with the target time range value of 2021.12-2 are selected from the index relationship table, and the target index that meets the conditions is determined from these indexes so that the document to be inserted can be saved to the target index.

[0086] Furthermore, the process of filtering the target index corresponding to the target time range value from the index relationship table includes: determining whether the target time range value has expired; if expired, the corresponding target index is an empty index; if not expired, determining whether an index corresponding to the target time range exists in the index relationship table; if not, creating a new index and using it as the target index; if it exists, obtaining all indexes corresponding to the target time range and their index relationship status from the index relationship table, and filtering out the target index whose index relationship status meets preset conditions. It can be understood that before the data writing client inserts document data into Elasticsearch, it searches for a suitable index in the index relationship table based on the document's generation time and index category. During the search, it also checks the index relationship status and ultimately returns a unique index name, i.e., the aforementioned target index. The data writing client uses this index name to insert document data. (See details...) Figure 7 As shown, Figure 7This application discloses a flowchart for obtaining the index name during a document insertion operation. First, a time range value is generated based on the document generation time and a preset time partitioning strategy. It then checks if the time range value has expired. If expired, the corresponding target index is an empty index, meaning no index name is returned. If not expired, it checks if an index corresponding to the time range exists in the index relationship table. If no index exists, it checks if there is an index in the index relationship status that is active and belongs to the same index category. If so, the target index corresponding to the active status is returned. If not, a new index is created and used as the target index. If an index corresponding to the time range exists, all indexes corresponding to the time range and their index relationship statuses are retrieved from the index relationship table, and the target indexes whose index relationship statuses meet preset conditions are selected. Specifically, after obtaining all indexes and their relationship statuses corresponding to the time range, the system first checks if there are any indexes in an active state; if so, it returns the corresponding index name. Next, it checks if there are any indexes in a writeable state; if so, it returns the corresponding index name. If neither active nor writeable indexes exist, the active index corresponding to the index category is used. If the corresponding index category does not yet have an active index, a new index is created. This approach ensures that document data is preferentially inserted into indexes where the document count is not exceeded, avoiding index fragmentation.

[0087] Step S26: When a document retrieval operation needs to be performed on the document data, the target retrieval time is determined, and the target time range value corresponding to the target retrieval time is determined using the preset time partitioning strategy; the target index corresponding to the target time range value is filtered from the index relationship table so as to obtain the corresponding index name according to the target index.

[0088] In this embodiment, if a document retrieval operation needs to be performed on document data, the corresponding target retrieval time is determined, i.e., which time period's document data needs to be retrieved. Then, a preset time partitioning strategy is used to determine the target time range corresponding to the target retrieval time. For example, if data for February 9, 2022 needs to be queried, the corresponding time range is February 1, 2022. The target index with a time range of February 1, 2022 is selected from the index relationship table to obtain the corresponding index name. That is, before the data retrieval client retrieves data from Elasticsearch, it searches for suitable indexes in the index relationship table based on the retrieval time range and index category. Ultimately, it returns an array of index names that meet the conditions. The data retrieval client uses this array of index names to perform data retrieval. It should be noted that the above search process also checks the status of the index relationship, excluding indexes with a status of "close" or "expire". (See details...) Figure 8 As shown, Figure 8 This application discloses a flowchart for obtaining index names during a document retrieval operation. First, an array of time range values ​​is generated based on the retrieval time range and a preset time partitioning strategy. Then, all index relationships corresponding to each time range value are sequentially searched, and the status of each index relationship is checked sequentially. Indexes with statuses of "close" and "expire" are excluded. Finally, an array of index names corresponding to the remaining indexes is returned.

[0089] For more detailed processing procedures of steps S21 and S22, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0090] As can be seen, in this embodiment, if a new index needs to be created, a time range value for the new index needs to be generated based on the current creation time and a preset time partitioning strategy to establish a connection between the index and the creation time, facilitating subsequent selection of suitable indexes based on the corresponding time range value. When a new document needs to be inserted, the document will be preferentially inserted into the index whose number of documents corresponding to the time range value has not exceeded the limit, i.e., the index corresponding to the first preset state "active" and the second preset state "writeable", to avoid index fragmentation. If there is no index corresponding to the time range value in the index relationship table, a new index will be created, and the document will be inserted into the new index. When a document needs to be retrieved, a time range value array is first generated based on the time range value of the retrieval time and the preset time partitioning strategy, and all index relationships corresponding to each time range value are searched in turn. After excluding indexes with the states "close" and "expire", an array of index names corresponding to the remaining indexes is returned.

[0091] The technical solution of this application will be explained in detail below through specific examples.

[0092] (I) Index Relationship Table

[0093] 1. Create the first index:

[0094] When the first index is created for a certain index category, the index name's time is the current time, the index relationship status is active, and the time range value is generated based on the current time. For example, if the current time is January 5, 2022, the corresponding time range value is 2022.01-1. See Table 1 for details:

[0095] Table 1

[0096] event_common 2022.01-1 event_common-2022.01.05-0001 active

[0097] Documents created between January 1, 2022 and January 10, 2022 will be saved in the index event_common-2022.01.05-0001.

[0098] 2. If the number of indexed documents exceeds the maximum number of indexed documents, a new index will be generated:

[0099] After running for a few days, the number of indexed documents exceeds the maximum, so a new index is generated. The previous index relationship status becomes writable, and the new index relationship status becomes active. For example, if the current time is January 9, 2022, the corresponding time range is 2022.01-1. See Table 2 for details.

[0100] Table 2

[0101]

[0102]

[0103] If a document is inserted at this point and the time range matches 2022.01-1, it will be saved to the index event_common-2022.01.09-0001.

[0104] 3. Inserting a document whose time range values ​​do not match any index relationships:

[0105] If an inserted document cannot find a matching index relationship based on its time range value, then the document is saved to the index corresponding to the active index, and a new index relationship is added using the document's time range value and the name of the active index. The index relationship status is then set to writable. For example, if the inserted document's time is 2021.12.15, the corresponding time range is 2021.12-2. See Table 3 for details.

[0106] Table 3

[0107] event_common 2022.01-1 event_common-2022.01.05-0001 writable [[ID=3N]]event_common 2022.01-1 event_common-2022.01.09-0001 active event_common 2021.12-2 event_common-2022.01.09-0001 writable

[0108] Save the document to event_common-2022.01.09-0001 and add an index relationship with a time range of 2021.12-2.

[0109] 4. Insert a document whose time range matches at least one of the multiple index relationship states that is writable, but has no active state:

[0110] In this scenario, the document was generated at a previous time, and the index corresponding to its time range has reached the maximum number of documents, but has not yet exceeded 5% of the maximum number of documents. Therefore, it can continue to be saved in the corresponding index. For example, if the document was generated on January 10, 2022, the corresponding time range is January 1, 2022. See Table 4 for details.

[0111] Table 4

[0112] event_common 2022.01-1 event_common-2022.01.05-0001 writable event_common 2022.01-1 [[ID=N0]]event_common-202N.01.09-0001 writable event_common 2021.12-2 event_common-2022.01.09-0001 writable event_common 2022.01-2 event_common-2022.01.15-0001 active

[0113] In this case, the document can be saved to event_common-2022.01.09-0001 and event_common-2022.01.05-0001, and you can choose one of them according to the time sequence.

[0114] 5. Insert a document whose time range matching index relationship status is all readonly:

[0115] In this scenario, the document was generated at a previous time, and the index corresponding to its time range has reached the maximum number of documents, exceeding 5% of the maximum number of documents, so further writing is not possible. In this case, the document is saved to the index corresponding to the active index, and a new index relationship is added using the document's time range value and the name of the active index. For example, if the document's generation time is 2021.12.15, the corresponding time range is 2021.12-2. See Table 5 for details.

[0116] Table 5

[0117] event_common 2022.01-1 event_common-2022.01.05-0001 writable event_common 2022.01-1 event_common-2022.01.09-0001 readonly event_common 2021.12-2 event_common-2022.01.09-0001 readonly event_common 2022.01-2 event_common-2022.01.15-0001 active event_common 2021.12-2 event_common-2022.01.15-0001 writable

[0118] The time range corresponding to the document generation time, event_common-2022.01.09-0001, cannot be written. The data is saved to the currently active index event_common-2022.01.15-0001, and the time range 2021.12-2 is added to event_common-2022.01.15-0001.

[0119] 6. Insert a document whose time range matches the index status is closed:

[0120] In this case, the index corresponding to the time range value is closed, but the document still needs to be saved. Save the document to the active index, and add a new index relationship using the document's time range key and the name of the active index. The index relationship status is closed. For example, if the document's generation time is 2021.12.15, the corresponding time range value is 2021.12-2. See Table 6 for details.

[0121] Table 6

[0122] event_common 2022.01-1 event_common-2022.01.05-0001 writable event_common 2022.01-1 event_common-2022.01.09-0001 writable event_common 2021.12-2 event_common-2022.01.09-0001 close event_common 2022.01-2 event_common-2022.01.15-0001 active event_common 2021.12-2 event_common-2022.01.15-0001 close

[0123] The index for event_common-2022.01.09-0001, which corresponds to the document's generation time, is in a closed state. The data needs to be saved to the currently active index event_common-2022.01.15-0001, and the time range 2021.12-2 needs to be added to event_common-2022.01.15-0001. The index relationship status will then be closed.

[0124] (ii) Filtering indexes by time

[0125] With the index relationship table in place, queries can filter the appropriate index list based on time range values. For example, the current index relationship table is shown in Table 7:

[0126] Table 7

[0127]

[0128]

[0129] 1. To query data for February 9, 2022, the corresponding time range is 2022.02-1. The indexes found are: event_common-2022.02.02-0001 and event_common-2022.02.02-0002.

[0130] 2. We need to query data for the month from January 10, 2022 to February 9, 2022, corresponding to the time ranges of January 1, 2022, January 2, 2022, January 3, and February 1, 2022. The indexes retrieved are:

[0131] event_common-2022.01.05-0001, event_common-2022.01.09-0001,

[0132] event_common-2022.01.15-0001, event_common-2022.02.02-0001,

[0133] event_common-2022.02.02-0002

[0134] 3. When querying data for December 19, 2021, the corresponding time range is 2021.12-2. The indexes found are event_common-2022.01.09-0001 and event_common-2022.01.15-0001. However, because the corresponding index relationship status is closed, the index will not be returned and no data can be found.

[0135] 4. Querying data for the month from June 1, 2021 to June 30, 2021, with the corresponding time range keys being 2021.06-1, 2021.06-2, and 2021.06-3, the index retrieved is event_common-2021.06.15-0001. However, because the corresponding index relationship status is expired, the index will not actually be returned, and no data will be found.

[0136] See Figure 9 As shown in the figure, this application discloses an index management device, which includes:

[0137] The update judgment module 11 is used to call the target interface at preset intervals to query the current status information of each index and determine whether the current status information meets the preset status update conditions.

[0138] The relation table generation module 12 is used to update the index relation status of the corresponding index according to the judgment result, and generate an index relation table based on each index and the index relation status corresponding to each index.

[0139] The new index creation module 13 is used to generate the index name and time range value of the new index according to the current creation time if the judgment result determines that a new index needs to be created, and add the new index to the index relationship table.

[0140] The target operation execution module 14 is used to determine a target time range value based on the target time corresponding to the target operation when a target operation needs to be performed on document data, and to filter out the target index corresponding to the target time range value from the index relationship table so as to execute the target operation based on the target index; the target operation includes document insertion operation and document retrieval operation.

[0141] As can be seen, this application calls the target interface at preset intervals to query the current status information of each index and determines whether the current status information meets the preset status update conditions; updates the index relationship status of the corresponding index based on the judgment result, and generates an index relationship table based on each index and the index relationship status corresponding to each index; if the judgment result determines that a new index needs to be created, the index name and time range value of the new index are generated based on the current creation time, and the new index is added to the index relationship table; when a target operation needs to be performed on document data, the target time range value is determined based on the target time corresponding to the target operation, and the target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation. Therefore, this application periodically calls the target interface to query the current status information of each index, updates the index relationship status of the corresponding index when the current status information meets the preset status update conditions, and generates an index relationship table based on each index and the corresponding index relationship status, which facilitates the lifecycle management of the index. When a new index needs to be created, the index name and time range value are generated based on the current creation time. This means that the application can associate the created index with a time range value. When document insertion and retrieval operations are performed on document data, the target time corresponding to the operation is determined, and then the target time range value is determined. The target time range value is then used to filter matching target indexes from the index relationship table before proceeding with subsequent operations. In other words, this application implements index filtering based on time ranges to match suitable indexes, improving the flexibility of index matching.

[0142] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the index management method performed by the electronic device disclosed in any of the foregoing embodiments.

[0143] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0144] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0145] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0146] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the index management method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0147] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the method steps executed during the index management process disclosed in any of the foregoing embodiments.

[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0149] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0151] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The above provides a detailed description of the index management method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An index management method, characterized in that, include: At preset intervals, the target interface is called to query the current status information of each index, and it is determined whether the current status information meets the preset status update conditions. Update the index relationship status of the corresponding index based on the judgment result, and generate an index relationship table based on each index and the index relationship status corresponding to each index; If the determination result indicates that a new index needs to be created, then the index name and time range value of the new index are generated based on the current creation time, and the new index is added to the index relationship table; When a target operation needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation, and a target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index; the target operation includes document insertion operation and document retrieval operation; The index relationship states are sequentially classified into the first preset state, the second preset state, the third preset state, the fourth preset state, and the fifth preset state according to the index lifecycle. The index relationship conversion methods include conversion from the first preset state to the second preset state, conversion from the second preset state to the third, fourth, or fifth preset state, conversion from the third preset state to the fourth or fifth preset state, and conversion from the fourth preset state to the fifth preset state. The step of determining whether the current state information meets the preset state update conditions includes: Determine the maximum number of index documents, index closing period, and index expiration period configured in the preset status update conditions, and determine the index relationship status corresponding to each index; If the index relationship status of the index is a first preset status or a second preset status, then determine whether the number of index documents in the current status information exceeds the maximum number of index documents; the first preset status is used to characterize the index as a state where data is written preferentially, and the second preset status is used to characterize the index as a state where data can be written. If the index relationship status of the index is the second preset status or the third preset status, then it is determined whether the index creation time in the current status information exceeds the index closing period; the third preset status is used to indicate that the index is in a read-only state. If the index relationship status of the index is the second preset status, the third preset status, or the fourth preset status, then it is determined whether the index creation time in the current status information exceeds the index expiration period; the fourth preset status is used to indicate that the index is in a closed state. The step of updating the index relationship status of the corresponding index based on the judgment result includes: If the index relationship status of the index is a first preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to a second preset status, and a new index is created, and the index relationship status of the new index is set to the first preset status. If the index relationship status of the index is the second preset status, and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, then the index relationship status of the index is updated to the third preset status. If the index relationship status of the index is in the second preset status or the third preset status, and the index creation time corresponding to the index exceeds the index closing period, then the index relationship status of the index will be updated to the fourth preset status. If the index relationship status of the index is a second preset status, a third preset status, or a fourth preset status, and the index creation time corresponding to the index exceeds the index expiration period, then the index relationship status of the index is updated to a fifth preset status; the fifth preset status is used to indicate that the index is in an expired state.

2. The index management method according to claim 1, characterized in that, Also includes: Traverse the index relationship table to obtain the status of each index and its corresponding index relationship; If all index relationship states corresponding to the index are in the fourth preset state, then the target interface is called to close the index; If all index relationship states corresponding to the index are in the fifth preset state, then the target interface is called to delete the index.

3. The index management method according to claim 1 or 2, characterized in that, The step of generating the index name and time range value of the new index based on the current creation time includes: The index name of the new index is generated based on the current creation time, index category, and index sequence number; The time range value of the new index is generated based on the current creation time and using a preset time partitioning strategy; the preset time partitioning strategy is to divide each month into several time regions and assign a value to each time region to obtain the corresponding time range value.

4. The index management method according to claim 3, characterized in that, When a target operation needs to be performed on document data, a target time range value is determined based on the target time corresponding to the target operation, and a target index corresponding to the target time range value is selected from the index relationship table so that the target operation can be performed based on the target index, including: When a document insertion operation needs to be performed on document data, the generation time of the document to be inserted is determined, and the target time range value corresponding to the generation time is determined using the preset time partitioning strategy. The target index corresponding to the target time range value is selected from the index relationship table so that the document to be inserted can be saved to the target index; When it is necessary to perform a document retrieval operation on the document data, the target retrieval time is determined, and the target time range value corresponding to the target retrieval time value is determined by the preset time partitioning strategy. Target indexes corresponding to the target time range value are selected from the index relationship table so that the corresponding index name can be obtained based on the target index.

5. The index management method according to claim 4, characterized in that, The step of filtering the target index corresponding to the target time range value from the index relationship table includes: Determine whether the target time range value has expired. If it has expired, the corresponding target index is an empty index. If it has not expired, determine whether there is an index in the index relationship table that corresponds to the target time range value. If it does not exist, create a new index and use the new index as the target index; If they exist, all indices corresponding to the target time range value and their index relationship status are obtained from the index relationship table, and the target indexes whose index relationship status meets the preset conditions are selected.

6. An index management device, characterized in that, include: The update judgment module is used to call the target interface at preset intervals to query the current status information of each index and determine whether the current status information meets the preset status update conditions. The relation table generation module is used to update the index relation status of the corresponding index according to the judgment result, and generate an index relation table based on each index and the index relation status corresponding to each index. The new index creation module is used to generate the index name and time range value of the new index based on the current creation time if the judgment result determines that a new index needs to be created, and to add the new index to the index relationship table. The target operation execution module is used to determine a target time range value based on the target time corresponding to the target operation when a target operation needs to be performed on document data, and to filter out the target index corresponding to the target time range value from the index relationship table so as to execute the target operation based on the target index; the target operation includes document insertion operation and document retrieval operation; The index relationship states are sequentially classified into the first preset state, the second preset state, the third preset state, the fourth preset state, and the fifth preset state according to the index lifecycle. The index relationship conversion methods include conversion from the first preset state to the second preset state, conversion from the second preset state to the third, fourth, or fifth preset state, conversion from the third preset state to the fourth or fifth preset state, and conversion from the fourth preset state to the fifth preset state. The update judgment module is specifically used to determine the maximum number of index documents, the index closing period, and the index expiration period configured in the preset state update conditions, and to determine the index relationship status corresponding to each index; if the index relationship status of the index is a first preset status or a second preset status, then it is determined whether the number of index documents in the current status information exceeds the maximum number of index documents; the first preset status is used to indicate that the index is in a state where data is written preferentially, and the second preset status is used to indicate that the index is in a state where data can be written; if the index relationship status of the index is the second preset status or a third preset status, then it is determined whether the index creation time in the current status information exceeds the index closing period; the third preset status is used to indicate that the index is in a read-only status; if the index relationship status of the index is the second preset status, the third preset status, or a fourth preset status, then it is determined whether the index creation time in the current status information exceeds the index expiration period; the fourth preset status is used to indicate that the index is in a closed status; The relationship table generation module is specifically configured to: if the index relationship status of the index is in a first preset state and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, update the index relationship status of the index to a second preset state, create a new index, and set the index relationship status of the new index to the first preset state; if the index relationship status of the index is in the second preset state and the number of index documents corresponding to the index exceeds the maximum number of documents in the index, update the index relationship status of the index to a third preset state; if the index relationship status of the index is in the second or third preset state and the index creation time corresponding to the index exceeds the index closing period, update the index relationship status of the index to a fourth preset state; if the index relationship status of the index is in the second, third, or fourth preset state and the index creation time corresponding to the index exceeds the index expiration period, update the index relationship status of the index to a fifth preset state; the fifth preset state is used to indicate that the index is in an expired state.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the index management method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of the index management method as described in any one of claims 1 to 5.

Images