An automatic expansion method for Elasticsearch indexes in streaming data scenarios

Abstract

An automatic expansion method for an Elasticsearch index in a streaming data scenario includes: planning index capacity; setting routing rules; setting logical views; creating index templates; accessing real-time data; and reading data using logical views. The present invention consumes streaming data from a message queue and routes and forwards the data according to the set routing rules, so that the data can be automatically and evenly distributed to multiple indexes. At the same time, when creating an index, the use of index templates is combined with the use of index templates to bind the same field structure and alias to these indexes, thereby providing a unified query and access view for the business layer. This allows the solution to have rich application scenarios, high real-time performance, automatic on-demand expansion, horizontal linear expansion, and simple reading and writing methods when storing and using index data.

Description

Technical Field

[0001] The present invention relates to the field of search engines, and in particular to an automatic expansion method for Elasticsearch indexes in streaming data scenarios. Background Art

[0002] With the advent of the big data era, search engines have played an increasingly important role in software system architectures. Elasticsearch, a renowned open-source search engine server, is widely used in fields such as information retrieval, digital libraries, and IoT big data analysis. However, once an Elasticsearch data index is created, the number of primary shards remains immutable, resulting in an upper limit on the amount of data a single index can accommodate. To accommodate massive amounts of data, developers must either create an index with a large number of primary shards from the outset or pause data writing before an index reaches capacity, resuming data writing after a new index is created. This process requires manual intervention and fails to meet the practical needs of uninterrupted data access in streaming data scenarios.

[0003] Sharding tables and databases is a technical solution for horizontally scaling relational tables in the relational database field. By defining appropriate sharding rules for sharding, originally large table data can be distributed across multiple tables with the same field structure for decentralized storage, accelerating query speeds on relational tables. This decentralized table data can be accessed using a unified logical library. However, since sharding is a concept native to relational databases and each shard must be manually created before use, it cannot adapt to the on-demand automated creation of shards in streaming data scenarios. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide an automatic expansion method of Elasticsearch index in a streaming data scenario, which overcomes the above problems or at least partially solves the above problems.

[0005] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0006] An automatic expansion method for an Elasticsearch index in a streaming data scenario includes:

[0007] S100. Plan the index capacity. According to the real-time data increment, plan the data volume of a single index, determine the shard size of each index, and plan the index capacity.

[0008] S200. Set routing rules, configure routing logic for incoming real-time data, and distribute the data to multiple indexes in Elasticsearch.

[0009] S300. Set up a logical view, which provides a unified query and write entry for these expanded indexes. By using the logical view, multiple indexes can be operated from one logical entry.

[0010] S400 creates an index template, defines the index configuration information in the index template according to the business logic of real-time data, and when the index expansion is triggered, a new index is automatically created according to the configuration in the index template for capacity expansion;

[0011] S500 accesses real-time data and writes the real-time data into the corresponding index for storage according to the set view name;

[0012] S600. Read data using the logical view, and use the view name to query the connected real-time data to provide it to the business logic.

[0013] Furthermore, in S100, the index capacity is planned. The planned single index data capacity should be moderate, and the amount of data that each index can accommodate and the frequency of index expansion should be kept within a reasonable range to avoid a single index with too many shards resulting in excessive data fragmentation and reduced search performance, or a single index with too few shards resulting in the need for frequent index expansion.

[0014] Furthermore, in S200, the routing rules adopted include numerical range sharding and time range sharding; wherein, numerical range sharding is to perform data routing based on a certain numerical type field of real-time data, and each index only contains data within a certain data range; time range sharding is to perform data routing based on a certain time type field of real-time data, and each index only contains data within a certain time range.

[0015] Furthermore, in S300 , the logical graph is set. When the index is expanded horizontally, all the indexes generated by the expansion are directed to the same index alias. The index alias is designated as the name of the logical view.

[0016] Furthermore, an index template is created. The index template configuration items include the index regular expression, field structure, number of shards, and index alias. The index template predefines various parameters for index expansion. When writing data, the template will be applied to create the index only if the written index name matches the regular expression of the index template.

[0017] Furthermore, in S400, when creating an index template, it is necessary to create an index mapping structure based on the field information of the real-time data, specify the number of shards to be included in the created index based on the index size planned for capacity, create an index alias based on the configured logical view name, and automatically bind the index alias to all indexes created using this template.

[0018] Furthermore, in S500, real-time data is accessed. Real-time data has the characteristics of non-stop and uninterrupted. When real-time data is accessed, the data is automatically distributed according to the configured routing rules and stored in the corresponding index. When the existing index is not sufficient to store the currently written streaming data, the index template will be automatically used to trigger an index expansion.

[0019] Furthermore, in S600 , when reading data using a logical view, one logical view is associated with multiple indexes through index aliases, and a query on multiple indexes is completed by querying the view.

[0020] Furthermore, after data routing, indexes are automatically created on demand without manual intervention. The index expansion method is linear, and the data volume is not limited by the capacity of a single index.

[0021] Furthermore, when reading data using a logical view, the logical view treats the data as a unified whole from a business perspective. When writing to the view, the data is automatically routed to the corresponding index; when reading, all indexes generated by the expansion are automatically selected for search.

[0022] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0023] An automatic expansion method for an Elasticsearch index in a streaming data scenario includes:

[0024] Plan the index capacity. According to the incremental real-time data, plan the data size of a single index, determine the shard size of each index, and plan the index capacity. Set routing rules, configure routing logic for the accessed real-time data, and distribute the data to multiple indexes in Elasticsearch. Set up logical views. Through logical views, provide a unified query and write entry for these expanded indexes. By using logical views, the purpose of operating multiple indexes from one logical entry is achieved. Create index templates. Define the index configuration information in the index template according to the business logic of real-time data. When the index expansion is triggered, a new index will be automatically created according to the configuration in the index template to expand the capacity. Access real-time data and write the real-time data into the corresponding index according to the set view name for storage. Use logical views to read data and use the view name to query the accessed real-time data for use by the business logic.

[0025] Compared with the prior art, the technical solution provided by the present invention has the following advantages:

[0026] 1. Rich routing rules and applicable business scenarios. Configure different routing rules according to different business needs. When real-time data arrives, it will be distributed to different indexes according to different routing rules for storage;

[0027] 2. Compatible with streaming data writing scenarios. Streaming data is characterized by non-stop and uninterrupted operation. When distributing data, if this solution finds that an index to be written does not exist in the search engine, it will automatically create one using the index template when writing data. The entire data writing process does not require interruption or human intervention.

[0028] 3. New indexes are automatically created on demand. Since index templates are used when creating indexes, index creation does not require manual operation. Only the index name and the template are matched with the regular expression, and then the index is automatically created. The entire process is automated and created on demand.

[0029] 4. Index capacity scales linearly horizontally, with no upper limit on data volume. Since real-time data is automatically distributed to different indexes based on the configuration of routing rules, each index can only accommodate a certain range of data. The index template only needs to continuously create new indexes to accommodate a wider range of data, and the amount of data that can be accommodated is not limited by a single index;

[0030] 5. Easy reading and writing, providing a unified logical view. Although streaming data is distributed across different indexes, these indexes are bound to the same index alias. When querying or performing statistical analysis on this data, you only need to use the bound index alias, regardless of the specific index the data resides in. From a logical perspective, these indexes remain a unified entity, allowing you to directly read and write data using the same view name.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0033] Figure 1 This is a flowchart of a method for automatically expanding an Elasticsearch index in a streaming data scenario in Example 1 of the present invention;

[0034] Figure 2 This is an architecture diagram of a method for automatically expanding the index of streaming data generated by IoT devices in Example 1 of the present invention;

[0035] Figure 3This is an architecture diagram of a method for automatically expanding the index of streaming order data generated by an e-commerce platform in Example 2 of the present invention. DETAILED DESCRIPTION

[0036] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0037] In order to solve the problems existing in the prior art, an embodiment of the present invention provides an automatic expansion method for Elasticsearch indexes in a streaming data scenario.

[0038] Example 1

[0039] This embodiment discloses an automatic expansion method for Elasticsearch index in a streaming data scenario, such as Figure 1 ,include:

[0040] S100. Plan the index capacity. According to the real-time data increment, plan the data volume of a single index, determine the shard size of each index, and plan the index capacity.

[0041] In S100 of this embodiment, the index capacity is planned. The data capacity of each shard of a planned single index should not exceed 25GB. The amount of data that each index can accommodate and the frequency of index expansion are kept within a reasonable range to avoid excessive data fragmentation and reduced search performance due to the number of shards of a single index being too large, or the need for frequent index expansion due to the number of shards of a single index being too small.

[0042] S200. Set routing rules, configure routing logic for the incoming real-time data, and distribute the data to multiple indexes in Elasticsearch. In S200 of this embodiment, the routing rules used include value range sharding and time range sharding. Among them, value range sharding is to perform data routing based on a certain value type field of the real-time data, and each index only contains data within a certain data range; time range sharding is to perform data routing based on a certain time type field of the real-time data, and each index only contains data within a certain time range.

[0043] S300. Set up a logical view, which provides a unified query and write entry for these expanded indexes. By using the logical view, multiple indexes can be operated from one logical entry.

[0044] In S300 of this embodiment, the logical graph is set. When the index is expanded horizontally, the indexes generated by the expansion are all directed to the same index alias, and the index alias is designated as the name of the logical view.

[0045] S400. Create an index template, define the index configuration information in the index template according to the business logic of real-time data, and when the index expansion is triggered, a new index will be automatically created according to the configuration in the index template to expand the capacity; in S400 of this embodiment, an index template is created, and the index template configuration items include index regular expressions, field structures, number of shards, and index aliases. The index template predefines various parameters for index expansion; when writing data, the template will be applied to create the index only if the written index name matches the regular expression of the index template.

[0046] Specifically, when creating an index template, you need to create an index mapping structure based on the field information of the real-time data, specify the number of shards to be included in the created index based on the index size planned for capacity, create an index alias based on the configured logical view name, and automatically bind the index alias to all indexes created using this template.

[0047] S500. Access real-time data and write it into the corresponding index according to the configured view name for storage. In S500 of this embodiment, real-time data is accessed without downtime or interruption. When real-time data is accessed, data is automatically distributed according to the configured routing rules and stored in the corresponding index. When the existing index is insufficient to store the currently written streaming data, index expansion is automatically triggered using the index template. After data routing, indexes are automatically created on demand without manual intervention. Index expansion is linear, and data volume is not limited by the capacity of a single index.

[0048] S600. Read data using a logical view, using the view name to query the accessed real-time data and provide it to the business logic. In S600 of this embodiment, when reading data using a logical view, a logical view is associated with multiple indexes through index aliases, and the query of multiple indexes is completed by querying the view.

[0049] When reading data using a logical view, the logical view treats the data as a unified whole from a business perspective. When writing to the view, the data is automatically routed to the corresponding index; when reading, all indexes generated by the expansion are automatically selected for search.

[0050] In order to better understand this embodiment, the method for automatically expanding the index of streaming data generated by IoT devices and the method for automatically expanding the index of streaming order data generated by e-commerce platforms are described using the above method. Specifically, Figure 2 The figure below demonstrates how to automatically expand the index for streaming data generated by IoT devices. Since the amount of data generated by IoT devices per unit time is relatively fixed, data routing is performed based on a time range. The routing field is the data generation time field, createtime, and the data slicing step is the month. This means that the index will automatically expand once a month. Streaming sensor data consumed from the MQTT message queue is routed and distributed by the data router based on the data generation time, createtime. Each index only stores data for the current month. Index naming must conform to the regular expression "device-*" set in the index template. This automatically binds these indexes to the same field structure and alias "device-all." Finally, these indexes that have accessed streaming data use the logical view "device-all" to provide a unified query entry for business users.

[0051] like Figure 3 The figure below demonstrates how to automatically scale indexes for streaming order data generated by an e-commerce platform. Since e-commerce orders have peak and trough periods, directly using time periods for data routing can easily lead to uneven data distribution across multiple indexes. Therefore, data routing is performed based on numerical ranges. Capacity planning specifies that each index holds 20 million orders, and the order serial number field, "serial-number," is used for data slicing. This means that for every 20 million order entries, the index will be automatically scaled to store the new order. The data router distributes the data to the corresponding index based on the order serial number. The index template defines the regular expression "order-*," which means that when forwarding data, the index name must be prefixed with "order-." The index template also binds these indexes to the same alias, "order-all," to provide a query view of all order data.

[0052] This embodiment discloses an automatic expansion method for an Elasticsearch index in a streaming data scenario, including: planning index capacity, planning the data volume of a single index based on the incremental real-time data, determining the shard size of each index, and planning the index capacity; setting routing rules, configuring routing logic for the incoming real-time data, and distributing the data to multiple Elasticsearch indexes;

[0053] Set up the logical view, which provides a unified query and write entry for these expanded indexes. By using the logical view, the purpose of operating multiple indexes from one logical entry is achieved; create an index template, define the index configuration information in the index template according to the business logic of the real-time data, and when the index expansion is triggered, a new index will be automatically created according to the configuration in the index template to expand the capacity; access the real-time data, and write the real-time data into the corresponding index according to the set view name for storage; use the logical view to read the data, and use the view name to query the connected real-time data for use by the business logic.

[0054] Compared with the prior art, the technical solution provided by this embodiment has the following advantages:

[0055] 1. Rich routing rules and applicable business scenarios. Configure different routing rules according to different business needs. When real-time data arrives, it will be distributed to different indexes according to different routing rules for storage;

[0056] 2. Compatible with streaming data writing scenarios. Streaming data is characterized by non-stop and uninterrupted operation. When distributing data, if this solution finds that an index to be written does not exist in the search engine, it will automatically create one using the index template when writing data. The entire data writing process does not require interruption or human intervention.

[0057] 3. New indexes are automatically created on demand. Since index templates are used when creating indexes, index creation does not require manual operation. Only the index name and the template are matched with the regular expression, and then the index is automatically created. The entire process is automated and created on demand.

[0058] 4. Index capacity scales linearly horizontally, with no upper limit on data volume. Since real-time data is automatically distributed to different indexes based on the configuration of routing rules, each index can only accommodate a certain range of data. The index template only needs to continuously create new indexes to accommodate a wider range of data, and the amount of data that can be accommodated is not limited by a single index;

[0059] 5. Easy reading and writing, providing a unified logical view. Although streaming data is distributed across different indexes, these indexes are bound to the same index alias. When querying or performing statistical analysis on this data, you only need to use the bound index alias, regardless of the specific index the data resides in. From a logical perspective, these indexes remain a unified entity, allowing you to directly read and write data using the same view name.

[0060] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0061] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention comprises less than all the features of any individual disclosed embodiment. The appended claims are therefore hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0062] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein may be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described around their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person may implement the described functions in an adaptable manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of this disclosure.

[0063] The steps of the methods or algorithms described in conjunction with the embodiments herein may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. The software module may be located in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and storage medium may also be present in a user terminal as discrete components.

[0064] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or external to the processor. In the latter case, it is communicatively coupled to the processor via various means, which are well known in the art.

[0065] The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purposes of describing the above embodiments, but one of ordinary skill in the art will recognize that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to encompass all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent the term "comprising" is used in the specification or claims, the term is intended to be encompassed in a manner similar to the term "including," as explained in terms of "including," used as a transitional word in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

Claims

1. An automatic expansion method for Elasticsearch indexes in streaming data scenarios, characterized in that: include: S100. Plan the index capacity. According to the real-time data increment, plan the data volume of a single index, determine the shard size of each index, and plan the index capacity. S200. Set routing rules, configure routing logic for incoming real-time data, and distribute the data to multiple indexes in Elasticsearch. S300. Set up a logical view, which provides a unified query and write entry for these expanded indexes. By using the logical view, multiple indexes can be operated from one logical entry. S400 creates an index template, defines the index configuration information in the index template according to the business logic of real-time data, and when the index expansion is triggered, a new index is automatically created according to the configuration in the index template for capacity expansion; S500 accesses real-time data and writes the real-time data into the corresponding index for storage according to the set view name; S600. Read data using the logical view, and use the view name to query the connected real-time data to provide it to the business logic.

2. The automatic expansion method of an Elasticsearch index in a streaming data scenario according to claim 1, characterized in that: In S100, the index capacity is planned. The data capacity of each shard of a planned single index does not exceed 25GB. The amount of data that each index can accommodate and the frequency of index expansion are kept within a reasonable range to avoid excessive data fragmentation and reduced search performance due to the number of shards of a single index being too large, or the need for frequent index expansion due to the number of shards of a single index being too small.

3. The automatic expansion method of an Elasticsearch index in a streaming data scenario according to claim 1, characterized in that: In S200, the routing rules used include value range sharding and time range sharding; among them, value range sharding is to perform data routing based on a certain value type field of real-time data, and each index only contains data within a certain data range; time range sharding is to perform data routing based on a certain time type field of real-time data, and each index only contains data within a certain time range.

4. The method for automatically expanding the capacity of an Elasticsearch index in a streaming data scenario according to claim 1, wherein: In S300 , the logical view is set. When the index is expanded horizontally, the indexes generated by the expansion are all pointed to the same index alias, and the index alias is designated as the name of the logical view.

5. The automatic expansion method of an Elasticsearch index in a streaming data scenario according to claim 1, characterized in that: In S400, an index template is created. The index template configuration items include index regular expression, field structure, number of shards, and index alias. The index template predefines various parameters for index expansion. When writing data, the template will be applied to create the index only if the written index name matches the regular expression of the index template.

6. The method for automatically expanding the capacity of an Elasticsearch index in a streaming data scenario according to claim 5, wherein: When creating an index template, you need to create an index mapping structure based on the field information of real-time data, specify the number of shards to be included in the created index based on the planned index size, create an index alias based on the configured logical view name, and automatically bind the index alias to all indexes created using this template.

7. The method for automatically expanding the capacity of an Elasticsearch index in a streaming data scenario according to claim 1, wherein: In S500, real-time data is accessed. Real-time data has the characteristics of non-stop and uninterrupted. When real-time data is accessed, the data is automatically distributed according to the configured routing rules and stored in the corresponding index. When the existing index is not sufficient to store the currently written streaming data, the index template will be automatically used to trigger an index expansion.

8. The method for automatically expanding the capacity of an Elasticsearch index in a streaming data scenario according to claim 1, wherein: In S600 , when reading data using a logical view, one logical view is associated with multiple indexes through an index alias, and a query on multiple indexes is completed by querying the view.

9. The method for automatically expanding the capacity of an Elasticsearch index in a streaming data scenario according to claim 7, wherein: After data routing, indexes are automatically created on demand without manual intervention. Index expansion is linear, and the amount of data is not limited by the capacity of a single index.

10. According to the method for automatically expanding the Elasticsearch index in a streaming data scenario as described in claim 8, when reading data using a logical view, the logical view regards the data as a unified whole from a business perspective. When writing to the view, the data is automatically routed to the corresponding index; when reading, all indexes generated by the expansion are automatically selected for search.

Images