Index construction method, index construction system and index updating method

Abstract

The present application discloses an index construction method, an index construction system, and an index update method. The index construction method obtains data information that needs to be indexed, determines the number of ES indexes to be constructed based on the data volume of the data information; creates an ES index set, the number of ES indexes in the ES index set is equal to the number, and each ES index in the ES index set contains a single primary shard; determines the target data information corresponding to each ES index, and the target data information corresponding to each ES index in the ES index set constitutes the data information, thereby achieving the distinction of data information; writes the target data information corresponding to each ES index into the primary shard of the ES index. Since the corresponding ES index only contains a single primary shard, the target data information can be directly written into the primary shard of the corresponding ES index. It can be seen that the present application can solve the problem of slow ES index construction in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an index construction method, an index construction system, and an index update method. Background Art

[0002] Elasticsearch, short for ES index, is a distributed, scalable, real-time search and analytics engine. Due to its powerful indexing capabilities, ES indexes are widely used. However, existing technologies for building ES indexes are slow, limiting their development. Therefore, improving the speed of ES index building has become a key concern.

[0003] In summary, there is an urgent need for an index construction method to solve the problem of slow ES index construction in the existing technology. Summary of the Invention

[0004] In view of this, the present application provides an index construction method, an index construction system and an index update method to solve the problem of slow ES index construction in the prior art.

[0005] In order to achieve the above objectives, the following solutions are proposed:

[0006] An index building method, comprising:

[0007] Obtain the data information that needs to be indexed, and determine the number of ES indexes to be constructed based on the data volume of the data information;

[0008] Create an ES index set, where the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set contains a single primary shard;

[0009] Determine target data information corresponding to each ES index, where the target data information corresponding to each ES index in the ES index set constitutes the data information;

[0010] The target data information corresponding to each ES index is written into the primary shard of the ES index.

[0011] Optionally, creating an ES index set includes:

[0012] Use a preset index initialization template to set the initial value of each ES index in the ES index set. The index initialization template includes the number of shards, the number of replicas, and the corresponding ES construction node, wherein the number of replicas is zero and the number of shards is one, so that each ES index in the ES index set contains only a single shard, and the shard is the primary shard;

[0013] Writing the target data information corresponding to each ES index into the primary shard of the ES index includes:

[0014] The ES construction node corresponding to each ES index is used to write the target data information corresponding to each ES index into the primary shard of the ES index.

[0015] Optionally, using the ES construction node corresponding to each ES index to write the target data information corresponding to each ES index into the primary shard of the ES index includes:

[0016] Arrange the target data information into a data structure corresponding to the ES index;

[0017] The ES construction node corresponding to the ES index is used to write the sorted target data information into the ES index.

[0018] Optionally, after using the ES construction node corresponding to each ES index and writing the target data information corresponding to each ES index into the primary shard of the ES index, the method further includes:

[0019] Refresh the ES index and merge the segments of the ES index;

[0020] Add an ES query node corresponding to the ES index, and the ES construction node and the ES query node constitute an ES node set, and the ES node set corresponds to the ES index;

[0021] The number of replicas of the ES index is updated, where the updated number of replicas is equal to the number of the ES node set minus 1, so that the ES index obtains the same number of replica shards as the number of replicas, so that the number of shards of the ES index is equal to the number of nodes in the ES node set. The shards of the ES index include primary shards and replica shards.

[0022] Delete the correspondence between the ES index and the ES construction node, and reduce the number of copies of the ES index. The reduced number of copies and the number of replicated shards are equal to the number of ES query nodes minus 1, so that the number of shards of the ES index is equal to the number of ES query nodes.

[0023] Optionally, after writing the target data information corresponding to each ES index into the primary shard of the ES index, the method further includes:

[0024] Check the ES index to determine whether the amount of data in the ES index meets the preset index standard;

[0025] If the index standard is not met, the ES index is deleted and the process returns to the step of creating an ES index set.

[0026] Optionally, before creating an ES index collection, also include:

[0027] Determine whether there is an ES index that has not been successfully built;

[0028] If there is an ES index that has not been successfully built, delete the ES index that has not been successfully built and return to the step of creating an ES index set;

[0029] If there is no ES index that has not been successfully built, directly execute the steps to create an ES index collection.

[0030] An index building system includes an index creation end and a data writing end;

[0031] The index creation end obtains data information to be indexed, determines the number of ES indexes to be constructed based on the data volume of the data information, creates an ES index set, wherein the number of ES indexes in the ES index set is equal to the number, and each ES index in the ES index set contains a single primary shard; and determines target data information corresponding to each ES index, wherein the target data information corresponding to each ES index in the ES index set constitutes the data information;

[0032] The data writing end writes the target data information corresponding to each ES index into the primary shard of the ES index.

[0033] Optionally, the index creation end is further configured to, after determining target data information corresponding to each ES index:

[0034] Sending a data write instruction corresponding to each ES index to the data writing end, wherein the data write instruction includes an index identifier corresponding to the ES index and target data information;

[0035] The data writing end writes the target data information corresponding to each ES index into the primary shard of the ES index, including:

[0036] The target data information corresponding to the ES index having the index identifier is determined through the data write instruction, the target data information is organized into a data structure corresponding to the ES index, and the organized target data information is written into the ES index.

[0037] An index updating method, comprising:

[0038] Get the alias of the original index;

[0039] Update the alias of each ES index in the ES index set created by the above index building method to the alias of the original index, and close the original index.

[0040] Optionally, before updating the alias of each ES index in the ES index set created by the above index building method to the alias of the original index, the method further includes:

[0041] Use the ES index interface to determine the health status of the newly created ES index;

[0042] If the health status meets the update criteria, the newly created ES index set and the original index are queried with the same keyword to obtain the query results corresponding to the newly created ES index set and the query results corresponding to the original index;

[0043] Compare the query results corresponding to the newly created ES index set with the query results corresponding to the original index to obtain a comparison result;

[0044] If the comparison result meets the preset replacement standard, the process returns to execute the step of updating the alias of each ES index in the ES index set created by the above index building method to the alias of the original index and closing the original index.

[0045] It can be seen from the above technical solution that the index construction method provided by the present application obtains the data information that needs to be indexed and determines the number of ES indexes to be constructed according to the data volume of the data information. In this way, the data information corresponding to an ES index with multiple primary shards in the prior art is converted into data information corresponding to multiple ES indexes; based on this, multiple ES indexes containing a single primary shard can be created. The newly created multiple ES indexes constitute an ES index set, and the number of ES indexes in the ES index set is equal to the number. In this way, an ES index with multiple primary shards in the prior art is divided into multiple ES indexes with a single primary shard; after creating multiple indexes, it is necessary to determine the information to be written to each index. Therefore, the target data information corresponding to each ES index can be determined, and the target data information corresponding to each ES index in the ES index set constitutes the data information. In this way, the data information is distinguished, and the target data information corresponding to different ES indexes is distinguished; based on this, when the target data information is written to the corresponding ES index, since the corresponding ES index only contains a single primary shard, the target data information can be directly written to the primary shard of the corresponding ES index. Since the data writing process of different ES indexes can be carried out simultaneously, in this application, each target data information in the data information can be written into the corresponding ES index at the same time, saving the construction time of the ES index and improving the construction speed of the ES index.

[0046] Furthermore, since the ES index may sort the data information related to the query keyword by the degree of matching during the data query process, only the data information within the preset range of the sorting results will be displayed, and all the data information related to the query keyword in the ES index will not be displayed. Therefore, the data information that can be queried by an ES index is limited. Through the index construction method provided by the present application, the data information library corresponding to an ES index in the prior art is divided into multiple data information libraries corresponding to ES indexes, that is, the ES index in the present application is combined with the data information libraries corresponding to each ES index to form the data information library corresponding to an ES index in the prior art. When the same query keyword is used to index an ES index in the prior art and multiple ES indexes of the present application, the present application sorts the different ES indexes and the ES indexes of the prior art, but the number of data information that can be displayed by each ES index in the multiple indexes of the present application is the same as the number of data information that can be displayed by one ES index in the prior art. When the query keyword and the data information library are the same, the data information displayed by the present application is composed of multiple ES indexes, while the prior art is composed of only one ES index. Therefore, when the query keyword and the data information library are the same, the present application may obtain more data information.

[0047] In addition, when the ES index containing multiple primary shards and the ES index set constructed by applying the present application have the same data information base, in the query process of the prior art, it is necessary to query an ES index containing multiple primary shards in a larger data information base. At this time, it is necessary to distribute query requests to each primary shard through a coordination node of the ES index. However, if there are too many primary shards of the ES index, the pressure on the coordination node will be too great, the query speed will be too slow, and the query performance requirements will be too high. In the process of querying the ES index set constructed by applying the present application, each ES index has its corresponding coordination node, and the data information bases corresponding to multiple ES indexes can be queried at the same time. Based on this, the query performance requirements are lower and the query speed is also faster. Therefore, compared with the prior art, the query speed of the ES index set constructed by the present application is faster and the query performance requirements are lower. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0049] Figure 1 This is a flow chart of an index construction method disclosed in this application;

[0050] Figure 2 This is a flow chart of another index construction method disclosed in this application;

[0051] Figure 3 This is an architecture diagram of an index construction system disclosed in this application;

[0052] Figure 4 This is a signaling flow chart of an index construction system disclosed in this application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] Next, combine Figure 1 The index construction method of this application is introduced in detail, including the following steps:

[0055] Step S1: Obtain data information that needs to be indexed, and determine the number of ES indexes to be constructed based on the data volume of the data information.

[0056] Specifically, the data information that needs to be indexed is stored in a wide table, and all data information is recorded in the wide table, and the corresponding relationship between each data information is recorded. In this way, the logical complexity of the data can be simplified. For example, the data information that needs to be indexed can be the data information corresponding to each commodity. At this time, the data information corresponding to each commodity can be recorded in the wide table, wherein each commodity can be each mobile phone and each towel, and the data information can be the type and price of each commodity. Therefore, the correspondence between the type of each towel and the price of the towel can be stored in the wide table. Similarly, the correspondence between the type of mobile phone and the price of the mobile phone can be recorded in the wide table.

[0057] When determining the number of ES indexes to be constructed, the amount of data information to be indexed can be determined based on the data volume. For example, if a wide table records the data information of various products that need to be indexed, the number of ES indexes to be constructed can be determined based on the data volume of the products in the wide table.

[0058] Step S2: Create an ES index set, where the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set contains a single primary shard.

[0059] Specifically, the ES index set contains one or more ES indexes, and when building each ES index, it is set to contain only one primary shard. The initial state of the ES index can be set using machine shards, and real-time refresh can be turned off.

[0060] Step S3: Determine the target data information corresponding to each ES index. The target data information corresponding to each ES index in the ES index set constitutes the data information.

[0061] Specifically, the type of item corresponding to each data information can be determined first. For example, the data information contains various mobile phones and various towels. Items of different types can be stored in different ES indexes, and the data information corresponding to the same mobile phone is stored in the same ES index.

[0062] Each ES index in the ES index set can have a unique corresponding index identifier. Based on this, when writing the data information corresponding to each product into the wide table, the index identifier of the ES index corresponding to each product can be written at the same time. Therefore, the target data information corresponding to the ES index can be determined based on the index identifier.

[0063] Furthermore, after creating multiple ES indexes, each ES index can be assigned a serial number. At the same time, the data for each product can be written into a wide table, and each product can be assigned a serial number according to the order in which it appears in the wide table. Subsequently, based on the serial numbers of each product and the serial numbers of each ES index, the target data corresponding to each ES index is determined. The data corresponding to each ES index constitutes the data corresponding to each product. For example, after creating three ES indexes, each ES index can be assigned a serial number, resulting in ES Index 1, ES Index 2, and ES Index 3, which serve as index identifiers for the three ES indexes.

[0064] If a wide table contains 100 products, and the 100 products are numbered from 1 to 100, the first 33 products in the wide table can be used as the target data information corresponding to ES index 1; the 34th to 66th products in the wide table can be used as the target data information corresponding to ES index 2; and the last 34 products in the wide table can be used as the target data information corresponding to ES index 3.

[0065] Step S4: Write the target data information corresponding to each ES index into the primary shard of the ES index.

[0066] Specifically, the target data information corresponding to each ES index can be processed simultaneously. This process can be performed concurrently by multiple processes, each of which reads the target data information corresponding to each ES index from the wide table, processes the target data information into data packets, and then writes the data packets to the corresponding ES index.

[0067] It can be seen from the above technical solution that the index construction method provided by the present application obtains the data information that needs to be indexed and determines the number of ES indexes to be constructed according to the data volume of the data information. In this way, the data information corresponding to an ES index with multiple primary shards in the prior art is converted into data information corresponding to multiple ES indexes; based on this, multiple ES indexes containing a single primary shard can be created. The newly created multiple ES indexes constitute an ES index set, and the number of ES indexes in the ES index set is equal to the number. In this way, an ES index with multiple primary shards in the prior art is divided into multiple ES indexes with a single primary shard; after creating multiple indexes, it is necessary to determine the information to be written to each index. Therefore, the target data information corresponding to each ES index can be determined, and the target data information corresponding to each ES index in the ES index set constitutes the data information. In this way, the data information is distinguished, and the target data information corresponding to different ES indexes is distinguished; based on this, when the target data information is written to the corresponding ES index, since the corresponding ES index only contains a single primary shard, the target data information can be directly written to the primary shard of the corresponding ES index. Since the data writing process of different ES indexes can be carried out simultaneously, in this application, each target data information in the data information can be written into the corresponding ES index at the same time, saving the ES index construction time and improving the ES index construction speed.

[0068] Furthermore, since the ES index may sort the data information related to the query keyword by the degree of matching during the data query process, only the data information within the preset range of the sorting results will be displayed, and all the data information related to the query keyword in the ES index will not be displayed. Therefore, the data information that can be queried by an ES index is limited. Through the index construction method provided by the present application, the data information library corresponding to an ES index in the prior art is divided into multiple data information libraries corresponding to ES indexes, that is, the ES index in the present application is combined with the data information libraries corresponding to each ES index to form the data information library corresponding to an ES index in the prior art. When the same query keyword is used to index an ES index in the prior art and multiple ES indexes of the present application, the present application sorts the different ES indexes and the ES indexes of the prior art, but the number of data information that can be displayed by each ES index in the multiple indexes of the present application is the same as the number of data information that can be displayed by one ES index in the prior art. When the query keyword and the data information library are the same, the data information displayed by the present application is composed of multiple ES indexes, while the prior art is composed of only one ES index. Therefore, when the query keyword and the data information library are the same, the present application may obtain more data information.

[0069] In addition, when the ES index containing multiple primary shards and the ES index set constructed by applying the present application have the same data information base, in the query process of the prior art, it is necessary to query an ES index containing multiple primary shards in a larger data information base. At this time, it is necessary to distribute query requests to each primary shard through a coordination node of the ES index. However, if there are too many primary shards of the ES index, the pressure on the coordination node will be too great, the query speed will be too slow, and the query performance requirements will be too high. In the process of querying the ES index set constructed by applying the present application, each ES index has its corresponding coordination node, and the data information bases corresponding to multiple ES indexes can be queried at the same time. Based on this, the query performance requirements are lower and the query speed is also faster. Therefore, compared with the prior art, the query speed of the ES index set constructed by the present application is faster and the query performance requirements are lower.

[0070] In some embodiments of the present application, considering that some unsuccessful ES indexes may be stored in the memory, in order to avoid the unsuccessful ES indexes from affecting the newly created ES indexes, an ES index set may be created in step S2, where the number of ES indexes in the ES index set is equal to the number, and each ES index in the ES index set contains a single primary shard before adding a process of deleting the unsuccessful ES indexes. Figure 2 , the process is described in detail, the specific steps are as follows:

[0071] S5. Determine whether there is an ES index that has not been successfully constructed. If there is an ES index that has not been successfully constructed, execute step S6; if there is no ES index that has not been successfully constructed, return to execute step S2.

[0072] Specifically, there may be an existing ES index named "Product Index 1_3" that has been built and is in use. In this case, the name of the ES index to be created is "Product Index 1_4". Based on this, it can be determined whether an ES index named "Product Index 1_4" exists in memory. If so, the ES index named "Product Index 1_4" will be deleted and a new ES index named "Product Index 1_4" will be created. The newly created ES index is an index in the ES index set.

[0073] In addition, you can also directly call the ES interface to obtain the health status of each ES index. ES indexes with poor health status are ES indexes that have not been successfully built.

[0074] S6. Delete the ES index that was not successfully constructed and execute step S2.

[0075] Specifically, you can directly delete the ES index that was not successfully built to avoid memory usage.

[0076] It can be seen from the above technical solution that this embodiment adds an optional process of first determining whether there are any ES indexes that have not been successfully constructed and deleting the ES indexes that have not been successfully constructed before creating the ES index set. Through the above process, the creation speed of the ES index can be further improved, and the impact of the ES index that has not been successfully constructed on the newly created ES index can be avoided.

[0077] In order to create an ES index suitable for use, in some embodiments of the present application, an attempt is made to add a check process for the ES index. Based on this, after step S4, when the target data information corresponding to each ES index is written into the primary fragment of the ES index, an check process for the ES index can be added. Figure 2 , the inspection process is described in detail, the specific steps are as follows:

[0078] S7. Check the ES index to determine whether the amount of data in the ES index meets the preset index standard. If it does not meet the index standard, execute step S8; if it does, it indicates that the construction of the ES index has been completed.

[0079] Specifically, you can call the ES interface to obtain the data volume of the newly created ES index and compare it with the data volume of the ES index in use. If the difference is not large and within the acceptable range, it can be considered to meet the indexing standards. If it is not within the acceptable range, it does not meet the indexing standards. The indexing standards can include the data volume of the ES index in use and the acceptable range.

[0080] S8. Delete the ES index and return to step S2.

[0081] Specifically, the ES index that does not meet the indexing standard can be directly deleted, and a new ES index can be rebuilt until an ES index that meets the indexing standard is obtained.

[0082] It can be seen from the above technical solution that this embodiment adds a checking process for the ES index to obtain a usable ES index, thereby completing the construction of the ES index.

[0083] In some embodiments of the present application, step S2, creating an ES index set, wherein the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set contains a single primary shard, is described in detail. The specific steps are as follows:

[0084] S10. Use a preset index initialization template to set the initial value of each ES index in the ES index set. The index initialization template includes the number of shards, the number of replicas, and the corresponding ES construction nodes, wherein the number of replicas is zero and the number of shards is one, so that each ES index in the ES index set contains only a single shard, and the shard is the primary shard.

[0085] Specifically, an index initialization template can be pre-established, which records the initial settings and field mapping of the ES index, including the number of ES index shards, the number of replicas, and the corresponding ES construction nodes. Each index can correspond to a different index initialization template.

[0086] The index initialization template can be understood as a global buffer configuration for configuring the ES index. Buffering means that during the configuration of a newly created ES index, the configuration of the existing ES index will not change. However, the next newly created ES index may have a new ES index configuration. Global means that all ES indexes are initialized and configured according to the index initialization template.

[0087] At this time, you can set the refresh_interval of each ES index in the ES index set to -1, that is, turn off the automatic refresh of each ES index in the ES index set.

[0088] On this basis, the process of writing the target data information corresponding to each ES index into the primary shard of the ES index in step S4 is described in detail. The steps are as follows:

[0089] S20. Use the ES construction node corresponding to each ES index to write the target data information corresponding to each ES index into the primary shard of the ES index.

[0090] Specifically, the target data information may be processed into a data packet, and then, through the ES construction node corresponding to each ES index, the data packet may be written into the primary shard of the corresponding ES index.

[0091] It can be seen from the above technical solution that compared with the previous embodiment, this embodiment provides an optional method for creating an ES index set, and also provides an optional method for writing target data information into the corresponding ES index. Through the above two methods, the construction of the ES index can be better realized.

[0092] In some embodiments of the present application, step S20, using the ES construction node corresponding to each ES index to write the target data information corresponding to each ES index into the primary shard of the ES index, is described in detail. The specific steps are as follows:

[0093] S100. Organize the target data information into a data structure corresponding to the ES index.

[0094] Specifically, the data structures corresponding to ES indexes include sorted lists (Array / List), HashMap / TreeMap, Skip List, Trie, Double Array Trie, Ternary Search Tree, and Finite State Transducers (FST). Depending on the specific situation, the target data information can be organized into any of the above data structures that suits the specific situation.

[0095] Among them, the target data information corresponding to different ES indexes can be organized into different data structures or into the same data structure, which is not limited in this application.

[0096] S110 , constructing a node through the ES corresponding to the ES index, and writing the sorted target data information into the ES index.

[0097] Specifically, the ES construction node corresponding to each ES index can be used to write the corresponding sorted target data information into the primary shard of the ES index.

[0098] It can be seen from the above technical solution that this embodiment provides an optional method for writing target data information into the corresponding ES index. Through the above method, an ES index that conforms to the actual situation can be better constructed.

[0099] In some embodiments of the present application, it is considered that each ES index in the ES index set can be queried through a specific ES query node, while the ES construction node only participates in the construction of the ES index. Therefore, it is necessary to add an ES index migration process after writing the target data information corresponding to each ES index into the primary shard of the ES index using the ES construction node corresponding to each ES index in step S30. The following will describe this process in detail. The specific steps are as follows:

[0100] S40: Refresh the ES index and merge the segments of the ES index.

[0101] Specifically, the ES index is set to perform a refresh operation to flush the data in the ES memory buffer to the commit point; and the ES index is set to perform a flush operation to write the memory data to the disk; and then the ES index is set to perform a merge segment operation to merge all the data information recorded in the ES index into a data segment, so that subsequent queries of the ES index can be faster and achieve better query efficiency.

[0102] S50. Add an ES query node corresponding to the ES index, and the ES construction node and the ES query node form an ES node set, and the ES node set corresponds to the ES index.

[0103] Specifically, index.routing.allocation.include can be used to correspond the ES query node to each ES index. In this case, the nodes corresponding to the ES index include the ES build node and the ES query node.

[0104] For example, initially, you can use three ES build nodes, Node01, Node02, and Node03, to build each ES index in the ES index set. At this point, the CPU usage of these three ES build nodes will be very high. After the build is complete, Node01, Node02, Node03, Node04, Node05, and Node06 are all designated as ES nodes to form the ES node set. Node04, Node05, and Node06 are designated as ES query nodes. At this point, since the ES node set contains six ES nodes, the number of replicas can be increased to five.

[0105] S60. Update the number of copies of the ES index. The updated number of copies is equal to the number of the ES node set minus 1, so that the ES index can obtain the same number of replica shards as the number of copies, so that the number of shards of the ES index is equal to the number of nodes in the ES node set. The shards of the ES index include primary shards and replica shards.

[0106] When the nodes corresponding to the ES index have ES construction nodes and ES query nodes, ES nodes need to be allocated. However, at this time, since the number of copies of the ES index is zero during the construction process and the ES index only contains one primary shard, the number of copies of the ES index needs to be increased to achieve ES node allocation.

[0107] Specifically, the number of replicas of the ES index can be increased so that the number of shards after the replica shards are added to the primary shards is the same as the number of nodes after the ES build nodes are added to the ES query nodes.

[0108] S70. Delete the correspondence between the ES index and the ES construction node, and reduce the number of copies of the ES index. The reduced number of copies and the number of replicated shards are equal to the number of ES query nodes minus 1, so that the number of shards of the ES index is equal to the number of ES query nodes.

[0109] Specifically, using index.routing.allocation.exclude allows the ES index to be distributed only in ES query nodes. In this case, the number of shards of the ES index is greater than the number of ES query nodes, thus reducing the number of copies of the ES index and thus reducing the number of replicated shards.

[0110] For example, Node04, Node05, and Node06 can be ES query nodes. In this case, since the ES index corresponds to three ES query nodes, the number of replicas can be reduced to two.

[0111] It can be seen from the above technical solution that compared with the previous embodiment, this embodiment provides an optional method for migrating ES index shards. Through segment merging, the ES index query efficiency can be improved, and the ES query nodes and ES construction nodes can be distinguished to avoid affecting the constructed ES index when the ES index is subsequently constructed. The ES query nodes and ES construction nodes are not shared, which can further improve the query efficiency of the ES index and obtain a better ES index.

[0112] In addition, this embodiment increases the number of copies of the ES index, which can increase the query throughput. For example, if one copy can provide 10 people with simultaneous queries, then two copies can provide 20 people with simultaneous queries, thereby improving the query speed of the ES index.

[0113] In some embodiments of the present application, considering that each ES index in the ES index set of the present application is an independent ES index of a single shard, the construction of each ES index in the ES index set can be completed by multiple terminals through the index construction system, further speeding up the construction of the ES index. Figure 3 and Figure 4 The index building process of the index building system is described in detail.

[0114] Figure 3 An optional system architecture for implementing index construction provided in an embodiment of the present application may include: an index creation end 10 and a data writing end 20.

[0115] There may be one or more data writing terminals 20. Multiple ES indexes in an ES index set may share one data writing terminal 20, or one ES index may use one data writing terminal 20, or even the same ES index may use multiple data writing terminals 20.

[0116] The index creation end 10 and the data writing end 20 can exchange data through the network. The index creation end 10 and the data writing end 20 can be implemented by a single machine or by multiple machines.

[0117] The index creation end 10 and the data writing end 20 may be a server and a server cluster, etc.

[0118] In some embodiments of the present application, based on Figure 3 The system architecture shown in FIG. 1 is used to introduce in detail the optional signaling process of the index construction method provided in the embodiment of the present application. The specific process is as follows:

[0119] S80. The index creation terminal 10 obtains data information that needs to be indexed, and determines the number of ES indexes to be constructed according to the data volume of the data information.

[0120] S81. The index creation terminal 10 creates an ES index set, where the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set includes a single primary shard.

[0121] S82. The index creation end 10 determines target data information corresponding to each ES index. The target data information corresponding to each ES index in the ES index set constitutes the data information.

[0122] The above steps S80-S82 correspond one-to-one to steps S1-S3 in the above embodiment. Please refer to the above introduction for details and will not be repeated here.

[0123] S83. The data writing terminal 20 writes the target data information corresponding to each ES index into the primary fragment of the ES index.

[0124] Specifically, when the translog of the ES index is set to asynchronous submission, the shards of multiple data writing terminals 20 can be used to process the target data information corresponding to each ES index, and then the processed target data information can be written into the corresponding ES index using the bulk API (batch operation interface).

[0125] Among them, during the data writing process of multiple data writing terminals 20, global redis can be used to ensure effective coordination of the data writing process. Before that, it is also possible to check whether each data writing terminal 20 can complete effective coordination to speed up the construction speed of each ES index.

[0126] It should be noted that whether it is necessary to use multiple data writing terminals 20 to complete the construction of a single ES index can be determined based on actual conditions, including but not limited to the data volume of the target data information corresponding to each ES index.

[0127] In some embodiments of the present application, in order for the data writing end 20 to better determine the target data information corresponding to each ES index, the index creation end 10 can also be used to, in step S82, determine the target data information corresponding to each ES index at the index creation end 10, and after the target data information corresponding to each ES index in the ES index set constitutes the data information, add a sending process of sending the data write instruction. Next, we will combine Figure 4 , the process is described in detail, the specific steps are as follows:

[0128] S200: Send the data write instruction corresponding to each ES index to the data write end.

[0129] Specifically, when multiple data writers 20 are required to complete the construction of a single ES index, the index creation end 10 assigns each data writer 20 the target processing information to be completed. Each target processing information constitutes the data information. Thus, each data writer 20 can clearly understand the construction task through the data write instruction sent by the index creation end 10, namely, writing the target processing information into the ES index corresponding to the index identifier. The data write instruction includes the index identifier corresponding to the ES index and the target data information.

[0130] On this basis, the process of step S83, in which the data writing end 20 writes the target data information corresponding to each ES index into the primary shard of the ES index, is described in detail. The specific steps are as follows:

[0131] S201. Determine target data information corresponding to the ES index having the index identifier through the data write instruction, organize the target data information into a data structure corresponding to the ES index, and write the organized target data information into the ES index.

[0132] Specifically, the data writing end 20 obtains the data information to be processed through the data writing instruction, and obtains the index identifier corresponding to the ES index.

[0133] When multiple data writing ends 20 are used to complete data writing of a single ES index, the last data writing end 20 to complete data writing in each ES index can be used as the target data writing end 20. The target data writing end 20 sets the ES index to perform a refresh operation, sets the ES index to perform a flush operation, and then sets the ES index to perform a mergesegment operation.

[0134] Each data writing port 20 can independently determine whether it is the last port to complete data writing. For example, the ES index completes the writing of target data information through four data writing ports 20. When each data writing port 20 completes writing, it can count in Redis and determine whether the value in Redis is 4. If it is 4, the data writing port 20 is the target data writing port 20.

[0135] It can be seen from the above technical solution that this embodiment adds the sending and receiving of data write instructions. Through the above process, the index creation end 10 and the data writing end 20 can better complete data interaction, thereby better completing the construction of the ES index.

[0136] In some embodiments of this application, frequent updates to an existing ES index can degrade performance. Furthermore, updating an existing ES index can severely impact both the user's query experience and performance. Therefore, upon completing an ES index update, a new ES index containing the latest data can be rebuilt, and then the index alias can be switched to complete the ES index update. This process will be described in detail below.

[0137] S90. Obtain the alias of the original index.

[0138] Specifically, the original index is the index that is in use and needs to be updated.

[0139] S91. Update the alias of each ES index in the ES index set created in the embodiment of the present application to the alias of the original index, and close the original index.

[0140] Specifically, the alias of each ES index in the newly created ES index set is changed to the alias of the original index. Afterwards, the original index can be deleted or only closed.

[0141] To close the original index, you can first delete the number of copies of the original index through the ES interface to delete all the replica shards of the original index and retain only the primary shard of the original index. Then, you can close the index of the original index.

[0142] It can be seen from the above technical solution that this embodiment provides an index updating method, through which the index can be updated without affecting the user's query experience, thereby minimizing the economic losses caused by updating the index.

[0143] In addition, the embodiment of the present application has a faster construction speed when constructing the ES index, so that the index can be updated more frequently, and the update and maintenance of the ES index can be completed more in real time, achieving quasi-real-time data.

[0144] In some embodiments of the present application, before updating the alias of each ES index in the ES index set created by the embodiment of the present application to the alias of the original index and closing the original index in step S91, the health of the newly created ES index can also be checked, and some data with a high number of queries can be pre-placed in the cache to achieve a more seamless index update. This process will be described in detail below, with the specific steps as follows:

[0145] S300. Determine the health status of the newly created ES index through the ES index interface. If the health status meets the update criteria, execute step S301. If the health status does not meet the update criteria, directly rebuild the ES index.

[0146] Specifically, the health status of the ES index can be directly obtained through the ES interface. The update criterion is whether the health status is healthy. When the ES index is healthy, step S301 can be executed.

[0147] S301. Use the same keyword to query the newly created ES index set and the original index respectively, and obtain the query results corresponding to the newly created ES index set and the query results corresponding to the original index.

[0148] Specifically, you can select hot words with a high number of searches as keywords to query the newly created ES index set and the original index respectively.

[0149] S302. Compare the query results corresponding to the newly created ES index set with the query results corresponding to the original index to obtain a comparison result. If the comparison result meets the preset replacement criteria, execute step S91; if the comparison result does not meet the preset replacement criteria, directly rebuild the ES index.

[0150] Specifically, the difference between the query result corresponding to the original index and the query result corresponding to the ES index set is calculated, and the ratio between the difference and the query result corresponding to the original index is used as the comparison result.

[0151] The replacement standard is that the comparison result must be less than a preset percentage. When it is below this percentage, the ES index set can be considered to have been successfully built. The percentage can be set based on actual conditions.

[0152] Through the above technical solution, this embodiment provides an optional method for determining whether a newly created ES index set has been successfully constructed. Through the above method, the health status of the newly created ES index set and the amount of data available for query can be confirmed. Among them, the query results corresponding to the newly created ES index set are the query results corresponding to the hot words with a high number of queries. Before the replacement, the hot words are queried, and some data with a high number of queries can be pre-placed in the cache to achieve a more seamless index update.

[0153] The index construction device provided in an embodiment of the present application is described below. The index construction device described below and the index construction method described above can be referenced to each other.

[0154] A data information acquisition unit is used to acquire the data information that needs to be indexed and determine the number of ES indexes to be constructed based on the amount of data in the data information;

[0155] An ES index set creation unit, configured to create an ES index set, wherein the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set includes a single primary shard;

[0156] a target data information determining unit, configured to determine target data information corresponding to each ES index, wherein the target data information corresponding to each ES index in the ES index set constitutes the data information;

[0157] The target data information writing unit is used to write the target data information corresponding to each ES index into the primary fragment of the ES index.

[0158] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0159] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0160] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0161] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application may be combined with each other. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an index, characterized in that: include: Obtain the data information that needs to be indexed, and determine the number of ES indexes to be constructed based on the data volume of the data information; Create an ES index set, where the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set contains a single primary shard; Determine target data information corresponding to each ES index, where the target data information corresponding to each ES index in the ES index set constitutes the data information; Write the target data information corresponding to each ES index into the primary shard of the ES index; Creating an ES index set includes: Use a preset index initialization template to set the initial value of each ES index in the ES index set. The index initialization template includes the number of shards, the number of replicas, and the corresponding ES construction node, wherein the number of replicas is zero and the number of shards is one, so that each ES index in the ES index set contains only a single shard, and the shard is the primary shard; Writing the target data information corresponding to each ES index into the primary shard of the ES index includes: Use the ES construction node corresponding to each ES index to write the target data information corresponding to each ES index into the primary shard of the ES index; Refresh the ES index and merge all target data information recorded in the ES index into one data segment; Add an ES query node corresponding to the ES index, and the ES construction node and the ES query node constitute an ES node set, and the ES node set corresponds to the ES index; The number of replicas of the ES index is updated, where the updated number of replicas is equal to the number of the ES node set minus 1, so that the ES index obtains the same number of replica shards as the number of replicas, so that the number of shards of the ES index is equal to the number of nodes in the ES node set. The shards of the ES index include primary shards and replica shards. Delete the correspondence between the ES index and the ES construction node, and reduce the number of copies of the ES index. The reduced number of copies and the number of replicated shards are equal to the number of ES query nodes minus 1, so that the number of shards of the ES index is equal to the number of ES query nodes.

2. The index construction method according to claim 1, characterized in that: The step of using the ES construction node corresponding to each ES index and writing the target data information corresponding to each ES index into the primary shard of the ES index includes: Arrange the target data information into a data structure corresponding to the ES index; The ES construction node corresponding to the ES index is used to write the sorted target data information into the ES index.

3. The index construction method according to claim 1, characterized in that: After writing the target data information corresponding to each ES index into the primary fragment of the ES index, the method further includes: Check the ES index to determine whether the amount of data in the ES index meets the preset index standard; If the index standard is not met, the ES index is deleted and the process returns to the step of creating an ES index set.

4. The index construction method according to claim 1, characterized in that: Before creating the ES index collection, also include: Determine whether there is an ES index that has not been successfully built; If there is an ES index that has not been successfully built, delete the ES index that has not been successfully built and return to the step of creating an ES index set; If there is no ES index that has not been successfully built, directly execute the steps to create an ES index collection.

5. An index building system, characterized in that: Including index creation end and data writing end; The index creation end obtains the data information to be indexed and determines the number of ES indexes to be constructed based on the data volume of the data information; Creating an ES index set, where the number of ES indexes in the ES index set is equal to the quantity, and each ES index in the ES index set contains a single primary shard; and determining target data information corresponding to each ES index, where the target data information corresponding to each ES index in the ES index set constitutes the data information; The step of creating an ES index set includes: Use a preset index initialization template to set the initial value of each ES index in the ES index set. The index initialization template includes the number of shards, the number of replicas, and the corresponding ES construction node, wherein the number of replicas is zero and the number of shards is one, so that each ES index in the ES index set contains only a single shard, and the shard is the primary shard; The data writing end writes the target data information corresponding to each ES index into the primary shard of the ES index; Writing the target data information corresponding to each ES index into the primary shard of the ES index includes: Use the ES construction node corresponding to each ES index to write the target data information corresponding to each ES index into the primary shard of the ES index; Refresh the ES index and merge all target data information recorded in the ES index into one data segment; Add an ES query node corresponding to the ES index, and the ES construction node and the ES query node constitute an ES node set, and the ES node set corresponds to the ES index; The number of replicas of the ES index is updated, where the updated number of replicas is equal to the number of the ES node set minus 1, so that the ES index obtains the same number of replica shards as the number of replicas, so that the number of shards of the ES index is equal to the number of nodes in the ES node set. The shards of the ES index include primary shards and replica shards. Delete the correspondence between the ES index and the ES construction node, and reduce the number of copies of the ES index. The reduced number of copies and the number of replicated shards are equal to the number of ES query nodes minus 1, so that the number of shards of the ES index is equal to the number of ES query nodes.

6. The index building system according to claim 5, characterized in that: The index creation end is further configured to, after determining the target data information corresponding to each ES index: Sending a data write instruction corresponding to each ES index to the data writing end, wherein the data write instruction includes an index identifier corresponding to the ES index and target data information; The data writing end writes the target data information corresponding to each ES index into the primary shard of the ES index, including: The target data information corresponding to the ES index having the index identifier is determined through the data write instruction, the target data information is organized into a data structure corresponding to the ES index, and the organized target data information is written into the ES index.

7. An index updating method, characterized in that: include: Get the alias of the original index, where the original index is the index that is in use and to be updated; Update the alias of each ES index in the ES index set created by the index construction method of any one of claims 1-4 to the alias of the original index, and close the original index so that each ES index in the ES index set replaces the original index to complete the index update.

8. The index updating method according to claim 7, characterized in that: Before updating the alias of each ES index in the ES index set created by the index construction method according to any one of claims 1 to 4 to the alias of the original index, the method further includes: Use the ES index interface to determine the health status of the newly created ES index; If the health status meets the update criteria, the newly created ES index set and the original index are queried with the same keyword to obtain the query results corresponding to the newly created ES index set and the query results corresponding to the original index; Compare the query results corresponding to the newly created ES index set with the query results corresponding to the original index to obtain a comparison result; If the comparison result meets the preset replacement criteria, the process returns to executing the step of updating the alias of each ES index in the ES index set created by the index construction method of any one of claims 1 to 4 to the alias of the original index and closing the original index.

Images