Index update methods, apparatus, electronic devices and storage media

Abstract

This application discloses an index updating method, apparatus, electronic device, and storage medium, relating to the field of computer technology, and particularly to artificial intelligence fields such as intelligent search and deep learning. The specific implementation scheme is as follows: Obtain the current first streaming index and the current first batch index corresponding to the target index building task from the index container providing online retrieval services; update the first batch index according to the first streaming index to obtain a second batch index; update the second batch index into the index container, so that the second batch index replaces the first batch index to provide online retrieval services.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to the field of artificial intelligence such as intelligent search and deep learning, specifically to an index update method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of internet technology, online information is exploding, and users' demand for efficiently obtaining the information they need is increasing. As a crucial tool for information retrieval, search engines can return relevant web page links and their summary information based on the user's query intent, helping users quickly locate target content. To improve the accuracy of search results, search engines need to update their indexes in real time in many business scenarios. Summary of the Invention

[0003] This application provides an index updating method, apparatus, electronic device, and storage medium. The specific solutions are as follows:

[0004] According to one aspect of this application, an index update method is provided, comprising:

[0005] Obtain the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services;

[0006] The first batch index is updated based on the first streaming index to obtain the second batch index;

[0007] The second batch index is updated in the index container so that the second batch index replaces the first batch index to provide online search services.

[0008] According to another aspect of this application, an index updating apparatus is provided, comprising:

[0009] The acquisition module is used to acquire the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services;

[0010] The first update module is used to update the first batch index according to the first streaming index to obtain the second batch index;

[0011] The second update module is used to update the second batch index into the index container so that the second batch index replaces the first batch index to provide online search services.

[0012] According to another aspect of this application, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in the above embodiments.

[0016] According to another aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method described in the above embodiments.

[0017] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the above embodiments.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0020] Figure 1 A schematic flowchart illustrating an index update method provided in an embodiment of this application;

[0021] Figure 2 A flowchart illustrating an index update method provided in another embodiment of this application;

[0022] Figure 3 A flowchart illustrating an index update method provided in another embodiment of this application;

[0023] Figure 4 A flowchart illustrating an index update method provided in another embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an index update device provided in an embodiment of this application;

[0025] Figure 6 This is a block diagram of an electronic device used to implement the index update method of the embodiments of this application. Detailed Implementation

[0026] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] It should be noted that the acquisition, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0028] The index update method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0029] Figure 1 This is a flowchart illustrating an index update method provided in an embodiment of this application.

[0030] The index update method of this application embodiment can be executed by the index update device of this application embodiment, which can be configured in an electronic device.

[0031] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0032] For example, the index update method of this application embodiment can be applied to a search engine, which can adopt a four-layer architecture: resource layer (also known as resource layer), data logical sharding layer (also known as data slot layer), model partitioning layer (also known as model slice layer), and physical partitioning layer (also known as physical shard layer).

[0033] For example, the resource layer can use a Kafka cluster as a unified data source and label the data using a three-level bitmap tagging system. The three-level bitmap tagging system can include active, rebuild, and archive data. Among them, active data can be data from newly created indexes, rebuild data is data from data reconstruction, and archive data is data that has been archived, which can be understood as a batch of indexes that have already been built.

[0034] For example, in the data logical sharding layer, each data logical shard can correspond to multiple physical data units, such as one slot corresponding to 32 Kafka Partitions.

[0035] For example, in the model partitioning layer, each data logical partition may include an inverted index, a vector index, an attribute column store, etc., and one data logical partition can correspond to one data logical shard.

[0036] For example, the physical partitioning layer can be deployed on a Kubernetes cluster.

[0037] like Figure 1 As shown, the index update method includes:

[0038] Step 101: Obtain the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services.

[0039] An index container can be a logical unit in a search engine that manages the lifecycle of an index; it is a container used to hold the search engine's indexes.

[0040] For example, the index container includes the batch index and the streaming index that currently provide online retrieval services; that is, the index container provides online retrieval services through the current batch index and the current streaming index.

[0041] A batch index can be understood as an index generated by processing a large amount of data at once, providing most of the indexing capabilities. Additionally, a batch index can also be called a full index.

[0042] Streaming indexes can be understood as indexes generated by data being processed continuously in the form of a stream. In other words, streaming indexes are built on streaming data and are constantly being updated. For example, when building a streaming index, data marked as active can be read to construct the streaming index.

[0043] In addition, the index data generated by batch indexing is data that search engines can directly load, while streaming indexing provides raw data, which is then processed by the search engine before it can use it.

[0044] In this application, the current first streaming index can be understood as the streaming index built from the last triggering of batch index building to the current triggering of batch index building, and the current first batch index is the batch index in the index container that currently provides online retrieval services. Building the batch index can be understood as updating the batch index.

[0045] In this application, the target index building task can be any index building task of the search engine. Executing an index building task once completes the construction of a batch of indexes in one go.

[0046] For example, the same index container can include batch indexes and streaming indexes for different services.

[0047] For example, before building the batch index, the current first streaming index and the current first batch index corresponding to the target index building task can be obtained from the index container.

[0048] For example, the batch index can be updated periodically, and when the update time for the batch index is reached, the current first streaming index and the current first batch index can be retrieved from the index container.

[0049] Step 102: Update the first batch index according to the first streaming index to obtain the second batch index.

[0050] Since streaming indexes are continuously built based on streaming data, the current first batch index can be updated based on the data in the first streaming index to obtain the second batch index.

[0051] For example, a batch index calculation can be performed on the data read from the first streaming index to obtain an incremental batch index, and then the incremental batch index can be merged into the first batch index to obtain a second batch index.

[0052] For example, the first streaming index is built based on data update messages from the 101st to the 200th, and the first batch index is built based on data update messages from the 1st to the 100th. Then, the first batch index can be updated based on the first streaming index, and the second batch index is built based on data update messages from the 1st to the 200th.

[0053] It is understandable that during the process of updating the batch index, the streaming index is constantly being built, so the streaming index obtained when the second batch index is obtained may be different from the first streaming index.

[0054] For example, when the first batch of indexes is updated, the first streaming index has built up to the 200th data update message. During the first batch of index updates, the streaming index is continuously being built. When the second batch of indexes is updated, the streaming index has built up to the 240th data update message.

[0055] Step 103: Update the second batch index into the index container so that the second batch index replaces the first batch index to provide online search services.

[0056] In this application, the newly updated second batch of indexes can be added to the index container, and the newly updated second batch of indexes can replace the first batch of indexes from the previous update to provide online retrieval services, thereby completing the search engine index update.

[0057] For example, after updating the second batch index to the index container, online retrieval services can be provided using the second batch index and the current streaming index.

[0058] In this embodiment, the current streaming index and batch index corresponding to the target index building task are obtained from the index container providing online retrieval services. Based on the streaming index currently providing online services, the batch index is updated to obtain the updated batch index. The updated batch index is then added to the index container, so that the updated second batch index replaces the current batch index in the index container to provide online retrieval services. Since streaming indexes update quickly and in real-time, but consume a lot of resources, while batch indexes provide faster retrieval services, but their offline update speed is slower, this approach continuously updates the batch index based on the current streaming index. The updated batch index, combined with the current streaming index, provides online retrieval services. This allows the index providing online retrieval services to achieve the update speed of streaming indexes while also enabling online retrieval services to reach the online access speed of batch indexes, thus improving the quality of retrieval services.

[0059] Figure 2 This is a flowchart illustrating an index update method provided in another embodiment of this application.

[0060] like Figure 2 As shown, the index update method includes:

[0061] Step 201: Obtain the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services.

[0062] Step 202: Update the first batch index according to the first streaming index to obtain the second batch index.

[0063] In this application, steps 201-202 can be implemented in any of the embodiments of this application, so they will not be described in detail here.

[0064] In some embodiments, when a node in the cluster is building an index and consuming data, it can first query the data from its local cache. If the corresponding data is not found in the local cache, it can query a remote in-memory database. This two-level caching improves index building efficiency.

[0065] For data in the local cache, a dual eviction strategy of LRU (Least Recently Used) and time window can be used to evict data. LRU evicts data that has not been used for a long time based on access timestamps, while time window eviction can be performed by setting a fixed time window (e.g., 5 minutes) and periodically scanning for data whose access count falls below a threshold within the window.

[0066] Step 203: Replace the first batch index in the index container with the second batch index.

[0067] In this application, the second batch of indexes can be loaded into the index container. After loading is complete, the first batch of indexes can be removed from the index container. Alternatively, the first batch of indexes can be removed from the index container first, and then the second batch of indexes can be loaded.

[0068] Step 204: Based on the second batch index, determine the third streaming index from the second streaming index, so as to provide online retrieval services by the second batch index and the third streaming index.

[0069] Since the second batch index is obtained by updating the first streaming index, and the first streaming index and the second batch index are built based on the same data update message, and a streaming index is built on the basis of the first streaming index during the process of updating the batch index, the third streaming index can be determined from the current second streaming index in the index container based on the second batch index.

[0070] Data update messages are the carriers of data changes and can be used to build indexes. For example, a data update message can be consumed to parse out the changed data, and an index can be built based on the parsed changed data.

[0071] The second streaming index is the current streaming index in the index container corresponding to the target index building task. It is the streaming index built upon the first streaming index during the time period from the start of updating the first batch of indexes to the completion of replacing them. Replacing the first batch of indexes can be understood as completing the loading of the second batch of indexes.

[0072] For example, after the second batch of index updates is completed, the index offset corresponding to the second batch of indexes can be recorded. Based on the index offset corresponding to the second batch of indexes, the location for building streaming index messages can be determined. Then, based on the location for building streaming index messages, the third streaming index can be determined from the second streaming index.

[0073] The index offset is used to mark the location of data update messages for which the second batch of indexes has been built.

[0074] The location of the streaming index message construction can refer to the location of the data update message corresponding to the index that the first streaming index is building when the first batch of indexes is triggered to update.

[0075] For example, an index preceding the index at the construction position of the streaming index message in the second streaming index can be deleted to obtain a third streaming index. In other words, an index built on the same data update message as the second batch index in the second streaming index can be removed to obtain a third streaming index.

[0076] For example, if the second batch index is built based on data update messages 1 to 100, then the index offset corresponding to the second batch index is 100. It can be determined that the streaming index message construction position is the 101st data update message. The index in the second streaming index is built from data update messages 1 to 110. Therefore, the indexes corresponding to data update messages 1 to 100 before the 101st message in the second streaming index can be deleted. The resulting third streaming index includes the indexes corresponding to data update messages 101 to 110.

[0077] Therefore, based on the index offset corresponding to the second batch index, the third streaming index is determined from the second streaming index. This allows for the accurate removal of index content that is duplicated with the second batch index from the second streaming index, ensuring data consistency.

[0078] For example, initially, the batch index is empty. When the streaming index is built from the 1st data update message to the 100th data update message, the first update of the batch index is triggered. The batch index updated based on this streaming index, that is, the batch index built based on the 1st to the 100th data update messages, can be loaded into the index container. During the update of the batch index, the streaming index continues to be built. At this time, the streaming index has built up to the 103rd data update message. Since the batch index in the index container is built from the 1st to the 100th data update message, while the streaming index is built from the 1st to the 103rd data update message, the indexes corresponding to the 1st to the 100th data update messages that are duplicates of the batch index in the streaming index can be deleted. The indexes corresponding to the 101st, 102nd, and 103rd data update messages can be retained. At this time, the index container provides online search services with the batch index built from the 1st to the 100th data update message and the streaming index built from the 101st to the 103rd data update message.

[0079] When the streaming index is built from the 101st to the 200th data update message, a second batch index update is triggered. Data corresponding to the 101st to 200th data update messages can then be read from the streaming index, and a batch index is built based on this data, resulting in an incremental batch index. This incremental batch index is then merged into the batch index obtained from the first update. The updated batch index is built based on the 1st to 200th data update messages, and this batch index is loaded into the index container. At this point, the streaming index in the index container has built up to the 210th data update message. The indexes corresponding to the 101st to 200th data update messages that are duplicates of the batch index are deleted from the streaming index, while the indexes corresponding to the 201st to 210th data update messages are retained. The index container then provides online retrieval services using the batch index built from the 1st to the 200th data update message and the streaming index built from the 201st to the 210th data update message.

[0080] In this embodiment, by using a second batch index to replace the first batch index in the index container, and by determining a third streaming index from the second streaming index based on the second batch index, online retrieval services are provided by the second batch index and the third streaming index. This avoids duplicate index content between the second batch index and the streaming index that provides online retrieval services, thereby improving retrieval efficiency.

[0081] In one embodiment of this application, the index container may contain multiple online service instances. The online service instances provide online retrieval services by retrieving the current batch index and the current streaming index from the container. To avoid affecting the online retrieval services, a new service instance can be created and started for any online service instance. The second batch index is loaded through the new service instance, and a third streaming index is constructed based on the second batch index. Once the second batch index and the third streaming index are loaded, the traffic of any online service instance is directed to the new service instance to provide online retrieval services through the new service instance, thereby realizing the online replacement of the batch index.

[0082] For example, the location of the streaming index message can be determined based on the index offset of the second batch index. The streaming index can be built starting from the location of the streaming index message construction until the location of the data update message of the streaming index in the online service instance is reached when the second batch index is loaded, thus obtaining the third streaming index.

[0083] For example, if there are 5 online service instances in the index container, 5 new service instances can be created and started. The batch index and streaming index can be loaded using the method described above. Then, traffic can be diverted from the online service instances to the new service instances, and the online service instances can be taken offline so that the new service instances can provide online search services.

[0084] Therefore, by creating a new service instance, once the new service instance has loaded the new batch index and streaming index, the traffic from the online service instance is directed to the new service instance. This allows for a seamless switch from providing online search services through the online service instance to providing online search services through the new service instance, without affecting the online search service itself.

[0085] In one embodiment of this application, updating the second batch index into the index container can also be done in the following way: the first batch index and the current streaming index can be removed from the index container, then the index container loads the second batch index, and starts building the streaming index from the data update message built from the second batch index, and provides online retrieval services based on the second batch index and the current streaming index.

[0086] To reduce the impact on online retrieval services during the replacement of batch indexes, for example, the index container may include multiple online service instances. The online service instances provide online services using the first batch index and the current streaming index. During the replacement, the multiple online service instances are replaced one by one, thereby unloading the first batch index and the current streaming index from one online service instance. During the loading of the second batch index, the remaining online service instances provide retrieval services, avoiding impact on the online retrieval services.

[0087] To ensure that online search services are not affected, optionally, when replacing online indexes using any of the above methods, the timing for replacing online batch indexes and streaming indexes can be determined based on QoS (Quality of Service).

[0088] For example, the replacement window can be divided into multiple time periods, each with a different bandwidth limit. The index replacement time can be determined based on the bandwidth limit and QoS of each time period.

[0089] For example, during peak hours, the bandwidth is limited to 1Gbps, resulting in significant online request latency. During off-peak hours (such as early morning), the bandwidth is released to 10Gbps, resulting in less online request latency. To avoid affecting online queries, the batch index and streaming index can be replaced during off-peak hours.

[0090] Figure 3 This is a flowchart illustrating an index update method provided in another embodiment of this application.

[0091] like Figure 3 As shown, the index update method includes:

[0092] Step 301: Obtain the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services.

[0093] In this application, step 301 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0094] Step 302: Determine whether the conditions for batch index update are met.

[0095] In this application, the batch index update conditions may include one or more of the following: reaching the periodic update time point, the current data volume of the streaming index is greater than a preset threshold, and the time elapsed since the last batch index update exceeds a time threshold.

[0096] For example, the condition for batch index update can be determined based on the first streaming index. For instance, if the amount of data in the first streaming index exceeds a preset threshold, it can be determined that the condition for batch index update is met.

[0097] Step 303: In response to the fulfillment of the batch index update condition, update the first batch index according to the first streaming index to obtain the second batch index.

[0098] In this application, if the batch index update conditions are met, the first batch index is updated according to the first streaming index to obtain the second batch index.

[0099] For example, the condition for batch index update can be determined based on the first streaming index. If the condition is met, the first batch index can then be updated based on the first streaming index. This not only avoids frequent updates, reduces fragmented operations, and saves resources, but also meets the need for customized batch index updates.

[0100] For example, if the time elapsed since the last batch index update exceeds the time threshold of 24 hours, or if the data volume of the first streaming index exceeds the preset threshold of 50GB, the first batch index can be updated based on the first streaming index to obtain the second batch index.

[0101] Step 304: Update the second batch index into the index container so that the second batch index replaces the first batch index to provide online search services.

[0102] In this application, step 304 can be implemented in any of the embodiments of this application, so it will not be described in detail here.

[0103] In this embodiment, by determining whether the batch index update conditions are met, the batch index is updated only when the batch index update conditions are met. This not only avoids frequent batch index updates, reduces fragmented operations, and saves resources, but also meets the needs of personalized batch index updates.

[0104] In one embodiment of this application, the first streaming index can be constructed from data in the physical data unit corresponding to the data logical partition bound to the target index construction task. The data logical partition bound to the target index construction task can be one or more, each data logical partition corresponds to a data logical shard, and each data logical shard is bound to a physical data unit. A data logical shard can be bound to one or more physical data units.

[0105] The following is combined with Figure 4 Explain the binding method between logical data fragments and physical data units. Figure 4 This is a flowchart illustrating an index update method provided in another embodiment of this application.

[0106] like Figure 4 As shown, the index update method also includes:

[0107] Step 401: Determine the first position information of the multiple virtual nodes of the data logical shard on the hash ring based on the identifiers of the multiple virtual nodes of the data logical shard.

[0108] In this application, a data logical shard can be divided into multiple virtual nodes. The first position information of these virtual nodes on the hash ring can be determined based on their identifiers. The first position information can be the hash value of the virtual node.

[0109] For example, the hash value of any virtual node can be calculated using a hash function based on the identifier of any virtual node in the data logical shard, and the virtual node can be placed on the hash ring based on the hash value of any virtual node, thereby mapping the virtual nodes of each data logical shard to the hash ring.

[0110] For example, if there are two data logical shards, and each data logical shard is divided into 100 virtual nodes, then 200 virtual nodes can be mapped onto a hash ring.

[0111] It should be noted that the number of virtual nodes corresponding to each data logical shard can be set according to actual needs, and this application does not limit this.

[0112] Step 402: Determine the second position information of the physical data unit on the hash ring based on the identifier of the physical data unit.

[0113] The second location information can be the hash value of the physical data unit.

[0114] In this application, there are multiple physical data units, and the physical data unit bound to any data logical segment can be one or more of the multiple physical data units.

[0115] For example, for any physical data unit, the hash value of any physical data unit can be determined based on the identifier of the physical data unit, and the physical data unit can be placed on the hash ring based on the hash value of the physical data unit, thereby mapping multiple physical data units to the hash ring.

[0116] For example, there are 64 physical data units, which can be mapped to a hash ring based on their hash values.

[0117] Step 403: Based on the first location information and the second location information, determine the first virtual node that is closest to the physical data unit in the target direction of the hash ring.

[0118] The target direction can be either clockwise or counterclockwise along the hash ring, without limitation.

[0119] In this application, for any physical data unit, the first virtual node that is closest to the physical data unit in the target direction on the hash ring can be determined based on the second location information of the first physical unit and the first location information of the virtual nodes of each data logical segment.

[0120] Step 404: Bind the physical data unit to the data logical fragment to which the first virtual node belongs.

[0121] In this application, for any physical data unit, any physical data unit can be bound to the data logical segment to which the first virtual node closest to the physical data unit belongs, thereby binding all physical data units to the corresponding data logical segments.

[0122] In this embodiment, by splitting the data logical shard into multiple virtual nodes and distributing the multiple virtual nodes of the data logical shard evenly on the hash ring, and binding the physical data unit to the data logical shard to which the nearest virtual node in the target direction of the hash ring belongs based on the position information of the physical data unit on the hash ring and the position information of the data logical shard on the hash ring, load balancing can be improved and hotspot issues can be reduced.

[0123] As the amount of data increases, it may be necessary to expand the capacity, such as by adding data logical shards. In one embodiment of this application, if a new data logical shard is added, the physical data units can be remapped in the following way:

[0124] Based on the identifiers of multiple virtual nodes in the new data logical shard, the third position information of multiple virtual nodes in the new data logical shard on the hash ring can be determined. Based on the first position information, the second position information, and the third position information, the second virtual node that is closest to the physical data unit in the target direction of the hash ring can be determined. If the second virtual node is inconsistent with the first virtual node, the physical data unit is unbound from the data logical shard to which the first virtual node belongs, and the physical data unit is bound to the data logical shard to which the second virtual node belongs.

[0125] For example, based on the identifiers of multiple virtual nodes in a new data logical shard, a hash value can be calculated using a hash function. Based on the hash value, the multiple virtual nodes of the new data logical shard can be placed on a hash ring. As the number of virtual nodes on the hash ring increases, the data logical shard to which the physical data unit belongs may change. If this changes, the data logical shard previously bound to the physical data unit can be unbound and bound to the data logical shard to which the current nearest virtual node belongs.

[0126] Therefore, if a new data logical shard is added, during expansion, the virtual nodes of the new data logical shard can be mapped onto the hash ring. For data logical shards whose binding relationship with the data logical shard is affected, they are re-bound. The number of physical data units that are re-bound is relatively small. Furthermore, by using the data logical shard as an intermediate bridge between the data logical partition and the physical data unit, when expanding, only the binding relationship is changed, and the data within the physical data unit is not moved.

[0127] When the amount of data decreases, in order to save resources, the scale can be reduced, such as by reducing the number of data logical shards. As the data logical shards are removed, the virtual nodes of the data logical shards on the hash ring will also be removed, thereby reducing the number of virtual nodes on the hash ring. Then the physical data units bound to the data logical shard need to be rebound to the data logical shard.

[0128] Based on this, in one embodiment of this application, if any data logical fragment is removed, the physical data unit can be remapped in the following manner:

[0129] Multiple virtual nodes of any data logical shard on the hash ring can be removed, and the third virtual node that is closest to the physical data unit corresponding to any data logical shard in the target direction of the hash ring can be determined. Then, the physical data unit corresponding to any data logical shard can be bound to the data logical shard to which the third virtual node belongs.

[0130] The physical data unit corresponding to any data logical segment refers to the physical data unit that was bound to the data logical segment before the data logical segment was removed.

[0131] For example, based on the position information of the virtual nodes corresponding to the remaining data logical shards on the hash ring and the position information of the physical data units corresponding to the removed data logical shards on the hash ring, the third virtual node that is closest to the physical data unit corresponding to the removed data logical shard in the target direction of the hash ring can be determined.

[0132] Therefore, if a data logical shard is removed, during scaling down, the virtual nodes of the removed data logical shard can be removed from the hash ring. For the physical data unit bound to the data logical shard, the data logical shard to which the nearest virtual node belongs is re-determined and bound. Thus, by using the data logical shard as an intermediate bridge between the data logical partition and the physical data unit, when scaling down, only the binding relationship is changed, and the data within the physical data unit is not moved.

[0133] To facilitate understanding of the binding method between the above-mentioned logical data fragmentation and physical data unit, an example is given below.

[0134] Taking a Kafka cluster as an example, for ease of description, logical data shards are called slots, and physical data units are called Kafka Partitions. Assume the initial state: 2 slots (S1, S2), 64 Kafka Partitions (P1 to P64); hash ring setup: using the SHA-256 hash function, with a ring range of 0 to 2. 265 -1, each slot is initially split into 100 virtual nodes.

[0135] (1) System startup initialization mapping

[0136] First, S1 is split into 100 virtual nodes, such as V_S1_1, V_S1_2, ..., V_S1_100. Each virtual node is hashed using a hash function, such as hash = SHA256(SlotID + virtual node number) % ring size, and then placed on the ring. Here, SlotID represents the identifier of the logical data shard.

[0137] Similarly, S2 is split into 100 virtual nodes, their hash values ​​are calculated, and they are placed on the hash ring. Thus, 200 virtual nodes are evenly distributed on the hash ring, with 100 virtual nodes each for S1 and S2.

[0138] Next, the Kafka Partitions are mapped to the hash ring: A hash value is calculated for each Kafka Partition based on its identifier; for example, the hash value of P1 is hash_P1 = SHA256(PartitionID) % ring size, and this hash value is placed directly on the hash ring. Thus, 64 Kafka Partitions are mapped to the hash ring. Since a Kafka Partition is a data unit, virtual nodes are not split.

[0139] Next, the Kafka Partitions are assigned to Slots: for each Kafka Partition, its position is found on the hash ring, and then the nearest virtual node is found clockwise. The Slot to which the virtual node belongs is responsible for this Kafka Partition.

[0140] For example: if the nearest virtual node clockwise for P1 is V_S1_50, then P1 is bound to S1; if the nearest virtual node clockwise for P2 is V_S2_30, then P2 is bound to S2. Repeat this process to allocate slots for all Kafka Partitions.

[0141] Then establish the initial binding relationship: Since the virtual nodes are evenly distributed, each slot is responsible for approximately 32 Kafka Partitions. The actual number may vary slightly due to hash deviations, such as S1 being responsible for 31 and S2 being responsible for 33.

[0142] The system can record metadata: Slot-Partition mapping tables, such as S1:[P1,P3,P5,...], S2:[P2,P4,P6,...].

[0143] As can be seen, compared with traditional consistent hashing, this application can ensure a more balanced distribution through virtual nodes.

[0144] (2) Dynamically add new slots (during expansion)

[0145] Adding a new slot S3 allows us to split slot S3 into 100 virtual nodes, such as V_S3_1, V_S3_2, ..., V_S3_100. The hash values ​​of these virtual nodes are then calculated and mapped onto a hash ring. At this point, the hash ring has 300 virtual nodes, with 100 virtual nodes each for S1, S2, and S3.

[0146] Then, the hash ring can be traversed to find the insertion position of the virtual node in S3.

[0147] For each virtual node in S3, identify the Kafka Partitions on the ring that are clockwise from the original slot virtual nodes. These Kafka Partitions originally belonged to S1 or S2, but are now closer to the virtual nodes in S3.

[0148] For example, if V_S3_1 is inserted between V_S1_20 and V_S1_21, then the Kafka Partition (such as P10) that was originally bound to V_S1_21 is now closer to V_S3_1, so P10 needs to be rebound to S3.

[0149] Next, update the metadata: unbind the affected Kafka partitions from their original slots and bind them to S3. For example, unbind P10 from S1 and bind it to S3, and unbind P15 from S2 and bind it to S3. Update the Slot-Partition mapping table, for example, S1: [original list minus moved items], S3: [newly added partition].

[0150] As you can see, when a new slot is added, only the binding relationship changes; the data within the Kafka Partition does not move. The physical location of the Kafka Partition remains unchanged.

[0151] Additionally, you can check the load on S3. If the initial distribution is uneven, dynamically adjust the number of virtual nodes on S3, for example, by increasing it to 120, to absorb more Kafka partitions.

[0152] System monitoring tools ensure that after migration, each slot is responsible for approximately 32 Kafka partitions (e.g., S1:21, S2:22, S3:21) to avoid hotspots.

[0153] In related technologies, when scaling up the underlying shards, all Kafka partitions need to be rehashed, resulting in a 50% data migration; here, only about 21 / 64≈33% of the Kafka partition bindings are moved, reducing the migration amount by nearly 34%.

[0154] (3) Dynamically remove slots (when shrinking)

[0155] For example, removing a slot, such as slot S2, and reassigning its Kafka Partition.

[0156] Remove slot S2 and all its virtual nodes (100) from the hash ring, reducing the number of virtual nodes on the hash ring to 200. These 200 virtual nodes are the 100 virtual nodes corresponding to S1 and S3 respectively.

[0157] The Kafka Partitions (e.g., 32) that slot S2 was originally responsible for are now unbound. On the hash ring, the nearest remaining virtual node (belonging to S1 or S3) of these Kafka Partitions is searched clockwise.

[0158] Update metadata: The Kafka partitions bound to slot S2 are reassigned to S1 or S3. The migration amount is approximately the number of Kafka partitions that were removed from the slot. It's clear that the data within the Kafka partitions remains unchanged; only the binding relationships are updated.

[0159] For example: P2 was originally bound to V_S2_10. After it is removed, V_S1_40 is found clockwise, and then P2 is bound to S1.

[0160] Additionally, the number of virtual nodes in other slots can be reduced (e.g., S1 and S3 can each be reduced to 90) to maintain load balance.

[0161] Here, the underlying model, the Slice layer, consumes slot data, and the slotID remains unchanged (e.g., S1 is always S1). When scaling up or down, it is only necessary to rebind and unbind the Slot and the underlying Kafka Partition, and update the metadata. Even if the Kafka Partition binding changes, the Slice layer is unaware of it, thus avoiding data reordering.

[0162] The data logical sharding and physical data unit binding method in this embodiment, along with virtual nodes and dynamic weights to ensure load balancing, minimizes migration strategies and reduces computational overhead. Compared to traditional consistent hashing, the improved scaling up / down time complexity is O(log n), where n is the number of data logical shards, making it highly efficient and dynamic. This method of independent scaling up and down of the data slot layer does not affect the resource layer (such as Kafka) and the model slice layer, achieving layered decoupling. Therefore, large scaling up and down of the data slot layer does not require reordering of the underlying data.

[0163] To improve query efficiency, in one embodiment of this application, if a data query request is received, the target query index can be determined from the second batch index and the current fourth streaming index based on the difference bitmap corresponding to the second batch index, and the query can be performed in the target query index to obtain the data query result.

[0164] The difference bitmap can be generated based on the differences between the second batch index and the first batch index. For example, a difference bitmap can be generated when creating the batch index to record changes from the previous version of the batch index.

[0165] The fourth streaming index can be a streaming index built on the first streaming index during the time period from the start of updating the first batch of indexes to the current data retrieval query request.

[0166] The target query index may include one or more of the second batch index and the fourth streaming index.

[0167] For example, if the difference bitmap determines whether the query is performed in the second batch index, the fourth streaming index, or both, then: if the query is performed in the second batch index, the second batch index is designated as the target query index; if the query is performed in the fourth streaming index, the fourth streaming index is designated as the target query index; and if the query is performed in both the second batch index and the fourth streaming index, both the second batch index and the fourth streaming index are designated as the target indexes for the query.

[0168] Therefore, by determining which index to query based on the difference bitmap corresponding to the second batch of indexes, query efficiency can be improved.

[0169] For example, the second batch index and the fourth streaming index are encoded as compressed bitmaps. If the target query index includes the second batch index and the fourth streaming index, queries are performed on the second batch index and the fourth streaming index respectively to obtain the first bitmap index and the second bitmap index. Bit operations are performed on the first bitmap index and the second bitmap index to obtain the data query results.

[0170] Therefore, by encoding the two types of indexes into compressed bitmaps, bitwise operations can be used to merge the query results of batch indexes and streaming indexes, which can improve the merging speed and further improve query efficiency.

[0171] Optionally, a data query scenario can be determined based on the data query request, a target query index can be determined based on the data query scenario, and a query can be performed in the target query index to obtain the data query results.

[0172] For example, in a data query scenario where the data is relatively new, such as a newly released mobile phone model, since streaming indexes are built in real-time while batch indexes are updated relatively slowly, the data for that phone model might not yet be included in the batch index. In this case, the query can be performed using the streaming index, meaning the current streaming index is designated as the target query index. Conversely, if the data being queried is relatively old, the query can be performed using the current batch index, meaning the current batch index is designated as the target query index.

[0173] Since there may be multiple index building tasks to be executed, in order to improve the execution efficiency and resource utilization, in one embodiment of this application, the target index building task is a task consisting of multiple index building tasks to be executed. The following method can be used to execute multiple index building tasks:

[0174] It can predict the resource requirements of multiple index building tasks to be executed. Based on the resources and resource requirements in each resource pool, it can sequentially determine the target resource pool corresponding to each index building task to be executed in the sorting results. It can then use the resources in the target resource pool to execute the index building task corresponding to the target resource pool.

[0175] Resource requirements can refer to the demand for one or more resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and memory.

[0176] For example, the resource requirements of any index building task to be executed can be predicted using a pre-trained resource prediction model based on data from multiple dimensions such as historical resource usage, index type, and data distribution characteristics.

[0177] For example, index building tasks to be executed can be placed into the first resource pool that can accommodate them, according to the sorting results.

[0178] For example, there are three resource pools: Pool1: CPU = 16 cores, memory = 64GB; Pool2: CPU = 8 cores, memory = 32GB; Pool3: CPU = 4 cores, memory = 16GB; The index building tasks to be executed are arranged in descending order of resource requirements: TaskA: CPU = 8 cores, memory = 32GB; TaskB: CPU = 4 cores, memory = 16GB; TaskC: CPU = 2 cores, memory = 8GB.

[0179] Scheduling process: Put TaskA into Pool1, try to put TaskB into Pool1 (remaining CPU = 8 cores, memory = 32GB, meeting resource requirements), try to put TaskC into Pool1 (remaining resources are sufficient), so that the resources in Pool1 can be used to execute TaskA, TaskB and TaskC in sequence.

[0180] Since different index building tasks may have different time requirements, such as some requiring the batch index to be updated within 1 day, while others require the batch index to be updated within 8 hours, based on this, alternatively, multiple index building tasks to be executed can be sorted in descending order according to resource and time requirements, and the target resource pool corresponding to each index building task to be executed in the sorted results can be determined in turn according to the resources, resource requirements and time requirements in each resource pool.

[0181] In this embodiment of the application, by predicting the resource requirements corresponding to multiple index building tasks to be executed, and scheduling and executing the index building tasks to be executed according to the resources and resource requirements in each resource pool, the task execution efficiency and resource utilization can be improved.

[0182] In one embodiment of this application, if a new node is detected joining the cluster, the number of data logical partitions can be determined based on the number of current nodes in the cluster, and new data logical partitions can be created based on the number of data logical partitions, and the new data logical partitions can be mapped to the new node.

[0183] For example, when a new node is added, a logical data partition can be created with 0.5 times the number of existing nodes.

[0184] Taking a data source deployed in a Kafka cluster as an example, the index update method in this application embodiment can combine streaming consumption of message queues with batch index reconstruction: when updating data, millisecond-level streaming index construction can be achieved through message middleware such as Kafka, and batch index reconstruction can be performed based on index offset. By predicting the resource requirements of index construction tasks, scheduling and executing index construction tasks can improve the utilization of computing resources.

[0185] Two modes are supported in the expansion scenario: when the data slots remain unchanged, lossless expansion can be performed using a consistent hashing algorithm; during full redistribution, new service instances are used to replace online service instances to ensure zero service interruption during the expansion period.

[0186] The index update method proposed in this application can be applied to scenarios such as e-commerce real-time search, financial risk control systems, and intelligent recommendation engines that require simultaneous processing of high-frequency incremental data and massive historical data.

[0187] To implement the above embodiments, this application also proposes an index update device. Figure 5 This is a schematic diagram of the structure of an index update device provided in an embodiment of this application.

[0188] like Figure 5 As shown, the index update device 500 includes:

[0189] The acquisition module 510 is used to acquire the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services;

[0190] The first update module 520 is used to update the first batch index according to the first streaming index to obtain the second batch index;

[0191] The second update module 530 is used to update the second batch index into the index container so that the second batch index replaces the first batch index to provide online retrieval services.

[0192] Optionally, the second update module 530 is used for:

[0193] The first batch index in the index container is replaced using the second batch index;

[0194] Based on the second batch index, a third streaming index is determined from the second streaming index to provide online retrieval services using the second batch index and the third streaming index;

[0195] The second streaming index is a streaming index built on the first streaming index during the time period from the start of updating the first batch index to the completion of replacing the first batch index.

[0196] Optionally, the second update module 530 is used for:

[0197] The location for building streaming index messages is determined based on the index offset corresponding to the second batch of indexes; wherein, the index offset is used to mark the location of data update messages for indexes already built in the second batch of indexes;

[0198] The third streaming index is determined from the second streaming index based on the location where the streaming index message is constructed.

[0199] Optionally, the second update module 530 is used for:

[0200] For any online service instance in the index container, create and start a new service instance;

[0201] The second batch index is loaded through the new service instance;

[0202] Based on the second batch index, construct the third streaming index;

[0203] In response to the completion of loading the second batch index and the third streaming index, traffic from any of the online service instances is diverted to the new service instance to provide online search services through the new service instance.

[0204] Optionally, the first update module 520 is used for:

[0205] Based on the first streaming index, determine whether the batch index update conditions are met;

[0206] In response to the fulfillment of the batch index update condition, the first batch index is updated according to the first streaming index to obtain the second batch index.

[0207] Optionally, the first streaming index is constructed from the data in the physical data unit corresponding to the data logical partition bound to the target index construction task; wherein, one data logical partition corresponds to one data logical shard, and the data logical shard is bound to a physical data unit.

[0208] The device may also include:

[0209] The first determining module is used to determine the first position information of the multiple virtual nodes of the data logical shard on the hash ring based on the identifiers of the multiple virtual nodes of the data logical shard.

[0210] The second determining module is used to determine the second position information of the physical data unit on the hash ring based on the identifier of the physical data unit;

[0211] The third determining module is used to determine, based on the first location information and the second location information, the first virtual node that is closest to the physical data unit in the target direction of the hash ring;

[0212] The binding module is used to bind the physical data unit to the data logical fragment to which the first virtual node belongs.

[0213] Optionally, the first determining module is further configured to, in response to the addition of a new data logical shard, determine the third position information of the multiple virtual nodes of the new data logical shard on the hash ring based on the identifiers of the multiple virtual nodes of the new data logical shard;

[0214] The first determining module is further configured to determine, based on the first location information, the second location information, and the third location information, the second virtual node that is closest to the physical data unit in the target direction of the hash ring;

[0215] The binding module is also configured to, in response to the inconsistency between the second virtual node and the first virtual node, unbind the physical data unit from the data logical fragment to which the first virtual node belongs, and bind the physical data unit to the data logical fragment to which the second virtual node belongs.

[0216] Optionally, the device may further include a removal module for removing multiple virtual nodes of any data logical shard on the hash ring in response to removing any data logical shard;

[0217] The second determining module is further configured to determine the third virtual node that is closest to the physical data unit corresponding to any data logical segment in the target direction of the hash ring;

[0218] The binding module is also used to bind the physical data unit corresponding to any of the data logical segments to the data logical segment to which the third virtual node belongs.

[0219] Optionally, the device may further include:

[0220] The third determining module is used to respond to a data query request by determining the target query index from the second batch index and the current fourth streaming index based on the difference bitmap corresponding to the second batch index; wherein the difference bitmap is generated based on the difference between the second batch index and the first batch index;

[0221] The query module is used to perform queries in the target query index to obtain data query results.

[0222] Optionally, the second batch index and the fourth streaming index are encoded as compressed bitmaps for use by the query module:

[0223] In response to the target query index including the second batch index and the fourth streaming index, queries are performed on the second batch index and the fourth streaming index respectively to obtain the first bitmap index and the second bitmap index;

[0224] Bitwise operations are performed on the first bitmap index and the second bitmap index to obtain the data query result.

[0225] Optionally, the target index building task is one of a plurality of index building tasks to be executed, and the apparatus may further include:

[0226] The prediction module is used to predict the resource requirements corresponding to the multiple index building tasks to be executed.

[0227] The sorting module is used to sort the multiple index building tasks to be executed in descending order according to the resource requirements, and obtain the sorting result;

[0228] The fourth determining module is used to determine the target resource pool corresponding to each index to be executed in the arrangement result to construct the task based on the resources in each resource pool and the resource requirements.

[0229] The execution module is used to utilize the resources in the target resource pool to execute the index construction task corresponding to the target resource pool.

[0230] It should be noted that the explanation of the aforementioned index update method embodiment also applies to the index update device of this embodiment, so it will not be repeated here.

[0231] In this embodiment, the current streaming index and batch index corresponding to the target index building task are obtained from the index container providing online retrieval services. Based on the streaming index currently providing online services, the batch index is updated to obtain the updated batch index. The updated batch index is then added to the index container, so that the updated second batch index replaces the current batch index in the index container to provide online retrieval services. Since streaming indexes update quickly and in real-time, but consume a lot of resources, while batch indexes provide faster retrieval services, but their offline update speed is slower, this approach continuously updates the batch index based on the current streaming index. The updated batch index, combined with the current streaming index, provides online retrieval services. This allows the index providing online retrieval services to achieve the update speed of streaming indexes while also enabling online retrieval services to reach the online access speed of batch indexes, thus improving the quality of retrieval services.

[0232] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0233] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0234] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or a computer program loaded from storage unit 608 into RAM (Random Access Memory) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O (Input / Output) interface 605 is also connected to bus 604.

[0235] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0236] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the index update method. For example, in some embodiments, the index update method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the index update method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the index update method by any other suitable means (e.g., by means of firmware).

[0237] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0238] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0239] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0240] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0241] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0242] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0243] According to embodiments of this application, this application also provides a computer program product that, when an instruction processor in the computer program product is executed, performs the index update method proposed in the above embodiments of this application.

[0244] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0245] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An index update method, comprising: Retrieve the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services; The first batch index is updated based on the first streaming index to obtain the second batch index; The second batch index is updated in the index container so that the second batch index replaces the first batch index to provide online search services; The first streaming index is constructed from the data in the physical data unit corresponding to the data logical partition bound to the target index construction task; wherein, one data logical partition corresponds to one data logical shard, and the data logical shard is bound to a physical data unit. The binding relationship between the data logical fragment and the physical data unit is determined in the following way: Based on the identifiers of the multiple virtual nodes of the data logical shard, determine the first position information of the multiple virtual nodes of the data logical shard on the hash ring; Based on the identifier of the physical data unit, determine the second position information of the physical data unit on the hash ring; Based on the first location information and the second location information, determine the first virtual node that is closest to the physical data unit in the target direction of the hash ring; The physical data unit is bound to the data logical segment to which the first virtual node belongs.

2. The method as described in claim 1, wherein, The step of updating the second batch index into the index container includes: The first batch index in the index container is replaced using the second batch index; Based on the second batch index, a third streaming index is determined from the second streaming index to provide online retrieval services using the second batch index and the third streaming index; The second streaming index is a streaming index built on the first streaming index during the time period from the start of updating the first batch index to the completion of replacing the first batch index.

3. The method as described in claim 2, wherein, The step of determining the third streaming index from the second streaming index based on the second batch index includes: The location for building streaming index messages is determined based on the index offset corresponding to the second batch of indexes; wherein, the index offset is used to mark the location of data update messages for indexes already built in the second batch of indexes; The third streaming index is determined from the second streaming index based on the location where the streaming index message is constructed.

4. The method of claim 1, wherein, The step of updating the second batch index into the index container includes: For any online service instance in the index container, create and start a new service instance; The second batch index is loaded through the new service instance; Based on the second batch index, construct the third streaming index; In response to the completion of loading the second batch index and the third streaming index, traffic from any of the online service instances is diverted to the new service instance to provide online search services through the new service instance.

5. The method of claim 1, wherein, The step of updating the first batch index based on the first streaming index to obtain the second batch index includes: Based on the first streaming index, determine whether the batch index update conditions are met; In response to the fulfillment of the batch index update condition, the first batch index is updated according to the first streaming index to obtain the second batch index.

6. The method of claim 1, further comprising: In response to the addition of a new data logical shard, the third position information of the multiple virtual nodes of the new data logical shard on the hash ring is determined based on the identifiers of the multiple virtual nodes of the new data logical shard. Based on the first location information, the second location information, and the third location information, determine the second virtual node that is closest to the physical data unit in the target direction of the hash ring; In response to the inconsistency between the second virtual node and the first virtual node, the physical data unit is unbound from the data logical fragment to which the first virtual node belongs, and the physical data unit is bound to the data logical fragment to which the second virtual node belongs.

7. The method of claim 1, further comprising: In response to removing any data logical shard, remove multiple virtual nodes of the any data logical shard on the hash ring; Determine the third virtual node that is closest to the physical data unit corresponding to any of the data logical fragments in the target direction of the hash ring; Bind the physical data unit corresponding to any of the data logical segments to the data logical segment to which the third virtual node belongs.

8. The method according to any one of claims 1-5, the method further comprising: In response to a data query request, the target query index is determined from the second batch index and the current fourth streaming index based on the difference bitmap corresponding to the second batch index; wherein the difference bitmap is generated based on the differences between the second batch index and the first batch index; Perform a query in the target query index to obtain the data query results.

9. The method of claim 8, wherein, The second batch index and the fourth streaming index are encoded as compressed bitmaps, and the query in the target query index to obtain data query results includes: In response to the target query index including the second batch index and the fourth streaming index, queries are performed on the second batch index and the fourth streaming index respectively to obtain the first bitmap index and the second bitmap index; Bitwise operations are performed on the first bitmap index and the second bitmap index to obtain the data query result.

10. The method according to any one of claims 1-5, wherein, The target index building task is one of multiple index building tasks to be executed, and the method further includes: Predict the resource requirements corresponding to the multiple index building tasks to be executed; Based on the resource requirements, the multiple index building tasks to be executed are sorted in descending order to obtain the sorting result; Based on the resources in each resource pool and the resource requirements, the target resource pool corresponding to each index construction task to be executed in the sorting result is determined in sequence; Utilize the resources in the target resource pool to execute the index building task corresponding to the target resource pool.

11. An index update apparatus, comprising: The acquisition module is used to acquire the current first streaming index and the current first batch index corresponding to the target index building task from the index container that provides online retrieval services; The first update module is used to update the first batch index according to the first streaming index to obtain the second batch index; The second update module is used to update the second batch index into the index container so that the second batch index replaces the first batch index to provide online retrieval services. The first streaming index is constructed from the data in the physical data unit corresponding to the data logical partition bound to the target index construction task; wherein, one data logical partition corresponds to one data logical shard, and the data logical shard is bound to a physical data unit; the device further includes: The first determining module is used to determine the first position information of the multiple virtual nodes of the data logical shard on the hash ring based on the identifiers of the multiple virtual nodes of the data logical shard. The second determining module is used to determine the second position information of the physical data unit on the hash ring based on the identifier of the physical data unit; The third determining module is used to determine, based on the first location information and the second location information, the first virtual node that is closest to the physical data unit in the target direction of the hash ring; The binding module is used to bind the physical data unit to the data logical fragment to which the first virtual node belongs.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images