Data synchronization method, medium, electronic device and product based on cloud search service

Abstract

A data synchronization method, medium, electronic equipment and product based on a cloud search service, relate to the technical field of computers, and through responding to the need of the primary shard to be migrated, determining the second index number corresponding to the second primary shard according to the first index number currently used by the first primary shard and the target step length related to the migration mode of the first primary shard, and then determining the file name corresponding to the index file written into the second primary shard based on the second index number, the file name corresponding to the index file written into the second primary shard can be defined using a larger second index number in the second primary shard when the first primary shard is migrated, thereby avoiding the file name of the index file written into the second primary shard from conflicting with the file name of the index file in the first primary shard.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data synchronization method, medium, electronic device, and product based on cloud search services. Background Technology

[0002] When a primary shard is migrated, a new primary shard creates a new index file. At this time, the filename of the new index file may conflict with the filename of the old primary shard's index file. This is especially problematic in scenarios using remote storage, where the new index file created by the new primary shard needs to be uploaded to remote storage. If two index files have the same filename but different data content, this can lead to serious issues. Therefore, ensuring that index filenames do not conflict is a critical technical problem that needs to be solved. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] Firstly, this disclosure provides a data synchronization method based on cloud search services, including:

[0005] In response to the need for migration of the first primary shard of the cloud search service, the first index number currently used by the first primary shard is determined. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard.

[0006] Based on the migration method of the first primary shard, a target step size is determined, and based on the target step size and the first index number, a second index number corresponding to the second primary shard used to replace the first primary shard is determined, wherein the target step size is used to make the second index number greater than the first index number;

[0007] During the migration from the first primary shard to the second primary shard, the file name corresponding to the index file written to the second primary shard is determined based on the second index number, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

[0008] Secondly, this disclosure provides a data synchronization device based on cloud search services, comprising:

[0009] The first determining module is configured to determine the first index number currently used by the first primary shard in response to the need for migration of the first primary shard of the cloud search service. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard.

[0010] The second determining module is configured to determine a target step size based on the migration method of the first primary shard, and to determine a second index number corresponding to the second primary shard used to replace the first primary shard based on the target step size and the first index number, wherein the target step size is used to make the second index number greater than the first index number.

[0011] The third determining module is configured to determine, based on the second index number, the file name corresponding to the index file written to the second primary shard during the migration process from the first primary shard to the second primary shard, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

[0012] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0013] Fourthly, this disclosure provides an electronic device, comprising:

[0014] A storage device on which computer programs are stored;

[0015] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.

[0016] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0017] Based on the above technical solution, in response to the need for migration of the primary shard in the cloud search service, the first index number currently used by the first primary shard is determined. According to the migration method of the first primary shard, the target step size is determined. Based on the target step size and the first index number, the second index number corresponding to the second primary shard is determined. Then, during the migration from the first primary shard to the second primary shard, the filename corresponding to the index file written to the second primary shard is determined based on the second index number. This ensures that the filename corresponding to the index file written to the second primary shard is different from the filename of the index file in the first primary shard. This not only avoids conflicts between the filename of the index file written to the second primary shard and the filename of the index file in the first primary shard, but also, in scenarios using remote storage, when the index file created by the second primary shard needs to be uploaded to remote storage, using a larger second index number to determine the filename ensures that there will be no conflicts with the filename in remote storage, thus guaranteeing that the index file created by the second primary shard can be successfully uploaded to remote storage.

[0018] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 This is an architecture diagram of a cloud search service illustrated according to some embodiments.

[0021] Figure 2 This is a flowchart illustrating a data synchronization method based on a cloud search service, according to some embodiments.

[0022] Figure 3 This is a schematic diagram of a pre-copy shown according to some embodiments.

[0023] Figure 4 This is a schematic diagram illustrating the startup of a second primary shard via a read / write engine, based on some embodiments.

[0024] Figure 5 This is a schematic diagram illustrating the startup of a replica shard via a read-only engine, based on some embodiments.

[0025] Figure 6 This is a schematic diagram of a data synchronization device based on a cloud search service, as shown in some embodiments.

[0026] Figure 7This is a schematic diagram of the structure of an electronic device according to some embodiments. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] The data synchronization method based on cloud search services provided in this disclosure is applicable to cloud search services. The cloud search service referred to in the embodiments of this disclosure can be understood as a one-stop information retrieval and analysis platform, applicable to business scenarios such as full-text search, vector search, hybrid search, AI search, and spatiotemporal retrieval. For example, in a log analysis system, the cloud search service can be used to store and query log data. In an e-commerce platform, the cloud search service can be used to provide product search functionality, offering a fast and accurate search experience. In a big data monitoring system, the cloud search service can be used to store and analyze monitoring data to help users promptly identify potential problems.

[0034] In practical applications, cloud search services can be deployed in a distributed manner across multiple nodes (such as cloud servers). Specifically, cloud search services can be deployed across multiple nodes using a storage-compute separation architecture. The following section, in conjunction with the appendix... Figure 1 The storage and compute separation architecture of cloud search services is explained in detail.

[0035] Figure 1 This is an architecture diagram of a cloud search service illustrated according to some embodiments. For example... Figure 1 As shown, the cloud search service comprises multiple compute nodes and storage nodes, which work collaboratively to provide distributed search and analysis capabilities. Compute nodes provide data computation capabilities, while storage nodes deploy remote storage to provide data storage capabilities. Figure 1 As shown, compute nodes 1, 2, and N serve as compute nodes for deploying primary and replica shards. P0, P1, ..., PN represent primary shards deployed on a single compute node, while R0, R1, ..., RN represent the corresponding replica shards. It's important to note that primary shards and their corresponding replica shards can be deployed on different nodes. Furthermore, the number of compute nodes and storage nodes can be elastically scaled up or down according to the needs of the cloud search service.

[0036] In cloud search services, an index is a data structure used to store and search documents. An index can store a massive amount of data that exceeds the hardware limitations of a single node. For example, an index with 1 billion documents might require 1TB of disk space, which may not be available on any single node, or the response time for processing search requests on a single node might be too slow. Therefore, in cloud search services, indexes are divided into multiple shards, such as... Figure 1 As shown, the index can be divided into N shards, P0, P1, ..., PN. Each shard stores a portion of the data in the index. Each shard is essentially a fully functional and independent "index," and each shard is deployed on different nodes.

[0037] Each shard can include a primary shard and replica shards. The primary shard stores the actual index files and is responsible for reading and writing them. The number of primary shards is defined when the index is created and cannot be changed after creation. Replica shards are copies of the primary shards. Each primary shard can have zero or more replica shards. Replica shards copy the index files stored in the primary shard and are responsible for reading them. Replica shards improve the high availability, fault tolerance, and query performance of cloud search services. Replica shards copy the index files of their corresponding primary shards to ensure redundancy and security. When a primary shard fails, the corresponding replica shard can be promoted to a new primary shard to continue providing search services.

[0038] exist Figure 1 In the compute-deposit separation architecture shown, the primary shard can store the index file in remote storage. This means that the compute node hosting the primary shard does not store the actual index file; instead, it maintains metadata indicating the storage path of the index file in remote storage. Similarly, in this architecture, the replica shard copies not the index file from the corresponding primary shard, but rather metadata indicating the storage path of the index file in remote storage. In other words, the replica shard also maintains metadata. When the primary or replica shard receives a query request from a client, it uses the metadata to determine the storage path of the data to be queried in remote storage and reads the data from remote storage using that path.

[0039] Therefore, under a storage-compute separation architecture, Figure 1 The compute nodes shown are responsible for data computation and do not actually store index files, while the storage nodes are responsible for actually storing index files. This enables the cloud search service to separate computation and storage, thereby reducing the local storage cost of compute nodes.

[0040] A client can send a write request to any compute node in the cloud search service. The compute node receiving the write request acts as a coordinating node, forwarding the write request to the corresponding primary shard's compute node. The primary shard processes the write request, writes the corresponding index file to the primary shard, and synchronizes the index file to the corresponding replica shard. After confirming successful write, the coordinating node returns a response to the client. Alternatively, while processing a write request, the primary shard can directly write the corresponding index file to remote storage and maintain metadata on the primary shard indicating the storage path of the index file in remote storage. The primary shard can also synchronously synchronize this metadata to the corresponding replica shard. When a primary or replica shard receives a query request from the client, it reads the required field values ​​from remote storage using the metadata, thus enabling the cloud search service to achieve storage-compute separation.

[0041] The data synchronization method based on cloud search service provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0042] Figure 2 This is a flowchart illustrating a data synchronization method based on a cloud search service, according to some embodiments. For example... Figure 2 As shown, this disclosure provides a data synchronization method based on cloud search services, specifically executed by a data synchronization device based on cloud search services, which can be implemented in software and / or hardware. Figure 2 As shown, the method may include the following steps.

[0043] In step 210, in response to the need for migration of the first primary shard of the cloud search service, the first index number currently used by the first primary shard is determined. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard.

[0044] Here, the first primary shard of the cloud search service can refer to any primary shard in the cloud search service. The migration of the first primary shard can be triggered by factors such as the node hosting the first primary shard becoming unavailable or a rebalancing of load. For example, if the node hosting the first primary shard exits, the replica shard corresponding to the first primary shard will be promoted to the second primary shard. Alternatively, if the node hosting the first primary shard runs out of disk space or requires load balancing, the first primary shard will migrate from one node to another.

[0045] During the migration of the first primary shard, it will be in a "migrating" or "partially available" state. The first primary shard still exists, but new write operations will be blocked and must wait for the migration to complete. The second primary shard is created on the new node to receive data migrated from the first primary shard. During the migration of the first primary shard, the second primary shard starts from a blank state or a replica shard state, then receives data from the first primary shard. Once data replication is complete, the second primary shard is marked as "active" and begins processing read and write requests. Therefore, when the first primary shard needs to be migrated, the second primary shard can be understood as a new primary shard formed by migrating the old first primary shard.

[0046] In other words, in this embodiment of the disclosure, the second primary shard is the primary shard that takes over the role of the old first primary shard. The second primary shard becomes the primary shard of the shard corresponding to the old first primary shard. The second primary shard is used to take over the working logic of the first primary shard and is responsible for handling read and write requests, while the old first primary shard will be deleted after verification.

[0047] The first index number currently used by the first primary shard refers to the index number currently being used by the first primary shard to determine the filename of the index file written to the first primary shard. In the cloud search service, the first primary shard maintains a segment counter to record changes in the number of segment files within the first primary shard. Whenever a new segment file is created in the first primary shard, the segment counter increments accordingly. Accordingly, the first index number currently used by the first primary shard could refer to the segment counter value when the first primary shard needs to be migrated.

[0048] It is important to note that in cloud search services, index data is segmented into multiple segment files. Each segment file is a complete and immutable index structure, possessing its own dictionary, inverted index, and document storage data structures. Therefore, in this embodiment, the index file can refer to each segment file.

[0049] In step 220, the target step size is determined according to the migration method of the first primary shard, and the second index number corresponding to the second primary shard used to replace the first primary shard is determined according to the target step size and the first index number. The target step size is used to make the second index number greater than the first index number.

[0050] Generally, when a second primary shard needs to be launched, an engine is used to initialize and run it. Therefore, as an example, the engine used to launch the second primary shard can determine the second index number corresponding to the second primary shard based on the first index number currently used by the first primary shard and the target step size. The target step size is related to the migration method of the primary shard and is used to ensure that the second index number is greater than the first index number.

[0051] For example, the second index number can be determined by the sum of the first index number and the target step size. For instance, assuming the first index number is 100 and the target step size is 200000, the second index number could be 200100. In this embodiment, when the first primary shard needs to be migrated, the second primary shard uses a larger second index number to determine the filename of the index file, thereby avoiding conflicts between the filename of the index file written to the second primary shard and the filename of the index file on the old first primary shard.

[0052] For example, the engine used to launch the second primary shard can be a read / write engine. In embodiments of this disclosure, a second index number used to determine the file name of the index file can be controlled at the engine level, thereby avoiding the generation of index files with the same file name but different file content.

[0053] It should be understood that the target step size is a dynamic step size, and the target step size will be different depending on the migration method of the first primary shard. In the embodiments of this disclosure, there are two migration methods for the first primary shard: one is that the replica shard corresponding to the first primary shard is upgraded to the second primary shard, and the other is that the first primary shard is directly migrated to another node.

[0054] For example, if the migration method of the first primary shard represents the upgrade of the replica shard corresponding to the first primary shard to the second primary shard, the target step size can be calculated using the tenure of the second primary shard. If the migration method of the first primary shard represents the direct migration of the first primary shard to another node, the target step size can be determined using the number of migrations already performed by the first primary shard. By using the target step size associated with the migration method of the first primary shard, the second index number can be dynamically adjusted. When the first primary shard is migrated, a larger second index number can be used to define the filename of the index file in the second primary shard, thus avoiding conflicts between the filename of the index file in the second primary shard and the filename of the index file in the old first primary shard.

[0055] It is worth noting that, such as Figure 1 As shown, when the cloud search service includes remote storage, the first primary shard uploads its index file to the remote storage. When the first primary shard migrates, the second primary shard generates a new index file and uploads it to the remote storage. However, since the old first primary shard continues to write data during the process of the second primary shard replacing the old one, directly using the first index number as the second index number for the second primary shard would result in index files with the same name existing in the remote storage. By using a second index number greater than the first index number, the generation of index files with the same name on the remote storage can be avoided, ensuring that the index file of the second primary shard can be successfully uploaded to the remote storage during the migration of the first primary shard, thus supporting a compute-storage separation architecture.

[0056] In step 230, during the migration from the first primary shard to the second primary shard, the filename corresponding to the index file written to the second primary shard is determined based on the second index number, so that the filename corresponding to the index file written to the second primary shard is different from the filename of the index file in the first primary shard.

[0057] Here, after determining the second index number, the filename of the index file written to the second primary shard can be determined using the second index number. For example, the second index number can be used as the initial number of the segment counter maintained by the second primary shard. Whenever a new index file is created in the second primary shard, the filename of that index file is determined based on the number recorded in the segment counter. Furthermore, the segment counter count is incremented accordingly. Consequently, the filename of subsequent index files written to the second primary shard will also be determined using a larger second index number.

[0058] It is worth noting that since the second index number is determined by the engine used to start the second primary shard, the second index number of each second primary shard is determined by the engine corresponding to the second primary shard. This is done in a decentralized manner. Furthermore, since the second index number is greater than the first index number, the filename of the index file written to the second primary shard will be different from the filename of the index file in the old first primary shard.

[0059] Therefore, in response to the need for migration of the primary shard in the cloud search service, the first index number currently used by the first primary shard is determined. Based on the migration method of the first primary shard, the target step size is determined. Based on the target step size and the first index number, the second index number corresponding to the second primary shard is determined. Then, during the migration from the first primary shard to the second primary shard, the filename corresponding to the index file written to the second primary shard is determined based on the second index number. This ensures that the filename corresponding to the index file written to the second primary shard is different from the filename of the index file in the first primary shard. This not only avoids conflicts between the filename of the index file written to the second primary shard and the filename of the index file in the first primary shard, but also, in scenarios using remote storage, when the index file created by the second primary shard needs to be uploaded to remote storage, using a larger second index number to determine the filename ensures that the filename in the remote storage will not conflict, thus ensuring that the index file created by the second primary shard can be successfully uploaded to the remote storage.

[0060] In some feasible implementations, in step 220, if the migration mode indicates that the replica shard corresponding to the first primary shard is upgraded to the second primary shard, the target step size is determined according to the tenure of the second primary shard, and the size of the target step size is positively correlated with the tenure of the second primary shard.

[0061] Here, the migration method for the first primary shard can include upgrading the replica shard corresponding to the first primary shard to the second primary shard. Accordingly, when the replica shard corresponding to the first primary shard is upgraded to the second primary shard, the target step size can be determined based on the term of the second primary shard using the engine used to start the second primary shard, and then the second index number corresponding to the second primary shard can be determined based on the first index number and the target step size.

[0062] The target step size is positively correlated with the primary term of the second primary shard. In other words, when the migration method of the first primary shard is to upgrade the replica shard corresponding to the first primary shard to the second primary shard, the target step size can be determined by the primary term of the second primary shard. Moreover, the longer the primary term of the second primary shard, the larger the target step size.

[0063] When the primary shard fails, the cloud search service uses an election algorithm to select a second primary shard from among the replica shards of the primary shard. Once a replica shard wins the election and becomes the second primary shard, its term begins. During its term, the second primary shard undertakes key responsibilities, such as handling client read and write requests, coordinating write requests, ensuring data consistency, and synchronizing write operation information to the corresponding replica shards to ensure that the data on the replica shards is updated in a timely manner and remains consistent with the second primary shard.

[0064] When a replica shard corresponding to the first primary shard is upgraded to the second primary shard, the term of the second primary shard, which is upgraded from the replica shard, will differ from the term of the first primary shard. For example, suppose shard A is the first primary shard with a term of 1, and shard B is a replica shard of the first primary shard. If shard A fails and shard B is upgraded to the second primary shard, then shard B's term will change to 2. If shard B fails after being upgraded to the second primary shard, and shard A is then switched back to the second primary shard, then shard A's term will change to 3.

[0065] In some embodiments, the sum of the first index number and the target step size can be determined as the second index number.

[0066] For example, the second index number can be obtained through a first calculation formula, which is:

[0067]

[0068] Where A1 is the second index number and B1 is the first index number. C1 is the baseline step size, and C1 is the term. The target step size.

[0069] It should be noted that the baseline step size can be a constant. For example, the baseline step size can be 100000. Of course, the baseline step size can also be other values, which can be set according to the actual situation.

[0070] Assume shard A is the first primary shard with a term of 1, and shard B is a replica shard of the first primary shard. Both the primary and replica shards have a first index number of 100. When shard A fails, shard B is promoted to the second primary shard, and its term changes to 2. Based on the first calculation formula, the second index number corresponding to shard B is... If shard B fails after being promoted to the second primary shard, and shard A is then switched back to the second primary shard, then the tenure of shard A will change to 3. At this point, based on the first calculation formula, the second index number corresponding to shard A will be... .

[0071] It should be understood that the first calculation formula provided in the above embodiments is an example of calculating the second index number. In other embodiments, other methods can also be used to calculate the second index number. For example, the product between the first index number and the target step size can be used as the second index number.

[0072] Therefore, when the migration method represents the upgrade of the replica shard corresponding to the first primary shard to the second primary shard, the target step size can be dynamically adjusted by the tenure of the second primary shard. This allows the second index number corresponding to the second primary shard to be much larger than the first index number, thereby avoiding conflicts between the file names of the index files in the second primary shard and the file names of the index files in the old first primary shard.

[0073] In some feasible implementations, in step 220, the migration mode represents the migration of the first primary shard to another node, and the target step size is determined based on the number of times the first primary shard has migrated to the other node. The size of the target step size is positively correlated with the number of times the first primary shard has migrated.

[0074] Here, the migration method for the first primary shard can include migrating the primary shard to another node. This other node can be a node that does not have a replica shard corresponding to the first primary shard deployed. Accordingly, when the first primary shard is migrated to another node, the engine used to start the second primary shard can determine the target step size based on the number of times the first primary shard has been migrated to the other node. Then, based on the first index number, the number of logs corresponding to the transaction logs included in the first primary shard, and the target step size, the second index number corresponding to the second primary shard can be determined.

[0075] The target step size is positively correlated with the number of times the first primary partition has been migrated. In other words, when the migration method of the first primary partition represents the migration of the first primary partition to another node, the target step size can be determined by the number of times the first primary partition has been migrated, and the larger the number of times the first primary partition has been migrated, the larger the target step size.

[0076] It's important to note that when the first primary shard is migrated to another node, the second primary shard is formed by migrating the old first primary shard. Therefore, the term of the second primary shard remains unchanged and is consistent with the term of the old first primary shard. Thus, the target step size can be dynamically adjusted based on the number of migrations of the first primary shard.

[0077] The number of times the first primary shard has been migrated refers to the number of times the first primary shard has migrated to another node. It's important to note that during the migration process, migration failures may occur. In this case, the migration process will be terminated and restarted until the first primary shard successfully migrates to the other node. Therefore, the number of times the first primary shard has been migrated to another node during this process is the number of times the first primary shard has been migrated.

[0078] For example, when the first primary shard is migrated for the first time, the number of migrations of the first primary shard can be 0. When the first primary shard fails to migrate to another node for the first time, the process of migrating the first primary shard to another node for the second time is started. When the first primary shard migrates to another node for the second time, the number of migrations of the first primary shard can be 1, and so on. As the number of migrations of the first primary shard increases, the target step size will also increase.

[0079] The transaction log (Translog) is a crucial component of the first primary shard, recording all changes made to the index data. During the migration of the first primary shard, the transaction log needs to be restored during peer recovery (the process of restoring a replica shard from the primary shard). Therefore, the number of transaction log entries must be considered when determining the second index number.

[0080] In some embodiments, the sum of the first index number, the number of logs, and the target step size can be used to determine the second index number.

[0081] For example, the second index number can be obtained through a second calculation formula, which is:

[0082]

[0083] Where A2 is the second index number, B2 is the first index number, and C2 is the number of logs. As the reference step size, This represents the number of migrations already made. The target step size.

[0084] It should be noted that the baseline step size can be a constant. For example, the baseline step size can be 100000. Of course, the baseline step size can also be other values, which can be set according to the actual situation.

[0085] Of course, the second calculation formula provided in the above embodiment is an example of calculating the second index number. In other embodiments, other methods can also be used to calculate the second index number. For example, the product between the first index number, the number of logs, and the target step size can be used as the second index number.

[0086] In some embodiments, during the migration of the first primary shard to another node, if the filename of the index file written to the second primary shard is the same as the filename of the index file in the first primary shard, the process of migrating the first primary shard to another node is terminated, the migration count of the first primary shard is incremented, and then the first primary shard is migrated to another node again based on the incremented migration count.

[0087] During the migration of the first primary shard to another node, filename conflicts may occur even if the second primary shard uses a larger second index number to define the filename of the index file written to it. This is because the first primary shard continues to write to its index file. To resolve this, during the migration, it's possible to check if the filename of the index file written to the second primary shard is the same as the filename of the index file in the first primary shard. If they are the same, the migration process is terminated, the migration count of the first primary shard is incremented, and the migration is restarted based on the incremented count. If they are different, the migration continues until completion.

[0088] It should be understood that a counter can be maintained to record the number of times the first primary shard has been migrated. The initial value of the counter can be 0. When it is detected that the file name of the index file written to the second primary shard is the same as the file name of the index file in the first primary shard, the process of migrating the first primary shard to another node is terminated, the counter value is incremented by 1, and then the process of migrating the first primary shard to another node is restarted.

[0089] Since the migration count is incremented by 1 each time the first primary shard is migrated to another node, the second index number used by the second primary shard will be larger. This significantly reduces the likelihood of filename conflicts when the first primary shard is migrated to another node. However, if a filename in the index file written to the second primary shard is still the same as the filename in the index file in the first primary shard, the migration process of the first primary shard to another node will continue to terminate, the migration count of the first primary shard will be incremented, and the next migration will be performed until the migration is successful.

[0090] Therefore, when the first primary shard is migrated to another node, the target step size can be dynamically adjusted by the number of times the first primary shard has been migrated. This allows the second index number corresponding to the second primary shard to be much larger than the first index number, thereby avoiding conflicts between the file names of the index files of the second primary shard and the index files of the first primary shard.

[0091] In some feasible implementations, when the first primary shard performs multiple segment file merging to obtain the target segment file, a pre-copy mechanism is used to copy the target segment file to the replica shard corresponding to the first primary shard, and when the target segment file has been copied to the replica shard, the target segment file is made visible to the first primary shard.

[0092] Here, segmented file replication is a novel data replication strategy that improves index throughput and resource utilization by copying segmented files from the index to replica shards. When a new segmented file is generated or an old segmented file is merged into the first primary shard, the new segmented file needs to be synchronized to the replica shards. When a node fails, the newly added replica shards need to rebuild the data by copying the segmented files from the first primary shard. Compared to document replication synchronization, segmented file replication significantly reduces memory and CPU overhead.

[0093] In cloud search services, there are two types of segment files: target segment files generated by merging segment files, and segment files built through refresh (which involves flushing the segment files in memory to the index), used for incremental indexing. For target segment files, assuming a target segment file is 1GB and the transmission bandwidth is 50MB / s, the copying time for the target segment file is at least 20 seconds. For larger target segment files, the copying time is even longer, resulting in a significant visibility delay between the first primary shard and the replica shards.

[0094] In this embodiment, when the first primary shard performs the operation of merging multiple segment files to obtain a target segment file, a pre-copy mechanism is used to copy the target segment file to the replica shard corresponding to the first primary shard. Furthermore, the target segment file is only made visible to the first primary shard after it has been copied to the replica shard, thereby reducing the visibility latency of the target segment file between the first primary shard and the replica shard. For segment files generated by Refresh, the segment copying process proceeds normally.

[0095] The pre-copy mechanism here refers to Lucene's IndexWriter.IndexReaderWarmer pre-copy mechanism (an interface that allows newly merged segment files to be warmed up before being committed to the index). The visibility of the target segment file to the first primary shard indicates that the first primary shard can use the target segment file to handle read and write requests.

[0096] It should be understood that the replica shard creates a listener on the first primary shard to detect whether a new segment file has been generated on the first primary shard. When the target segment file becomes visible to the first primary shard, the replica shard detects the presence of a target segment file on the first primary shard, and at this point, the replica shard needs to copy the target segment file. However, in this embodiment, the target segment file has already been pre-copied to the replica shard before it becomes visible to the first primary shard. Therefore, when the replica shard detects the presence of a target segment file on the first primary shard, it can directly load the target segment file without needing to copy it. Thus, the visibility latency of the target segment file in the first primary shard and the replica shard can be reduced to the millisecond level.

[0097] Figure 3 This is a schematic diagram illustrating a pre-copy according to some embodiments. For example... Figure 3 As shown, the first primary shard processes the write stream through the engine. The first primary shard includes segment file 4 (the target segment file), which is generated by merging segment file 1 and segment file 2, and segment file 3, generated through refresh. For segment file 4, a pre-copy mechanism is used to copy segment file 4 to the replica shard, and segment file 4 becomes visible to the first primary shard only after it has been completely copied to the replica shard. For segment file 3, it is copied to the replica shard through the normal segment copy process. In the replica shard, segment file 3 and segment file 4 are loaded into the engine.

[0098] Since segment file 4 becomes visible to the first primary shard only after it has been completely copied to the replica shard, the replica shard detects the presence of segment file 4 on the first primary shard. Because segment file 4 already exists locally on the replica shard, the replica shard will not copy segment file 4 again, but will directly make segment file 4 visible to the replica shard, thereby reducing the visibility delay of segment file 4 between the first primary shard and the replica shard.

[0099] Therefore, the pre-copy mechanism can reduce the visibility latency of the target segment file generated by merging between the first primary fragment and the replica fragment.

[0100] In some feasible implementations, when the first primary shard needs to be migrated, the second primary shard is started through the read / write engine so that the second primary shard is in a read / write state during the migration from the first primary shard to the second primary shard.

[0101] Here, the read / write engine can be any engine that supports read and write functions; for example, it could be an Internal Engine. When the first primary shard needs to be migrated, the second primary shard is started via the read / write engine. The second primary shard can then be in a read-write state during the migration of the first primary shard. Thus, when the first and second primary shards perform a handoff, the process is very brief and has almost no impact on the user's real-time writes.

[0102] It's important to note that if the second primary shard is started with a read-only engine, it will initially create a read-only engine solely for receiving segment files and translog files from the first primary shard. It will not replay the translog files. Only during the handoff phase between the first and second primary shards will the first primary shard initiate a new round of forced segment replication and then reset the read-only engine to a read-write engine. At this point, the first primary shard will accept new write operations. However, the handoff phase can cause interruptions in user write operations. If users write large amounts of data during the migration process, this can lead to excessively long handoff times, resulting in prolonged write interruptions and potentially causing write failures.

[0103] Figure 4 This is a schematic diagram illustrating the startup of a second primary shard via a read / write engine, based on some embodiments. For example... Figure 4 As shown, the read / write engine of the first primary shard is responsible for handling the write stream, writing the data of the write stream into memory and recording it in the transaction log. In addition, the read / write engine of the first primary shard will periodically perform a refresh operation to write the data in memory into the disk, forming segmented files (such as segmented file 1 and segmented file 2).

[0104] When the first primary shard needs to be migrated, a segmented replication operation is first performed, copying segment files (e.g., segment file 1, segment file 2) to the second primary shard. The second primary shard loads the acquired segment files into its read / write engine and updates the segment file information. Then, the first primary shard creates a transaction log snapshot and sends it to the second primary shard. The second primary shard receives the transaction log snapshot and replays it to apply it to its transaction log. Thus, the second primary shard will have the same uncommitted transaction records as the first primary shard. Finally, a handoff operation is performed, specifically a complete relocation notification operation on the first primary shard. The first primary shard transfers control to the new second primary shard. Upon receiving control, the second primary shard also executes a complete relocation notification operation, confirming that it is ready to work independently. During the handoff phase, the first primary shard blocks new write operations to ensure that all uncommitted transactions have been synchronized to the second primary shard.

[0105] In other words, when the first primary shard needs to be migrated, starting the second primary shard through the read-write engine eliminates the need for forced segmented copying compared to starting the second primary shard through the read-only engine. Instead, a simple completeRelocation operation is sufficient, making the handoff process extremely short and virtually unaffected by the user's real-time writes.

[0106] Therefore, through the above implementation method, it can be ensured that the first primary shard can migrate smoothly without blocking the user's real-time writes.

[0107] In some feasible implementations, when the index file is copied from the first primary shard to the corresponding replica shard, the replica shard is started with a read-only engine so that the replica shard is in a read-only state.

[0108] Here, when the first primary shard copies the index file to the replica shard corresponding to the first primary shard, the replica shard is started through the read-only engine.

[0109] It should be understood that starting the replica shard with a read-only engine, so that the replica shard is in a read-only state, can prevent the replica shard from performing index operations (referring to the replica shard participating in the document writing process).

[0110] Figure 5 This is a schematic diagram illustrating the startup of a replica shard via a read-only engine, based on some embodiments. For example... Figure 5 As shown, when the first primary shard copies segment files to the replica shard, the replica shard is started through the read-only engine, and the segment files (such as segment file 1, segment file 2, and segment file 3) are copied to the replica shard.

[0111] In some feasible implementations, the first primary shard can be controlled to upload the index file written in the first primary shard to remote storage, which is used for the first primary shard or the replica shard corresponding to the first primary shard to read the required index file from the remote storage.

[0112] Here, as Figure 1 As shown, the cloud search service includes remote storage. The first primary shard can upload the index file to the remote storage and maintain the metadata corresponding to the index file in the first primary shard. The metadata is used to indicate the storage path of the index data in the remote storage.

[0113] It's important to note that because the first primary shard uploads the index file to remote storage, the compute node hosting the first primary shard does not need to mount a local disk for data storage, thus reducing local storage costs. For the compute node, maintaining a small amount of metadata is sufficient. When a query request is received, the first primary shard or replica shard uses the metadata to determine the storage path of the data requested in the remote storage. The first primary shard or replica shard then retrieves the data from the remote storage using that storage path. For the storage node, data is stored remotely; the computation is handled by the compute node. Therefore, the cloud search service achieves separation of computation and storage.

[0114] The primary shard can also synchronize metadata to the corresponding replica shard through segment replication. A client can send a query request to any node in the cloud search service. The node receiving the query request acts as a coordinating node, randomly forwarding the request to the corresponding primary or replica shard. When the primary or replica shard receives the query request, it accesses the index file in remote storage based on the metadata maintained by the primary or replica shard, and determines the field value corresponding to the query request from the index file.

[0115] Fields are the basic data units in a document, used to store and index specific document data. Each field has a name and one or more values. Fields define the structure and content of a document, facilitating searching, indexing, and querying. Field values ​​refer to the specific data values ​​stored in each field within the document. For example, in a user information document, the value of the "name" field could be a specific name, and the value of the "age" field could be a numerical age, etc.

[0116] In this embodiment of the disclosure, the first primary shard can write the index file to remote storage. Both the first primary shard and the replica shard support direct access to remote storage to read the field values ​​corresponding to the query request, thereby greatly reducing the cost of local storage.

[0117] It is worth noting that since the second index number is determined by the engine used to start the second primary shard, the second index number of each second primary shard is determined by the engine corresponding to the second primary shard. This is done in a decentralized manner. Furthermore, since the second index number is greater than the first index number, the filename of the index file written to remote storage by the second primary shard will be different from the filename of the index file written to remote storage by the old first primary shard.

[0118] Because the filename of the index data written to the remote storage by the second primary shard is different from the filename of the index file written to the remote storage by the old first primary shard, it is possible to support the first primary shard and the corresponding replica shard to directly query the index file in the remote storage to obtain field values.

[0119] Figure 6 This is a schematic diagram illustrating the structure of a data synchronization device based on a cloud search service, according to some embodiments. For example... Figure 6 As shown, this disclosure provides a data synchronization device 500 based on cloud search services, which includes:

[0120] The first determining module 501 is configured to determine the first index number currently used by the first primary shard in response to the need for migration of the first primary shard of the cloud search service. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard.

[0121] The second determining module 502 is configured to determine a target step size based on the migration method of the first primary shard, and to determine a second index number corresponding to the second primary shard used to replace the first primary shard based on the target step size and the first index number, wherein the target step size is used to make the second index number greater than the first index number.

[0122] The third determining module 503 is configured to determine the file name corresponding to the index file written to the second primary shard based on the second index number during the migration process from the first primary shard to the second primary shard, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

[0123] Optionally, the second determining module 502 is specifically configured as follows:

[0124] When the migration method represents the upgrade of the replica shard corresponding to the first primary shard to the second primary shard, the target step size is determined according to the tenure of the second primary shard, and the size of the target step size is positively correlated with the tenure of the second primary shard.

[0125] Optionally, the second determining module 502 is specifically configured as follows:

[0126] The migration method represents the migration of the first primary shard to another node. The target step size is determined based on the number of times the first primary shard has migrated to the other node. The size of the target step size is positively correlated with the number of times the first primary shard has migrated.

[0127] Optionally, the data synchronization device 500 based on cloud search service further includes:

[0128] The termination module is configured to terminate the migration of the first primary shard to the other node if, during the migration process of the first primary shard to the other node, the file name of the index file written to the second primary shard is the same as the file name of the index file in the first primary shard, and increment the number of migrations of the first primary shard.

[0129] The re-migration module is configured to re-migrate the first primary shard to the other node based on the increased number of migrations.

[0130] Optionally, the data synchronization device 500 based on cloud search service further includes:

[0131] The copy module is configured to, when the first primary shard performs multiple segment file merging to obtain a target segment file, copy the target segment file to the replica shard corresponding to the first primary shard through a pre-copy mechanism, and control the target segment file to be visible to the first primary shard after the target segment file has been copied to the replica shard.

[0132] Optionally, the data synchronization device 500 based on cloud search service further includes:

[0133] The first startup module is configured to start the second primary shard through the read / write engine when the first primary shard needs to be migrated, so that the second primary shard is in a readable and writable state during the migration from the first primary shard to the second primary shard.

[0134] The second startup module is configured to start the replica shard through a read-only engine when the first primary shard copies the index file to the replica shard corresponding to the first primary shard, so that the replica shard is in a read-only state.

[0135] Optionally, the data synchronization device 500 based on cloud search service further includes:

[0136] The upload module is configured to control the first primary shard to upload the index file written in the first primary shard to remote storage, the remote storage being used by the first primary shard or the replica shard corresponding to the first primary shard to read the required index file from the remote storage.

[0137] The functional logic executed by each functional module in the aforementioned data synchronization device 500 based on cloud search services has been explained in detail in the section on methods, and will not be repeated here.

[0138] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an electronic device (e.g., a server) 600 suitable for implementing embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0139] like Figure 7 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0140] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0141] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0142] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0143] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0146] In response to the need for migration of the first primary shard of the cloud search service, the first index number currently used by the first primary shard is determined. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard.

[0147] Based on the migration method of the first primary shard, a target step size is determined, and based on the target step size and the first index number, a second index number corresponding to the second primary shard used to replace the first primary shard is determined, wherein the target step size is used to make the second index number greater than the first index number;

[0148] During the migration from the first primary shard to the second primary shard, the file name corresponding to the index file written to the second primary shard is determined based on the second index number, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

[0149] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0151] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0152] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0155] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0156] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A data synchronization method based on cloud search services, characterized in that, include: In response to the need for migration of the first primary shard of the cloud search service, the first index number currently used by the first primary shard is determined. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard. Based on the migration method of the first primary shard, a target step size is determined, and based on the target step size and the first index number, a second index number corresponding to the second primary shard used to replace the first primary shard is determined, wherein the target step size is used to make the second index number greater than the first index number; During the migration from the first primary shard to the second primary shard, the file name corresponding to the index file written to the second primary shard is determined based on the second index number, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

2. The method according to claim 1, characterized in that, The step of determining the target step size based on the migration method of the first primary partition includes: When the migration method represents the upgrade of the replica shard corresponding to the first primary shard to the second primary shard, the target step size is determined according to the tenure of the second primary shard, and the size of the target step size is positively correlated with the tenure of the second primary shard.

3. The method according to claim 1, characterized in that, The step of determining the target step size based on the migration method of the first primary partition includes: The migration method represents the migration of the first primary shard to another node. The target step size is determined based on the number of times the first primary shard has migrated to the other node. The size of the target step size is positively correlated with the number of times the first primary shard has migrated.

4. The method according to claim 3, characterized in that, The method further includes: During the migration of the first primary shard to the other node, if the file name of the index file written to the second primary shard is the same as the file name of the index file in the first primary shard, then the process of migrating the first primary shard to the other node is terminated, and the migration count of the first primary shard is incremented. Based on the increased number of migrations, the first primary shard is migrated again to the other node.

5. The method according to any one of claims 1 to 4, characterized in that, The index file is a segmented file, and the method further includes: When the first primary shard performs multiple segment file merging to obtain the target segment file, the target segment file is copied to the replica shard corresponding to the first primary shard through a pre-copy mechanism. After the target segment file has been copied to the replica shard, the target segment file is made visible to the first primary shard.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: If the first primary shard needs to be migrated, the second primary shard is started through the read / write engine so that the second primary shard is in a read / write state during the migration from the first primary shard to the second primary shard; When the first primary shard copies the index file to the corresponding replica shard, the replica shard is started using a read-only engine to put the replica shard into a read-only state.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The first primary shard is controlled to upload the index file written in the first primary shard to remote storage, which is used for the first primary shard or the replica shard corresponding to the first primary shard to read the required index file from the remote storage.

8. A data synchronization device based on cloud search service, characterized in that, include: The first determining module is configured to determine the first index number currently used by the first primary shard in response to the need for migration of the first primary shard of the cloud search service. The first index number is used by the first primary shard to determine the file name corresponding to the index file written to the first primary shard. The second determining module is configured to determine a target step size based on the migration method of the first primary shard, and to determine a second index number corresponding to the second primary shard used to replace the first primary shard based on the target step size and the first index number, wherein the target step size is used to make the second index number greater than the first index number. The third determining module is configured to determine, based on the second index number, the file name corresponding to the index file written to the second primary shard during the migration process from the first primary shard to the second primary shard, so that the file name corresponding to the index file written to the second primary shard is different from the file name of the index file of the first primary shard.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing device, it implements the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images