Sharding processing method and device based on distributed storage system, equipment and storage medium

Abstract

The present disclosure provides a sharding processing method and device based on a distributed storage system, equipment and a storage medium, relates to the technical field of computers, in particular to the field of big data, distributed data and distributed storage systems. The specific implementation scheme is: in response to the sharding synchronization instruction, synchronizing the source shard indicated by the sharding synchronization instruction to the target shard indicated by the sharding synchronization instruction. During the synchronization process, if the operation request of the user is obtained, the operation information indicated by the operation request of the user can be written into the preset temporary storage space. After completing the synchronization from the source shard to the target shard, the operation information in the temporary storage space can be merged into the target shard.

Description

Technical Field

[0001] The present disclosure relates to big data, distributed data, and distributed storage systems in computer technology, and in particular to a sharding processing method, apparatus, device, and storage medium based on a distributed storage system. Background Art

[0002] A distributed storage system includes buckets. A bucket can contain multiple shards. Objects are stored in the bucket's shards. If the number of shards is not properly planned when creating a bucket, the bucket's efficiency may be affected as the number of objects in the bucket increases. Therefore, it is often necessary to perform a resharding operation on a bucket to synchronize a shard to another shard. During the synchronization process, the bucket receives user requests for operations on objects in the shards.

[0003] Furthermore, there is an urgent need for a solution that can promptly process user operation requests during shard synchronization. Summary of the Invention

[0004] The present disclosure provides a sharding processing method, apparatus, device, and storage medium based on a distributed storage system.

[0005] According to a first aspect of the present disclosure, a shard processing method based on a distributed storage system is provided, comprising:

[0006] In response to a shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard, storing operation information indicated by the user's operation request in a preset temporary storage space; wherein the shard synchronization instruction instructs synchronization of the source shard to the target shard; the operation request instructs to operate on the source shard or the target shard; and the operation information represents the content of the operation indicated by the operation request;

[0007] The operation information in the preset temporary storage space is merged into the target shard.

[0008] According to a second aspect of the present disclosure, a shard processing device based on a distributed storage system is provided, comprising:

[0009] a synchronization unit, configured to, in response to a shard synchronization instruction, store operation information indicated by a user's operation request in a preset temporary storage space during synchronization of a source shard indicated by the shard synchronization instruction to a target shard; wherein the shard synchronization instruction instructs synchronization of the source shard to the target shard; the operation request instructs operation on the source shard or the target shard; and the operation information represents the content of the operation indicated by the operation request;

[0010] A merging unit is used to merge the operation information in the preset temporary storage space into the target slice.

[0011] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0012] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a scene graph that can implement the embodiments of the present disclosure;

[0018] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0022] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0023] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0024] Figure 8 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0025] Figure 9 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0026] Figure 10 is a schematic block diagram of an electronic device that can implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] A distributed storage system includes buckets. A bucket can have multiple shards. Objects can be stored in the shards of a bucket. If the number of shards is not properly planned when creating a data bucket, the efficiency of the bucket may be affected as the number of objects in the bucket increases. Therefore, it is often necessary to perform a resharding operation on the bucket to increase the number of shards and thus reduce the size of each shard. However, currently, during the entire resharding process, all write operations to the source shard will be blocked, resulting in the corresponding bucket being prohibited from writing data. For a bucket containing 100 million objects, the blocking time for this write operation will exceed 1 hour. This is unacceptable to most users.

[0029] Currently, the implementation of reshard operations mainly includes the following three steps:

[0030] First, lock the entire bucket and prohibit write operations on the bucket. Create the corresponding number of target shards based on the newly configured number of shards.

[0031] Second, list processing is performed on the objects in the source shard, and the data in the list in the source shard is written to the corresponding target shard. This list processing can obtain a list of common objects, a list of version numbers, a list of olh objects, etc. in the source shard.

[0032] Third, the bucket lock is released. If reharding succeeds, the source shard is deleted. The list processing targets in the bucket are directed to the target shard. If reharding fails, the target shard is deleted.

[0033] Based on the above steps, in order to ensure that users can still perform write operations after locking the bucket during the rehard process, a log-based optimization method has been proposed. This method can first synchronize objects on the source shard to the target shard during the rehard process. During this synchronization process, if the user performs a write operation, the source shard will record a log. The source shard will then continue to block the user's write operation. However, after the synchronization of the source shard data is completed, the objects involved can be copied to the target shard by replaying the log file, thus completing the entire rehard process. However, the log usually contains a large amount of data. When performing a write operation, it is usually accompanied by a query operation. The other data in the log information can easily affect the real-time performance of the query process, resulting in blocking the user's write operation.

[0034] In response to the above problems, this paper proposes a sharding processing method based on a distributed storage system, such as Figure 1 As shown, it may include a source shard 101, a temporary storage space 102 established in the memory area of ​​the source shard 101, and a target shard 103. A synchronization operation 104 and a merge operation 105 may also be included between the source shard and the target shard. Specifically, when it is determined that the source shard 101 needs to be synchronized to the target shard 103, the synchronization operation 104 is executed, and the temporary storage space 102 is established. During the execution of the synchronization operation 104, the user's operation request will be recorded in the temporary storage space 102. After the execution of the synchronization operation 104 is completed, the merge operation 105 will be executed. The merge operation 105 is used to merge the information recorded in the temporary storage space 102 into the target shard 103. The setting of the temporary storage space can realize user write operations, thereby further shortening the query efficiency during the user write process and shortening the user blocking time.

[0035] The present disclosure provides a shard processing method, apparatus, device, and storage medium based on a distributed storage system, which are applied to big data, distributed data, and distributed storage systems in the computer field to reduce the user's blocking time during the shard replication process and improve the user experience.

[0036] It should be noted that the data in this embodiment is not targeted at a specific user and does not reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0037] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to make readers more deeply understand the implementation principle of this disclosure, the following Figure 3-Figure 7 right Figure 2 The illustrated embodiment is further refined.

[0039] Figure 2 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 2 As shown, the present disclosure provides a sharding processing method based on a distributed storage system, the method comprising:

[0040] 201. In response to a shard synchronization instruction, while synchronizing the source shard indicated by the shard synchronization instruction to the target shard, store operation information indicated by the user's operation request in a preset temporary storage space. The shard synchronization instruction instructs synchronization of the source shard to the target shard. The operation request instructs an operation to be performed on the source shard or the target shard. The operation information represents the content of the operation indicated by the operation request.

[0041] In this embodiment, after the number of target shards is set, multiple corresponding shard synchronization instructions may be generated. The shard synchronization instructions may be used to instruct the synchronization of source shards to target shards. Alternatively, the shard synchronization instructions may be used to instruct the synchronization of objects in the source shards to the target shards. Alternatively, the shard synchronization instructions may be used to instruct the synchronization of some or all objects in one or more source shards to one or more target shards.

[0042] In response to the shard synchronization instruction, the source shard indicated by the shard synchronization instruction may be locked. The lock is used to indicate to the user that the source shard is currently unable to perform a write operation when the user requests to obtain information in the source shard.

[0043] Optionally, in response to the shard synchronization instruction, a temporary storage space may be created within the storage space where the source shard is located. Optionally, setting the temporary storage space within the memory space of the source shard may facilitate the synchronous release of the source shard and the temporary storage space. Optionally, the size of the temporary storage space may be set according to the size of the source shard. Optionally, the size of the temporary storage space may also be determined based on historical access information of the source shard. Optionally, setting the size of the temporary storage space based on the size of the source shard may ensure that there is sufficient space in the temporary storage space and that the space may be used as little as possible, thereby improving space utilization.

[0044] In response to the shard synchronization instruction, the source shard indicated by the shard synchronization instruction is synchronized to the target shard indicated by the shard synchronization instruction. During the synchronization process, if the user's operation request is obtained, since the source shard has been locked, the source shard will not be able to respond to the operation request to perform a write operation. Therefore, when the operation request includes a write operation, the operation information indicated by the user's operation request can be written into the temporary storage space. Optionally, the operation information can include the operation content in the operation request. For example, when the operation request is a modification operation, the operation content can include the index value of the object to be modified, and the data of the modified object. Optionally, when the user's operation request is a query, the source shard can still perform a query operation. However, since the source shard cannot be written, it is necessary to synchronize the query data in the temporary storage space to ensure that the feedback data is the latest data.

[0045] 202. Merge the operation information in the preset temporary storage space into the target shard.

[0046] In this embodiment, after synchronization of the source shard with the target shard is complete, the operation information in the temporary storage space can be merged into the target shard to ensure that the data stored in the target shard is the latest data at the current moment. Optionally, during this merging process, the operation information in the temporary storage space can be obtained one by one and then merged into the target shard one by one.

[0047] In this embodiment, by setting up a temporary storage space, during the synchronization process between the source shard and the target shard, operation information of user operation requests that cannot be written due to the source shard lock can be written to the temporary storage space. This ensures that the user operation request is processed during the synchronization process between the source shard and the target shard, avoids blocking the user operation, and improves the user experience. Furthermore, the normal implementation of the user operation can further ensure the efficiency of the user's business execution.

[0048] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 3 As shown, the present disclosure provides a sharding processing method based on a distributed storage system. In the specific process of storing the operation information in the user's operation request into a preset temporary storage space, the method includes:

[0049] 301. Generate operation information and a timestamp of an object corresponding to the operation information according to the operation request.

[0050] In this embodiment, when the operation request is received, corresponding operation information can be generated in response to the operation request. This operation information can specifically include the index value of the object to be operated on and the data of the object to be operated on after the operation. Furthermore, during the process of generating the operation information, a timestamp of the object corresponding to the operation information can also be generated. Optionally, the operation can include operations such as adding, deleting, and modifying that require performing a write operation on the source shard.

[0051] Specifically, according to different operation requests such as adding, deleting, and modifying, the timestamps of the generated operation information and the objects corresponding to the operation information can be as follows:

[0052] First, if the operation request represents a modification of an object in the source or target shard, operation information is generated, including the index value of the object in the shard indicated by the operation request, the object's data, and a timestamp for the object. By generating operation information containing the modified object's index value and data, as well as a timestamp, for modification operations, the modification can be better recorded and the timestamp can be used to ensure the validity of the data during the update process.

[0053] Second, if the operation request indicates the deletion of an object in the source or target shard, operation information is generated, including the index value of the object in the shard indicated by the operation request, the empty data of the object, and the timestamp of the object. By setting the data to empty, a special record of the deletion operation is implemented, ensuring effective deletion during the data merge process.

[0054] Third, if the operation request indicates adding an object to the source or target shard, operation information is generated, including the index value of the newly added object indicated by the operation request, the data of the newly added object, and a timestamp for the object. By recording the newly added object, incremental processing of the data is guaranteed, thereby ensuring the validity of the user operation.

[0055] 302. Store the operation information and timestamp in a preset temporary storage space.

[0056] In this embodiment, after the operation information and timestamp are generated, the operation information and timestamp can be written to the temporary storage space. Optionally, the temporary storage space can include a data table. The data table can include three items: an index value, a numerical value, and a timestamp. Based on these three items, the operation information and timestamp can be written to the data table. Optionally, the temporary storage space can include a data structure. Each data structure can include three data items: an index value, a numerical value, and a timestamp. The data structure can be generated based on the operation information and timestamp, and the data structure can be stored in the temporary storage space.

[0057] In this embodiment, a data file containing an object index, object data, and an object timestamp is generated based on a user's operation request and written to a temporary storage space, thereby recording user operations in the temporary storage space. Furthermore, the timestamp further ensures the timeliness of the data, improving data storage efficiency.

[0058] Figure 4 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 4 As shown, the present disclosure provides a shard processing method based on a distributed storage system. In the process of merging operation information in a temporary storage space into a target shard, it is necessary to ensure effective merging of data by matching the index value and timestamp of the object to avoid data duplication or abnormal data overwriting. The method specifically includes:

[0059] 401. For the current object in the temporary storage space, according to the index value of the current object, search in the target shard for a target object having the same index value as the index value of the current object.

[0060] In this embodiment, the current object to be merged can first be obtained from the temporary storage space. Optionally, the current object can include an index value, data, and a timestamp. Based on the index value of the current object, a target object with the same index value as the current object can be searched in the target shard. Optionally, in the target shard, the target object can also include an index value, data, and a timestamp.

[0061] Optionally, for the objects modified and deleted in step 301 , a target object with the same index value can usually be matched, and then the subsequent merging in step 402 is performed.

[0062] Alternatively, for the newly added object in step 301, it may not be possible to find a corresponding target object. Therefore, for such newly added objects, if no corresponding target object can be found, the current object can be directly written to the target shard. If the newly added object has a target object with the same index as the newly added object, the newly added object is merged according to the timestamp in step 402.

[0063] 402. Merge the current object and the target object according to the timestamp of the current object and the timestamp of the target object.

[0064] In this embodiment, after determining the current object and the target object, the timestamp of the current object and the timestamp of the target object can be obtained respectively. Then, based on the two timestamps, the object corresponding to the latest timestamp can be retained, thereby merging the current object and the target object.

[0065] Optionally, based on the comparison of the timestamps, a specific process of implementing the merge may include the following steps:

[0066] 4021. If it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, the current object is deleted from the temporary storage space.

[0067] In this step, if it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, it means that the current object is no longer the latest data. At this time, the current object can be directly deleted in the temporary storage space to prevent the old data corresponding to the current object from overwriting the new data in the target object. For example, the data corresponding to the index value k1 in the source shard can be v1. The index value of the current object is k1, and the data is v2. The index value of the target object is k1, and the data is v3. When the timestamp of the current object is earlier than or equal to the timestamp of the target object, the data v3 of the target object is retained in the target shard. Among them, data v2 can be the operation data generated by the user performing the modification operation during the synchronization from the source shard to the target shard. Data v3 can be the operation data generated by the modification performed directly in the target shard when the user performs the modification operation after the synchronization from the source shard to the target shard is completed.

[0068] 4022. If it is determined that the timestamp of the current object is later than the timestamp of the target object, merge the data of the current object into the target object and delete the current object from the temporary storage space.

[0069] In this step, if it is determined that the timestamp of the current object is later than the timestamp of the target object, the data in the current object is the latest data. In this case, the data of the current object needs to be merged into the data of the target object to ensure that the target object has the latest data. Optionally, the timestamp of the current object can also be merged into the timestamp of the target object to ensure that the update time of the data remains consistent with the timestamp. After the target object is merged, the current object in the temporary storage space can be deleted to avoid repeated merging after the current object is repeatedly retrieved. Optionally, this merge can specifically include two situations: normal data merging and data deletion corresponding to empty data. For example, in the source shard, the data corresponding to the index value k1 can be v1. The current object has an index value k1 and data v2. The target object has an index value k1 and data v1. If the timestamp of the current object is later than the timestamp of the target object, the data v2 in the current object will be written to the target object. The data v2 can be the operation data generated by the user performing a modification operation during the synchronization process between the source shard and the target shard.

[0070] The two processing methods 4021 and 4022 based on the timestamp comparison results can more accurately merge the current object into the target object, thereby ensuring that each object in the target shard is the latest data and improving the effectiveness of the target shard.

[0071] Optionally, for both modification and deletion operations, since the deletion operation is recorded as a special modification operation in the temporary storage space, it is necessary to process the modification and deletion operations separately in the merging phase. The specific process may include:

[0072] 40221. If the data of the current object is empty, delete the target object from the target shard.

[0073] In this step, if the data of the current object is empty, it means that the current object corresponds to a delete operation. Therefore, when merging the current object into the target shard, the target shard executes the delete operation and deletes the target object.

[0074] 40222. If the data of the current object is not empty, the data of the current object will overwrite the data of the target object.

[0075] In this step, if the data of the current object is not empty, it means that the current object corresponds to a modification operation. Therefore, if the data of the current object is the latest data, the data of the current object can be directly overwritten into the data of the target object of the target shard.

[0076] Through steps 40221 and 40222, both modification and deletion operations are processed during the merge process. This method allows the deletion operation to be recorded as a special modification operation in the temporary storage space, thereby improving the efficiency of recording deletion operations in the temporary storage space and ensuring the accuracy of the deletion operation during the merge process.

[0077] In this embodiment, by matching objects with the same index value in the target shard and the temporary storage space and comparing the timestamps of the two objects, the two objects are merged, thereby ensuring that the data in the target shard is the latest data.

[0078] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 5 As shown, the present disclosure provides a shard processing method based on a distributed storage system, which implements the process of synchronizing the source shard to the target shard and merging the temporary storage space into the target shard. When the user performs a query operation, data query can be implemented based on the combination of the source shard and the temporary storage space or the target shard and the temporary storage space. The method includes:

[0079] 501. In response to a first query instruction, obtain a first query result based on the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space. The first query instruction is received during synchronization of the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

[0080] In this embodiment, during the synchronization of a source shard to a target shard, if the user operation received is a query operation, the operation instruction corresponding to the query operation can be recorded as a first query instruction. This first query instruction can include an index value of the object to be queried. Based on this index value, a search can be performed in both the source shard and the temporary storage space to obtain at least one object. Based on this at least one object, the latest object can be determined by comparing timestamps, and this object can be fed back to the user as the first query result.

[0081] In one implementation, the query process may include: first, searching the source shard for an object corresponding to the index value to obtain a first object. Then, searching the temporary storage space for an object corresponding to the index value. If the object corresponding to the index value does not exist in the temporary storage space, the first object is fed back to the user as the first query result. If the object corresponding to the index value exists in the temporary storage space, the object is recorded as the second object. The timestamps of the first and second objects are compared, and the object with the latest timestamp is fed back to the user as the first query result.

[0082] In another implementation, the query process may include first searching for the object corresponding to the index value in the temporary storage space. If the object exists, the object is fed back to the user as the first query result. If the object does not exist, the source shard is further searched, and the object obtained from the search is fed back to the user as the first query result.

[0083] 502. In response to a second query instruction, obtain a second query result based on the timestamps of the objects in the target shard and the timestamps of the objects in the temporary storage space. The second query instruction is received during the process of merging the operation information in the preset temporary storage space into the target shard, and the first query instruction indicates the index value of the object to be queried.

[0084] In this embodiment, during the process of merging the operation information in the temporary storage space into the target shard, if the user operation received is a query operation, the operation instruction corresponding to the query operation can be recorded as a second query instruction. This second query instruction can include an index value of the object to be queried. Based on this index value, a search can be performed in both the target shard and the temporary storage space to obtain at least one object. Based on this at least one object, the latest object can be determined by comparing timestamps, and this object can be fed back to the user as the second query result.

[0085] The query process may include: first, searching the target shard for the object corresponding to the index value to obtain a first object. Then, searching the temporary storage space for the object corresponding to the index value. If the object corresponding to the index value does not exist in the temporary storage space, the first object is fed back to the user as the second query result. If the object corresponding to the index value exists in the temporary storage space, the object is recorded as the second object. The timestamps of the first and second objects are compared, and the object with the latest timestamp is fed back to the user as the second query result.

[0086] In this embodiment, the latest objects are retrieved by querying the source shard and temporary storage space during synchronization of the source shard with the target shard, and by querying the target shard and temporary storage space during merging operation information in the temporary storage space into the target shard. Because the temporary storage space is only used to record user operations, increasing the query time of the temporary storage space does not take up much time, ensuring efficient response to user query requests, thereby ensuring the efficiency of processing other operations during business processing and user experience.

[0087] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 6 As shown, the present disclosure provides a shard processing method based on a distributed storage system, which implements specific steps for processing when a user operation request is received during the process of merging temporary storage space into a target shard. The method includes:

[0088] 601. During the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the received user processing request indicates modification or addition of an object, the object indicated by the processing request is adjusted in the target shard. The processing request indicates operation on the target shard.

[0089] In this embodiment, during the process of merging the operation information in the preset temporary storage space into the target shard, if a user operation request such as adding an object or modifying an object is received, the object can be processed directly on the target shard. For example, if the operation is to add an object, the added object can be directly added to the target shard. For another example, if the operation is to modify an object, the target object of the modified object in the target shard can be determined based on the index value of the modified object. The data of the modified object can then be written into the data of the target object. At the same time, the timestamp of the modified object can also be recorded in the target object.

[0090] 602. In the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the user's processing request received represents the deletion of an object, the object indicated by the processing request is deleted in the target shard, and the object indicated by the processing request is deleted in the preset temporary storage space.

[0091] In this embodiment, in the process of merging the operation information in the preset temporary storage space into the target shard, if a deletion request from the user is received, then based on the deletion request, the deletion object indicated by the deletion request can be deleted in the target shard. Optionally, in order to avoid the object with the same index value as the deletion object in the temporary storage space from being rewritten to the target shard after the deletion object in the target shard is deleted, before deleting the deletion object in the target shard, a query can be performed in the temporary storage space based on the index value of the deletion object. If there is an object with the same index value as the deletion object in the temporary storage space, the object with the same index value as the deletion object in the temporary storage space is deleted first. Thereafter, the deletion object in the target shard is deleted.

[0092] In this embodiment, by processing the user's operation request during the process of merging the operation information in the preset temporary storage space into the target shard, it is further ensured that the data in the target shard is the latest data, and the correctness of the data merging in the temporary storage space is ensured.

[0093] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, as shown in Figure 7 As shown, the present disclosure provides a shard processing method based on a distributed storage system. If an exception occurs during the synchronization of a source shard to a target shard, or an exception occurs during the merging of a temporary storage space into a target shard, exception handling is required. The method includes:

[0094] 701. If it is determined that the process of synchronizing the source shard to the target shard fails, the operation information in the preset temporary storage space is merged into the source shard.

[0095] In this embodiment, if an exception occurs during the synchronization of the source shard to the target shard, it can be determined that the synchronization process of the source shard to the target shard has failed. At this time, in order to ensure the security of the data, it is necessary to clear the target shard to prevent the abnormally copied data from being retained. At the same time, since the user operation during the copy process has modified the data in the source shard, it is necessary to merge the operation information in the temporary storage space into the source shard to ensure that the user operation will be retained. After the data in the temporary storage space is merged into the source shard, the operation of synchronizing the source shard to the target shard can be executed again in response to the shard synchronization instruction.

[0096] 702. If it is determined that the process of merging the temporary storage space into the target shard fails, re-execute the process of merging the temporary storage space into the target shard.

[0097] In this embodiment, if the process of merging the temporary storage space into the target shard fails after the source shard is synchronized to the target shard, the source shard does not need to be processed again because all the data in the source shard has already been synchronized to the target shard. Furthermore, after the synchronization is complete, the user's operations have already been recorded in the target shard. Therefore, only the process of merging the temporary storage space into the target shard needs to be restarted. This process is used to ensure that the user's operations are not lost during the synchronization process of the source shard to the target shard.

[0098] In this embodiment, by handling exceptions during the process of synchronizing the source shard to the target shard, and handling exceptions after the process of synchronizing the source shard to the target shard, the data security in the shards and the effectiveness of user operations during the shard synchronization process are further guaranteed.

[0099] Figure 8 is a schematic diagram according to the seventh embodiment of the present disclosure, as shown in Figure 8 As shown, the present disclosure provides a shard processing device 800 based on a distributed storage system, comprising:

[0100] The synchronization unit 801 is configured to store operation information indicated by a user's operation request in a preset temporary storage space in response to a shard synchronization instruction while synchronizing the source shard indicated by the shard synchronization instruction to the target shard. The shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates performing an operation on the source shard or the target shard; and the operation information indicates the content of the operation indicated by the operation request.

[0101] The merging unit 802 is configured to merge the operation information in the preset temporary storage space into the target shard.

[0102] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0103] Figure 9 is a schematic diagram according to an eighth embodiment of the present disclosure, as shown in Figure 9 As shown, the present disclosure provides a shard processing device 900 based on a distributed storage system, comprising:

[0104] The synchronization unit 910 is configured to store the operation information indicated by the user's operation request in a preset temporary storage space in response to the shard synchronization instruction while synchronizing the source shard indicated by the shard synchronization instruction to the target shard. The shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates performing an operation on the source shard or the target shard; and the operation information indicates the content of the operation indicated by the operation request.

[0105] The merging unit 920 is configured to merge the operation information in the preset temporary storage space into the target shard.

[0106] Optionally, the synchronization unit 910 includes:

[0107] A generating module 911 is configured to generate operation information and a timestamp of an object corresponding to the operation information according to an operation request;

[0108] The storage module 912 is used to store the operation information and timestamp in a preset temporary storage space.

[0109] Optionally, the generating module 911 includes:

[0110] The first generation submodule 9111 is used to generate operation information including the index value of the object in the shard indicated by the operation request, the data of the object, and the timestamp of the object when the operation request represents the modification of the object on the source shard or the target shard.

[0111] Optionally, the generating module 911 includes:

[0112] The second generation submodule 9112 is used to generate operation information including the index value of the object in the shard indicated by the operation request, the empty data of the object, and the timestamp of the object when the operation request represents the deletion of the object in the source shard or the target shard.

[0113] Optionally, the generating module 911 includes:

[0114] The third generation submodule 9113 is used to generate operation information including the index value of the newly added object indicated by the operation request, the data of the newly added object, and the timestamp of the object when the operation request indicates adding an object in the source shard or the target shard.

[0115] Optionally, the operation information includes an index value of the object and data of the object, and the object has a timestamp; the merging unit 920 includes:

[0116] A search module 921 is configured to search for a target object having the same index value as the current object in the temporary storage space according to the index value of the current object in the target shard;

[0117] The merging module 922 is configured to merge the current object with the target object according to the timestamp of the current object and the timestamp of the target object.

[0118] Optionally, the merging module 922 includes:

[0119] A first merging submodule 9221 is configured to delete the current object from the temporary storage space if it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object;

[0120] The second merging submodule 9222 is configured to merge the data of the current object into the target object and delete the current object from the temporary storage space if it is determined that the timestamp of the current object is later than the timestamp of the target object.

[0121] Optionally, the second merging submodule 9222 specifically includes:

[0122] If the data of the current object is empty, the target object is deleted from the target shard.

[0123] If the data of the current object is not empty, the data of the current object will overwrite the data of the target object.

[0124] Optionally, the operation information includes an index value and data of the object, where the object has a timestamp; further includes:

[0125] The first operation unit 930 is used to obtain a first query result in response to a first query instruction based on the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space; wherein the first query instruction is received in the process of synchronizing the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

[0126] 1 Optionally, the operation information includes the index value and data of the object, and the object has a timestamp; further includes:

[0127] The second operation unit 940 is used to obtain a second query result in response to a second query instruction based on the timestamp of the object in the target shard and the timestamp of the object in the temporary storage space; wherein the second query instruction is received in the process of merging the operation information in the preset temporary storage space into the target shard, and the first query instruction indicates the index value of the object to be queried.

[0128] Optionally, it also includes:

[0129] The first operation unit 930 is further configured to, during the process of merging the operation information in the preset temporary storage space into the target shard, adjust the object indicated by the processing request in the target shard if it is determined that the received user processing request indicates modification or addition of an object; wherein the processing request indicates an operation to be performed on the target shard;

[0130] The second operation unit 940 is also used to delete the object indicated by the processing request in the target shard and delete the object indicated by the processing request in the preset temporary storage space if it is determined that the received user processing request represents the deletion of the object during the process of merging the operation information in the preset temporary storage space into the target shard.

[0131] Optionally, the synchronization unit 910 further includes:

[0132] The creation module 913 is configured to create a preset temporary storage space that matches the size of the source shard according to the size of the source shard.

[0133] Optionally, it also includes:

[0134] The exception handling unit 950 is configured to merge the operation information in the preset temporary storage space into the source shard if it is determined that the synchronization process of the source shard to the target shard fails.

[0135] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0138] Figure 10is a schematic block diagram of an electronic device in which embodiments of the present disclosure may be implemented. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0140] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the sharding processing method based on a distributed storage system. For example, in some embodiments, the sharding processing method based on a distributed storage system can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the sharding processing method based on the distributed storage system described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a sharding processing method based on a distributed storage system in any other appropriate manner (eg, by means of firmware).

[0142] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0148] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0149] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A sharding processing method based on a distributed storage system, comprising: In response to a shard synchronization instruction, during the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard, operation information indicated by the user's operation request is stored in a preset temporary storage space; wherein the shard synchronization instruction instructs synchronization of the source shard to the target shard; the operation request instructs to perform an operation on the source shard or the target shard; the operation information represents the operation content indicated by the operation request; and the temporary storage space is established in the memory area of ​​the source shard; The operation information in the preset temporary storage space is merged into the target shard.

2. The method according to claim 1, wherein storing the operation information indicated by the user's operation request in a preset temporary storage space comprises: Generate the operation information and a timestamp of an object corresponding to the operation information according to the operation request; The operation information and the timestamp are stored in the preset temporary storage space.

3. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request represents modification of an object in the source shard or the target shard, operation information including the index value of the object in the shard indicated by the operation request and the data of the object is generated, and a timestamp of the object is generated.

4. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request indicates deletion of an object in the source shard or the target shard, operation information including the index value of the object in the shard indicated by the operation request and empty data of the object is generated, and a timestamp of the object is generated.

5. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request indicates adding an object to the source shard or the target shard, operation information including an index value of the newly added object indicated by the operation request and data of the newly added object is generated, and a timestamp of the object is generated.

6. The method according to claim 5, wherein the operation information includes an index value of an object and data of the object, and the object has a timestamp; merging the operation information in the preset temporary storage space into the target shard comprises: For the current object in the temporary storage space, according to the index value of the current object, searching in the target shard for a target object having the same index value as the index value of the current object; The current object and the target object are merged according to the timestamp of the current object and the timestamp of the target object.

7. The method according to claim 6, wherein merging the current object with the target object according to the timestamp of the current object and the timestamp of the target object comprises: If it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, deleting the current object from the temporary storage space; If it is determined that the timestamp of the current object is later than the timestamp of the target object, the data of the current object is merged into the target object, and the current object is deleted from the temporary storage space.

8. The method according to claim 7, wherein merging the data of the current object into the target object comprises: If the data of the current object is empty, deleting the target object from the target slice; If the data of the current object is not empty, the data of the target object is overwritten by the data of the current object.

9. The method according to any one of claims 1 to 8, wherein the operation information includes an index value of the object and data of the object, and the object has a timestamp; the method further comprises: In response to a first query instruction, a first query result is obtained based on the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space; wherein the first query instruction is received during the process of synchronizing the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

10. The method according to any one of claims 1 to 8, wherein the operation information includes an index value of the object and data of the object, and the object has a timestamp; the method further comprises: In response to a second query instruction, a second query result is obtained based on the timestamp of the object in the target shard and the timestamp of the object in the temporary storage space; wherein, the second query instruction is received in the process of merging the operation information in the preset temporary storage space into the target shard, and the second query instruction indicates the index value of the object to be queried.

11. The method according to any one of claims 1 to 8, further comprising: In the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the received user processing request indicates modification of an object or addition of an object, adjusting the object indicated by the processing request in the target shard; wherein the processing request indicates operation on the target shard; In the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the processing request received from the user represents the deletion of an object, the object indicated by the processing request is deleted in the target shard, and the object indicated by the processing request is deleted in the preset temporary storage space.

12. The method according to any one of claims 1 to 8, further comprising: According to the size of the source shard, the preset temporary storage space matching the size of the source shard is created.

13. The method according to any one of claims 1 to 8, further comprising: If it is determined that the process of synchronizing the source shard to the target shard fails, the operation information in the preset temporary storage space is merged into the source shard.

14. A shard processing device based on a distributed storage system, comprising: A synchronization unit, configured to, in response to a shard synchronization instruction, store operation information indicated by a user's operation request in a preset temporary storage space during synchronization of a source shard indicated by the shard synchronization instruction to a target shard; wherein the shard synchronization instruction instructs synchronization of the source shard to the target shard; the operation request instructs to perform an operation on the source shard or the target shard; the operation information represents the operation content indicated by the operation request; and the temporary storage space is established in a memory area of ​​the source shard; A merging unit is used to merge the operation information in the preset temporary storage space into the target slice.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

17. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 13.

Images