Fragmentation processing method and device based on distributed storage system, equipment and storage medium

Abstract

The invention provides a fragmentation processing method and device based on a distributed storage system, equipment and a storage medium, and relates to the technical field of computers, in particular to the fields of big data, distributed data and distributed storage systems. According to the specific implementation scheme, in response to a fragment synchronization instruction, a source fragment indicated by the fragment synchronization instruction is synchronized to a target fragment indicated by the fragment synchronization instruction. In the synchronization process, if the operation request of the user is obtained, the operation information indicated by the operation request of the user can be written into the preset temporary storage space. After synchronization from the source fragment to the target fragment is completed, the operation information in the temporary storage space can be merged into the target fragment.

Description

Technical Field

[0001] The present disclosure relates to big data, distributed data, and distributed storage systems in computer technology, and in particular to a sharding processing method, apparatus, device, and storage medium based on a distributed storage system. Background Art

[0002] The distributed storage system includes a bucket. Multiple shards can be set in a bucket. Objects can be stored in the shards of a bucket. If the number of shards is not properly planned when creating a data bucket, the efficiency of the bucket may be affected as the number of objects in the bucket increases. Therefore, it is often necessary to perform a resharding operation in a bucket to synchronize a shard to another shard. During the synchronization process, the bucket will receive user operation requests for objects in the shard.

[0003] Furthermore, there is an urgent need for a solution that can promptly process user operation requests during shard synchronization. Summary of the invention

[0004] The present disclosure provides a sharding processing method, apparatus, device and storage medium based on a distributed storage system.

[0005] According to a first aspect of the present disclosure, a shard processing method based on a distributed storage system is provided, comprising:

[0006] In response to the shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard, the operation information indicated by the user's operation request is stored in a preset temporary storage space; wherein the shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates operating the source shard or the target shard; and the operation information represents the operation content indicated by the operation request;

[0007] The operation information in the preset temporary storage space is merged into the target slice.

[0008] According to a second aspect of the present disclosure, a shard processing device based on a distributed storage system is provided, comprising:

[0009] A synchronization unit, configured to store operation information indicated by a user's operation request in a preset temporary storage space in response to a shard synchronization instruction, during the process of synchronizing a source shard indicated by the shard synchronization instruction to a target shard; wherein the shard synchronization instruction indicates synchronizing a source shard to a target shard; the operation request indicates operating a source shard or a target shard; and the operation information represents the operation content indicated by the operation request;

[0010] A merging unit is used to merge the operation information in the preset temporary storage space into the target slice.

[0011] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0012] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method of any one of claims 1-13.

[0013] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0014] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the method described in the first aspect.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0017] Figure 1 is a scene graph that can implement the embodiments of the present disclosure;

[0018] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0022] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0023] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0024] Figure 8 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0025] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0026] Fig.10 is a schematic block diagram of an electronic device that can implement an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] The distributed storage system includes buckets. Multiple shards can be set in a bucket. Objects can be stored in the shards of a bucket. If the number of shards is not properly planned when creating a data bucket, the efficiency of the bucket may be affected as the number of objects in the bucket increases. Therefore, it is often necessary to perform a resharding operation in the bucket to increase the number of shards and thus reduce the size of each shard. However, currently, during the entire stage of the resharding process, all write operations to the source shard will be blocked, resulting in the corresponding bucket being prohibited from writing data. For a bucket containing 100 million objects, the blocking time for the write operation will exceed 1 hour. This is unacceptable to most users.

[0029] Currently, the implementation of reshard operation mainly includes the following three steps:

[0030] First, lock the entire bucket and prohibit write operations on the bucket. Create the corresponding number of target shards according to the number of newly configured shards.

[0031] Second, list processing is performed on the objects in the source shard, and the data in the list in the source shard is written into the corresponding target shard. The list processing can obtain a list of common objects, a list of version numbers, a list of olh objects, etc. in the source shard.

[0032] Third, release the lock of the bucket. If rehard succeeds, delete the source shard. And point the list processing target in the bucket to the target shard. If rehard fails, delete the target shard.

[0033] Based on the above steps, in order to achieve that users can still perform write operations after locking the bucket during the rehard process, an optimization method based on recording logs is currently proposed. This method can synchronize objects on the source shard to the target shard during the rehard process. During the synchronization process, if the user has a write operation, the source shard will record a log. Then, the source shard will continue to block the user's write operation. However, after the synchronization of the source shard data is completed, the objects involved can be copied to the target shard by replaying the log file, thereby completing the entire rehard process. However, the log usually contains a large amount of data. When performing a write operation, it is usually accompanied by a query operation. Other data in the log information is likely to affect the real-time performance during the query process, resulting in blocking of user write operations.

[0034] In view of the above problems, the present disclosure proposes a sharding processing method based on a distributed storage system, such as Figure 1 As shown, it may include a source shard 101, a temporary storage space 102 established in the memory area of ​​the source shard 101, and a target shard 103. A synchronization operation 104 and a merge operation 105 may also be included between the source shard and the target shard. Specifically, when it is determined that the source shard 101 needs to be synchronized to the target shard 103, the synchronization operation 104 is executed, and the temporary storage space 102 is established. During the execution of the synchronization operation 104, the user's operation request will be recorded in the temporary storage space 102. After the execution of the synchronization operation 104 is completed, the merge operation 105 will be executed. The merge operation 105 is used to merge the information recorded in the temporary storage space 102 into the target shard 103. The setting of the temporary storage space can realize the user write operation, thereby further shortening the query efficiency during the user write process and shortening the user blocking time.

[0035] The present invention discloses a shard processing method, apparatus, device and storage medium based on a distributed storage system, which are applied to big data, distributed data and distributed storage systems in the computer field to reduce the blocking time of users in the shard replication process and improve the user experience.

[0036] It should be noted that the data in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 3-Figure 7 right Figure 2 The illustrated embodiment is further refined.

[0039] Figure 2 is a schematic diagram according to the first embodiment of the present disclosure, such as Figure 2 As shown, the present disclosure provides a sharding processing method based on a distributed storage system, the method comprising:

[0040] 201. In response to a shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard, the operation information indicated by the user's operation request is stored in a preset temporary storage space. The shard synchronization instruction indicates that the source shard is synchronized to the target shard. The operation request indicates that an operation is performed on the source shard or the target shard. The operation information represents the operation content indicated by the operation request.

[0041] In this embodiment, after the setting of the number of target shards is completed, multiple shard synchronization instructions can be generated accordingly. The shard synchronization instruction can be used to instruct the synchronization of the source shard to the target shard. Optionally, the shard synchronization instruction can be used to synchronize the objects in the source shard to the target shard. Optionally, the shard synchronization instruction can be used to instruct the synchronization of some or all objects in one or more source shards to one or more target shards.

[0042] In response to the shard synchronization instruction, the source shard indicated by the shard synchronization instruction may be locked. The lock is used to indicate to the user that the source shard cannot currently perform a write operation when the user requests to obtain information in the source shard.

[0043] Optionally, in response to the shard synchronization instruction, a temporary storage space may be created in the storage space where the source shard is located. Optionally, setting the temporary storage space in the memory space of the source shard may facilitate the synchronous release of the source shard and the temporary storage space. Optionally, the size of the temporary storage space may be set according to the size of the source shard. Optionally, the size of the temporary storage space may also be determined based on the historical access information of the source shard. Optionally, setting the size of the temporary storage space according to the size of the source shard may ensure that there is enough space in the temporary storage space, and that the space may be used as little as possible, thereby improving space utilization.

[0044] In response to the shard synchronization instruction, the source shard indicated by the shard synchronization instruction is synchronized to the target shard indicated by the shard synchronization instruction. During the synchronization process, if the user's operation request is obtained, since the source shard has been locked, the source shard will not be able to respond to the operation request to perform a write operation. Therefore, when the operation request contains a write operation, the operation information indicated by the user's operation request can be written into the temporary storage space. Optionally, the operation information may include the operation content in the operation request. For example, when the operation request is a modification operation, the operation content may include the index value of the object to be modified, and the data of the modified object. Optionally, when the user's operation request is a query, the source shard can still perform a query operation. However, since the source shard cannot be written, it is necessary to synchronously query the data in the temporary storage space to ensure that the feedback data is the latest data.

[0045] 202. Merge the operation information in the preset temporary storage space into the target shard.

[0046] In this embodiment, after the synchronization from the source shard to the target shard is completed, the operation information in the temporary storage space can be merged into the target shard to ensure that the data stored in the target shard is the latest data at the current moment. Optionally, in the merging process, the operation information in the temporary storage space can be obtained one by one, and the operation information can be merged into the target shard one by one.

[0047] In this embodiment, by setting a temporary storage space, in the process of synchronizing the source shard to the target shard, the operation information of the user operation request that cannot be written due to the source shard lock can be written to the temporary storage space, thereby ensuring the processing of the user's operation request in the process of synchronizing the source shard to the target shard, avoiding the blocking of the user operation, and improving the user experience. Furthermore, the normal implementation of the user operation can further ensure the user's business execution efficiency.

[0048] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure, such as Figure 3 As shown, the present disclosure provides a sharding processing method based on a distributed storage system. In the specific process of storing the operation information in the user's operation request into a preset temporary storage space, the method includes:

[0049] 301. Generate operation information and a timestamp of an object corresponding to the operation information according to the operation request.

[0050] In this embodiment, when the operation request is obtained, corresponding operation information can be generated in response to the operation request. The operation information can specifically include the index value of the object to be operated, and the data of the object to be operated after the operation. In addition, in the process of generating the operation information, a timestamp of the object corresponding to the operation information can also be generated. Optionally, the operation can include operations such as adding, deleting, and modifying that require performing write operations in the source shard.

[0051] Specifically, according to different operation requests such as adding, deleting, and modifying, the timestamps of the generated operation information and the objects corresponding to the operation information may be as follows:

[0052] First, if the operation request represents the modification of an object in the source shard or the target shard, then the operation information including the index value of the object in the shard indicated by the operation request and the data of the object is generated, and the timestamp of the object is generated. By generating the operation information including the index value and data of the modified object and the timestamp for the modification operation, the record of the modification can be better realized, and the validity of the data during the update process can be guaranteed through the timestamp.

[0053] Second, if the operation request indicates the deletion of an object in the source shard or the target shard, the operation information including the index value of the object in the shard indicated by the operation request and the empty data of the object is generated, and the timestamp of the object is generated. By setting the data to empty data, a special record of the deletion operation is realized, thereby ensuring effective deletion during the data merging process.

[0054] Third, if the operation request indicates adding an object to the source shard or the target shard, then the operation information including the index value of the newly added object indicated by the operation request, the data of the newly added object, and the timestamp of the object are generated. By recording the newly added object, the incremental processing of the data is guaranteed, thereby ensuring the validity of the user operation.

[0055] 302. Store the operation information and timestamp in a preset temporary storage space.

[0056] In this embodiment, after the generation of the operation information and the timestamp is completed, the operation information and the timestamp can be written into the temporary storage space. Optionally, the temporary storage space may include a data table. The data table may include three items: index value, value, and timestamp. Based on the three items, the operation information and the timestamp can be written into the data table. Optionally, the temporary storage space may include a data structure. Each data structure may include three data items: index value, value, and timestamp. The data structure can be generated according to the operation information and the timestamp, and the data structure can be stored in the temporary storage space.

[0057] In this embodiment, by generating an index value of an object, object data, and an object timestamp according to a user's operation request and writing them into a temporary storage space, the user operation is recorded in the temporary storage space. Furthermore, the timestamp further ensures the timeliness of the data and improves the storage effect of the data.

[0058] Figure 4 is a schematic diagram according to the third embodiment of the present disclosure, Figure 4 As shown, the present disclosure provides a shard processing method based on a distributed storage system. In the process of merging the operation information in the temporary storage space into the target shard, it is necessary to ensure the effective merging of data by matching the index value and timestamp of the object to avoid data duplication or abnormal data overwriting. The method specifically includes:

[0059] 401. For the current object in the temporary storage space, according to the index value of the current object, a target object having the same index value as the index value of the current object is searched in the target shard.

[0060] In this embodiment, the current object to be merged can first be obtained from the temporary storage space. Optionally, the current object can include an index value, data, and a timestamp. According to the index value of the current object, a target object having the same index value as the current object can be searched in the target slice. Optionally, in the target slice, the target object can also include an index value, data, and a timestamp.

[0061] Optionally, for the objects modified and deleted in 301 , a target object with the same index value can usually be matched, and then the subsequent merging in step 402 is performed.

[0062] Optionally, for the newly added object in 301, it may be that the corresponding target object cannot be matched. Therefore, for such newly added objects, when the corresponding target object cannot be matched, the current object can be directly written into the target shard. If the newly added object has a target object with the same index as the newly added object, the newly added object is merged according to the timestamp in step 402.

[0063] 402. According to the timestamp of the current object and the timestamp of the target object, merge the current object and the target object.

[0064] In this embodiment, after determining the current object and the target object, the timestamp of the current object and the timestamp of the target object can be obtained respectively. Then, based on the two timestamps, the object corresponding to the latest timestamp can be retained, thereby merging the current object and the target object.

[0065] Optionally, based on the comparison of the timestamps, the specific process of implementing the merging may include the following steps:

[0066] 4021. If it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, the current object is deleted from the temporary storage space.

[0067] In this step, if it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, it means that the current object is no longer the latest data. At this time, the current object can be directly deleted in the temporary storage space to avoid the old data corresponding to the current object from overwriting the new data in the target object. For example, the data corresponding to the index value k1 in the source shard can be v1. The index value of the current object is k1, and the data is v2. The index value of the target object is k1, and the data is v3. When the timestamp of the current object is earlier than or equal to the timestamp of the target object, the data v3 of the target object is retained in the target shard. Among them, data v2 can be the operation data generated by the user performing the modification operation during the synchronization from the source shard to the target shard. Data v3 can be the operation data generated by the modification directly performed in the target shard when the user performs the modification operation after the synchronization from the source shard to the target shard is completed.

[0068] 4022. If it is determined that the timestamp of the current object is later than the timestamp of the target object, merge the data of the current object into the target object, and delete the current object from the temporary storage space.

[0069] In this step, if it is determined that the timestamp of the current object is later than the timestamp of the target object, it means that the data in the current object is the latest data. At this time, the data of the current object needs to be merged into the data of the target object so that the target object has the latest data. Optionally, the timestamp of the current object can also be merged into the timestamp of the target object so that the update time of the data remains corresponding to the timestamp. After the merging of the target object is completed, the current object in the temporary storage space can be deleted, so as to avoid repeated merging after the current object is repeatedly acquired. Optionally, the merging can specifically include two situations: normal data merging and data deletion corresponding to empty data. For example, the data corresponding to the index value k1 in the source shard can be v1. The index value of the current object is k1, and the data is v2. The index value of the target object is k1, and the data is v1. In the case where the timestamp of the current object is later than the timestamp of the target object, the data v2 in the current object will be written into the target object. Among them, the data v2 can be the operation data generated by the user performing the modification operation during the synchronization of the source shard to the target shard.

[0070] Through the two processing methods 4021 and 4022 based on the timestamp comparison results, the merging of the current object into the target object can be more accurately achieved, thereby ensuring that each object in the target shard is the latest data and improving the effectiveness of the target shard.

[0071] Optionally, for the modification and deletion operations, since the deletion operation is recorded as a special modification operation in the temporary storage space, it is necessary to process the modification and deletion operations separately in the merging stage. The specific process may include:

[0072] 40221. If the data of the current object is empty, delete the target object from the target shard.

[0073] In this step, if the data of the current object is empty, it means that the current object corresponds to a deletion operation. Therefore, in the process of merging the current object into the target shard, the target shard executes the deletion operation and deletes the target object.

[0074] 40222. If the data of the current object is not empty, the data of the current object will overwrite the data of the target object.

[0075] In this step, if the data of the current object is not empty, it means that the current object corresponds to a modification operation. Therefore, if the data of the current object is the latest data, the data of the current object can be directly overwritten to the data of the target object of the target slice.

[0076] Through the above steps 40221 and 40222, the processing of the two operations of modification and deletion in the merging process is realized. This method allows the deletion operation to be recorded as a special modification operation in the temporary storage space, thereby improving the recording efficiency of the deletion operation in the temporary storage space and ensuring the execution accuracy of the deletion operation in the merging process.

[0077] In this embodiment, by matching the object with the same index value in the target shard and the temporary storage space and comparing the timestamps of the two objects, the two objects are merged, thereby ensuring that the data in the target shard is the latest data.

[0078] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure, Figure 5 As shown, the present disclosure provides a shard processing method based on a distributed storage system, which realizes that in the process of synchronizing the source shard to the target shard and merging the temporary storage space into the target shard, when the user operates a query operation, data query can be realized according to the combination of the source shard and the temporary storage space or the target shard and the temporary storage space. The method includes:

[0079] 501. In response to a first query instruction, a first query result is obtained according to the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space. The first query instruction is received during the process of synchronizing the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

[0080] In this embodiment, in the process of synchronizing the source shard to the target shard, if the received user operation is a query operation, the operation instruction corresponding to the query operation can be recorded as a first query instruction. The first query instruction may include an index value of the object to be queried. Based on the index value, the source shard and the temporary storage space can be searched respectively to obtain at least one object. Based on the at least one object, the latest object can be further determined by comparing the timestamps, so that the object is fed back to the user as the first query result.

[0081] In one implementation, the query process may include: first, querying the corresponding object according to the index value in the source shard to obtain the first object. Then, querying the corresponding object according to the index value in the temporary storage space. If the object corresponding to the index value does not exist in the temporary storage space, the first object is fed back to the user as the first query result. If the object corresponding to the index value exists in the temporary storage space, the object is recorded as the second object. The timestamps of the first object and the second object are compared, and the object with the latest timestamp is fed back to the user as the first query result.

[0082] In another implementation, the query process may include: first querying the corresponding object according to the index value in the temporary storage space. If the object exists, the object is fed back to the user as the first query result. If the object does not exist, the source shard is continued to be queried, and the object obtained by the query is fed back to the user as the first query result.

[0083] 502. In response to the second query instruction, a second query result is obtained according to the timestamp of the object in the target shard and the timestamp of the object in the temporary storage space. The second query instruction is received during the process of merging the operation information in the preset temporary storage space into the target shard, and the first query instruction indicates the index value of the object to be queried.

[0084] In this embodiment, in the process of merging the operation information in the temporary storage space into the target shard, if the received user operation is a query operation, the operation instruction corresponding to the query operation can be recorded as a second query instruction. The second query instruction may include an index value of the object to be queried. Based on the index value, the target shard and the temporary storage space can be searched respectively to obtain at least one object. Based on the at least one object, the latest object can be further determined by comparing the timestamps, so that the object is fed back to the user as the second query result.

[0085] The query process may include: first, querying the corresponding object according to the index value in the target shard to obtain the first object. Then, querying the corresponding object according to the index value in the temporary storage space. If the object corresponding to the index value does not exist in the temporary storage space, the first object is fed back to the user as the first query result. If the object corresponding to the index value exists in the temporary storage space, the object is recorded as the second object. The timestamps of the first object and the second object are compared, and the object with the latest timestamp is fed back to the user as the second query result.

[0086] In this embodiment, the latest object is obtained by querying the source shard and the temporary storage space in the process of synchronizing the source shard to the target shard, and querying the target shard and the temporary storage space in the process of merging the operation information in the temporary storage space into the target shard. Since the temporary storage space is only used to record the user's operation, adding the query of the temporary storage space does not take up much time, which ensures the response efficiency of the user's query request, and then ensures the processing efficiency of other operations of the user in the business processing process, and ensures the user experience.

[0087] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 6 As shown, the present disclosure provides a shard processing method based on a distributed storage system, which implements specific steps of processing when receiving a user's operation request during the process of merging a temporary storage space into a target shard. The method includes:

[0088] 601. In the process of merging the operation information in the preset temporary storage space into the target slice, if it is determined that the received user processing request represents modifying an object or adding an object, the object indicated by the processing request is adjusted in the target slice. The processing request indicates to operate on the target slice.

[0089] In this embodiment, in the process of merging the operation information in the preset temporary storage space into the target slice, if a user's operation request such as adding an object or modifying an object is received, the object can be directly processed on the target slice. For example, when the operation is to add an object, the newly added object can be directly added to the target slice. For another example, when the operation is to modify an object, the target object of the modified object in the target slice can be determined based on the index value of the modified object. The data of the modified object is written into the data of the target object. At the same time, the timestamp of the modified object can also be recorded in the target object.

[0090] 602. In the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the received user processing request represents the deletion of an object, the object indicated by the processing request is deleted in the target shard, and the object indicated by the processing request is deleted in the preset temporary storage space.

[0091] In this embodiment, in the process of merging the operation information in the preset temporary storage space into the target shard, if a deletion request from the user is received, then based on the deletion request, the deletion object indicated by the deletion request can be deleted in the target shard. Optionally, in order to avoid the object with the same index value as the deletion object in the temporary storage space from being rewritten into the target shard after the deletion object in the target shard is deleted, before deleting the deletion object in the target shard, a query can be performed in the temporary storage space according to the index value of the deletion object. If there is an object with the same index value as the deletion object in the temporary storage space, the object with the same index value as the deletion object in the temporary storage space is deleted first. Thereafter, the deletion object in the target shard is deleted.

[0092] In this embodiment, by processing the user's operation request during the process of merging the operation information in the preset temporary storage space into the target shard, it is further ensured that the data in the target shard is the latest data, and the correctness of the data merging in the temporary storage space is ensured.

[0093] Figure 7 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 7 As shown, the present disclosure provides a shard processing method based on a distributed storage system. If an exception occurs during the process of synchronizing a source shard to a target shard, or an exception occurs during the process of merging a temporary storage space to a target shard, exception processing is required. The method includes:

[0094] 701. If it is determined that the process of synchronizing the source shard to the target shard fails, the operation information in the preset temporary storage space is merged into the source shard.

[0095] In this embodiment, if an exception occurs during the process of synchronizing the source shard to the target shard, it can be determined that the process of synchronizing the source shard to the target shard has failed. At this time, in order to ensure the security of the data, the target shard needs to be cleared to avoid the retention of abnormally copied data. At the same time, since the user operation during the copying process has modified the data in the source shard, it is necessary to merge the operation information in the temporary storage space into the source shard to ensure that the user's operation will be retained. After the merger of the data in the temporary storage space to the source shard is completed, the operation of synchronizing the source shard to the target shard can be executed again in response to the shard synchronization instruction.

[0096] 702. If it is determined that the process of merging the temporary storage space into the target shard fails, re-execute the process of merging the temporary storage space into the target shard.

[0097] In this embodiment, if the process of merging the temporary storage space to the target shard fails after the process of synchronizing the source shard to the target shard is completed, then since the data in the source shard has been synchronized to the target shard, it is no longer necessary to process the source shard, and after the synchronization is completed, the user's operation has been recorded in the target shard, so it is only necessary to restart the process of merging the temporary storage space to the target shard. This process is used to ensure that the user's operation will not be lost during the process of synchronizing the source shard to the target shard.

[0098] In this embodiment, by handling exceptions during the process of synchronizing the source shard to the target shard and after the process of synchronizing the source shard to the target shard, the data security in the shard and the effectiveness of user operations during the shard synchronization process are further guaranteed.

[0099] Figure 8 is a schematic diagram according to the seventh embodiment of the present disclosure, Figure 8 As shown, the present disclosure provides a shard processing device 800 based on a distributed storage system, comprising:

[0100] The synchronization unit 801 is used to store the operation information indicated by the user's operation request in a preset temporary storage space in response to the shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard; wherein the shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates operating the source shard or the target shard; and the operation information represents the operation content indicated by the operation request;

[0101] The merging unit 802 is used to merge the operation information in the preset temporary storage space into the target slice.

[0102] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0103] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure, Fig. 9 As shown, the present disclosure provides a shard processing device 900 based on a distributed storage system, comprising:

[0104] The synchronization unit 910 is used to store the operation information indicated by the user's operation request in a preset temporary storage space in response to the shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard; wherein the shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates operating the source shard or the target shard; and the operation information represents the operation content indicated by the operation request;

[0105] The merging unit 920 is used to merge the operation information in the preset temporary storage space into the target slice.

[0106] Optionally, the synchronization unit 910 includes:

[0107] A generating module 911, used to generate operation information and a timestamp of an object corresponding to the operation information according to an operation request;

[0108] The storage module 912 is used to store the operation information and timestamp in a preset temporary storage space.

[0109] Optionally, the generating module 911 includes:

[0110] The first generation submodule 9111 is used to generate operation information including the index value of the object in the shard indicated by the operation request, the data of the object, and the timestamp of the object when the operation request represents the modification of the object to the source shard or the target shard.

[0111] Optionally, the generating module 911 includes:

[0112] The second generation submodule 9112 is used to generate operation information including the index value of the object in the shard indicated by the operation request, the empty data of the object, and the timestamp of the object when the operation request represents the deletion of the object in the source shard or the target shard.

[0113] Optionally, the generating module 911 includes:

[0114] The third generation submodule 9113 is used to generate operation information including the index value of the newly added object indicated by the operation request, the data of the newly added object, and the timestamp of the object when the operation request indicates adding an object in the source shard or the target shard.

[0115] Optionally, the operation information includes an index value of the object and data of the object, and the object has a timestamp; the merging unit 920 includes:

[0116] A search module 921 is used to search for a target object having the same index value as the index value of the current object in the target shard according to the index value of the current object in the temporary storage space;

[0117] The merging module 922 is used to merge the current object and the target object according to the timestamp of the current object and the timestamp of the target object.

[0118] Optionally, the merging module 922 includes:

[0119] A first merging submodule 9221, configured to delete the current object from the temporary storage space if it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object;

[0120] The second merging submodule 9222 is configured to merge the data of the current object into the target object and delete the current object from the temporary storage space if it is determined that the timestamp of the current object is later than the timestamp of the target object.

[0121] Optionally, the second merging submodule 9222 specifically includes:

[0122] If the data of the current object is empty, the target object is deleted from the target shard.

[0123] If the data of the current object is not empty, the data of the current object will overwrite the data of the target object.

[0124] Optionally, the operation information includes an index value of the object and data of the object, and the object has a timestamp; further includes:

[0125] The first operation unit 930 is used to obtain a first query result in response to a first query instruction based on the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space; wherein the first query instruction is received in the process of synchronizing the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

[0126] 1 Optionally, the operation information includes an index value of the object and data of the object, and the object has a timestamp; and also includes:

[0127] The second operation unit 940 is used to obtain a second query result in response to a second query instruction based on the timestamp of the object in the target shard and the timestamp of the object in the temporary storage space; wherein the second query instruction is received in the process of merging the operation information in the preset temporary storage space into the target shard, and the first query instruction indicates the index value of the object to be queried.

[0128] Optionally, it also includes:

[0129] The first operation unit 930 is further configured to adjust the object indicated by the processing request in the target slice if it is determined that the received processing request from the user represents modification of the object or addition of the object during the process of merging the operation information in the preset temporary storage space into the target slice; wherein the processing request indicates operation on the target slice;

[0130] The second operation unit 940 is also used to delete the object indicated by the processing request in the target shard and delete the object indicated by the processing request in the preset temporary storage space if it is determined that the received user processing request represents the deletion of the object during the process of merging the operation information in the preset temporary storage space into the target shard.

[0131] Optionally, the synchronization unit 910 further includes:

[0132] The creation module 913 is used to create a preset temporary storage space matching the size of the source shard according to the size of the source shard.

[0133] Optionally, it also includes:

[0134] The exception handling unit 950 is used to merge the operation information in the preset temporary storage space into the source shard when it is determined that the process of synchronizing the source shard to the target shard fails.

[0135] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0138] Fig.10Schematic block diagram of an electronic device that can implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0140] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a sharding processing method based on a distributed storage system. For example, in some embodiments, the sharding processing method based on a distributed storage system may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the sharding processing method based on the distributed storage system described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a sharding processing method based on a distributed storage system in any other appropriate manner (eg, by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A sharding processing method based on a distributed storage system, comprising: In response to the shard synchronization instruction, in the process of synchronizing the source shard indicated by the shard synchronization instruction to the target shard, the operation information indicated by the user's operation request is stored in a preset temporary storage space; wherein the shard synchronization instruction indicates synchronizing the source shard to the target shard; the operation request indicates operating the source shard or the target shard; and the operation information represents the operation content indicated by the operation request; The operation information in the preset temporary storage space is merged into the target slice.

2. The method according to claim 1, wherein storing the operation information indicated by the user's operation request in a preset temporary storage space comprises: Generate the operation information and a timestamp of an object corresponding to the operation information according to the operation request; The operation information and the timestamp are stored in the preset temporary storage space.

3. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request represents modification of an object in a source shard or a target shard, operation information including an index value of the object in the shard indicated by the operation request and data of the object is generated, and a timestamp of the object is generated.

4. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request represents deletion of an object in the source shard or the target shard, operation information including the index value of the object in the shard indicated by the operation request and empty data of the object is generated, and a timestamp of the object is generated.

5. The method according to claim 2, wherein generating the operation information and the timestamp of the object corresponding to the operation information according to the operation request comprises: If the operation request indicates adding an object to the source shard or the target shard, operation information including an index value of the newly added object indicated by the operation request and data of the newly added object is generated, and a timestamp of the object is generated.

6. According to the method according to any one of claims 1 to 5, the operation information includes an index value of the object and data of the object, and the object has a timestamp; merging the operation information in the preset temporary storage space into the target shard comprises: For the current object in the temporary storage space, according to the index value of the current object, searching in the target shard for a target object having the same index value as the index value of the current object; The current object and the target object are merged according to the timestamp of the current object and the timestamp of the target object.

7. The method according to claim 6, wherein the merging of the current object and the target object according to the timestamp of the current object and the timestamp of the target object comprises: If it is determined that the timestamp of the current object is earlier than or equal to the timestamp of the target object, deleting the current object from the temporary storage space; If it is determined that the timestamp of the current object is later than the timestamp of the target object, the data of the current object is merged into the target object, and the current object is deleted from the temporary storage space.

8. The method according to claim 7, wherein merging the data of the current object into the target object comprises: If the data of the current object is empty, the target object is deleted from the target slice. If the data of the current object is not empty, the data of the target object is overwritten by the data of the current object.

9. The method according to any one of claims 1 to 8, wherein the operation information includes an index value of the object and data of the object, and the object has a timestamp; the method further comprises: In response to a first query instruction, a first query result is obtained according to the timestamp of the object in the source shard and the timestamp of the object in the temporary storage space; wherein the first query instruction is received during the process of synchronizing the source shard to the target shard, and the first query instruction indicates the index value of the object to be queried.

10. The method according to any one of claims 1 to 8, wherein the operation information includes an index value of the object and data of the object, and the object has a timestamp; the method further comprises: In response to a second query instruction, a second query result is obtained based on the timestamp of the object in the target shard and the timestamp of the object in the temporary storage space; wherein the second query instruction is received in the process of merging the operation information in the preset temporary storage space into the target shard, and the second query instruction indicates the index value of the object to be queried.

11. The method according to any one of claims 1 to 10, further comprising: In the process of merging the operation information in the preset temporary storage space into the target slice, if it is determined that the received user processing request represents modification of an object or addition of an object, the object indicated by the processing request is adjusted in the target slice; wherein the processing request indicates operation on the target slice; In the process of merging the operation information in the preset temporary storage space into the target shard, if it is determined that the processing request received from the user represents the deletion of an object, the object indicated by the processing request is deleted in the target shard, and the object indicated by the processing request is deleted in the preset temporary storage space.

12. The method according to any one of claims 1 to 11, further comprising: According to the size of the source shard, the preset temporary storage space matching the size of the source shard is created.

13. The method according to any one of claims 1 to 12, further comprising: If it is determined that the process of synchronizing the source shard to the target shard fails, the operation information in the preset temporary storage space is merged into the source shard.

14. A shard processing device based on a distributed storage system, comprising: A synchronization unit, configured to store operation information indicated by a user's operation request in a preset temporary storage space in response to a shard synchronization instruction, during the process of synchronizing a source shard indicated by the shard synchronization instruction to a target shard; wherein the shard synchronization instruction indicates synchronizing a source shard to a target shard; the operation request indicates operating a source shard or a target shard; and the operation information represents the operation content indicated by the operation request; A merging unit is used to merge the operation information in the preset temporary storage space into the target slice.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

17. A computer program product, comprising a computer program, which implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.

Images