Methods, apparatus, and devices for processing replication operations based on distributed object storage

Abstract

This disclosure provides a method, apparatus, and device for processing replication operations based on distributed object storage, relating to artificial intelligence, and particularly to the technical fields of cloud computing, cloud storage, and content delivery networks. The specific implementation scheme is as follows: In response to a processing request sent by a terminal device, the metadata of the header object data of the source object indicated by the processing request is read and saved to a preset file; wherein, the processing request is used to instruct a replication operation on the source object; the source object includes header object data and multiple fragment data, the header object data having metadata, the metadata representing the descriptive information of the source object; replication operations are performed on the fragment data; if it is determined that the replication operations on each fragment data are successful, the name of the preset file is set to the name of the target object to obtain the target object. This reduces the amount of data processing, improves the speed of object replication operations, and reduces the consumption of hardware resources.

Description

Technical Field

[0001] This disclosure relates to the technical fields of cloud computing, cloud storage, and content delivery networks in artificial intelligence, and in particular to a method, apparatus, and device for processing replication operations based on distributed object storage. Background Technology

[0002] In distributed object storage, various processing operations are required for objects, often necessitating object copying. During a copy operation, the source object is read, and the read source data is stored in the target bucket to obtain the target object.

[0003] However, for distributed object storage, the object copying operation needs to be performed at a faster speed and the hardware resource consumption needs to be reduced. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and device for processing copy operations based on distributed object storage to improve the speed of object copy operations and reduce the consumption of hardware resources.

[0005] According to a first aspect of this disclosure, a method for processing replication operations based on distributed object storage is provided, comprising:

[0006] In response to a processing request sent by a terminal device, the metadata of the header object data of the source object indicated by the processing request is read and the metadata is saved to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata represents the descriptive information of the source object;

[0007] The fragmented data is copied.

[0008] If it is determined that the copying operation of each of the aforementioned data segments is successful, then the name of the preset file is determined as the name of the target object, so as to obtain the target object.

[0009] According to a second aspect of this disclosure, a replication operation processing apparatus based on distributed object storage is provided, comprising:

[0010] A storage unit is configured to respond to a processing request sent by a terminal device, read the metadata of the header object data of the source object indicated by the processing request, and save the metadata to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata characterizes the descriptive information of the source object;

[0011] The processing unit is used to perform copying operations on the fragmented data;

[0012] The first determining unit is configured to, if it is determined that the copying operation of each of the data segments is successful, determine the name of the preset file as the name of the target object, so as to obtain the target object.

[0013] According to a third aspect of this disclosure, a server is provided, comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect.

[0017] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method provided in the first aspect.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in the first aspect. The electronic device is the aforementioned server.

[0019] According to a sixth aspect of this disclosure, a distributed storage system is provided, comprising: a plurality of servers provided in a third aspect.

[0020] According to the scheme of this disclosure, in response to a processing request sent by a terminal device, the metadata of the header object data of the source object indicated by the processing request is read and saved to a preset file; the fragmented data is copied; if it is determined that the copying operation of each fragmented data is successful, the name of the preset file is set to the name of the target object to obtain the target object. Since the copying operation of each fragmented data of the source object has been successfully processed, and the header object data of the source object has been mapped to the target object, the target object can be obtained, and the copying process of the source object is completed. Furthermore, in this embodiment, only the copying operation of each fragmented data of the source object needs to be processed, and the copying operation of the header object data of the source object does not need to be processed; thereby reducing the amount of data processing, improving the speed of object copying operation, and reducing the occupation of hardware resources.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a scene diagram that can implement the embodiments of this disclosure;

[0024] Figure 2 This is a schematic diagram based on the first embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram according to the second embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram according to the third embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0029] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] In distributed object storage, various processing operations are required for objects, often necessitating object copying. During a copy operation, the source object is read from the source bucket, and then the read source data is stored in the target bucket to obtain the target object.

[0032] The object includes a header object and at least one shard. The header object itself is also a shard.

[0033] Figure 1The scene diagrams can be implemented according to the embodiments of this disclosure. Figure 1 As shown, the distributed object storage involves a distributed object storage system 100, which includes at least one server 101. The server 101 can perform object replication operations.

[0034] In one example, a complete copy of all data from the source object is performed. The source object is read from the source bucket, and then a complete copy of all data from the source object is performed to obtain the target object; the obtained target object is then stored in the target bucket.

[0035] However, the above method requires a complete copy of all the data of the source object, which consumes a lot of resources to complete the data copying operation, thus requiring more bandwidth resources and input / output (IO) operations; and it takes a long time to complete the copying operation.

[0036] In one example, a renaming operation is performed on an object. This involves processing the header data of the source object to rename it. After renaming the source object, the target object is obtained, allowing for copying. This involves operations on both the source and target buckets. When cutting and pasting the source object based on the source bucket, if the processing speed is fast, target header data can be generated. This target header data can point to the source data, thus obtaining the target object. Then, the source header data of the original object is deleted.

[0037] However, the above method involves the cutting and pasting process. If the target header object data has been successfully generated but the source object's header data has not been successfully deleted; or if the source object's header data has been successfully deleted but the target header object data has not been successfully generated, the atomicity and consistency of the object cannot be guaranteed, and the copy operation will fail.

[0038] In one example, for the source object, each fragment of data is copied one by one. During the process, each fragment has a count parameter `refcount`; when a fragment is copied, the count parameter `refcount` for that fragment is incremented by 1. After copying each fragment, the process waits for the result of that fragment's copy before moving on to the next fragment. This process ultimately yields the target object, completing the copy operation of the source object.

[0039] However, the above method requires copying each fragment of the source object one by one. Only after obtaining the copying result of the current fragment will the next fragment be processed, making the overall copying operation time-consuming and slow. Furthermore, since the header data in the source object is also a fragment, the header data and each fragment of the source object need to be copied to obtain the target object. The large amount of data to be processed further reduces the speed of the copying operation and requires significant hardware resources.

[0040] It is evident that performing object copying operations requires improving the speed of the object copying operation and reducing the consumption of hardware resources.

[0041] This disclosure provides a method, apparatus, and device for processing replication operations based on distributed object storage, which can be applied to fields such as cloud computing, cloud storage, and content delivery networks in artificial intelligence, in order to improve the speed of object replication operations and reduce the consumption of hardware resources.

[0042] Figure 2 This is a schematic diagram based on the first embodiment of the present disclosure, as shown below. Figure 2 As shown, the replication operation processing method based on distributed object storage provided in this embodiment includes:

[0043] S201. In response to a processing request sent by a terminal device, read the metadata of the header object data of the source object indicated by the processing request, and save the metadata to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata represents the descriptive information of the source object.

[0044] For example, in this embodiment, the executing entity may be a server, a replication operation processing device or device based on distributed object storage, or other devices or devices that can implement this embodiment. In this embodiment, the executing entity is described as a server.

[0045] The terminal device receives a processing request instructing a copy operation to be performed on the source object. This copy operation refers to the copy operation in distributed object storage.

[0046] In one example, when a user is processing an object in a distributed object storage, and determines that a copy operation of the source object is required, the user inputs a processing request to the terminal device through the user interface; then, the terminal device receives the processing request input by the user, which is used to instruct the copy operation of the source object indicated by the user.

[0047] In another example, when a terminal device determines that a copy operation of the source object is required while processing business in a distributed object storage, the terminal device generates a processing request that instructs the copy operation of the source object corresponding to the business.

[0048] The terminal device then sends the processing request to the server. After receiving the processing request, the server determines the source object indicated by the processing request.

[0049] The source object includes header object data and multiple fragment data. The header object data contains metadata, which represents the descriptive information of the source object. For example, the descriptive information represented by the metadata includes the storage path of each fragment data under the source data, the identifier of each fragment data under the source data, the order relationship between the fragment data under the source data, and so on.

[0050] The header object data in the source object is also a fragmented data.

[0051] After the server identifies the source object indicated by the processing request, it saves the metadata of the source object's header data to a pre-defined file. In this process, it does not need to directly copy the source object's header data (i.e., the source object's source header data fragment), but only pre-extracts the metadata from the source object's header data; this reduces the amount of data processing.

[0052] S202, Perform copy operation on the fragmented data.

[0053] For example, the server performs a copy operation on each fragment of the source data separately, but the server does not need to perform a copy operation on the header object data of the source data.

[0054] S203. If it is determined that the copying operation of each data segment is successful, the name of the preset file is set as the name of the target object to obtain the target object.

[0055] For example, when the server determines that the copying operations of each fragment of the source data have been successfully processed, the server renames the preset file that previously stored the metadata of the source object's header data to the name of the target object, thereby mapping the source object's header data to the target object. Since the copying operations of each fragment of the source object have been successfully processed, and the source object's header data has been mapped to the target object, the target object can be obtained, and the copying process of the source object is complete. Furthermore, in this embodiment, only the copying operations of each fragment of the source object need to be processed; the copying operation of the source object's header data is not required.

[0056] In this embodiment, in response to a processing request sent by the terminal device, the metadata of the header object data of the source object indicated by the processing request is read and saved to a preset file; the fragmented data is copied; if it is determined that the copying operation of each fragmented data is successful, the preset file is named as the target object to obtain the target object. Since the copying operation of each fragmented data of the source object has been successfully processed, and the header object data of the source object has been mapped to the target object, the target object can be obtained, and the copying process of the source object is completed. Furthermore, in this embodiment, only the copying operation of each fragmented data of the source object needs to be processed, and the copying operation of the header object data of the source object is not required; thus, the amount of data processing is reduced, the speed of object copying operation can be improved, and the hardware resource consumption is reduced.

[0057] To help readers gain a deeper understanding of the implementation principles of this disclosure, the following will be discussed in conjunction with... Figure 3 right Figure 2 The illustrated embodiments are further refined.

[0058] Figure 3 This is a schematic diagram based on the second embodiment of the present disclosure, as shown below. Figure 3 As shown, the replication operation processing method based on distributed object storage provided in this embodiment includes:

[0059] S301. Obtain the disk processing capacity information of the servers in the current distributed storage system.

[0060] For example, in this embodiment, the executing entity may be a server, a replication operation processing device or device based on distributed object storage, or other devices or devices that can implement this embodiment. In this embodiment, the executing entity is described as a server.

[0061] An asynchronous input / output (AIO) library is provided, which includes queues. These queues are used to cache operation requests corresponding to fragmented data that require copying. The maximum storage capacity N of the AIO library needs to be configured; where N represents the maximum number of operation requests that the queues in the AIO library can hold. N is a positive integer greater than or equal to 1.

[0062] Because servers in different distributed storage systems have varying disk processing capabilities, each server can obtain its own disk processing capability information. This information includes the following: the number of disks in the server, the maximum data storage space of each disk, the remaining data storage space of each disk, and so on.

[0063] S302. Based on the preset correspondence between disk processing capacity information and maximum storage capacity, determine the maximum storage capacity corresponding to the disk processing capacity information; wherein, the maximum storage capacity represents the maximum number of operation requests that can be stored in the queue of the asynchronous input / output public library.

[0064] For example, a correspondence is pre-set in the server, which represents the correspondence between disk processing capacity information and the maximum number of storage units. Then, the server can determine the maximum number of storage units corresponding to the disk processing capacity information based on the correspondence obtained in step S301.

[0065] Furthermore, based on the server's disk processing capacity information, the maximum number of storage requests for operation requests can be reasonably set for the server's asynchronous input / output public library.

[0066] S303. In response to a processing request sent by a terminal device, read the metadata of the header object data of the source object indicated by the processing request, and save the metadata to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata represents the descriptive information of the source object.

[0067] In one example, the metadata includes at least one of the following: the object name of the fragmented data, the identifier of the fragmented data, the storage path of the fragmented data, and the order relationship between the fragmented data.

[0068] For example, the terminal device receives a processing request that instructs a copy operation to be performed on the source object, where the copy operation refers to the copy operation in a distributed object storage.

[0069] The terminal device then sends the processing request to the server. After receiving the processing request, the server determines the source object indicated by the processing request.

[0070] The source object includes header object data and multiple fragment data. The header object data contains metadata, which represents the descriptive information of the source object. For example, the descriptive information represented by the metadata includes the object name of each fragment of the source data, the storage path of each fragment of the source data, the identifier of each fragment of the source data, the order relationship between the fragments of the source data, and so on.

[0071] The header object data in the source object is also a fragmented data.

[0072] After the server determines the source object indicated by the processing request, the server saves the metadata of the source object's header object data to a preset file.

[0073] S304. Generate operation requests for fragmented data, wherein the operation requests are used to instruct the fragmented data to be copied, and input the operation requests corresponding to each of the N fragmented data into the queue in the asynchronous input / output public library; wherein the queue is used to store the operation requests corresponding to each of the N fragmented data, N is the maximum storage capacity of the current server, the maximum storage capacity represents the maximum number of operation requests that the queue in the asynchronous input / output public library can store, and N is a positive integer greater than or equal to 1.

[0074] For example, the server performs a copy operation on the fragmented data of the source data, but the server does not need to perform a copy operation on the header object data of the source data.

[0075] For each data shard, the server generates an operation request for that shard, and thus obtains the operation request for each shard.

[0076] When processing fragmented data, the server caches operation requests for fragmented data based on a queue in the asynchronous input / output public library, and processes the operation requests corresponding to each fragment based on this queue. First, based on the maximum storage capacity N of the current server, the server inputs the operation requests corresponding to each of the N fragmented data into the queue in the asynchronous input / output public library for caching.

[0077] S305. Repeat the following steps until the preset conditions are met: retrieve operation requests from the queue, and perform copy operations on the shard data corresponding to the retrieved operation requests; if it is determined that the processing of the M retrieved operation requests is successful at the same time, add the operation requests corresponding to the M shard data to the queue; where M is a positive integer greater than or equal to 1 and less than or equal to N.

[0078] In one example, the preset condition is that the copy operation of each shard of data has been completed; or, the preset condition is that feedback information indicating that the copy operation failed has been obtained.

[0079] For example, after step S304, the server retrieves operation requests from the queue in the asynchronous input / output public library. Based on the retrieved operation request for each data segment, the server performs a copy operation on each data segment. If the server determines that the processing of a retrieved operation request is successful, it determines that the copy operation on the data segment corresponding to that operation request is successful. If the server determines that the processing of M retrieved operation requests is successful simultaneously, it needs to add M new operation requests corresponding to each data segment to the queue of the asynchronous input / output public library. This process is repeated until the copy operation processing of all data segments of the source data is completed; or until feedback information indicating that the copy operation processing has failed is obtained, whereby the feedback message indicates that the copy operation processing of the data segment corresponding to an operation request being processed has failed.

[0080] The copying operation of the source object is considered successful only if all copying operations on each fragment of the source data are successfully completed. If the copying operation of even one fragment fails, the copying of the source object is considered to have failed, and there is no need to process the unprocessed fragments of the source object.

[0081] For example, the source object has a header object and A shards, where A is a positive integer greater than or equal to 1. The server generates an operation request for each of the A shards and inputs the operation requests corresponding to each of the N shards into a queue in the asynchronous input / output public library, where N is the maximum number of bytes the server can store. N is a positive integer greater than or equal to 1 and less than or equal to A.

[0082] Then, the server retrieves operation requests from the queue and performs copy operations on each of the N data shards in the queue. If it is determined that the processing of M operation requests corresponding to each of the N data shards is successful simultaneously, the server then extracts the operation requests corresponding to the new N data shards from the A data shards and adds them to the queue. M is a positive integer greater than or equal to 1 and less than or equal to N.

[0083] Repeat the above process until all A shards of the source data have been copied; or until feedback information indicating that the copy operation failed is received.

[0084] By providing an asynchronous input / output (AIO) library and setting up a queue within it to store operation requests for sharded data, this queue can hold a maximum of N operation requests. When it is determined that M operation requests have been processed simultaneously, M operation requests are added to the queue. This eliminates the need to wait for the copying result of each shard before processing the next; subsequent shards can be processed as soon as the copying of multiple shards is complete. This improves the processing speed of the copy operation, reduces latency, and consequently reduces hardware resource consumption. Furthermore, through this process, it is equivalent to changing from synchronous to asynchronous on the object storage gateway side (i.e., the program on the server that receives processing requests for source objects) and to parallel operation on the storage pool service side (i.e., the target disk of the server).

[0085] In one example, each data shard is set with a count parameter, which represents the number of times the allocated data is copied. In step S305, if it is determined that the processing of the M retrieved operation requests is successful simultaneously, the count parameter of the data shard corresponding to the successfully processed operation request is incremented by one.

[0086] The copy operation is used to characterize the establishment of a correspondence between the fragmented data and the target object.

[0087] In one example, in step S305, if it is determined that the processing of the M retrieved operation requests is successful at the same time, the object name of the fragment data corresponding to the successfully processed operation request is added to a preset array.

[0088] For example, step S305 can be implemented in the following manner: Each piece of source data is set with a count parameter refcount, which represents the number of times the allocated data is copied. The header object data of the source data may not have any technical parameters set.

[0089] In this embodiment, the server also needs to include the header data of the source object in the processing of the fragmented data. The header data of the source object also has a count parameter refcount.

[0090] After step S304, the server retrieves operation requests from the queue in the asynchronous input / output public library. Based on the retrieved operation request for each data segment, the server performs a copy operation on each data segment. In one example, when performing a copy operation on each data segment, the server can establish a correspondence between the data segment and the target object; thus, instead of directly copying the data segments, it points the source data segments to the target object by establishing a correspondence, reducing the amount of data processing and speeding up the object copying operation.

[0091] If the server determines that the requested operation was successfully processed, it increments the counter parameter of the fragment data corresponding to the successfully processed operation request, thereby recording the number of times the fragment data was copied. Instead of directly copying the fragment data, the server records the number of times the fragment data of the source data has been copied through the counter parameter, reducing the amount of data processing. At this point, the server determines that the copy operation of the fragment data corresponding to the operation request was successfully processed.

[0092] Furthermore, when the server determines that the processing of the invoked operation request is successful, since the sharded data has object names, it also needs to add the object name of the sharded data corresponding to the successfully processed operation request to a preset array. This is so that the count parameter of the sharded data corresponding to the object name can be decremented by one later based on the object name in the preset array. If the server determines that the processing of the M invoked operation requests is successful simultaneously, it needs to add the operation requests corresponding to the M sharded data to the queue of the asynchronous input / output public library. The above process is repeated until the copying operation of all sharded data of the source data is completed; or until feedback information indicating that the copying operation of the sharded data corresponding to the currently processed operation request has failed.

[0093] For example, the source object has a header object and A shards, where A is a positive integer greater than or equal to 1. The server generates an operation request for each of the A shards and inputs the operation requests corresponding to each of the N shards into a queue in the asynchronous input / output public library, where N is the maximum number of bytes the server can store. N is a positive integer greater than or equal to 1 and less than or equal to A.

[0094] Then, the server retrieves operation requests from the queue and performs copy operations on each of the N data shards in the queue. If it is determined that the processing of M operation requests corresponding to each of the N data shards is successful simultaneously, the refcount parameter of each shard corresponding to the successfully processed operation request is incremented by 1. Furthermore, the server adds the object name of the shard corresponding to the successfully processed operation request to a preset array. Then, the server extracts the operation requests corresponding to the new N data shards from the A data shards and adds them to the queue. M is a positive integer greater than or equal to 1 and less than or equal to N.

[0095] Repeat the above process until all A shards of the source data have been copied; or until feedback information indicating that the copy operation failed is received.

[0096] S306. If it is determined that the copying operation of each data segment is successful, the name of the preset file is set as the name of the target object to obtain the target object.

[0097] For example, after step S305, when the server determines that the copying operations of each fragment of the source data have been successfully processed, the server renames the preset file that previously stored the metadata of the source object's header object data to the name of the target object, thereby mapping the header object data of the source object to the target object. Since the copying operations of each fragment of the source object have been successfully processed, and the header object data of the source object has been mapped to the target object, the target object can be obtained, and the copying process of the source object is completed. Furthermore, in this embodiment, only the copying operations of each fragment of the source object need to be processed; the copying operation of the source object's header object data is not required.

[0098] After the server obtains the target object, it can also delete the header data of the source object.

[0099] S307. If it is determined that the processing of the operation request has failed, then it is determined that the copy operation of the source object has failed; decrement the count parameter of the fragment data corresponding to the object name in the preset array by one.

[0100] For example, after step S305, if any fragment data copying operation fails, it is determined that the source object copying operation has failed, and feedback information indicating that the copying operation has failed will be obtained.

[0101] At this point, since the object names of the shards corresponding to the successfully processed operation requests have already been added to the preset array, the server can retrieve the object names from the preset array. For each retrieved object name, the server decrements the refcount parameter of the corresponding shard data by one. That is, the refcount parameter is decremented by 1. The server can wait for all operation requests retrieved from the queue to complete processing by calling the drain_ios function, and then iterate through all the object names of the successfully processed shard data in the preset array, and then decrement the refcount parameter of each shard data by 1. Thus, since the preset array stores only the object names of successfully processed shard data, the server can decrement the refcount parameter of the shard data corresponding to each object name in the preset array by 1, allowing it to revert to the state before the source data was processed.

[0102] During a delete operation, the server only needs to decrement the count parameter of the data segment corresponding to each object name in the preset array by 1 to complete the deletion operation; this reduces the amount of data processing and the amount of hardware resources used for data storage.

[0103] S308. Generate an error code, wherein the error code is used to indicate that the copying operation on the source object has failed; return the error code to the terminal device.

[0104] In one example, the error code includes operation data for each shard of data in the source object where the copy operation failed. Alternatively, the error code includes operation data for the last shard of data in the source object where the copy operation failed.

[0105] For example, after step S307, when the server determines that the copying operation of the source data has failed, the server can send an error code to the terminal device, thereby informing the terminal device that the copying operation of the source data has failed. This allows the terminal device to initiate a processing request again, or for the terminal device to take other measures to process the source data.

[0106] The error code includes the operation data of the last fragment of data in the source object whose copy operation failed. Thus, the terminal device is informed of the failure of the copy operation on the source data only through the error code, without the need to provide more information to the terminal device, thereby saving data transmission resources.

[0107] Alternatively, the error code can include the operation data for each fragment of data in the source object that failed to be processed, thereby informing the terminal device of the operation data for each fragment of data that failed to be processed; this helps the terminal device determine a solution.

[0108] In one example, after step S306, the following procedure may also be performed:

[0109] In response to a read command, which instructs the reader to read a target object, the system determines the corresponding fragment data for the target object based on a mapping relationship between the target object and fragment data. Then, it reads the fragment data corresponding to the target object from the preset file containing the target object's name, according to the storage path of the fragment data in that file.

[0110] For example, after step S306, the target object is obtained. Since the correspondence between the target object and the fragment data was previously established by incrementing the refcount parameter of each fragment of the source object by 1, the fragment data copying operation of the source object has been completed, and there is no need to directly copy the fragment data. Therefore, when the target object needs to be read, after receiving the read instruction, the server determines the fragment data under the target object to be read based on the correspondence between the target object and the fragment data.

[0111] Because the target object has header object data, which is obtained by renaming the source object's header object data, the metadata in the header object data is stored in a pre-defined file under the target object's name. The metadata includes the object name of each data segment under the source data, the storage path of each data segment under the source data, the identifier of each data segment under the source data, the order relationship between the data segments under the source data, and so on. The information included in the metadata allows the server to read the target object.

[0112] Furthermore, the server can determine the storage path of the target object's fragmented data from the metadata stored in the pre-defined file; based on the storage path of each fragment, the identifier of each fragment, and the order relationship between the fragments, the server reads the allocated data. Thus, the server can read the target object.

[0113] This embodiment, based on the above embodiments, provides an asynchronous input / output public library. A queue is set up in this library to store operation requests for fragmented data. This queue can hold a maximum of N operation requests. When it is determined that M operation requests have been processed simultaneously, M operation requests are added to the queue. It eliminates the need to wait for the copying result of each fragment before processing the next; subsequent fragments can be processed as soon as multiple fragment copies are completed. This improves the processing speed of the copy operation, reduces time consumption, and reduces hardware resource consumption. Furthermore, the counter parameter of the fragment corresponding to the successfully processed operation request is incremented by one to record the number of fragment copyings; thus, the fragment data is not directly copied. In one example, for the header object data of the source object, the header object data fragment is separated from the metadata; it is not necessary to copy the header object data fragment, but only to operate on the refcount parameter of the header object data and rename the metadata of the header object data; and to operate on the refcount parameter of the other fragments of the source pair; this reduces data processing, improves the speed of copying operations, and further reduces the consumption of hardware resources.

[0114] Figure 4 This is a schematic diagram based on the third embodiment of the present disclosure, as shown below. Figure 4 As shown, the replication operation processing device 400 based on distributed object storage provided in this embodiment includes:

[0115] Storage unit 401 is used to respond to a processing request sent by a terminal device, read the metadata of the header object data of the source object indicated by the processing request, and save the metadata to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata represents the descriptive information of the source object.

[0116] Processing unit 402 is used to perform copying operations on fragmented data.

[0117] The first determining unit 403 is used to determine the name of the target object as the name of the preset file if it is determined that the copying operation of each piece of data is successful, so as to obtain the target object.

[0118] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0119] Figure 5 This is a schematic diagram based on the fourth embodiment of the present disclosure, as shown below. Figure 5As shown, the replication operation processing device 500 based on distributed object storage provided in this embodiment includes:

[0120] Storage unit 501 is used to respond to a processing request sent by a terminal device, read the metadata of the header object data of the source object indicated by the processing request, and save the metadata to a preset file; wherein, the processing request is used to instruct a copy operation on the source object; the source object includes header object data and multiple fragment data, the header object data has metadata, and the metadata represents the descriptive information of the source object.

[0121] The processing unit 502 is used to perform copy operations on the fragmented data.

[0122] The first determining unit 503 is used to determine the name of the target object as the name of the preset file if it is determined that the copying operation of each piece of data is successful, so as to obtain the target object.

[0123] In one example, processing unit 502 includes:

[0124] The generation module 5021 is used to generate operation requests for the fragmented data. The operation requests are used to instruct the fragmented data to be copied. The operation requests corresponding to each of the N fragmented data are input into the queue in the asynchronous input / output public library. The queue is used to store the operation requests corresponding to each of the N fragmented data. N is the maximum storage capacity of the current server. The maximum storage capacity represents the maximum number of operation requests that the queue in the asynchronous input / output public library can hold. N is a positive integer greater than or equal to 1.

[0125] Execution module 5022 is used to repeat the following processing module and add module until the preset conditions are met.

[0126] The first processing module 5023 is used to retrieve operation requests from the queue and perform copy operations on the sharded data corresponding to the retrieved operation requests.

[0127] Add module 5024, which is used to add M new operation requests corresponding to each of the M data segments to the queue if it is determined that the processing of the M operation requests is successful at the same time; where M is a positive integer greater than or equal to 1 and less than or equal to N.

[0128] In one example, each data shard is configured with a count parameter, which represents the number of times the allocated data is copied; the processing unit 502 also includes:

[0129] The second processing module 5025 is used to increment the count parameter of the segment data corresponding to the successfully processed operation request if it is determined that the processing of the M operation requests is successful at the same time before adding the operation requests corresponding to the M newly added segment data to the queue.

[0130] In one example, the copy operation is used to characterize the establishment of a correspondence between fragmented data and the target object.

[0131] In one example, the preset condition is that the copy operation of each shard of data has been completed; or, the preset condition is that feedback information indicating that the copy operation failed has been obtained.

[0132] In one example, each data shard is configured with a count parameter, which represents the number of times the allocated data is copied; the processing unit 502 also includes:

[0133] The third processing module 5026 is used to add the object name of the sliced ​​data corresponding to the successfully processed operation request to a preset array if it is determined that the processing of the M operation requests is successful at the same time.

[0134] In one example, the apparatus provided in this embodiment further includes:

[0135] The second determining unit 504 is used to determine that the copying operation of the source object has failed if the processing of the operation request for retrieval fails; and to decrement the count parameter of the fragment data corresponding to the object name in the preset array by one.

[0136] In one example, the apparatus provided in this embodiment further includes:

[0137] The feedback unit 505 is used to generate an error code after the second determining unit 504 determines that the copying operation on the source object has failed, wherein the error code is used to characterize the failure of the copying operation on the source object; and to return the error code to the terminal device.

[0138] In one example, the error code includes operation data for each shard of data in the source object where the copy operation failed to process; or, the error code includes operation data for the last shard of data in the source object where the copy operation failed to process.

[0139] In one example, the apparatus provided in this embodiment further includes:

[0140] The acquisition unit 506 is used to acquire disk processing capability information of the servers in the current distributed storage system.

[0141] The third determining unit 507 is used to determine the maximum number of storage corresponding to the disk processing capacity information based on the preset correspondence between disk processing capacity information and the maximum number of storage; wherein, the maximum number of storage represents the maximum number of operation requests that can be stored in the queue of the asynchronous input / output public library.

[0142] In one example, the apparatus provided in this embodiment further includes:

[0143] The fourth determining unit 508 is used to respond to a read instruction, wherein the read instruction is used to indicate the reading of a target object, and to determine the fragment data corresponding to the target object to be read according to the correspondence relationship, wherein the correspondence relationship is the correspondence relationship between the target object and the fragment data.

[0144] The reading unit 509 is used to read the fragment data corresponding to the target object according to the storage path of the fragment data in the preset file under the name of the target object.

[0145] In one example, the metadata includes at least one of the following: the object name of the fragmented data, the identifier of the fragmented data, the storage path of the fragmented data, and the order relationship between the fragmented data.

[0146] The apparatus in this embodiment can execute the technical solutions in the above method. Its specific implementation process and technical principles are the same, and will not be repeated here.

[0147] Figure 6 This is a schematic diagram based on the fifth embodiment of the present disclosure, as shown below. Figure 6 As shown, the electronic device 600 in this embodiment may include a processor 601 and a memory 602.

[0148] Memory 602 is used to store programs. Memory 602 may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; memory may also include non-volatile memory, such as flash memory. Memory 602 is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc. The computer programs, computer instructions, etc., can be partitioned and stored in one or more memories 602. Furthermore, the computer programs, computer instructions, data, etc., can be accessed by processor 601.

[0149] The aforementioned computer programs and instructions can be stored in one or more partitions of memory 602. Furthermore, the aforementioned computer programs and instructions can be invoked by processor 601.

[0150] The processor 601 is configured to execute the computer program stored in the memory 602 to implement the various steps in the methods described in the above embodiments.

[0151] For details, please refer to the relevant descriptions in the preceding method embodiments.

[0152] The processor 601 and the memory 602 can be independent structures or integrated structures. When the processor 601 and the memory 602 are independent structures, the memory 602 and the processor 601 can be coupled together via bus 603.

[0153] The electronic device in this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principle are the same, and will not be repeated here.

[0154] According to embodiments of this disclosure, this disclosure also provides a server, a readable storage medium, and a computer program product.

[0155] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the scheme provided in any of the above embodiments.

[0156] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0157] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device 700 is a server. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0158] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0159] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0160] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the copy operation processing based on distributed object storage. For example, in some embodiments, the copy operation processing based on distributed object storage can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the copy operation processing based on distributed object storage described above can be performed. Alternatively, in other embodiments, computing unit 701 may be configured to perform copy operation processing based on distributed object storage by any other suitable means (e.g., by means of firmware).

[0161] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0165] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0166] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0167] According to embodiments of this disclosure, this disclosure also provides a distributed storage system, in which the server provided in the above embodiments is provided.

[0168] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing a replication operation based on a distributed object storage, comprising: reading metadata of head object data of a source object indicated by a processing request in response to the processing request, and saving the metadata into a preset file, wherein the processing request is used to indicate a replication operation on the source object, the source object comprises the head object data and a plurality of shard data, the head object data has metadata, the metadata represents description information of the source object, and the metadata comprises at least one of an object name of the shard data, an identification of the shard data, a storage path of the shard data, and an order relationship between the shard data; processing the replication operation on the shard data, wherein the replication operation processing is used to represent a correspondence between the shard data and a target object, and the head object data is not processed by the replication operation; if it is determined that the replication operation on each of the shard data is successful, determining a name of the preset file as a name of the target object to obtain the target object.

2. The method of claim 1, wherein, processing the replication operation on the shard data, comprising: generating an operation request of the shard data, wherein the operation request is used to indicate the replication operation on the shard data, and N operation requests corresponding to N shard data are input into a queue in an asynchronous input / output common library, wherein the queue is used to store the N operation requests corresponding to the N shard data, N is a maximum storage number of a current server, the maximum storage number represents a maximum number of operation requests that can be stored in the queue in the asynchronous input / output common library, and N is a positive integer greater than or equal to 1; repeating the following steps until a preset condition is reached: calling an operation request from the queue, processing the shard data corresponding to the called operation request according to the called operation request, and adding M operation requests corresponding to M shard data into the queue if it is determined that the M operation requests are processed successfully at the same time, wherein M is a positive integer greater than or equal to 1 and less than or equal to N.

3. The method of claim 2, wherein, Each of the shard data is provided with a count parameter, and the count parameter represents a number of times that the shard data is replicated; before the M operation requests corresponding to the M shard data are added into the queue, the method further comprises: if it is determined that the M operation requests are processed successfully at the same time, increasing the count parameter of the shard data corresponding to the processed operation request by one.

4. The method of claim 2 or 3, wherein, The preset condition is that the replication operation on each of the shard data is completed, or the preset condition is that feedback information of a failed replication operation is obtained.

5. The method of claim 2 or 3, wherein, Each of the shard data is provided with a count parameter, and the count parameter represents a number of times that the shard data is replicated; the method further comprises: if it is determined that the M operation requests are processed successfully at the same time, adding an object name of the shard data corresponding to the processed operation request into a preset array. 6.The method of claim 5, further comprising: if it is determined that the processing of the called operation request fails, determining that the replication operation on the source object fails; decreasing the count parameter of the shard data corresponding to the object name in the preset array by one.

7. The method of claim 6, after determining that the copy operation on the source object fails, further comprising: generating an error code, wherein the error code is used to represent that the copy operation on the source object fails; returning the error code to the terminal device.

8. The method of claim 7, wherein, the error code includes operation data of each piece of shard data in the source object for which the copy operation processing fails; or the error code includes operation data of the last piece of shard data in the source object for which the copy operation processing fails.

9. The method of any one of claims 2-3, 6-8, further comprising: obtaining disk processing capability information of a server in a current distributed storage system; determining a maximum storage number corresponding to the disk processing capability information according to a preset correspondence between disk processing capability information and maximum storage numbers, wherein the maximum storage number represents a maximum number of operation requests that a queue in an asynchronous input / output common library can store.

10. The method of any one of claims 1-3, 6-8, further comprising: in response to a read instruction, wherein the read instruction is used to instruct to read the target object, determining shard data corresponding to the target object to be read according to a correspondence, wherein the correspondence is a correspondence between the target object and the shard data; reading the shard data corresponding to the target object to be read according to a storage path of the shard data in a preset file under a name of the target object.

11. A device for processing a copy operation based on distributed object storage, comprising: a storage unit configured to read metadata of head object data of a source object indicated by a processing request in response to the processing request sent by a terminal device and save the metadata into a preset file, wherein the processing request is used to instruct to perform a copy operation on the source object; the source object includes the head object data and a plurality of pieces of shard data, the head object data has metadata representing description information of the source object, and the metadata includes at least one of an object name of the shard data, an identifier of the shard data, a storage path of the shard data, and an order relationship between the pieces of shard data; a processing unit configured to perform copy operation processing on the pieces of shard data, wherein the copy operation processing is used to represent establishing a correspondence between the pieces of shard data and a target object, and the head object data is not subjected to the copy operation processing; a first determination unit configured to, if it is determined that the copy operation processing on each of the pieces of shard data is successful, determine a name of the preset file as a name of the target object to obtain the target object. the processing unit comprises:

12. The apparatus of claim 11, wherein, ​ The generating module is configured to generate operation requests of the piece of data, wherein the operation requests are used to indicate a copy operation processing of the piece of data, and input the operation requests corresponding to the N pieces of data respectively into a queue in an asynchronous input / output common library; the queue is used to store the operation requests corresponding to the N pieces of data respectively, N is a maximum storage number of a current server, the maximum storage number represents a maximum number of the operation requests that can be stored in the queue in the asynchronous input / output common library, and N is a positive integer greater than or equal to 1; The executing module is configured to repeat the processing module and the adding module until a preset condition is reached. The first processing module is configured to retrieve the operation requests from the queue, and perform the copy operation processing on the piece of data corresponding to the operation request according to the retrieved operation request. The adding module is configured to add the operation requests corresponding to the M pieces of data respectively into the queue if it is determined that the processing of the M operation requests is successful simultaneously, wherein M is a positive integer greater than or equal to 1 and less than or equal to N.

13. The apparatus of claim 12, wherein, Each piece of data is provided with a count parameter, and the count parameter represents a number of times of copy of the piece of data. The processing unit further comprises: The second processing module is configured to perform a plus one operation on the count parameter of the piece of data corresponding to the operation request if it is determined that the processing of the M operation requests is successful simultaneously before adding the operation requests corresponding to the M pieces of data respectively into the queue.

14. The apparatus of claim 12 or 13, wherein, The preset condition is that the copy operation processing of each piece of data is completed, or the preset condition is that feedback information of a copy operation processing failure is obtained.

15. The apparatus of claim 12 or 13, wherein, Each piece of data is provided with a count parameter, and the count parameter represents a number of times of copy of the piece of data. The processing unit further comprises: The third processing module is configured to add the object name of the piece of data corresponding to the operation request into a preset array if it is determined that the processing of the M operation requests is successful simultaneously.

16. The apparatus of claim 15, further comprising: The second determining unit is configured to determine that the copy operation of the source object fails if it is determined that the processing of the operation request fails. The count parameter of the piece of data corresponding to the object name in the preset array is decremented by one.

17. The apparatus of claim 16, further comprising: The feedback unit is configured to generate an error code after the second determining unit determines that the copy operation of the source object fails, wherein the error code is used to indicate that the copy operation of the source object fails, and the error code is returned to the terminal device.

18. The apparatus of claim 17, wherein, The error code comprises operation data of each piece of data of the source object for which the copy operation processing fails. Alternatively, the error code comprises operation data of the last piece of data of the source object for which the copy operation processing fails.

19. The apparatus of any of claims 12-13, 16-18, further comprising: The obtaining unit is configured to obtain disk processing capability information of a server in a current distributed storage system. A third determining unit is configured to determine a maximum storage number corresponding to the disk processing capacity information according to a preset correspondence between the disk processing capacity information and the maximum storage number, wherein the maximum storage number represents a maximum number of operation requests that can be stored in a queue in an asynchronous input / output common library.

20. The apparatus of any of claims 11-13, 16-18, further comprising: A fourth determining unit is configured to respond to a read instruction, wherein the read instruction is used to instruct to read the target object, and determine the shard data corresponding to the target object to be read according to a correspondence, wherein the correspondence is a correspondence between the target object and the shard data; A reading unit is configured to read the shard data corresponding to the target object to be read according to a storage path of the shard data in a preset file under a name of the target object.

21. A server, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method of any one of claims 1-10.

24. A distributed storage system comprising: A plurality of servers as claimed in claim 21.

Images