Object storage system, data processing method and related device

Abstract

Provided in the present application is an object storage system. The object storage system comprises a service layer, a persistence layer and a metadata layer, wherein the service layer is used for receiving a batch upload request which is sent by a client and indicates a plurality of first objects to be uploaded, and is used for sending data of the plurality of first objects to the persistence layer, generating metadata of the plurality of first objects, and sending the metadata of the plurality of first objects to the metadata layer; the persistence layer is used for storing the data of the plurality of first objects; and the metadata layer is used for storing the metadata of the plurality of first objects in the persistence layer, and sending first indication information to the service layer when it is determined that the metadata of at least one first object has failed to be stored in the persistence layer, with the first indication information being used for indicating that garbage collection is to be performed on the plurality of first objects. In this way, in an object storage system, by means of ensuring that metadata of a plurality of objects is atomically stored in a persistence layer, the atomicity of a multi-object batch operation is achieved.

Description

Object storage system, data processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 28, 2023, with application number 202311092359.1 and application name “A method, device and other equipment for object operation”, and claims priority to the Chinese patent application filed with the China Patent Office on December 29, 2023, with application number 202311865861.1 and application name “An object storage system, data processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to an object storage system, a data processing method, and related equipment. Background Art

[0003] Object Storage Service (OBS) is an object-based storage service that provides massive, secure, low-cost, and highly reliable unstructured storage services. It eliminates capacity limitations and offers multiple storage types to meet the demands of various business scenarios.

[0004] Atomicity guarantees mean that one or more operations in a transaction will either all succeed or all fail, with no intermediate states. In object storage, a batch operation may contain multiple operations for uploading or deleting objects. Atomicity guarantees that all of these operations will either all succeed or all fail.

[0005] However, in object storage, there is no atomicity guarantee for multiple object uploads or deletions. This lack of atomicity can lead to inconsistent user experiences, such as a user successfully accessing metadata but the corresponding data not being uploaded successfully, resulting in data access failure.

[0006] Summary of the Invention

[0007] This application provides an object storage system for achieving atomicity in batch operations on multiple objects by ensuring the atomic storage of metadata for multiple objects in a persistence layer. Furthermore, this application provides a data processing method, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0008] In a first aspect, the present application provides an object storage system, which is used to provide object storage services through at least one bucket. The object storage system includes a service layer, a persistence layer, and a metadata layer, wherein the service layer is used to receive a batch upload request sent by a client, the batch upload request indicating multiple first objects to be uploaded; and sending the data of the multiple first objects to the persistence layer, and generating metadata of the multiple first objects, and sending the metadata of the multiple first objects to the metadata layer; the persistence layer is used to store the data of the multiple first objects; the metadata layer is used to store the metadata of the multiple first objects to the persistence layer, and, upon determining that the metadata of at least one first object fails to be stored in the persistence layer, send a first indication information to the service layer, the first indication information being used to instruct garbage collection of the multiple first objects.

[0009] In this embodiment, after receiving a batch upload request, the service layer first stores the data of the multiple first objects indicated by the batch upload request in the persistence layer. The metadata layer then atomically stores the metadata of the multiple first objects in the persistence layer. Only when the metadata of each of the multiple first objects is successfully stored in the persistence layer does the service layer return a message to the client indicating that the batch upload of the objects has been successful. If the metadata of at least one of the multiple first objects fails to be stored in the persistence layer, the service layer instructs the data of the multiple first objects to be garbage collected, i.e., falls back to the state where the multiple first objects have not been uploaded. In this way, based on the separate writing of data and metadata to the persistence layer in the object storage system, the atomicity of batch uploading of multiple objects is achieved by ensuring that the metadata of multiple objects is atomically stored in the persistence layer.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the service layer is also used to determine the partition keys of multiple first objects, and the partition keys are used to indicate the partitions of the metadata of the multiple first objects; the metadata layer is specifically used to store the metadata of the multiple first objects to the persistence layer according to the partition keys of the multiple first objects.

[0011] In this implementation, the metadata layer needs to determine the partition keys of the multiple first objects before atomically storing the metadata of the multiple first objects in the persistence layer. After determining the partition keys of the multiple first objects and obtaining the partitions of the metadata of the multiple first objects, the metadata of the multiple first objects is stored in the persistence layer using different atomicity strategies to ensure the atomicity of the batch upload of objects.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the metadata layer is used to store the metadata of the first object belonging to the same partition to the persistence layer through a write-ahead log; and to store the metadata of the first object belonging to different partitions to the persistence layer through a distributed transaction.

[0013] In this implementation, in the batch object upload operation, the object metadata of the same partition is guaranteed to be atomic in batches through pre-write logs. For the writing of object metadata of different partitions, each partition in the different partitions is treated as a whole in the distributed transaction to achieve batch object atomic operation and greatly reduce the number of distributed transaction participants, which helps to stabilize the performance of distributed transactions.

[0014] In combination with the first aspect, in some implementations of the first aspect, the service layer is configured to determine the partition keys of the multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong.

[0015] In this implementation, users can customize partition key rules when creating buckets, so that users can use partition key rules to divide the distribution relationship of objects.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the service layer is further used to receive a bucket creation request sent by the client before determining the partition keys of multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong, where the bucket creation request includes the bucket name of the target bucket and the target partition key rule, and sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; the metadata layer is also used to store the first mapping relationship.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the service layer is further used to receive a batch deletion request sent by the client, the batch deletion request indicates multiple second objects to be deleted, and sends metadata of the multiple second objects to the metadata layer, the metadata of the multiple second objects includes a deletion mark; the metadata layer is also used to store the metadata of the multiple second objects to the persistence layer, and determine that the metadata of at least one second object fails to be stored in the persistence layer, and send a second indication information to the service layer, the second indication information is used to indicate to abandon garbage collection of data of the multiple second objects.

[0018] In this implementation, after receiving a batch delete request, the service layer atomically stores the metadata of the multiple second objects indicated by the batch delete request in the persistence layer. When the metadata of each of the multiple second objects is successfully stored in the persistence layer, the service layer returns a message to the client indicating that the batch object deletion is successful. If the metadata of at least one of the multiple second objects fails to be stored in the persistence layer, the service layer instructs the client to abandon garbage collection of the data of the multiple first objects. In this way, based on the separate writing of data and metadata to the persistence layer in the object storage system, the atomicity of batch deletion of multiple objects is achieved by ensuring the atomic storage of metadata of multiple objects in the persistence layer.

[0019] In combination with the first aspect, in some implementations of the first aspect, the service layer is further configured to store the first mapping relationship in the partition cache.

[0020] In combination with the first aspect, in some implementations of the first aspect, the service layer is used to query the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition keys of the multiple first objects.

[0021] In this implementation, the service layer stores the partition key rule in the partition cache to avoid accessing the bucket metadata for each partition key read request, reducing the access overhead to the metadata layer to improve the partition key reading performance.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the service layer is also used to query the partition key rules of the buckets to which multiple first objects belong in the metadata layer when a query on the first mapping relationship stored in the partition cache misses, so as to determine the partition keys of the multiple first objects.

[0023] In a second aspect, a data processing method is provided, which is applied to an object storage system, which is used to provide object storage services through at least one bucket. The object storage system includes a service layer, a persistence layer and a metadata layer. The method includes: the service layer receives a batch upload request sent by a client, and the batch upload request indicates multiple first objects to be uploaded; the service layer sends the data of the multiple first objects to the persistence layer; the persistence layer stores the data of the multiple first objects; the service layer generates metadata for the multiple first objects and sends the metadata of the multiple first objects to the metadata layer; the metadata layer stores the metadata of the multiple first objects in the persistence layer; the metadata layer determines that the storage of the metadata of at least one first object in the persistence layer fails, and sends first indication information to the service layer, and the first indication information is used to instruct garbage collection of the multiple first objects.

[0024] In combination with the second aspect, in some implementations of the second aspect, the service layer determines partition keys of the multiple first objects, where the partition keys are used to indicate partitions of metadata of the multiple first objects.

[0025] In combination with the second aspect, in certain implementations of the second aspect, the metadata layer stores the metadata of the first object belonging to the same partition to the persistence layer through a write-ahead log; the metadata layer stores the metadata of the first object belonging to different partitions to the persistence layer through a distributed transaction.

[0026] In combination with the second aspect, in certain implementations of the second aspect, the service layer determines the partition keys of the multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong.

[0027] In combination with the second aspect, in certain implementations of the second aspect, before determining the partition keys of multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong, the service layer receives a bucket creation request sent by the client, where the bucket creation request includes the bucket name of the target bucket and the target partition key rule; the service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; and the metadata layer stores the first mapping relationship.

[0028] In combination with the second aspect, in certain implementations of the second aspect, the service layer receives a batch deletion request sent by the client, and the batch deletion request indicates multiple second objects to be deleted; the service layer sends the metadata of the multiple second objects to the metadata layer, and the metadata of the multiple second objects includes a deletion mark; the metadata layer stores the metadata of the multiple second objects to the persistence layer; the metadata layer determines that the storage of the metadata of at least one second object to the persistence layer fails, and sends a second indication information to the service layer, and the second indication information is used to indicate that garbage collection of the data of the multiple second objects is abandoned.

[0029] In combination with the second aspect, in some implementations of the second aspect, the service layer stores the first mapping relationship in the partition cache.

[0030] In combination with the second aspect, in some implementations of the second aspect, the service layer queries the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache to determine the partition keys of the multiple first objects.

[0031] In combination with the second aspect, in certain implementations of the second aspect, when the service layer fails to hit the first mapping relationship query stored in the partition cache, the partition key rules of the buckets to which multiple first objects belong are queried in the metadata layer to determine the partition keys of the multiple first objects.

[0032] Regarding the technical principles and beneficial effects of the second aspect, please refer to the relevant description of the first aspect mentioned above, and no further details will be given here.

[0033] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method of the above-mentioned second aspect or any one of the implementation methods of the second aspect.

[0034] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computer device cluster, enables the computer device cluster to execute the method of the second aspect or any one of the implementations of the second aspect.

[0035] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method of the above-mentioned second aspect or any one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0037] FIG1 is a schematic diagram of a scenario of an object storage system involved in this application.

[0038] FIG2 is a schematic diagram of the architecture of an object storage system involved in this application.

[0039] FIG3 is a schematic diagram of the structure of a range partition involved in this application.

[0040] FIG4 is a hardware structure diagram of an object storage system involved in this application.

[0041] FIG5 is a schematic diagram of a scenario of an object storage system provided by this application.

[0042] FIG6 is one of the flow charts of a data processing method provided in this application.

[0043] FIG7 is a schematic diagram of a batch operation configuration interface provided by this application.

[0044] FIG8 is one of the flowcharts of a data processing method provided in this application.

[0045] FIG9 is a flow chart of a distributed transaction provided by this application.

[0046] FIG10 is one of the flowcharts of a data processing method provided in this application.

[0047] FIG11 is a flowchart of a data processing method provided in the present application.

[0048] FIG12 is a schematic diagram of the structure of an object storage system provided by this application.

[0049] FIG13 is a schematic diagram of the structure of a computing device provided by the present application.

[0050] FIG14 is a schematic diagram of the structure of a computing device cluster provided by this application.

[0051] FIG15 is a schematic diagram of a structure of a network connection between computing devices provided by the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0053] In the description of the embodiments of this application, unless otherwise specified, the term "and / or" is used herein to describe an association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " herein indicates that the associated objects are in an "or" relationship, for example, A / B means A or B. A and B can be singular or plural.

[0054] In the description of the embodiments of the present application, unless otherwise specified, the term "multiple" herein refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0055] In addition, to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0056] At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. In order to make the technical solutions provided by the present application clearer, the relevant terms are first explained before the specific description of the technical solutions provided by the present application.

[0058] (1) Object storage service (OBS): It stores data in the form of objects in object storage nodes. Object storage nodes provide a flat storage method based on buckets and objects. All objects in the bucket are at the same logical level, eliminating the multi-level tree directory structure in the file storage service node. The object storage system is used to provide users with operations that conform to object semantics (i.e., object services), such as: uploading objects, downloading objects, listing each object in the bucket, copying objects, moving objects, renaming objects, deleting objects, deleting each object in the bucket, deleting multiple specified objects, etc. Among them, the object storage node provides an object interface, which can indicate operations that conform to object semantics. For example, the object interface can be a representational state transfer (REST) ​​application programming interface (API), and users can access data on the object storage service node through the REST API interface. It should be noted that when users access data through the object interface, the data can be called an object.

[0059] (2) Object: An object is the basic unit of data storage in an object storage system. An object is actually a collection of a file's data and its related attribute information (metadata). Data uploaded by tenants to OBS is stored in buckets as objects. Objects consist of three parts: a key, metadata, and data. The key, or object name, is a UTF-8-encoded character sequence with a length greater than 0 and no more than 1024. Each object in a bucket has a unique object key.

[0060] (3) Bucket: A bucket is a container for storing objects in OBS. Object storage provides a flat storage method based on buckets and objects. All objects in a bucket are at the same logical level, eliminating the multi-level tree directory structure in the file system. Each bucket has its own storage category, access rights, region, and other attributes. Tenants can create buckets with different storage categories and access rights and configure more advanced attributes to meet the storage requirements of different scenarios.

[0061] (4) Object storage device (OSD): OSD is the basic storage unit of the object storage system, which is set on the physical disk. Specifically, it is a fixed-size storage space of the physical disk. The object storage system manages the physical disks of multiple computing devices in the form of OSD.

[0062] (5) Range partitioning: For clusters storing metadata, partitioning is often used to dynamically split metadata to accommodate the growing number of metadata entries. Each partition manages different data entries, and each partition is served by a server. The relationship between partitions and servers is dynamically assigned by the cluster management of the distributed metadata storage system. In partitioning, the partition is usually determined by the value of a metadata attribute, or a sequential combination of several attributes. This value is called the partition key (also called partition key or shard key).

[0063] In the object storage service, tenants can log in to the cloud management platform on the public cloud access page with a pre-registered account and password, and after successful login, select and purchase the corresponding public cloud services on the public cloud access page, such as OBS, virtual machine services, container services, etc. For OBS, tenants can further create multiple buckets through the configuration interface or API on the public cloud access page provided by the cloud platform. There is no limit on the total number and size of objects stored in each bucket, and tenants do not need to consider the scalability of data. OBS is a service based on REST API and hypertext transport protocol (HTTP) or hypertext transfer protocol secure (HTTPS). Tenants can locate bucket resources through uniform resource identifier (URI), which is also called access domain name in this application, or domain name for short. Among them, in OBS, the bucket name is globally unique and cannot be modified, that is, the bucket created by the tenant cannot be the same as the name of other buckets created by the tenant, nor can it be the same as the bucket name created by other tenants.

[0064] Figure 1 shows a schematic diagram of at least one application scenario in an embodiment of the present application. As shown in Figure 1, each bucket can include multiple objects, and the objects between buckets are isolated from each other. The tenant logs in to the cloud management platform 10 through the client, selects and purchases the cloud service of the object storage service on the cloud management platform 10. After the purchase, the tenant can store objects based on the object storage service provided by the object storage system. Among them, the cloud management platform 10 is mainly used to manage the infrastructure for running the object storage service. Exemplarily, the infrastructure for running the object storage service may include multiple data centers set up in different regions, and each data center includes multiple servers. The data center can provide basic resources for the object storage service, such as computing resources, storage resources, etc. Therefore, when the tenant purchases and uses the object storage service, he mainly pays for the resources used. Specifically, the object storage service provides the domain name of the bucket, and the tenant can access the domain name of the bucket through the client to upload data to the bucket or download data from the bucket, wherein the uploaded data is stored in the bucket in the form of objects.

[0065] When a tenant uploads an object, they can specify the object's storage category. If not specified, the default is the same as the bucket's storage category. After uploading, the object's storage category can be modified, and the bucket can be accessed through a client. The client can be a browser used locally by the tenant or a dedicated client provided by the cloud platform. For example, a tenant can access a bucket through a browser set up on a local computer. After entering their account and passing verification, the tenant can use the browser to upload objects to the bucket, or perform operations such as modifying and deleting objects in the bucket. The local computer is connected to the internet.

[0066] 2 , the object storage system 200 can provide object storage services for the client 20. Referring to FIG2 , the object storage system 200 includes a service layer 21, a metadata layer 22, and a persistence layer 23.

[0067] Among them, the service layer 21 is used to interact with the client 20 and provide foreground services to the outside world. The foreground services can be services for the client 20 to request to read or write data. The client 20 can be deployed on a terminal device, and multiple applications can also be deployed on the terminal device. Taking application 201 as an example, application 201 can be deployed in the user's terminal device. For example, the terminal device can be a mobile phone, a laptop, a tablet computer, a PDA, a wireless terminal in a smart city, or a wireless device in a smart home. This application does not limit the specific type of terminal device. The client 20 can interact with application 201. For example, application 201 can send an object upload request to the client 20 when triggered by a tenant. After receiving the object upload request, the client 20 can send an object upload request to application 201 according to the object upload request.

[0068] Tenants can use their accounts on client 20 to log in to the cloud management platform, create buckets, configure bucket names, and obtain bucket domain names through the cloud management platform. After detecting the tenant's operations (such as creating a bucket, configuring a bucket name, etc.), the cloud management platform can send a creation instruction to the service layer 21 through the switching device. The creation instruction includes information such as the bucket name and bucket domain name, notifying the object storage service node to create the bucket and save the bucket name, bucket domain name, and other information. After the bucket is created, the tenant can also access the bucket domain name by operating client 20, locate the bucket managed by the service layer 21, and upload data (such as uploading objects) and download data (such as downloading objects) in the bucket.

[0069] The following is specifically about the service layer 21. The service layer 21 includes a gateway 211, a metadata interface 212, and a persistence layer interface 213. Among them, the gateway 211 is responsible for the access, authentication, authorization and protocol processing of the object storage. The metadata interface 212 is the interface of the metadata layer 22. Through the metadata interface 212, the metadata corresponding to the data written by the tenant can be written into the metadata layer 22. The metadata is finally persisted in the persistence layer 23. In the persistence layer 23, the metadata is generally stored in the form of copies. By storing multiple copies of the data (generally three copies), when a copy is lost, it can be copied from other copies. The persistence layer interface 213 is the interface of the persistence layer 23. Through the persistence layer interface 213, the data that the tenant needs to write can be written into the persistence layer 23. Among them, in the persistence layer 23, the data is generally stored in the form of erasure coding (EC). Redundancy is obtained by encoding the original data, and the data and redundancy are stored together in the persistence layer 23 to achieve the purpose of fault tolerance.

[0070] Object metadata is a distributed storage engine based on range partitioning and is generally sorted lexicographically. The metadata layer 22 includes bucket metadata 221, object metadata 222, and a persistence interface 223. For clusters storing metadata, partitioning is often used to dynamically split metadata to accommodate the growing number of metadata entries. Each partition (also called a partition) manages different data entries. Each partition is served by a server, and the relationship between partitions and servers is dynamically assigned by the cluster management of the distributed metadata storage system. Dynamic partitioning and dynamic partition affiliation enable dynamic horizontal scalability of the distributed metadata storage system. Partitioning technology typically uses a fixed value of a metadata attribute, a sequential combination of several attributes, or a custom rule to determine the partition. This value is called the partition key, and the rule is stored in the bucket metadata 221. During addition, the partition key of each data entry is queried in the range partition to uniquely determine the partition to which the data entry belongs. Bucket metadata 221 also records bucket metadata information, such as the region to which the bucket belongs. Object metadata 222 records object metadata, such as its attributes, size, upload time, and policy information. Object metadata 222 also includes a write-ahead log (WAL). This WAL requires that all modifications to the storage system be written to the log before being submitted, ensuring the atomicity and durability of the object storage system. Assuming the hard drive data is intact, the WAL allows the storage system to recover to its pre-crash state after a crash, guided by the log, to avoid data loss. This technology is widely used to store metadata information in storage systems (file, object, database, and column).

[0071] In order to ensure that the metadata of the object can be arranged in the natural order of the partition key, distributed storage systems usually adopt range partitioning technology. Among them, in the partitioning technology, the value of a column of the data table, or the sequential combination value of several column values, is usually used to determine the partition. As shown in the user metadata table in Table 1, the combination of "bucket name", "object name" and "object version number" in the metadata table uniquely identifies an object, that is, the object identifier is {bucket name, object name, object version number}. The combination value of the three column fields {Bn, On, ver} uniquely identifies an object. The object entry key value = {Bn, On, ver}. In order to adapt to the continuous growth of the object data scale, the combination value of the column field {Bn, On} prefixed by the primary index entry key value is usually used as the partition key, which facilitates indexing partitions according to the primary index entry key value. Although each object entry has a partition key, the partition key is not required to be unique. In the example in Table 1:

[0072] The partition key of the first row of data entry = {Bn = "A", On = "0109"};

[0073] The partition key of the data entry in row 2 = {Bn = "A", On = "0109"}. The partition key is the same as that in row 1, but the entry key values ​​are different.

[0074] The partition key of the data entry in row 3 = {Bn = "B", On = "0201"}.

[0075] Table 1: User metadata table

[0076] The partition key of the second row of data entry in Table 1 is the same as the partition key of the first row of data entry. Therefore, the metadata represented by the first row of data entry and the metadata represented by the second row of data entry belong to the same partition. However, the identifier of the object represented by the first row of data entry and the identifier of the object represented by the second row of data entry are different.

[0077] For metadata at the boundary of a partition, the partition to which the metadata belongs is determined by the principle of left-closed and right-open or left-open and right-closed. For example, FIG3 shows the structure of a range partition. Bn in Table 1 represents the name of the bucket, and On represents the name of the object. For metadata at the boundary of a partition, if the principle of left-closed and right-open is used to determine the partition to which the metadata belongs, the numerical range of the partition key of partition 1 is: ({Bn: Min; on: min}, {Bn: "A"; on: "w"}], and the numerical range of the partition key of partition 2 is: ({Bn: "A"; on: "w"}, {Bn: "C"; on: "x"}].

[0078] In the persistence layer 23, the object storage device OSD control unit configures the physical disks as multiple object storage devices (OSDs). This application uses OSDs as the basic storage unit as an example; any storage space that can serve as a basic storage unit is applicable to this application. After receiving the creation instruction for bucket 1, the cloud management platform creates bucket 1 and creates bucket 1 on physical disk 1 and physical disk 2. Bucket 1's OSDs are distributed across physical disk 1 and physical disk 2, and bucket 1's bucket name, bucket domain name, and other information are stored.

[0079] Exemplarily, the service layer 21 , metadata layer 22 and persistence layer 23 may be implemented by software or hardware.

[0080] When implemented through hardware, the service layer 21, metadata layer 22, and persistence layer 23 can all be implemented through at least one computing device, such as through a server. Next, referring to Figure 4, the hardware structure of the object storage system 200 of the present application is introduced. When the above-mentioned service layer 21, metadata layer 22, and persistence layer 23 are implemented through a server, the service layer 21 is deployed on the service node 41, the metadata layer 22 is deployed on the metadata node 42, and the persistence layer 23 is deployed on the storage node 43. The hardware structure of the service node 41 is introduced below. The various components shown in Figure 4 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. As shown in Figure 4, the service node 41 may include: a processor 411, a memory 412, and a communication interface 413. Among them, the processor 411, the memory 412, and the communication interface 413 can be connected to each other through a bus 414, or can be connected to each other in other ways.

[0081] Among them, processor 411 is the control center of the service node. Processor 411 can be a general-purpose central processing unit (CPU) or other general-purpose processors, among which the general-purpose processor can be a microprocessor or any conventional processor. For example, processor 411 can include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a controller, etc. The controller in processor 411 is the nerve center and command center of the service node. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution. Optionally, a memory can also be provided in processor 411 for storing instructions and data. Exemplarily, processor 411 can include one or more CPUs, such as CPU1 and CPU2 shown in Figure 4.

[0082] The memory 412 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical storage, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. In the embodiment of the present application, the memory 412 can store information such as computer instructions.

[0083] The communication interface 413 may be a transceiver module for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. The communication interface 413 may receive instructions, messages, or data. The transceiver module may be a device such as a transceiver or a transceiver. Alternatively, the communication interface 413 may be a transceiver circuit located within the processor 201, for implementing signal input and signal output of the processor. The communication interface 413 may be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface, or the communication interface 413 may be a wireless interface.

[0084] Bus 414 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, FIG4 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0085] In addition, the storage node 43 may include a memory 431. The memory 431 may include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0086] It should be noted that the service node 41 shown in FIG4 is merely an example of a service node. The service node may have more or fewer components than those shown in FIG4 , may combine two or more components, or may have different component configurations. In actual applications, the object storage system 200 may also be applicable to other scenarios. For example, in other possible application scenarios, the object storage system 200 may provide object storage services to a larger number of clients, or to other types of clients. Alternatively, the object storage system 200 may also include devices with other functions to support this embodiment, and this is not a limitation.

[0087] In standard object storage, the object's Application Programming Interface (API) uses a single object as the operation granularity, such as uploading an object, deleting an object, and a single object can achieve atomicity guarantees. For the write operation of a single object, a tenant's operation is divided into two operations within the storage system: data writing and metadata writing. The result of the tenant's write operation is determined by whether the metadata is written successfully. After the tenant's specific data is written to the persistence layer 21, the generated metadata is processed by the metadata layer 22. After receiving the request, the metadata layer 22 records the metadata operation and metadata in the WAL log. After the write is successful, the operation result is returned to the service layer 21. After receiving the response that the metadata is written successfully, the service layer 21 returns the final operation result to the user. Therefore, the result of the user's write operation is usually determined by whether the metadata is written successfully.

[0088] However, in real-world applications, tenants may require batch operations on multiple objects. Object storage doesn't guarantee atomicity for batch operations like uploading or deleting multiple objects. This lack of atomicity can lead to inconsistent user experiences. For example, a user may successfully access metadata, but data access may fail because the corresponding data was not uploaded successfully.

[0089] To facilitate understanding, at least one application scenario of an embodiment of the present application is first introduced below.

[0090] Exemplarily, FIG5 shows a schematic diagram of at least one application scenario in an embodiment of the present application. As shown in FIG5 , a tenant uses an account to log in to the cloud management platform on the client 20, and uploads multiple high-definition picture objects to bucket 1 in batches through the cloud management platform. A single picture object and the description information object of the picture object are written twice into bucket 1. Alternatively, thumbnail objects of different resolutions for the same picture object need to be written separately into bucket 1. However, due to the lack of atomicity guarantee for batch operations on multiple objects in the object storage system 200, taking the case where the picture object and the description information object of the picture object are written twice into bucket 1 as an example, if the description information of the picture object is written successfully, but the picture object fails to be written, the tenant can only access the description information of the picture object when accessing it, but cannot access the picture object, resulting in tenant access failure and poor tenant experience.

[0091] In order to solve the problem of no atomicity guarantee in batch operations of multiple objects, an embodiment of the present application proposes a data processing method, which is based on the object storage system, in which data and metadata are written separately, and the atomicity of batch operations of multiple objects is achieved through the atomic writing guarantee of metadata of multiple objects.

[0092] Next, based on the above description, a data processing method provided by an embodiment of the present application is introduced. It is understandable that this method is proposed based on the above description, and part or all of the content of this method can be referred to the above description.

[0093] For example, FIG6 illustrates a flow chart of a data processing method provided in an embodiment of the present application. It will be appreciated that the method can be implemented in any suitable device, apparatus, platform, or device cluster with computing, processing, and storage capabilities. As shown in FIG6 , the method may include steps S601 to S606.

[0094] Step S601: The service layer receives a batch upload request sent by the client.

[0095] In this embodiment, the service layer of an object storage system receives a batch upload request from a client, the batch upload request indicating multiple first objects to be uploaded, wherein the batch upload request carries the multiple first objects to be uploaded. The service layer of the object storage system may be service layer 21, the client may be client 20, the persistence layer may be persistence layer 23, and the metadata layer may be metadata layer 22. This example is also used below for explanation.

[0096] Optionally, the gateway 211 of the service layer 21 receives the batch upload request and sends the multiple first objects to be uploaded carried in the batch upload request to the persistence interface 212 , and the persistence interface 212 is responsible for writing the data of the multiple first objects into the persistence layer 23 .

[0097] Optionally, as shown in FIG7 , the client provides a batch operation configuration interface 700, which is used to receive the user's batch operation configuration instructions for objects, so that the service layer performs batch operation processing and atomicity configuration based on the batch operation configuration instructions. The items that can be configured by the user in the batch operation configuration interface 700 include batch processing operations, storage categories, atomicity, uploaded objects, etc. Among them, the storage category can be divided into standard storage, low-frequency access storage, and archive storage, which are not specifically limited in the embodiment of the present application. In this embodiment, the user can choose to upload objects in batches. When selecting objects to be uploaded in batches, the user can set whether the uploaded objects in batches need to be atomically guaranteed by turning on or off atomicity.

[0098] Step S602: The service layer sends data of multiple first objects to the persistence layer.

[0099] In this embodiment, after receiving a batch upload request from a client, the service layer sends the data for the multiple objects to the persistence layer in response to the batch upload request. Objects consist of three parts: a key value, metadata, and data. The data is the data content of the file. After receiving the batch upload request, the service layer first sends the data for the multiple first objects to the persistence layer.

[0100] Step S603: The persistence layer stores data of the plurality of first objects.

[0101] In this embodiment, after receiving the data of the multiple first objects, the persistence layer stores the data of the multiple first objects.

[0102] Step S604: The service layer generates metadata for multiple first objects.

[0103] In this embodiment, after the persistence layer stores the data of the multiple first objects, it sends the storage address of the data of the multiple first objects to the service layer, and the service layer generates metadata of the multiple first objects based on the storage address of the data and the object names of the multiple first objects.

[0104] Specifically, object metadata is descriptive information about the object, including system metadata and user metadata. System metadata is automatically generated by the object storage system and used when processing object data. User metadata is specified by the user when uploading objects in batches. It is user-defined object description information and may include object attributes, upload time, policy information, storage address, etc., which is not specifically limited in this embodiment of the application. The user has enabled atomicity in batch operation configuration interface 700. Configuring batch upload of objects requires atomicity guarantees. The metadata layer atomically writes the metadata of multiple first objects to the persistence layer based on this configuration information.

[0105] Step S605: The service layer sends metadata of multiple first objects.

[0106] In this embodiment, the service layer generates metadata of the multiple first objects according to the storage address of the storage data and the object names of the multiple first objects, and then sends the metadata of the multiple first objects to the metadata layer.

[0107] Step S606: The metadata layer stores metadata of the plurality of first objects in the persistence layer.

[0108] In this embodiment, after receiving the metadata of the plurality of first objects, the metadata layer stores the metadata of the plurality of first objects in the persistence layer.

[0109] Step S607: The metadata layer determines that the metadata of at least one first object fails to be stored in the persistence layer, and sends first indication information to the service layer.

[0110] In this embodiment, the metadata layer determines that the metadata of at least one first object has failed to be stored in the persistence layer, and sends a first indication message to the service layer, where the first indication message is used to instruct the garbage collection of the multiple first objects. That is, the data of the multiple first objects stored in the persistence layer are garbage collected, and the state is restored to the state where the multiple first objects have not been uploaded. If the metadata layer determines that the metadata of at least one first object has been successfully stored in the persistence layer, the persistence layer sends the metadata persistence address to the metadata layer, the metadata layer records the metadata persistence address and sends it to the service layer, and the service layer returns to the client that the batch upload of objects is successful.

[0111] Thus, in this embodiment, based on the separation of data and metadata in the object storage system and writing them into the persistence layer, the atomicity of batch uploading of multiple objects is achieved by ensuring the atomic storage of metadata of multiple objects into the persistence layer.

[0112] In some possible implementations, as shown in FIG8 , executing step S606 may specifically include steps S801 to S802 .

[0113] In this implementation, before the metadata layer stores metadata of the multiple first objects in the persistence layer, it first determines the partition keys of the multiple first objects. After determining the partition keys of the multiple first objects and obtaining the partitions of the metadata of the multiple first objects, the metadata of the multiple first objects is stored in the persistence layer using different atomicity strategies.

[0114] Specifically, partitioning technology typically uses the value of a metadata attribute, or a sequential combination of several attributes, to determine a partition. This value is called a partition key. By querying each object's partition key, the partition containing the object's metadata is uniquely determined. In this implementation, the service layer determines the partitions for the metadata of multiple first objects based on their partition keys, sends the metadata of first objects belonging to the same partition to that partition, and the metadata layer of that partition atomically stores the metadata of the multiple first objects in the persistence layer.

[0115] Step S801: The metadata layer stores metadata of a first object belonging to the same partition to the persistence layer through a write-ahead log.

[0116] In this implementation, after the service layer determines the partition keys of the multiple first objects, it then stores the metadata of the first objects belonging to the same partition in the persistence layer via a write-ahead log. The metadata layer of each partition records metadata operations and metadata records within the current partition in a write-ahead log, writes them to the cache, and persists them to the persistence layer once the cache is full.

[0117] Specifically, the write-ahead log of each partition is only responsible for recording the update operations of the metadata belonging to the current partition. For the metadata of the first object belonging to the same partition, the metadata of the first object of the partition is not written directly to the persistence layer, but is first written to the write-ahead log of the metadata layer. If at least one of the metadata of the first objects in the partition fails to be written to the write-ahead log, the record in the write-ahead log will be ignored; if the metadata of each of the metadata of the first objects in the partition is successfully written to the write-ahead log, the metadata of the first object in the partition will be stored in the persistence layer according to the record in the log.

[0118] Specifically, the write-ahead log includes a redo log and an undo log. When the metadata of at least one of the first objects in the partition fails to be written to the write-ahead log, the undo log is used to restore the data to the state before the batch upload of the multiple first objects, that is, roll-forward recovery is performed to ensure the atomicity of the metadata write of the first objects in the partition.

[0119] Step S802: The metadata layer stores metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0120] In this implementation, after the service layer determines the partition keys of the multiple first objects, it then stores the metadata of the first objects belonging to different partitions in the persistence layer based on a distributed transaction. Each of the different partitions participates in the distributed transaction as a transaction participant, thereby storing the metadata of the first objects in the different partitions in the persistence layer.

[0121] Optionally, the distributed transaction may be a two-phase commit protocol (2PC), a three-phase commit protocol (2PC), or a compensating transaction (try-confirm-cancel, TCC), which is not specifically limited in the embodiments of the present application.

[0122] In one example, a transaction is a reliable, independent unit of work consisting of a set of operations, in which all operations either succeed or fail. Transactions have four basic properties: atomicity, consistency, isolation, and durability, also referred to as the ACID properties of transactions. Distributed transactions include transaction managers, transaction coordinators, and transaction participants. In this embodiment, the service layer can be a transaction manager, which acts as a global scheduler and is responsible for deciding the commit and rollback of the entire distributed transaction. The target participating partition can be the transaction coordinator, and the transaction participants can be each participating partition, which is responsible for the commit and rollback of its own local transaction.

[0123] In one example, a two-phase transaction, also known as a two-phase commit protocol, divides the entire transaction process into two phases: the prepare phase and the commit phase. The 2 refers to the two phases, P stands for prepare phase, and C stands for commit phase. During the prepare phase, the transaction manager sends a prepare message to each participant. Each participant executes the transaction locally and writes local undo / redo logs. During this phase, the transaction is not committed. During the commit phase, if the transaction manager receives a participant execution failure or timeout message, it directly sends a rollback message to each participant. Otherwise, it sends a commit message. Transaction participants commit or rollback based on the transaction manager's instructions and release lock resources used during the transaction.

[0124] As shown in FIG9 , executing step S802 may specifically include steps S901 to S906 , wherein steps S901 to S905 are phase one of the two-phase transaction, and steps S906 to S908 are phase two of the two-phase transaction.

[0125] Step S901: The service layer applies for a distributed transaction.

[0126] Specifically, the service layer applies for a distributed transaction, which is successfully created and generates a transaction identifier, also known as a transaction ID. The transaction identifier uniquely identifies a distributed transaction and can be a unique number assigned to the distributed transaction or another identifier that uniquely represents the distributed transaction.

[0127] Step S902: The service layer sends a batch upload object preparation operation to each participating partition.

[0128] Specifically, after a distributed transaction is successfully created, the service layer sends a batch upload object preparation operation to each partition (also called participating partitions). This is the transaction inquiry phase in phase one. The service layer sends the transaction content to each participating partition, inquiring whether the participating partition can execute the transaction commit operation, and then begins waiting for responses from each participating partition. The transaction content is the metadata of the first object in each partition, and the metadata layer of each participating partition stores the metadata of the first object in that participating partition.

[0129] Step S903: Each participating partition locks the key value of the first object of the participating partition.

[0130] Specifically, after each participating partition receives the metadata for the first object in that partition, before executing a transaction, it locks the key value of the first object in that partition to prevent concurrent reads and writes to the first object, which could affect the execution of subsequent transactions. It also locally registers a transaction identifier, which is identical to the identifier generated when the distributed transaction is successfully created and is used to uniquely identify the distributed transaction. The lock indicates that the first object in that partition is currently being written.

[0131] Alternatively, the lock may be a single bit, where a bit value of 1 indicates that the lock indicates that the first object of the participating partition is being written, and a bit value of 0 indicates that the lock indicates that the first object of the participating partition has been written. Alternatively, a bit value of 1 indicates that the lock indicates that the first object of the participating partition has been written, and a bit value of 0 indicates that the lock indicates that the first object of the participating partition is being written.

[0132] Step S904: The metadata layer of the participating partition stores the metadata of the first object belonging to the same partition to the persistence layer through the write-ahead log.

[0133] Specifically, after each participating partition locks the key value of the first object of the participating partition, the metadata layer of each participating partition writes the metadata of the first object of the participating partition into the pre-write log. The writing of the metadata of the first object in the participating partition is the same as step S801. The metadata layer stores the metadata of the first object belonging to the same partition to the persistence layer through the pre-write log, which will not be repeated here.

[0134] Step S905: Each participating partition feeds back a response to the batch upload object preparation operation.

[0135] Specifically, after each participating partition stores the metadata of the first object of the participating partition to the persistence layer, it feeds back the response of the batch upload preparation operation to the service layer. If the metadata of the first object in each participating partition is successfully stored in the persistence layer, it feeds back the agreement to submit the request to the service layer and continues to execute the transaction in the second phase; if the metadata of the first object in at least one participating partition in each participating partition fails to be stored in the persistence layer and the submission request is not fed back to the service layer, the transaction is rolled back.

[0136] Step S906: Select a target participating partition from each participating partition and send a submission request.

[0137] Specifically, after receiving the response of the batch upload object preparation operation from each participating partition, the service layer selects a target participating partition from each participating partition as the transaction coordinator and sends a commit request.

[0138] Step S907: Each participating partition stores the write-ahead log of the metadata layer of the participating partition to the persistence layer and releases the lock.

[0139] Specifically, the target participating partition sends a prepare-commit request to the other participating partitions. Upon receiving the prepare-commit request, each participating partition formally executes the transaction commit operation, storing the write-ahead log of the participating partition's metadata layer to the persistence layer. After storing the write-ahead log of the participating partition's metadata layer to the persistence layer, the lock on the key value of the first object in the participating partition is released. Furthermore, all other transaction resources occupied during the entire transaction are released.

[0140] Step S908: The target participating partition sends a confirmation request.

[0141] Specifically, after each participating partition completes the asynchronous transaction commit, the target participating partition sends a confirmation request to the service layer, completing the distributed transaction. Simultaneously, the service layer sends a commit-end request to each participating partition and cancels the transaction identifier for the distributed transaction. This completes the distributed transaction, storing the metadata of the first object belonging to each partition in the persistence layer.

[0142] Thus, in this embodiment, in the operation of uploading objects in batches, the object metadata of the same partition is guaranteed to be atomic in batches through pre-write logs. For the writing of object metadata of different partitions, each partition in the different partitions is treated as a whole to participate in the distributed transaction, thereby realizing atomic operation of batch objects and greatly reducing the number of distributed transaction participants, which helps to stabilize the performance of distributed transactions.

[0143] In some possible implementations, as shown in FIG10 , the data processing method further includes steps S1001 to S1004 .

[0144] Step S1001: The service layer receives a batch deletion request sent by the client.

[0145] In this embodiment, the service layer receives a batch deletion request from the client, where the batch deletion request is used to instruct deletion of multiple second objects, wherein the batch deletion request may carry the object names of the multiple second objects to be deleted.

[0146] Step S1002: The service layer sends metadata of multiple second objects to the metadata layer.

[0147] In this embodiment, after receiving a batch delete request from a client, the service layer generates metadata for multiple second objects based on the object names of the multiple second objects to be deleted, and sends the metadata for the multiple second objects to the metadata layer in response to the batch delete request. The metadata for the multiple second objects includes a delete marker, which includes a key value of the second object to be deleted, indicating that the object should be deleted.

[0148] Step S1003: The metadata layer stores metadata of the plurality of second objects in the persistence layer.

[0149] In this embodiment, after receiving the metadata of the second object, the metadata layer stores the metadata of the multiple second objects in the persistence layer. Specifically, the service layer first determines the partition keys of the multiple second objects. The metadata layer stores the metadata of the second objects belonging to the same partition in the persistence layer via a write-ahead log. The metadata layer then stores the metadata of the second objects belonging to different partitions in the persistence layer via a distributed transaction. The specific steps are executed in the same way as steps S801 and S802 and are not further described here.

[0150] Step S1004: The metadata layer determines that the metadata of at least one second object fails to be stored in the persistence layer, and sends second indication information to the service layer.

[0151] In this embodiment, the metadata layer determines that the metadata of at least one second object has failed to be stored in the persistence layer, and sends a second indication message to the service layer. The second indication message indicates that garbage collection of the data of the multiple second objects has been abandoned, that is, deletion of the multiple second objects has been abandoned. If the metadata layer determines that the metadata of at least one second object has been successfully stored in the persistence layer, the service layer returns a message to the client indicating that the batch deletion of the objects has been successful.

[0152] Specifically, after the persistence layer stores the metadata for multiple second objects, it marks the multiple second objects as unavailable. This does not immediately delete the data itself from the persistence layer. Distributed storage systems typically employ garbage collection (GC). The system periodically scans storage devices, marks deleted or expired data as garbage, and deletes or moves it to a garbage collection area for recycling. GC can be implemented through either periodic cleanup or triggered cleanup.

[0153] Thus, in this embodiment, based on the separation of data and metadata in the object storage system and writing them into the persistence layer, the atomicity of batch deletion of multiple objects is achieved by ensuring the atomic storage of metadata of multiple objects into the persistence layer.

[0154] In some possible implementations, as shown in FIG11 , the service layer may specifically include steps S1101 to S1105 when determining the partition keys of the multiple first objects.

[0155] Step S1101: The service layer receives a bucket creation request sent by the client.

[0156] In this implementation, users can customize partition key rules when creating a bucket. The service layer receives a bucket creation request from the client, which may include the bucket name, region, and partition key rule. The partition key rule is user-configured and predefined. For example, it may consist of the bucket name prefix and the object name prefix.

[0157] In one example, the partition key rule might consist of the three-character prefix of the bucket name and the four-character prefix of the object name, such as {[{type:bucketprefix,len:3},{type:objprefix,len:4}]}. This indicates that the partition key consists of the three-character prefix of the bucket name and the four-character prefix of the object name. After configuring the partition key rule, metadata for different objects can be placed in the same partition or in different partitions.

[0158] Optionally, users can choose not to configure the partition key rule. After the service layer receives the bucket creation request, it parses the partition key rule as the default type, such as {type:null}. The service layer then uses the bucket name and object name as the partition key.

[0159] In step S1102 , the service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer.

[0160] In this implementation, after receiving the target bucket name and target partition key rule included in the create bucket request sent by the client, the service layer sends the first mapping relationship between the target bucket name and the target partition key rule to the metadata layer.

[0161] Step S1103: The metadata layer stores the first mapping relationship.

[0162] In this implementation, after receiving the first mapping relationship between the bucket target bucket name and the target partition key rule, the metadata layer stores the first mapping relationship in the bucket metadata, which can be bucket metadata 222.

[0163] Step S1104: The service layer stores the first mapping relationship in the partition cache.

[0164] In this implementation, the service layer includes a partitioned cache for caching user-configured partition key rules. Because the partition key rules in the metadata layer's bucket metadata cannot be modified, a new partitioned cache is added to the service layer to prevent each partition key read request from accessing the bucket metadata. After the metadata layer successfully stores the first mapping relationship in the bucket metadata, it sends a storage success message to the service layer. Upon receiving this storage success message, the service layer stores the first mapping relationship in the partitioned cache.

[0165] In step S1105 , the service layer searches the first mapping relationship stored in the partition cache for the partition key rule of the bucket to which the multiple first objects belong, and determines the partition keys of the multiple first objects.

[0166] In this implementation, when determining the partition keys of multiple objects, the service layer first searches the first mapping relationship stored in the partition cache of the service layer for the partition key rules of the buckets to which the multiple first objects belong, thereby determining the partition keys of the multiple first objects. Based on the searched partition keys of the multiple first objects, the service layer determines the partition to which each first object belongs.

[0167] Step S1106: When the service layer does not hit a query on the first mapping relationship stored in the partition cache, the service layer queries the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

[0168] In this implementation, if the service layer does not hit the first mapping relationship query stored in the partition cache, it continues to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

[0169] In step S1107 , the service layer stores the second mapping relationship between the bucket names of the buckets to which the multiple first objects belong and the partition key rules of the buckets to which the first objects belong in the partition cache.

[0170] In this implementation, after querying the metadata layer for the partition key rules of the buckets to which the multiple first objects belong, the service layer stores the second mapping relationship between the bucket names of the multiple first objects and the partition key rules of the buckets to which the first objects belong in the partition cache. This allows the service layer to access the partition cache and query the partition key rules of the buckets to which the multiple first objects belong in the next read or write request for the partition key.

[0171] In this way, in this implementation, users can configure partition key rules when creating buckets, and the service layer stores the partition key rules in the partition cache, avoiding access to the bucket metadata for each partition key read request, reducing the access overhead to the metadata layer, and improving the reading performance of the partition key.

[0172] In each of the above embodiments, the data processing method in the object storage system 200 may be software configured on a computing device or computing device cluster. By running this software on the computing device or computing device cluster, the computing device or computing device cluster can implement the functions of the above-described object storage system 200. The object storage system 200 involved in batch object operations is described in detail below.

[0173] As shown in FIG12 , the object storage system 200 includes a service layer 21, a metadata layer 22, and a persistence layer 23. Among them:

[0174] The service layer 21 is configured to receive a batch upload request sent by a client, the batch upload request indicating multiple first objects to be uploaded; send the data of the multiple first objects to the persistence layer; generate metadata of the multiple first objects, and send the metadata of the multiple first objects to the metadata layer;

[0175] The persistence layer 23 is used to store data of multiple first objects;

[0176] The metadata layer 22 is used to store metadata of multiple first objects in the persistence layer; if it is determined that the metadata of at least one first object fails to be stored in the persistence layer, a first indication message is sent to the service layer, and the first indication message is used to indicate garbage collection of the multiple first objects.

[0177] In some possible implementations, the service layer 21 is further configured to determine partition keys of the plurality of first objects, where the partition keys are used to indicate partitions of metadata of the plurality of first objects;

[0178] The metadata layer 22 is specifically configured to store metadata of the plurality of first objects in the persistence layer according to the partition keys of the plurality of first objects.

[0179] In some possible implementations, the metadata layer 22 is configured to store metadata of first objects belonging to the same partition to the persistence layer through a write-ahead log; and store metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0180] In some possible implementations, the service layer 21 is configured to determine the partition keys of the multiple first objects according to a partition key rule of the bucket to which the multiple first objects belong.

[0181] In some possible implementations, the service layer 21 is further configured to, before determining the partition keys of the plurality of first objects based on the partition key rules of the buckets to which the plurality of first objects belong, receive a create bucket request from a client, the create bucket request including a bucket name of a target bucket and a target partition key rule; and send a first mapping between the bucket name of the target bucket and the target partition key rule to the metadata layer.

[0182] The metadata layer 22 is also used to store the first mapping relationship.

[0183] In some possible implementations, the service layer 21 is further configured to receive a batch deletion request sent by a client, the batch deletion request indicating multiple second objects to be deleted; and send metadata of the multiple second objects to the metadata layer, the metadata of the multiple second objects including deletion marks.

[0184] The metadata layer 22 is also used to store the metadata of multiple second objects in the persistence layer; if it is determined that the metadata of at least one second object fails to be stored in the persistence layer, a second indication message is sent to the service layer, and the second indication message is used to indicate that garbage collection of the data of multiple second objects is abandoned.

[0185] In some possible implementations, the service layer 21 is further configured to store the first mapping relationship in the partition cache.

[0186] In some possible implementations, the service layer 21 is configured to query the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition keys of the multiple first objects.

[0187] In some possible implementations, the service layer 21 is further configured to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer when a query of the first mapping relationship stored in the partition cache fails to hit the partition key, so as to determine the partition keys of the multiple first objects.

[0188] The service layer 21, metadata layer 22, and persistence layer 23 can all be implemented in software or hardware. For example, the implementation of the service layer 21 will be described below using the service layer 21 as an example. Similarly, the implementation of the metadata layer 22 and persistence layer 23 can refer to the implementation of the service layer 21.

[0189] As an example of a software functional unit, the service layer 21 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the service layer 21 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple geographically close data centers. Generally, a region may include multiple AZs.

[0190] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0191] As an example of a hardware functional unit, a module can include at least one computing device, such as a server. Alternatively, service layer 21 can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD can be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0192] The multiple computing devices included in service layer 21 can be distributed in the same region or in different regions. They can also be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in service layer 21 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0193] It should be noted that, in other embodiments, the service layer 21 can be used to execute any step in the data processing method, the metadata layer 22 can be used to execute any step in the data processing method, and the persistence layer 23 can be used to execute any step in the data processing method. The steps that the service layer 21, metadata layer 22 and persistence layer 23 are responsible for implementing can be specified as needed. The full functions of the object storage system 200 are realized by respectively implementing different steps in the data processing method through the service layer 21, metadata layer 22 and persistence layer 23.

[0194] Figure 13 shows a structural diagram of a computing device, on which the above-mentioned object storage system 200 can be deployed. The computing device can be a computing device in a cloud environment (such as a server), or a computing device in an edge environment, or a terminal device, etc., which can be specifically used to implement the functions of the service layer 21, metadata layer 22 and persistence layer 23 in the object storage system 200 in the embodiments shown in Figures 6, 8 to 11 above.

[0195] As shown in Figure 13, computing device 100 includes a bus 102, a processor 104, a memory 106, a storage 107, and a communication interface 108. Processor 104, memory 106, storage 107, and communication interface 108 communicate with each other via bus 102. Computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0196] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG13 shows only one line, but this does not imply a single bus or type of bus. Bus 102 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0197] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0198] The memory 106 may include a volatile memory, such as a random access memory (RAM).

[0199] The memory 107 may include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0200] Executable program codes are stored in memory 106 and memory 107. Processor 104 executes the executable program codes to implement the functions of the aforementioned service layer 21, metadata layer 22, and persistence layer 23, respectively, thereby implementing the data processing method. In other words, memory 106 stores instructions for executing the data processing method.

[0201] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0202] Specifically, when implementing the embodiments shown in Figures 6 and 8 to 11, and when the object storage system 200 described in the embodiments shown in Figures 6 and 8 to 11 is implemented by software, the software or program code required to execute the functions of the object storage system 200 in Figures 6 and 8 to 11 is stored in the memory 106 and the memory 107. The interaction between the object storage system 200 and other devices is implemented through the communication interface 103. The processor 104 is used to execute instructions in the memory 106 and the memory 107 to implement the method executed by the object storage system 200.

[0203] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0204] As shown in Figure 14, the computing device cluster includes at least one computing device 100. The memory 106 and the memory 107 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0205] In some possible implementations, the memory 106 and the memory 107 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data processing method.

[0206] It should be noted that the memory 106 and memory 107 in different computing devices 100 in the computing device cluster can store different instructions, each used to execute a portion of the functions of the object storage system. In other words, the instructions stored in the memory 106 and memory 107 in different computing devices 100 can implement the functions of one or more modules in the service layer 21, metadata layer 22, and persistence layer 23.

[0207] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG15 shows a possible implementation. As shown in FIG15 , two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is made through a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the service layer 21. FIG15 takes the example of the memory 106 in the computing device 100A storing instructions for executing the functions of the service layer 21. At the same time, the memory 107 in the computing device 100B stores instructions for executing the functions of the metadata layer 22 and the persistence layer 23. FIG15 takes the example of the memory 107 in the computing device 100B storing instructions for executing the functions of the metadata layer 22 and the persistence layer 23.

[0208] It should be understood that the functions of the computing device 100A shown in FIG15 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0209] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the computer program product causes the at least one computing device to perform a data processing method or a data processing method.

[0210] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An object storage system, characterized in that: The object storage system is used to provide object storage services through at least one bucket, and the object storage system includes a service layer, a persistence layer, and a metadata layer, wherein: The service layer is configured to receive a batch upload request sent by a client, wherein the batch upload request indicates a plurality of first objects to be uploaded; send the data of the plurality of first objects to the persistence layer; generate metadata of the plurality of first objects, and send the metadata of the plurality of first objects to the metadata layer; The persistence layer is used to store data of the multiple first objects; The metadata layer is used to store the metadata of the multiple first objects in the persistence layer; when it is determined that the metadata of at least one first object fails to be stored in the persistence layer, a first indication message is sent to the service layer, where the first indication message is used to indicate garbage collection of the multiple first objects.

2. The object storage system according to claim 1, characterized in that: The service layer is further used to determine a partition key of the plurality of first objects, where the partition key is used to indicate a partition of the metadata of the plurality of first objects; The metadata layer is specifically configured to store the metadata of the multiple first objects in the persistence layer according to the partition keys of the multiple first objects.

3. The object storage system according to claim 1 or 2, characterized in that: The metadata layer is used to store metadata of first objects belonging to the same partition to the persistence layer through a pre-write log; and store metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

4. The object storage system according to claim 2 or 3, characterized in that: The service layer is used to determine the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong.

5. The object storage system according to claim 4, characterized in that: The service layer is further configured to receive a bucket creation request sent by the client before determining the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong, wherein the bucket creation request includes a bucket name of a target bucket and a target partition key rule; and send a first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; The metadata layer is also used to store the first mapping relationship.

6. The object storage system according to any one of claims 1 to 5, characterized in that: The service layer is further configured to receive a batch deletion request sent by the client, wherein the batch deletion request indicates a plurality of second objects to be deleted; and send metadata of the plurality of second objects to the metadata layer, wherein the metadata of the plurality of second objects includes a deletion mark; The metadata layer is further used to store the metadata of the plurality of second objects in the persistence layer; It is determined that the metadata of at least one second object fails to be stored in the persistence layer, and second indication information is sent to the service layer, where the second indication information is used to indicate that garbage collection of data of the multiple second objects is abandoned.

7. The object storage system according to claim 5, characterized in that: The service layer also includes a partition cache, The service layer is further used to store the first mapping relationship in the partition cache.

8. The object storage system according to any one of claims 1 or 8, characterized in that: The service layer is used to query the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition key of the multiple first objects.

9. The object storage system according to claim 8, characterized in that: The service layer is further used to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects when the first mapping relationship query stored in the partition cache does not hit.

10. A data processing method, characterized in that: The method is applied to an object storage system, the object storage system is used to provide an object storage service through at least one bucket, the object storage system includes a service layer, a persistence layer and a metadata layer, and the method includes: The service layer receives a batch upload request sent by a client, where the batch upload request indicates a plurality of first objects to be uploaded; The service layer sends the data of the plurality of first objects to the persistence layer; The persistence layer stores data of the plurality of first objects; The service layer generates metadata of the plurality of first objects, and sends the metadata of the plurality of first objects to the metadata layer; The metadata layer stores the metadata of the plurality of first objects in the persistence layer; The metadata layer determines that metadata of at least one first object fails to be stored in the persistence layer, and sends first indication information to the service layer, where the first indication information is used to indicate garbage collection of the multiple first objects.

11. The method according to claim 10, characterized in that The method further comprises: The service layer determines a partition key of the plurality of first objects, the partition key being used to indicate a partition of metadata of the plurality of first objects.

12. The method according to claim 10 or 11, characterized in that: The metadata layer stores the metadata of the plurality of first objects in the persistence layer, including: The metadata layer stores the metadata of the first object belonging to the same partition to the persistence layer through a write-ahead log; The metadata layer stores metadata of first objects belonging to different partitions to the persistence layer through distributed transactions.

13. The method according to claim 11 or 12, characterized in that: The service layer determines the partition keys of the plurality of first objects, including: The service layer determines the partition keys of the multiple first objects according to the partition key rule of the bucket to which the multiple first objects belong.

14. The method according to claim 13, characterized in that The method further comprises: The service layer receives a bucket creation request sent by the client before determining the partition keys of the multiple first objects according to the partition key rule of the bucket to which the multiple first objects belong, wherein the bucket creation request includes a bucket name of a target bucket and a target partition key rule; The service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; The metadata layer stores the first mapping relationship.

15. The method according to any one of claims 10 to 14, characterized in that The method further comprises: The service layer receives a batch deletion request sent by the client, where the batch deletion request indicates a plurality of second objects to be deleted; The service layer sends the metadata of the plurality of second objects to the metadata layer, wherein the metadata of the plurality of second objects includes a deletion mark; The metadata layer stores the metadata of the plurality of second objects in the persistence layer; The metadata layer determines that the metadata of at least one second object fails to be stored in the persistence layer, and sends second indication information to the service layer, where the second indication information is used to indicate that garbage collection of data of the multiple second objects is abandoned.

16. The method according to claim 14, characterized in that The service layer also includes a partition cache, and the method further includes: The service layer stores the first mapping relationship in the partition cache.

17. The method according to any one of claims 10 or 16, characterized in that: The service layer determines the partition keys of the plurality of first objects, including: The service layer queries the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determines the partition keys of the multiple first objects.

18. The object storage system according to claim 17, characterized in that: The method further comprises: When the service layer fails to query the first mapping relationship stored in the partition cache, the service layer queries the partition key rule of the bucket to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

19. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images