Object storage system, data processing method and related equipment

Abstract

The invention provides an object storage system, the object storage system comprises a service layer, a persistence layer and a metadata layer, the service layer is used for receiving a batch uploading request sent by a client, and the batch uploading request indicates a plurality of first objects to be uploaded; furthermore, the data of the plurality of first objects are sent to a persistence layer, metadata of the plurality of first objects are generated, and the metadata of the plurality of first objects are sent to a metadata layer; the persistence layer is used for storing data of a plurality of first objects; the metadata layer is used for storing the metadata of the plurality of first objects to the persistence layer and sending first indication information to the service layer when it is determined that the metadata of at least one first object fails to be stored to the persistence layer, and the first indication information is used for indicating garbage collection of the plurality of first objects. Therefore, in the object storage system, the atomicity of the multi-object batch operation is realized by ensuring that the plurality of object metadata are atomically stored in the persistence layer.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on August 28, 2023, with application number 202311092359.1, and the priority of the Chinese patent application with the invention name “A method, device and other equipment for object operation”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to an object storage system, a data processing method and related equipment. Background Art

[0003] Object Storage Service (OBS) is an object-based storage service that provides massive, secure, low-cost, and highly reliable unstructured storage services. There is no need to consider capacity limitations when using it, and it provides multiple storage types to meet the needs of various business scenarios.

[0004] Atomicity guarantees mean that one or more operations in a transaction are either all executed successfully or all executed failed, with no intermediate state. In object storage, a batch operation may contain multiple operations to upload or delete objects. Atomicity guarantees mean that these multiple operations are either all executed successfully or all executed failed.

[0005] However, in object storage, there is no atomicity guarantee for multiple operations of uploading or deleting objects. This lack of atomicity may lead to inconsistent user experience, such as a user successfully accessing metadata, but the corresponding data is not uploaded successfully, resulting in a failure to access the data. Summary of the invention

[0006] The present application provides an object storage system for implementing atomicity of multi-object batch operations by ensuring atomicity of metadata of multiple objects stored in a persistence layer in batch object operations. In addition, the present application also provides a data processing method, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0007] In a first aspect, the present application provides an object storage system, which is used to provide object storage services through at least one bucket, and the object storage system includes a service layer, a persistence layer and a metadata layer, wherein the service layer is used to receive a batch upload request sent by a client, the batch upload request indicates multiple first objects to be uploaded; and send the data of the multiple first objects to the persistence layer, and generate metadata of the multiple first objects, and send the metadata of the multiple first objects to the metadata layer; the persistence layer is used to store the data of the multiple first objects; the metadata layer is used to store the metadata of the multiple first objects to the persistence layer, and determine that the storage of the metadata of at least one first object to the persistence layer fails, and send a first indication information to the service layer, the first indication information is used to indicate garbage collection of the multiple first objects.

[0008] In this embodiment, after receiving a batch upload request, the service layer first stores the data of the multiple first objects indicated by the batch upload request to the persistence layer, and then the metadata layer atomically stores the metadata of the multiple first objects to the persistence layer. When the metadata of each of the multiple first objects is successfully stored in the persistence layer, the service layer returns to the client that the batch upload of objects is successful; if the metadata of at least one of the multiple first objects fails to be stored in the persistence layer, the service layer instructs the data of the multiple first objects to be garbage collected, that is, to fall back to the state where the multiple first objects are not uploaded. In this way, based on the separate writing of data and metadata to the persistence layer in the object storage system, the atomicity of batch uploading of multiple objects is achieved by ensuring that the metadata of multiple objects are atomically stored in the persistence layer.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the service layer is also used to determine the partition keys of the multiple first objects, and the partition keys are used to indicate the partitions of the metadata of the multiple first objects; the metadata layer is specifically used to store the metadata of the multiple first objects to the persistence layer according to the partition keys of the multiple first objects.

[0010] In this implementation, the metadata layer needs to determine the partition keys of the multiple first objects before atomically storing the metadata of the multiple first objects in the persistence layer. After determining the partition keys of the multiple first objects to obtain the partitions of the metadata of the multiple first objects, different atomicity strategies are used to store the metadata of the multiple first objects in the persistence layer to ensure the atomicity of the batch upload objects.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the metadata layer is used to store metadata of first objects belonging to the same partition to the persistence layer through a write-ahead log; and to store metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0012] In this implementation, in the batch upload object operation, the object metadata of the same partition is guaranteed to be atomic in batches through a pre-write log. For the writing of object metadata of different partitions, each partition in the different partitions participates in the distributed transaction as a whole, realizing the atomic operation of batch objects and greatly reducing the number of distributed transaction participants, which helps to stabilize the performance of distributed transactions.

[0013] In combination with the first aspect, in some implementations of the first aspect, the service layer is used to determine the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong.

[0014] In this implementation, the user can customize the partition key rule when creating a bucket, so that the user can use the partition key rule to divide the distribution relationship of objects.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the service layer is also used to receive a bucket creation request sent by a client before determining the partition keys of multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong, wherein the bucket creation request includes the bucket name of the target bucket and the target partition key rule, and sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; the metadata layer is also used to store the first mapping relationship.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the service layer is further used to receive a batch deletion request sent by a client, the batch deletion request indicates multiple second objects to be deleted, and the metadata of the multiple second objects is sent to the metadata layer, the metadata of the multiple second objects includes a deletion mark; the metadata layer is also used to store the metadata of the multiple second objects to the persistence layer, and determine that the storage of the metadata of at least one second object to the persistence layer fails, and send a second indication information to the service layer, the second indication information is used to indicate to abandon garbage collection of the data of the multiple second objects.

[0017] In this implementation, after receiving a batch deletion request, the service layer atomically stores the metadata of multiple second objects indicated by the batch deletion request to the persistence layer. When the metadata of each of the multiple second objects is successfully stored in the persistence layer, the service layer returns a message to the client indicating that the batch deletion of the objects is successful. If the metadata of at least one of the multiple second objects fails to be stored in the persistence layer, the service layer indicates that the data of the multiple first objects should be abandoned for garbage collection. In this way, based on the separate writing of data and metadata to the persistence layer in the object storage system, the atomicity of batch deletion of multiple objects is achieved by ensuring that the metadata of multiple objects are atomically stored in the persistence layer.

[0018] In combination with the first aspect, in some implementations of the first aspect, the service layer is further used to store the first mapping relationship in the partition cache.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the service layer is used to query the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition keys of the multiple first objects.

[0020] In this implementation, the service layer stores the partition key rule in the partition cache to avoid accessing the bucket metadata for each partition key read request, thereby reducing the access overhead to the metadata layer to improve the partition key reading performance.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the service layer is also used to query the partition key rules of the buckets to which multiple first objects belong in the metadata layer when a query on the first mapping relationship stored in the partition cache misses, so as to determine the partition keys of the multiple first objects.

[0022] In a second aspect, a data processing method is provided, which is applied to an object storage system, wherein the object storage system is used to provide object storage services through at least one bucket, and the object storage system includes a service layer, a persistence layer and a metadata layer, and the method includes: the service layer receives a batch upload request sent by a client, and the batch upload request indicates multiple first objects to be uploaded; the service layer sends the data of the multiple first objects to the persistence layer; the persistence layer stores the data of the multiple first objects; the service layer generates metadata for the multiple first objects and sends the metadata for the multiple first objects to the metadata layer; the metadata layer stores the metadata for the multiple first objects in the persistence layer; the metadata layer determines that the storage of metadata of at least one first object in the persistence layer fails, and sends first indication information to the service layer, and the first indication information is used to indicate garbage collection of the multiple first objects.

[0023] In combination with the second aspect, in some implementations of the second aspect, the service layer determines a partition key of the plurality of first objects, where the partition key is used to indicate a partition of metadata of the plurality of first objects.

[0024] In combination with the second aspect, in certain implementations of the second aspect, the metadata layer stores metadata of first objects belonging to the same partition to the persistence layer through a write-ahead log; the metadata layer stores metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0025] In combination with the second aspect, in some implementations of the second aspect, the service layer determines the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong.

[0026] In combination with the second aspect, in certain implementations of the second aspect, before determining the partition keys of multiple first objects based on the partition key rules of the buckets to which the multiple first objects belong, the service layer receives a bucket creation request sent by the client, wherein the bucket creation request includes the bucket name of the target bucket and the target partition key rule; the service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; and the metadata layer stores the first mapping relationship.

[0027] In combination with the second aspect, in certain implementations of the second aspect, the service layer receives a batch deletion request sent by the client, and the batch deletion request indicates multiple second objects to be deleted; the service layer sends metadata of the multiple second objects to the metadata layer, and the metadata of the multiple second objects includes deletion marks; the metadata layer stores the metadata of the multiple second objects to the persistence layer; the metadata layer determines that the storage of metadata of at least one second object to the persistence layer fails, and sends second indication information to the service layer, and the second indication information is used to indicate that garbage collection of data of the multiple second objects is abandoned.

[0028] In combination with the second aspect, in certain implementations of the second aspect, the service layer stores the first mapping relationship in the partition cache.

[0029] In combination with the second aspect, in certain implementations of the second aspect, the service layer queries the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache to determine the partition keys of the multiple first objects.

[0030] In combination with the second aspect, in certain implementations of the second aspect, when the service layer fails to hit the first mapping relationship query stored in the partition cache, the partition key rules of the buckets to which the multiple first objects belong are queried in the metadata layer to determine the partition keys of the multiple first objects.

[0031] Regarding the technical principles and beneficial effects of the second aspect, reference may be made to the relevant description of the first aspect, which will not be repeated here.

[0032] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method of the above-mentioned second aspect or any one of the implementation methods of the second aspect.

[0033] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computer device cluster, enables the computer device cluster to execute the method of the second aspect or any one of the implementation modes of the second aspect.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method of the above-mentioned second aspect or any one of the implementation modes of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0036] Figure 1 This is a scenario diagram of an object storage system involved in this application.

[0037] Figure 2 This is a schematic diagram of the architecture of an object storage system involved in this application.

[0038] Figure 3 It is a structural diagram of a range partition involved in this application.

[0039] Figure 4 This is a hardware structure diagram of an object storage system involved in this application.

[0040] Figure 5 This is a scenario diagram of an object storage system provided by this application.

[0041] Figure 6 This is one of the flow charts of a data processing method provided by this application.

[0042] Figure 7 It is a schematic diagram of a batch operation configuration interface provided by this application.

[0043] Figure 8 This is one of the flow charts of a data processing method provided by this application.

[0044] Fig. 9 This is a flow chart of a distributed transaction provided by this application.

[0045] Fig.10 This is one of the flow charts of a data processing method provided by this application.

[0046] Fig.11 This is one of the flow charts of a data processing method provided by this application.

[0047] Fig.12 It is a structural diagram of an object storage system provided by this application.

[0048] Fig.13 It is a structural schematic diagram of a computing device provided by this application.

[0049] Fig.14 It is a structural diagram of a computing device cluster provided by this application.

[0050] Fig.15 It is a structural diagram of a network connection between computing devices provided by the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0052] In the description of the embodiments of the present application, unless otherwise specified, the term "and / or" herein is a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " herein indicates that the associated objects are in an or relationship, for example, A / B means A or B. A and B may be singular or plural.

[0053] In the description of the embodiments of the present application, unless otherwise specified, the term "multiple" herein refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0054] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0055] Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. In order to make the technical solution provided by the present application clearer, the relevant terms are first explained before the technical solution provided by the present application is specifically described.

[0057] (1) Object storage service (OBS): It stores data in the form of objects in object storage nodes. Object storage nodes provide a flat storage method based on buckets and objects. All objects in a bucket are at the same logical level, eliminating the multi-level tree directory structure in the file storage service node. The object storage system is used to provide users with operations that conform to object semantics (i.e., object services), such as: uploading objects, downloading objects, listing each object in a bucket, copying objects, moving objects, renaming objects, deleting objects, deleting each object in a bucket, deleting specified multiple objects, etc. Among them, the object storage node provides an object interface, and the object interface can indicate operations that conform to object semantics. For example, the object interface can be a representational state transfer (REST) ​​application programming interface (API), and users can access data on the object storage service node through the REST API interface. It should be noted that when users access data through the object interface, the data can be called an object.

[0058] (2) Object: An object is the basic unit of data storage in an object storage system. An object is actually a collection of a file's data and its related attribute information (metadata). The data uploaded by tenants to OBS is stored in buckets in the form of objects. Objects consist of three parts: key, metadata, and data. The key, or object name, is a UTF-8-encoded character sequence with a length greater than 0 and less than 1024. Each object in a bucket has a unique object key.

[0059] (3) Bucket: A bucket is a container for storing objects in OBS. Object storage provides a flat storage method based on buckets and objects. All objects in a bucket are at the same logical level, eliminating the multi-level tree directory structure in the file system. Each bucket has its own storage category, access rights, region, and other attributes. Tenants can create buckets with different storage categories and access rights, and configure more advanced attributes to meet the storage requirements of different scenarios.

[0060] (4) Object storage device (OSD): OSD is the basic storage unit of the object storage system. It is set on the physical disk and is specifically a fixed-size storage space of the physical disk. The object storage system manages the physical disks of multiple computing devices in the form of OSD.

[0061] (5) Range partition: For clusters storing metadata, in order to adapt to the growing number of metadata entries, partitioning technology is usually used to dynamically split metadata. Each partition manages different data entries. Each partition is served by a server, and the relationship between partitions and servers is dynamically specified by the cluster management of the distributed metadata storage system. In partitioning technology, the value of a certain attribute of metadata, or the sequential combination of several attributes, is usually used to determine the partition. This value is called the partition key (also called partition key or shard key).

[0062] In the object storage service, tenants can log in to the cloud management platform on the public cloud access page with a pre-registered account and password, and after successful login, select and purchase the corresponding public cloud services on the public cloud access page, such as OBS, virtual machine services, container services, etc. For OBS, tenants can further create multiple buckets through the configuration interface or API on the public cloud access page provided by the cloud platform. There is no limit on the total number and size of objects stored in each bucket. Tenants do not need to consider the scalability of data. OBS is a service based on REST API and hypertext transfer protocol (HTTP) or hypertext transfer protocol secure (HTTPS). Tenants can locate bucket resources through uniform resource identifiers (URIs), which are also referred to as access domain names in this application, or domain names for short. Among them, in OBS, bucket names are globally unique and cannot be modified, that is, buckets created by tenants cannot have the same name as other buckets they have created, nor can they have the same name as buckets created by other tenants.

[0063] Figure 1 A schematic diagram showing at least one application scenario in an embodiment of the present application is shown. Figure 1As shown, each bucket can include multiple objects, and the objects between buckets are isolated from each other. The tenant logs in to the cloud management platform 10 through the client, selects and purchases the cloud service of the object storage service on the cloud management platform 10. After the purchase, the tenant can store objects based on the object storage service provided by the object storage system. Among them, the cloud management platform 10 is mainly used to manage the infrastructure for running the object storage service. Exemplarily, the infrastructure for running the object storage service may include multiple data centers set up in different regions, and each data center includes multiple servers. The data center can provide basic resources for the object storage service, such as computing resources, storage resources, etc. Therefore, when the tenant purchases and uses the object storage service, it mainly pays for the resources used. Specifically, the object storage service provides the domain name of the bucket, and the tenant can access the domain name of the bucket through the client to upload data to the bucket or download data from the bucket, wherein the uploaded data is stored in the bucket in the form of objects.

[0064] When a tenant uploads an object, he or she can specify the storage category of the object. If not specified, the default is the same as the storage category of the bucket. After uploading, the storage category of the object can be modified, and the bucket can be accessed through the client. The client can be a browser used locally by the tenant or a dedicated client provided by the cloud platform. For example, a tenant can access a bucket through a browser set up in a local computer. After entering the account and passing the verification, the tenant can use the browser to upload objects to the bucket, or modify or delete objects in the bucket, etc., where the local computer is connected to the Internet.

[0065] Combined with the following Figure 2 The system architecture applicable to the embodiment of the present application is described. The object storage system 200 can provide object storage services for the client 20. Figure 2 The object storage system 200 includes a service layer 21, a metadata layer 22 and a persistence layer 23.

[0066] Among them, the service layer 21 is used to interact with the client 20 and provide foreground services to the outside. The foreground services can be services for the client 20 to request to read or write data. The client 20 can be deployed on a terminal device, and multiple applications can also be deployed on the terminal device. Taking application 201 as an example, application 201 can be deployed in the user's terminal device. For example, the terminal device can be a mobile phone, a laptop, a tablet computer, a PDA, a wireless terminal in a smart city, and a wireless device in a smart home. This application does not limit the specific type of terminal device. The client 20 can interact with application 201. For example, application 201 can send an object upload request to the client 20 under the triggering of the tenant. After receiving the object upload request, the client 20 can send an object upload request to application 201 according to the object upload request.

[0067] Tenants can use their accounts on client 20 to log in to the cloud management platform, create buckets, configure bucket names, and obtain bucket domain names through the cloud management platform. After detecting tenant operations (such as creating buckets, configuring bucket names, etc.), the cloud management platform can send creation instructions to the service layer 21 through the switching device. The creation instructions include information such as the bucket name and bucket domain name, and notify the object storage service node to create a bucket and save the bucket name, bucket domain name, and other information. After creating a bucket, the tenant can also access the bucket domain name by operating the client 20, locate the bucket managed by the service layer 21, and upload data (such as uploading objects) and download data (such as downloading objects) in the bucket.

[0068] The following is specific to the service layer 21, which includes a gateway 211, a metadata interface 212, and a persistence layer interface 213. Among them, the gateway 211 is responsible for the access, authentication, authorization and protocol processing of the object storage. The metadata interface 212 is the interface of the metadata layer 22. Through the metadata interface 212, the metadata corresponding to the data written by the tenant can be written into the metadata layer 22, and the metadata is finally persisted in the persistence layer 23. In the persistence layer 23, the metadata is generally stored in the form of copies. By storing multiple copies of the data (generally three copies), when a copy is lost, it can be copied from other copies. The persistence layer interface 213 is the interface of the persistence layer 23. Through the persistence layer interface 213, the data that the tenant needs to write can be written into the persistence layer 23. In the persistence layer 23, the data is generally stored in the form of erasure coding (EC). Redundancy is obtained by encoding the original data, and the data and redundancy are stored together in the persistence layer 23 to achieve the purpose of fault tolerance.

[0069] Object metadata is a distributed storage engine based on range partition, and is generally sorted in lexicographic order. The metadata layer 22 includes bucket metadata 221, object metadata 222, and persistence interface 223. For clusters storing metadata, in order to adapt to the growing demand for the number of metadata entries, partitioning technology is usually used to dynamically split the metadata, and each partition (also called partition) after the split manages different data entries. Among them, each partition is served by a certain server, and the ownership relationship between the partition and the server is dynamically specified by the cluster management of the distributed metadata storage system. Through dynamic partitioning and dynamic partition ownership, dynamic horizontal expansion of the distributed metadata storage system is achieved. In partitioning technology, the value of a certain attribute of metadata, or the sequential combination value of several attributes, or a custom rule is usually used to determine the partition. This value can be called a partition key. The rule is stored in the bucket metadata 221. When adding, deleting, modifying, and checking, the partition where the data entry is located is uniquely determined by querying the partition key of each data entry in the range partition. In addition, the bucket metadata 221 also records the metadata information of the bucket, such as the region to which the bucket belongs. Object metadata 222 records the metadata of the object, such as the object's attributes, size, upload time, policy information, etc. Object metadata 222 also includes a write ahead log (WAL). The write ahead log requires that all modification operations of the storage system must be written to the log before submission, which can ensure the atomicity and persistence of the object storage system. In the case that the hard disk data is not damaged, the write ahead log allows the storage system to be restored to the state before the crash under the guidance of the log after a crash, avoiding data loss. This technology is widely used to store metadata information in (file, object, database, column) storage systems.

[0070] In order to ensure that the metadata of the object can be arranged in the natural order of the partition key, the distributed storage system usually adopts range partitioning technology. Among them, in the partitioning technology, the value of a column of the data table, or the sequential combination of several column values, is usually used to determine the partition. As shown in the user metadata table in Table 1, the combination of "bucket name", "object name" and "object version number" in the metadata table uniquely identifies an object, that is, the object identifier is {bucket name, object name, object version number}. The combination of the three column domains {Bn, On, ver} uniquely identifies an object. The object entry key value = {Bn, On, ver}. In order to adapt to the continuous growth of the object data scale, the combination of the column domain {Bn, On} prefix of the primary index entry key value is usually used as the partition key, which is convenient for indexing partitions according to the primary index entry key value. Although each object entry has a partition key, the partition key is not required to be unique. In the example in Table 1:

[0071] The partition key of the first row of data entry = {Bn = "A", On = "0109"};

[0072] The partition key of the data entry in row 2 = {Bn = "A", On = "0109"}, the partition key is the same as the first row, but the entry key value is different;

[0073] The partition key of the data entry in row 3 = {Bn = "B", On = "0201"}.

[0074] Table 1: User metadata table

[0075] Bucket Name (Bn) Object Name,On Version (Version,ver) "A” "0019” 1 "B” "0019” 2 "C” "0201” 1

[0076] The partition key of the second row of data entry in Table 1 is the same as the partition key of the first row of data entry, so the metadata represented by the first row of data entry and the metadata represented by the second row of data entry belong to the same partition. However, the identifier of the object represented by the first row of data entry and the identifier of the object represented by the second row of data entry are different.

[0077] For metadata at the partition boundary, the partition to which the metadata belongs is determined using the principle of left-closed and right-open or left-open and right-closed. Figure 3 The structure of range partition is shown. In Table 1, Bn represents the name of the bucket, and On represents the name of the object. For metadata at the boundary of the partition, if the left-closed and right-opened principle is used to determine the partition to which the metadata belongs, the value range of the partition key of partition 1 is: ({Bn: Min; on: min}, {Bn: "A"; on: "w"}], and the value range of the partition key of partition 2 is: ({Bn: "A"; on: "w"}, {Bn: "C"; on: "x"}].

[0078] In the persistence layer 23, the object storage device OSD control unit sets the physical disk as multiple object storage devices OSD. This application only takes OSD as the basic storage unit as an example. All storage spaces that can be used as basic storage units are applicable to this application. After receiving the creation instruction of bucket 1, the cloud management platform creates bucket 1 and creates bucket 1 on physical disk 1 and physical disk 2. The OSD of bucket 1 is distributed in physical disk 1 and physical disk 2, and the bucket name, bucket domain name and other information of bucket 1 are saved.

[0079] Exemplarily, the service layer 21, metadata layer 22 and persistence layer 23 may be implemented by software or hardware.

[0080] When implemented by hardware, the service layer 21, the metadata layer 22 and the persistence layer 23 can all be implemented by at least one computing device, such as a server. Figure 4, the hardware structure of the object storage system 200 of the present application is introduced. When the service layer 21, metadata layer 22 and persistence layer 23 are implemented by a server, the service layer 21 is deployed on the service node 41, the metadata layer 22 is deployed on the metadata node 42, and the persistence layer 23 is deployed on the storage node 43. The hardware structure of the service node 41 is introduced below. Figure 4 The various components shown may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits. Figure 4 As shown, the service node 41 may include: a processor 411, a memory 412, and a communication interface 413. The processor 411, the memory 412, and the communication interface 413 may be connected to each other via a bus 414, or in other ways.

[0081] Among them, processor 411 is the control center of the service node. Processor 411 can be a general-purpose central processing unit (CPU) or other general-purpose processors, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc. For example, processor 411 may include an application processor (AP), a graphics processor (GPU), an image signal processor (ISP), a controller, etc. The controller in processor 411 is the nerve center and command center of the service node. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution. Optionally, a memory can also be set in processor 411 for storing instructions and data. Exemplarily, processor 411 may include one or more CPUs, such as Figure 4 CPU1 and CPU2 are shown in FIG.

[0082] The memory 412 includes, but is not limited to, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical memory, or any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In the embodiment of the present application, the memory 412 can store information such as computer instructions.

[0083] The communication interface 413 may be a transceiver module, which is used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. The communication interface 413 may receive instructions, messages or data, etc. The transceiver module may be a device such as a transceiver or a transceiver. Optionally, the communication interface 413 may also be a transceiver circuit located in the processor 201, which is used to implement signal input and signal output of the processor. The communication interface 413 may be a wired interface (port), such as a fiber distributed data interface (FDDI), a gigabit Ethernet (GE) interface, or the communication interface 413 may also be a wireless interface.

[0084] The bus 414 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] In addition, the storage node 43 may include a memory 431. The memory 431 may include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0086] It should be noted that Figure 4 The service node 41 shown is only an example of a service node, and the service node may have Figure 4 More or fewer components shown in the figure may be combined with two or more components, or may have different component configurations. In actual application, the object storage system 200 may also be applicable to other scenarios. For example, in other possible application scenarios, the object storage system 200 may provide object storage services for a larger number of clients, or provide object storage services for other types of clients. Alternatively, the object storage system 200 may also include devices with other functions to support this embodiment, which is not limited thereto.

[0087] In standard object storage, the object's application programming interface (API) uses a single object as the operation granularity, such as uploading an object, deleting an object, and a single object can achieve atomicity guarantee. For the write operation of a single object, an operation of a tenant is divided into two operations inside the storage system: data writing and metadata writing. The result of the tenant's write operation is determined by whether the metadata is written successfully. After the tenant's specific data is written to the persistence layer 21, the generated metadata is processed by the metadata layer 22. After receiving the request, the metadata layer 22 records the metadata operation and metadata in the WAL log. After the write is successful, the operation result is returned to the service layer 21. After receiving the response that the metadata is written successfully, the service layer 21 returns the final operation result to the user. Therefore, the result of the user's write operation is usually determined by whether the metadata is written successfully.

[0088] However, in actual applications, tenants may have demands for batch operations on multiple objects. In object storage, there is no atomicity guarantee for batch operations of uploading or deleting multiple objects. This lack of atomicity may lead to inconsistent user experience, such as a user successfully accessing metadata, but failing to access data because the corresponding data was not uploaded successfully.

[0089] To facilitate understanding, at least one application scenario of an embodiment of the present application is first introduced below.

[0090] For example, Figure 5 What is shown is a schematic diagram of at least one application scenario in an embodiment of the present application. Figure 5 As shown, the tenant uses an account to log in to the cloud management platform on the client 20, and uploads multiple high-definition picture objects to bucket 1 in batches through the cloud management platform. A single picture object and the description information object of the picture object are written twice into bucket 1. Alternatively, thumbnail objects of different resolutions for the same picture object need to be written into bucket 1 separately. However, due to the lack of atomicity guarantee for multi-object batch operations in the object storage system 200, taking the case where the picture object and the description information object of the picture object are written twice into bucket 1 as an example, if the description information of the picture object is written successfully, but the picture object fails to be written, the tenant can only access the description information of the picture object when accessing it, but cannot access the picture object, resulting in tenant access failure and poor tenant experience.

[0091] In order to solve the problem of no atomicity guarantee for batch operations on multiple objects, an embodiment of the present application proposes a data processing method, based on the object storage system, data and metadata are written separately, and the atomicity of batch operations on multiple objects is achieved through the atomic writing guarantee of metadata of multiple objects.

[0092] Next, based on the above description, a data processing method provided by an embodiment of the present application is introduced. It can be understood that the method is proposed based on the above description, and part or all of the contents of the method can be referred to the above description.

[0093] For example, Figure 6 FIG. 1 is a flow chart of a data processing method provided by an embodiment of the present application. It is understood that the method can be implemented in any suitable device, equipment, platform or device cluster with computing, processing and storage capabilities. Figure 6 As shown, the method may include steps S601 to S606.

[0094] Step S601: the service layer receives a batch upload request sent by the client.

[0095] In this embodiment, the service layer of the object storage system receives a batch upload request from a client, the batch upload request is used to indicate multiple first objects to be uploaded, wherein the batch upload request carries the multiple first objects to be uploaded. The service layer of the object storage system may be the service layer 21, the client may be the client 20, the persistence layer may be the persistence layer 23, and the metadata layer may be the metadata layer 22, which is also used as an example for explanation below.

[0096] Optionally, the gateway 211 of the service layer 21 receives the batch upload request and sends the multiple first objects to be uploaded carried in the batch upload request to the persistence interface 212 , and the persistence interface 212 is responsible for writing the data of the multiple first objects into the persistence layer 23 .

[0097] Alternatively, if Figure 7 As shown, the client provides a batch operation configuration interface 700, which is used to receive the user's batch operation configuration instructions for the object, so that the service layer performs batch operation processing and atomicity configuration based on the batch operation configuration instructions. The items that can be configured by the user in the batch operation configuration interface 700 include batch processing operations, storage categories, atomicity, uploaded objects, etc. Among them, the storage category can be divided into standard storage, low-frequency access storage, and archive storage, which are not specifically limited in the embodiments of the present application. In this embodiment, the user can choose to upload objects in batches. When selecting objects to upload in batches, the user can turn on or off atomicity to set whether the objects to upload in batches need to be guaranteed by atomicity.

[0098] Step S602: the service layer sends data of multiple first objects to the persistence layer.

[0099] In this embodiment, after receiving the batch upload request from the client, the service layer sends the data of the multiple objects to the persistence layer in response to the batch upload request. The object includes three parts: key value, metadata and data, where the data is the data content of the file. After receiving the batch upload request, the service layer first sends the data of the multiple first objects to the persistence layer.

[0100] Step S603: the persistence layer stores data of multiple first objects.

[0101] In this embodiment, after receiving the data of the multiple first objects, the persistence layer stores the data of the multiple first objects.

[0102] Step S604: the service layer generates metadata for multiple first objects.

[0103] In this embodiment, after the persistence layer stores the data of the multiple first objects, it sends the storage address of the data of the multiple first objects to the service layer, and the service layer generates metadata of the multiple first objects according to the storage address of the data and the object names of the multiple first objects.

[0104] Specifically, the metadata of an object is the description information of the object, including system metadata and user metadata. The system metadata is automatically generated by the object storage system and is used when processing object data. User metadata is specified by the user when uploading objects in batches. It is user-defined object description information, which may specifically include object attributes, upload time, policy information, storage address, etc., which are not specifically limited in the embodiments of the present application. The user turns on atomicity in the batch operation configuration interface 700. The configuration of batch upload objects requires atomicity guarantee. The metadata layer atomically writes the metadata of multiple first objects to the persistence layer according to the configuration information.

[0105] Step S605: the service layer sends metadata of multiple first objects.

[0106] In this embodiment, the service layer generates metadata of the multiple first objects according to the storage address of the storage data and the object names of the multiple first objects, and then sends the metadata of the multiple first objects to the metadata layer.

[0107] Step S606: The metadata layer stores metadata of the plurality of first objects in the persistence layer.

[0108] In this embodiment, after receiving the metadata of the plurality of first objects, the metadata layer stores the metadata of the plurality of first objects in the persistence layer.

[0109] Step S607: The metadata layer determines that the metadata of at least one first object fails to be stored in the persistence layer, and sends first indication information to the service layer.

[0110] In this embodiment, the metadata layer determines that the metadata of at least one first object has failed to be stored in the persistence layer, and sends a first indication message to the service layer, where the first indication message is used to indicate that the multiple first objects are to be garbage collected. That is, the data of the multiple first objects that have been stored in the persistence layer are garbage collected, and the multiple first objects are rolled back to a state where the multiple first objects have not been uploaded. If the metadata layer determines that the metadata of at least one first object has been successfully stored in the persistence layer, the persistence layer sends the metadata persistence address to the metadata layer, and the metadata layer records the metadata persistence address and sends it to the service layer, which returns to the client that the batch upload of objects is successful.

[0111] Thus, in this embodiment, based on the separate writing of data and metadata into the persistence layer in the object storage system, the atomicity of batch uploading of multiple objects is achieved by ensuring the atomicity of metadata of multiple objects stored in the persistence layer.

[0112] In some possible implementations, such as Figure 8 As shown, executing step S606 may specifically include steps S801 to S802.

[0113] In this implementation, before the metadata layer stores the metadata of the multiple first objects in the persistence layer, the partition keys of the multiple first objects are first determined. After the partition keys of the multiple first objects are determined to obtain the partitions of the metadata of the multiple first objects, different atomicity strategies are used to store the metadata of the multiple first objects in the persistence layer.

[0114] Specifically, in partitioning technology, the value of a certain attribute of metadata, or the sequential combination of several attributes, is usually used to determine the partition. This value is called a partition key. The partition where the metadata of each object is located is uniquely determined by querying the partition key of each object. In this implementation, the service layer determines the partition of the metadata of multiple first objects based on the partition keys of multiple first objects, sends the metadata of first objects belonging to the same partition to the partition, and the metadata layer of this partition atomically stores the metadata of multiple first objects to the persistence layer.

[0115] Step S801: The metadata layer stores metadata of a first object belonging to the same partition to the persistence layer through a write-ahead log.

[0116] In this implementation, after the service layer determines the partition keys of the multiple first objects, the metadata of the first objects belonging to the same partition is stored in the persistence layer through the pre-write log. The pre-write log of the metadata layer of each partition records the metadata operations and metadata records in the current partition, and then writes them into the cache. After the cache is full, it is persisted to the persistence layer.

[0117] Specifically, the write-ahead log of each partition is only responsible for recording the update operations of the metadata belonging to the current partition. For the metadata of the first object belonging to the same partition, the metadata of the first object of the partition is not written directly to the persistence layer, but is first written to the write-ahead log of the metadata layer. If at least one of the metadata of the first objects in the partition fails to be written to the write-ahead log, the record in the write-ahead log will be ignored; if the metadata of each of the first objects in the partition is successfully written to the write-ahead log, the metadata of the first object in the partition is stored in the persistence layer according to the records in the log.

[0118] Specifically, the write-ahead log includes a redo log and an undo log. When the metadata of at least one of the first objects in the partition fails to be written into the write-ahead log, the metadata is restored to the state before the multiple first objects are uploaded in batches based on the undo log, that is, roll-forward recovery is performed to ensure the atomicity of the metadata writing of the first objects in the partition.

[0119] Step S802: The metadata layer stores metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0120] In this implementation, after the service layer determines the partition keys of the multiple first objects, the metadata of the first objects belonging to different partitions is stored in the persistence layer based on the distributed transaction. Each partition in the different partitions participates in the distributed transaction as a transaction participant, so that the metadata of the first objects in the different partitions is stored in the persistence layer.

[0121] Optionally, the distributed transaction may be a two-phase transaction (two-phase commit protocol, 2PC), a three-phase transaction (three-phase commit protocol, 2PC) and a compensating transaction (try-confirm-cancel, TCC), which is not specifically limited in the embodiments of the present application.

[0122] In one example, a transaction is a reliable, independent unit of work consisting of a set of operations, in which all operations either succeed or fail. Transactions have four basic properties: atomicity, consistency, isolation, and durability, also referred to as the ACID properties of transactions. Distributed transactions include transaction managers, transaction coordinators, and transaction participants. In this embodiment, the service layer can be a transaction manager, which acts as a global scheduler and is responsible for deciding the commit and rollback of the entire distributed transaction. The target participating partition can be a transaction coordinator, and the transaction participants can be each participating partition, which is responsible for the commit and rollback of its own local transaction.

[0123] In one example, a two-phase transaction is used as an example. A two-phase transaction is a two-phase commit protocol, which divides the entire transaction process into two phases: prepare phase and commit phase. 2 refers to the two phases, P refers to the prepare phase, and C refers to the commit phase. In the prepare phase, the transaction manager sends a prepare message to each participant. Each transaction participant executes the transaction locally and writes the local undo / redo log. At this time, the transaction is not committed. In the commit phase, if the transaction manager receives a participant execution failure or timeout message, it directly sends a rollback message to each transaction participant; otherwise, it sends a commit message; the transaction participant commits or rolls back operations according to the execution of the transaction manager, and releases the lock resources used in the transaction processing process.

[0124] like Fig. 9 As shown, executing step S802 may specifically include steps S901 to S906, wherein steps S901 to S905 are stage one of the two-stage transaction, and steps S906 to S908 are stage two of the two-stage transaction.

[0125] Step S901: The service layer applies for a distributed transaction.

[0126] Specifically, the service layer applies for a distributed transaction, the distributed transaction is successfully created and a transaction identifier, also called a transaction ID, is generated. The transaction identifier is used to uniquely identify a distributed transaction, and the transaction identifier can be a unique number assigned to the distributed transaction, or other identifiers that can uniquely represent the distributed transaction.

[0127] Step S902: the service layer sends a batch upload object preparation operation to each participating partition.

[0128] Specifically, after the distributed transaction is successfully created, the service layer sends a batch upload object preparation operation to each partition of different partitions (also called participating partitions), that is, the transaction inquiry phase of phase one, in which the service layer sends the transaction content to each participating partition, inquiring whether each participating partition can perform the transaction submission operation, and starts waiting for the response of each participating partition. Among them, the transaction content is the metadata of the first object of each partition, and the metadata layer of each participating partition stores the metadata of the first object of the participating partition.

[0129] Step S903: Each participating partition locks the key value of the first object of the participating partition.

[0130] Specifically, after each participating partition receives the metadata of the first object of the participating partition, each participating partition locks the key value of the first object of the participating partition before executing the transaction to prevent concurrent reading and writing of the first object, which affects the execution process of subsequent transactions, and registers the transaction identifier locally. The transaction identifier is the same as the identifier generated when the distributed transaction is successfully created, and is used to uniquely identify the distributed transaction. The lock is used to indicate that the first object of the participating partition is currently being written.

[0131] Optionally, the lock may be a single bit, and when the bit is a bit value of 1, it indicates that the lock indicates that the first object of the participating partition is being written, and when the bit is a bit value of 0, it indicates that the lock indicates that the first object of the participating partition has been written. Alternatively, when the bit is a bit value of 1, it indicates that the lock indicates that the first object of the participating partition has been written, and when the bit is a bit value of 0, it indicates that the lock indicates that the first object of the participating partition is being written.

[0132] Step S904: the metadata layer of the participating partition stores the metadata of the first object belonging to the same partition to the persistence layer through the write-ahead log.

[0133] Specifically, after each participating partition locks the key value of the first object of the participating partition, the metadata layer of each participating partition writes the metadata of the first object of the participating partition into the pre-write log. The writing of the metadata of the first object in the participating partition is the same as step S801. The metadata layer stores the metadata of the first object belonging to the same partition to the persistence layer through the pre-write log, which will not be repeated here.

[0134] Step S905: Each participating partition feeds back a response to the batch upload object preparation operation.

[0135] Specifically, after each participating partition stores the metadata of the first object of the participating partition to the persistence layer, it feeds back a response to the batch upload preparation operation to the service layer. If the metadata of the first object in each participating partition is successfully stored in the persistence layer, feedback is given to the service layer to agree to submit the request, and the transaction is submitted in the second phase. If the metadata of the first object in at least one participating partition in each participating partition fails to be stored in the persistence layer and the submission request is not fed back to the service layer, the transaction is rolled back.

[0136] Step S906: Select a target participating partition from each participating partition and send a submission request.

[0137] Specifically, after receiving the responses of the preparation operation of batch uploading objects from each participating partition, the service layer selects a target participating partition from each participating partition as the transaction coordinator and sends a commit request.

[0138] Step S907: Each participating partition stores the write-ahead log of the metadata layer of the participating partition to the persistence layer and releases the lock.

[0139] Specifically, the target participating partition sends a prepare-to-commit request to other participating partitions. After receiving the prepare-to-commit request, each participating partition formally executes the transaction commit operation, that is, stores the pre-write log of the metadata layer of the participating partition to the persistence layer. After storing the pre-write log of the metadata layer of the participating partition to the persistence layer, the lock of the key value of the first object in the participating partition is released. In addition, other transaction resources occupied during the entire transaction are released.

[0140] Step S908: The target participating partition sends a confirmation request.

[0141] Specifically, after each participating partition completes the asynchronous transaction submission, the target participating partition sends a confirmation request to the service layer to complete the distributed transaction. At the same time, the service layer sends an end submission request to each participating partition and cancels the transaction identifier of the distributed transaction, that is, the metadata layer completes the storage of the metadata of the first object belonging to different partitions to the persistence layer through the distributed transaction.

[0142] Thus, in this embodiment, in the batch object upload operation, the object metadata of the same partition is guaranteed to be atomically operated in batches through a pre-write log, and for the writing of object metadata of different partitions, each partition in the different partitions participates in the distributed transaction as a whole, thereby realizing atomic operation of batch objects and greatly reducing the number of distributed transaction participants, which helps to stabilize the performance of distributed transactions.

[0143] In some possible implementations, such as Fig.10 As shown, the data processing method also includes steps S1001 to S1004.

[0144] Step S1001: The service layer receives a batch deletion request sent by a client.

[0145] In this embodiment, the service layer receives a batch deletion request from the client, where the batch deletion request is used to instruct deletion of multiple second objects, wherein the batch deletion request may carry the object names of the multiple second objects to be deleted.

[0146] Step S1002: the service layer sends metadata of multiple second objects to the metadata layer.

[0147] In this embodiment, after receiving the batch deletion request from the client, the service layer generates metadata of multiple second objects from the object names of the multiple second objects to be deleted in response to the batch deletion request, and sends the metadata of the second objects to the metadata layer. The metadata of the multiple second objects includes a delete marker, and the delete marker includes the key value of the second object to be deleted, which is used to indicate the deletion of the object.

[0148] Step S1003: the metadata layer stores metadata of the plurality of second objects in the persistence layer.

[0149] In this embodiment, after receiving the metadata of the second object, the metadata layer stores the metadata of multiple second objects in the persistence layer. Specifically, the service layer first determines the partition keys of the multiple second objects, and the metadata layer stores the metadata of the second objects belonging to the same partition in the persistence layer through the pre-write log, and the metadata layer stores the metadata of the second objects belonging to different partitions in the persistence layer through the distributed transaction. The execution of the specific steps is the same as step S801 and step S802, which will not be repeated here.

[0150] Step S1004: The metadata layer determines that the metadata of at least one second object fails to be stored in the persistence layer, and sends second indication information to the service layer.

[0151] In this embodiment, the metadata layer determines that the metadata of at least one second object has failed to be stored in the persistence layer, and sends a second indication message to the service layer. The second indication message is used to indicate that garbage collection of the data of the multiple second objects is abandoned, that is, the deletion of the multiple second objects is abandoned. If the metadata layer determines that the metadata of at least one second object is successfully stored in the persistence layer, the service layer returns a message to the client that the batch deletion of objects is successful.

[0152] Specifically, after the persistence layer stores the metadata of the multiple second objects, the multiple second objects are marked as unavailable in the persistence layer, but the data itself is not immediately deleted from the persistence layer. In a distributed storage system, garbage collection (GC) is usually used. The system periodically scans the storage device, marks the deleted or expired data as garbage data, and deletes it or moves it to the garbage collection area to recycle the garbage data. GC can be implemented through regular cleaning or triggered cleaning.

[0153] Thus, in this embodiment, based on the data and metadata in the object storage system being written separately to the persistence layer, the atomicity of batch deletion of multiple objects is achieved by ensuring the atomicity of metadata of multiple objects being stored in the persistence layer.

[0154] In some possible implementations, such as Fig.11 As shown, when the service layer determines the partition keys of the multiple first objects, it can specifically include steps S1101 to S1105.

[0155] Step S1101: The service layer receives a bucket creation request sent by a client.

[0156] In this implementation, users can customize partition key rules when creating a bucket. The service layer receives a bucket creation request sent by the client, which can carry the bucket name, region, and partition key rules. The partition key rule is configured by the user and is a rule predefined by the user. For example, it can be composed of a bucket name prefix and an object name prefix.

[0157] In an example, the partition key rule can be composed of the 3-character prefix of the bucket name and the 4-character prefix of the object name, such as {[{type:bucketprefix,len:3},{type:objprefix,len:4}]}, which means that the partition key is composed of the 3-character prefix of the bucket name and the 4-character prefix of the bucket name. After the partition key rule is configured, the metadata of different objects can be located in the same partition or in different partitions.

[0158] Optionally, the user can choose not to configure the partition key rule. After receiving the request to create a bucket, the service layer parses the partition key rule as the default type, such as {type:null}, and the service layer uses the bucket name and object name as the partition key.

[0159] Step S1102: The service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer.

[0160] In this implementation, after the service layer receives the target bucket name and the target partition key rule included in the bucket creation request sent by the client, it sends the first mapping relationship between the target bucket name and the target partition key rule to the metadata layer.

[0161] Step S1103: the metadata layer stores the first mapping relationship.

[0162] In this implementation, after receiving the first mapping relationship between the bucket target bucket name and the target partition key rule, the metadata layer stores the first mapping relationship in the bucket metadata, and the bucket metadata may be the bucket metadata 222 .

[0163] Step S1104: the service layer stores the first mapping relationship in the partition cache.

[0164] In this implementation, the service layer includes a partition cache for caching the partition key rules configured by the user. Since the partition key rules in the bucket metadata of the metadata layer are not allowed to be modified, a partition cache is newly added to the service layer to avoid accessing the bucket metadata for each partition key read request. After the metadata layer successfully stores the first mapping relationship in the bucket metadata, it sends a storage success message to the service layer. After receiving the storage success message, the service layer stores the first mapping relationship in the partition cache.

[0165] Step S1105: The service layer queries the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determines the partition keys of the multiple first objects.

[0166] In this implementation, when determining the partition keys of multiple objects, the service layer first queries the partition key rules of the buckets to which the multiple first objects belong in the first mapping relationship stored in the partition cache by the service layer to determine the partition keys of the multiple first objects. According to the queried partition keys of the multiple first objects, the partition to which each first object belongs is determined.

[0167] Step S1106: When the service layer fails to find a hit in the query of the first mapping relationship stored in the partition cache, the service layer queries the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

[0168] In this implementation, if the service layer does not hit the first mapping relationship query stored in the partition cache, it continues to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

[0169] Step S1107: The service layer stores the second mapping relationship between the bucket names of the buckets to which the multiple first objects belong and the partition key rules of the buckets to which the first objects belong in the partition cache.

[0170] In this implementation, after querying the partition key rules of the buckets to which the multiple first objects belong in the metadata layer, the service layer stores the second mapping relationship between the bucket names of the multiple first objects and the partition key rules of the buckets to which the first objects belong in the partition cache, so that the partition cache can be accessed in the next read and write request of the partition key to query the partition key rules of the buckets to which the multiple first objects belong.

[0171] In this way, in this implementation, users can configure partition key rules when creating buckets, and the service layer stores the partition key rules in the partition cache to avoid accessing the bucket metadata for each partition key read request, thereby reducing the access overhead to the metadata layer to improve the reading performance of the partition key.

[0172] In the above embodiments, the data processing method in the object storage system 200 may be software configured on a computing device or a computing device cluster, and by running the software on the computing device or the computing device cluster, the computing device or the computing device cluster may implement the functions of the above object storage system 200. The object storage system 200 involved in the process of batch object operation is described in detail below.

[0173] like Fig.12 As shown, the object storage system 200 includes a service layer 21, a metadata layer 22 and a persistence layer 23. Among them:

[0174] The service layer 21 is used to receive a batch upload request sent by a client, where the batch upload request indicates multiple first objects to be uploaded; send the data of the multiple first objects to the persistence layer; generate metadata of the multiple first objects, and send the metadata of the multiple first objects to the metadata layer;

[0175] The persistence layer 23 is used to store data of multiple first objects;

[0176] The metadata layer 22 is used to store the metadata of multiple first objects in the persistence layer; if it is determined that the metadata of at least one first object fails to be stored in the persistence layer, a first indication message is sent to the service layer, and the first indication message is used to indicate garbage collection of the multiple first objects.

[0177] In some possible implementations, the service layer 21 is further used to determine a partition key of the plurality of first objects, where the partition key is used to indicate a partition of the metadata of the plurality of first objects;

[0178] The metadata layer 22 is specifically configured to store metadata of the plurality of first objects in a persistence layer according to the partition keys of the plurality of first objects.

[0179] In some possible implementations, the metadata layer 22 is used to store metadata of first objects belonging to the same partition to the persistence layer through a write-ahead log; and to store metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

[0180] In some possible implementations, the service layer 21 is configured to determine the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong.

[0181] In some possible implementations, the service layer 21 is further configured to receive a bucket creation request sent by a client before determining the partition keys of the plurality of first objects according to the partition key rules of the buckets to which the plurality of first objects belong, the bucket creation request including the bucket name of the target bucket and the target partition key rule; and send the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer;

[0182] The metadata layer 22 is also used to store the first mapping relationship.

[0183] In some possible implementations, the service layer 21 is further configured to receive a batch deletion request sent by a client, the batch deletion request indicating multiple second objects to be deleted; send metadata of the multiple second objects to the metadata layer, the metadata of the multiple second objects including deletion marks;

[0184] The metadata layer 22 is also used to store the metadata of multiple second objects to the persistence layer; if it is determined that the metadata of at least one second object fails to be stored in the persistence layer, a second indication message is sent to the service layer, and the second indication message is used to indicate that garbage collection of the data of the multiple second objects is abandoned.

[0185] In some possible implementations, the service layer 21 is further configured to store the first mapping relationship in the partition cache.

[0186] In some possible implementations, the service layer 21 is used to query the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition keys of the multiple first objects.

[0187] In some possible implementations, the service layer 21 is further used to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects when the first mapping relationship query stored in the partition cache does not hit.

[0188] Among them, the service layer 21, the metadata layer 22 and the persistence layer 23 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the service layer 21 is introduced below by taking the service layer 21 as an example. Similarly, the implementation of the metadata layer 22 and the persistence layer 23 can refer to the implementation of the service layer 21.

[0189] As an example of a software functional unit, the service layer 21 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above-mentioned computing instance may be one or more. For example, the service layer 21 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same AZ or in different AZs, and each AZ includes a data center or multiple data centers with similar geographical locations. Among them, usually a region can include multiple AZs.

[0190] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0191] As an example of a hardware functional unit, the service layer 21 may include at least one computing device, such as a server, etc. Alternatively, the service layer 21 may also be a device implemented using an application-specific integrated circuit ASIC, or a programmable logic device PLD, etc. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0192] The multiple computing devices included in the service layer 21 can be distributed in the same region or in different regions. The multiple computing devices included in the service layer 21 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the service layer 21 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0193] It should be noted that, in other embodiments, the service layer 21 can be used to execute any step in the data processing method, the metadata layer 22 can be used to execute any step in the data processing method, and the persistence layer 23 can be used to execute any step in the data processing method. The steps that the service layer 21, the metadata layer 22, and the persistence layer 23 are responsible for implementing can be specified as needed. The full functions of the object storage system 200 are realized by respectively implementing different steps in the data processing method through the service layer 21, the metadata layer 22, and the persistence layer 23.

[0194] Fig.13 The structure diagram of a computing device is shown. The object storage system 200 can be deployed on the computing device. The computing device can be a computing device in a cloud environment (such as a server), or a computing device in an edge environment, or a terminal device, etc., which can be used to implement the above Figure 6 , Figures 8 to 11 The functions of the service layer 21, metadata layer 22 and persistence layer 23 in the object storage system 200 in the illustrated embodiment.

[0195] like Fig.13 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, a storage 107, and a communication interface 108. The processor 104, the memory 106, the storage 107, and the communication interface 108 communicate through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0196] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0197] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0198] The memory 106 may include a volatile memory, such as a random access memory (RAM).

[0199] The memory 107 may include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0200] The memory 106 and the memory 107 store executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned service layer 21, metadata layer 22 and persistence layer 23, thereby implementing the data processing method. That is, the memory 106 stores instructions for executing the data processing method.

[0201] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0202] Specifically, in implementing Figure 6 , Figures 8 to 11 In the case of the embodiment shown, and Figure 6 , Figures 8 to 11 In the case where the object storage system 200 described in the illustrated embodiment is implemented by software, the execution Figure 6 , Figures 8 to 11 The software or program code required for the functions of the object storage system 200 is stored in the memory 106 and the memory 107. The interaction between the object storage system 200 and other devices is realized through the communication interface 103. The processor 104 is used to execute the instructions in the memory 106 and the memory 107 to implement the method executed by the object storage system 200.

[0203] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0204] like Fig.14 As shown, the computing device cluster includes at least one computing device 100. The memory 106 and the memory 107 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data processing method.

[0205] In some possible implementations, the memory 106 and the memory 107 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the data processing method.

[0206] It should be noted that the memory 106 and the memory 107 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the object storage system. That is, the instructions stored in the memory 106 and the memory 107 in different computing devices 100 can implement the functions of one or more modules in the service layer 21, the metadata layer 22 and the persistence layer 23.

[0207] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.15 A possible implementation is shown. Fig.15 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the service layer 21. Fig.15 In the example, the memory 106 in the computing device 100A stores instructions for executing the functions of the service layer 21. Meanwhile, the memory 107 in the computing device 100B stores instructions for executing the functions of the metadata layer 22 and the persistence layer 23. Fig.15 In the example, the memory 107 in the computing device 100B stores instructions for executing the functions of the metadata layer 22 and the persistence layer 23.

[0208] It should be understood that Fig.15The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0209] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes a data processing method or a data processing method.

[0210] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to execute a data processing method, or instructs a computing device to execute a data processing method.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An object storage system, characterized in that: The object storage system is used to provide object storage services through at least one bucket, and the object storage system includes a service layer, a persistence layer, and a metadata layer, wherein: The service layer is configured to receive a batch upload request sent by a client, wherein the batch upload request indicates a plurality of first objects to be uploaded; send the data of the plurality of first objects to the persistence layer; generate metadata of the plurality of first objects, and send the metadata of the plurality of first objects to the metadata layer; The persistence layer is used to store data of the multiple first objects; The metadata layer is used to store the metadata of the multiple first objects in the persistence layer; when it is determined that the metadata of at least one first object fails to be stored in the persistence layer, a first indication message is sent to the service layer, where the first indication message is used to indicate garbage collection of the multiple first objects.

2. The object storage system according to claim 1, characterized in that: The service layer is further used to determine a partition key of the plurality of first objects, where the partition key is used to indicate a partition of the metadata of the plurality of first objects; The metadata layer is specifically configured to store the metadata of the multiple first objects in the persistence layer according to the partition keys of the multiple first objects.

3. The object storage system according to claim 1 or 2, characterized in that: The metadata layer is used to store metadata of first objects belonging to the same partition to the persistence layer through a pre-write log; and store metadata of first objects belonging to different partitions to the persistence layer through a distributed transaction.

4. The object storage system according to claim 2 or 3, characterized in that: The service layer is used to determine the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong.

5. The object storage system according to claim 4, characterized in that: The service layer is further configured to receive a bucket creation request sent by the client before determining the partition keys of the multiple first objects according to the partition key rules of the buckets to which the multiple first objects belong, wherein the bucket creation request includes a bucket name of a target bucket and a target partition key rule; and send a first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; The metadata layer is also used to store the first mapping relationship.

6. The object storage system according to any one of claims 1 to 5, characterized in that: The service layer is further configured to receive a batch deletion request sent by the client, wherein the batch deletion request indicates a plurality of second objects to be deleted; and send metadata of the plurality of second objects to the metadata layer, wherein the metadata of the plurality of second objects includes a deletion mark; The metadata layer is further used to store the metadata of the plurality of second objects in the persistence layer; It is determined that the metadata of at least one second object fails to be stored in the persistence layer, and second indication information is sent to the service layer, where the second indication information is used to indicate that garbage collection of data of the multiple second objects is abandoned.

7. The object storage system according to claim 5, characterized in that: The service layer also includes a partition cache, The service layer is further used to store the first mapping relationship in the partition cache.

8. The object storage system according to any one of claims 1 or 8, characterized in that: The service layer is used to query the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determine the partition key of the multiple first objects.

9. The object storage system according to claim 8, characterized in that: The service layer is further used to query the partition key rules of the buckets to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects when the first mapping relationship query stored in the partition cache does not hit.

10. A data processing method, characterized in that: The method is applied to an object storage system, the object storage system is used to provide an object storage service through at least one bucket, the object storage system includes a service layer, a persistence layer and a metadata layer, and the method includes: The service layer receives a batch upload request sent by a client, where the batch upload request indicates a plurality of first objects to be uploaded; The service layer sends the data of the plurality of first objects to the persistence layer; The persistence layer stores data of the plurality of first objects; The service layer generates metadata of the plurality of first objects, and sends the metadata of the plurality of first objects to the metadata layer; The metadata layer stores the metadata of the plurality of first objects in the persistence layer; The metadata layer determines that metadata of at least one first object fails to be stored in the persistence layer, and sends first indication information to the service layer, where the first indication information is used to indicate garbage collection of the multiple first objects.

11. The method according to claim 10, characterized in that The method further comprises: The service layer determines a partition key of the plurality of first objects, the partition key being used to indicate a partition of metadata of the plurality of first objects.

12. The method according to claim 10 or 11, characterized in that: The metadata layer stores the metadata of the plurality of first objects in the persistence layer, including: The metadata layer stores the metadata of the first object belonging to the same partition to the persistence layer through a write-ahead log; The metadata layer stores metadata of first objects belonging to different partitions to the persistence layer through distributed transactions.

13. The method according to claim 11 or 12, characterized in that: The service layer determines the partition keys of the plurality of first objects, including: The service layer determines the partition keys of the multiple first objects according to the partition key rule of the bucket to which the multiple first objects belong.

14. The method according to claim 13, characterized in that The method further comprises: The service layer receives a bucket creation request sent by the client before determining the partition keys of the multiple first objects according to the partition key rule of the bucket to which the multiple first objects belong, wherein the bucket creation request includes a bucket name of a target bucket and a target partition key rule; The service layer sends the first mapping relationship between the bucket name of the target bucket and the target partition key rule to the metadata layer; The metadata layer stores the first mapping relationship.

15. The method according to any one of claims 10 to 14, characterized in that The method further comprises: The service layer receives a batch deletion request sent by the client, where the batch deletion request indicates a plurality of second objects to be deleted; The service layer sends the metadata of the plurality of second objects to the metadata layer, wherein the metadata of the plurality of second objects includes a deletion mark; The metadata layer stores the metadata of the plurality of second objects in the persistence layer; The metadata layer determines that the metadata of at least one second object fails to be stored in the persistence layer, and sends second indication information to the service layer, where the second indication information is used to indicate that garbage collection of data of the multiple second objects is abandoned.

16. The method according to claim 14, characterized in that The service layer also includes a partition cache, and the method further includes: The service layer stores the first mapping relationship in the partition cache.

17. The method according to any one of claims 10 or 16, characterized in that: The service layer determines the partition keys of the plurality of first objects, including: The service layer queries the partition key rule of the bucket to which the multiple first objects belong in the first mapping relationship stored in the partition cache, and determines the partition keys of the multiple first objects.

18. The object storage system according to claim 17, characterized in that: The method further comprises: When the service layer fails to query the first mapping relationship stored in the partition cache, the service layer queries the partition key rule of the bucket to which the multiple first objects belong in the metadata layer to determine the partition keys of the multiple first objects.

19. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

20. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images