Object data processing method, electronic device, storage medium, and program product

Abstract

The present application provides a kind of object data processing method, electronic equipment, storage medium and program product.The method comprises: after starting multi-version function for object data, determine the version number and generation of object data, the generation is used to indicate the operation serial number of the object data;After determining the different versions of multiple object data to be written, without imposing write lock on the different versions of multiple object data to be written, based on the version number of the multiple object data, the multiple object data with the same generation are concurrently written to database.

Description

Technical Field

[0001] This application relates to the technical field of object storage, and more particularly to a method for processing object data, an electronic device, a storage medium, and a program product. Background Technology

[0002] In related technologies, when an access node performs write operations on multiple versions of object data, it is necessary to write the multiple versions of object data sequentially through serialization. Furthermore, the object data to be written must be locked before it can be written. Therefore, the writing speed and efficiency of object data are reduced to some extent. Summary of the Invention

[0003] This application provides a method for processing object data, an electronic device, a storage medium, and a program product that can improve the writing speed and efficiency of multiple versions of object data.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for processing object data, the method comprising: After enabling the multi-version function for object data, determine the version number and generation of the object data, whereby the generation is used to represent the operation sequence number of the object data; After determining multiple object data of different versions to be written, without applying write locks to the multiple object data of different versions to be written, the multiple object data of the same generation are concurrently written to the database based on the version number of the multiple object data.

[0005] This application also provides an electronic device, which includes a processor and a memory for storing a computer program that can run on the processor; wherein the processor is used to run the computer program to perform any of the above-described object data processing methods.

[0006] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described object data processing methods.

[0007] This application also provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement any of the above-described object data processing methods.

[0008] The embodiments of this application have the following beneficial effects: It can be seen that the embodiments of this application can concurrently write multiple versions of object data to the database without applying a write lock to the object data to be written. Compared with the related technologies, which serialize the writing of different versions of data after locking the data, the writing speed and efficiency of object data can be improved. Attached Figure Description

[0009] Figure 1 This is a flowchart of a method for processing object data according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the object data processing apparatus according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0010] In related technologies, when an access node performs write operations on multiple versions of object data, it is necessary to write multiple versions of object data sequentially through serialization. Furthermore, it is necessary to first lock the object data to be written, and then perform the steps of allocating the version number of the object data, writing the data, and updating the latest version number of the object data. This reduces the writing speed and efficiency of object data to a certain extent, and the writing performance of object data will be significantly affected.

[0011] In related technologies, when reading object data across multiple versions, the latest version identification number (Identity, Document, ID) is first obtained, and then the key-value (KV) data is read based on this ID. For performance reasons, no read lock is used when reading the version ID, which can negatively impact data read consistency.

[0012] In related technologies, the processing logic for multi-version object data is relatively dispersed, and a large amount of object data processing occurs at the access node. This introduces a large number of unnecessary remote procedure calls (RPC) processes and bandwidth consumption, resulting in complex processes and high coupling.

[0013] To address the technical problems existing in related technologies, this application proposes a technical solution based on embodiments of the present application. This application proposes a method for processing object data, an electronic device, a storage medium, and a program product, which can fully realize the multi-version feature of object storage based on metadata services. This application aims to meet users' needs for managing and operating different versions of objects, improve the read and write performance of multi-version object storage data, and ensure the consistency of object data across different versions during write, update, and delete operations.

[0014] In an exemplary application scenario of this application, the Share Nothing architecture general-purpose KV storage system is a strongly consistent distributed storage system that provides services through a table interface. The general-purpose KV storage cluster consists of a coordinating node, metadata management nodes, and data replica nodes, providing data read / write (IO) services. Internally, tables are divided into multiple partitions according to range or hash methods. Each partition is responsible for a portion of the table data, providing data read / write services. Data reliability is guaranteed through consensus algorithms such as Multi Paxos or Raft. When version control (multi-versioning) is enabled in the object storage system, the system assigns a unique ID value to each version of all files or objects at the bucket level, and the content and permissions of existing objects in the bucket remain unchanged. Enabling version control also prevents accidental overwriting or deletion of objects and allows querying and restoring historical versions of objects.

[0015] This application's embodiments can be applied to object storage architectures, which are mainly implemented through three services: gateway service (implemented at the access layer), metadata service, and data persistence service. In the metadata service, NoSQL or NewSQL type table-structure databases can be used to provide efficient and reliable key-value storage. In this application's embodiments, a table structure (TableSchema) can be designed in the metadata service to support multi-version characteristics.

[0016] At the table structure level, this application embodiment designs a completely new architecture for a general KV storage engine that implements multi-version features. On the basis of the ID of the basic object data, it adds fields such as timestamp, universally unique identifier (Uuid), epoch, and reserved flag.

[0017] Here, the ID of the object data is the basic identifier of the object data and belongs to the key field in the table structure. The timestamp is the timestamp information generated by the gateway layer based on the current time. It can be used to distinguish the chronological order of data and is part of the version number. It also belongs to the key field in the table structure.

[0018] For example, the timestamp is 64 bits of data. The Uuid is a random value generated by the gateway layer to ensure the uniqueness of the version number. It is part of the version number and belongs to the Key field in the table structure. For example, the Uuid is 128 bits of data.

[0019] Flag can be part of a reserved bitmap, which can be 8 to 16 bits of data. Flag can use 1 bit of data from the bitmap. Flag can indicate whether to add a delete marker and belongs to the Value field in the table structure.

[0020] The initial value of the generation can be set according to the needs of the actual scenario. When the user enables or suspends the multi-version feature for the object data in the bucket, the generation value can be incremented by 1 each time an operation is performed on the object data. The generation can be 8 digits and the value of the generation can be a number between 0 and 255. The generation is used to compare the order of data of different versions of objects and belongs to the Value field in the table structure.

[0021] For example, data for different versions of an object may include invalid (NULL) versions of the object data. Here, NULL versions of the object data represent the object data stored in the database when multi-versioning is not enabled for the object data.

[0022] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0023] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0024] The object data processing method provided in this application includes a series of steps, but the object data processing method provided in this application is not limited to the steps described. Similarly, the object data processing apparatus provided in this application includes a series of modules, but the apparatus provided in this application is not limited to the modules explicitly described, and may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0025] This application provides a method for processing object data. Figure 1 This is a flowchart of the object data processing method according to an embodiment of this application, such as... Figure 1 As shown, the process includes: Step 101: After enabling the multi-version feature for object data, determine the version number and generation of the object data. The generation is used to represent the operation sequence number of the object data.

[0026] In this embodiment of the application, after enabling the multi-version function for object data, operations such as writing, reading, deleting, and updating can be performed on object data of multiple versions.

[0027] The version number of object data is used to uniquely identify the version of object data.

[0028] For example, the version number of object data can consist of a timestamp and a Uuid. In a real-world scenario, the gateway layer generates a timestamp based on the local clock for multiple versions of object data to identify the order of the versions (following the first-come, first-served principle), and can also generate a Uuid to ensure the uniqueness of the version number.

[0029] The operation sequence number of object data refers to the order in which operations are performed on different versions of object data with the same identifier. In practical scenarios, the metadata service layer can determine the operation order of multiple object data based on their generation. For example, when multiple object data of different versions have the same generation, parallel operations can be performed on multiple object data.

[0030] Step 102: After determining the multiple object data of different versions to be written, without applying write locks to the multiple object data of different versions to be written, write the multiple object data of the same generation concurrently to the database based on the version number of the multiple object data.

[0031] In this embodiment, when multi-version functionality is enabled for object data, if a write command for the object data is received, multiple versions of object data to be written can be obtained from the write command. The identifier for the object data can be an ID, name, or other identifier. For example, the identifier for the object data can consist of the ID of the bucket storing the object data and the ID of the object data itself. The bucket ID can be used as a prefix in the identifier for the object data.

[0032] Before enabling multi-version support for object data (i.e., when multi-version support is not enabled), object data in the database can be NULL-version object data. For example, before enabling multi-version support for object data, the identifiers of two object data in the database can be Aa and Ba, where A represents the bucket ID and a represents the object data ID. The table structure of object data before enabling multi-version support for object data can be the table structure shown in Table 1.

[0033] Table 1

[0034] The table structure of object data includes the metadata information of object data. After enabling the multi-version function for object data, it is necessary to modify the metadata information in the table structure (for example, modify the metadata information of the bucket with ID A). When it is determined that an operation needs to be performed on the object data (such as read, write, update, delete, etc.), the generation of object data can be changed. At this time, the updated metadata information can be notified to all access nodes in the gateway layer.

[0035] After enabling the multi-version feature for object data, NULL version object data can be converted into one of the different versions of object data. That is, different versions of object data can include the converted NULL version of object data.

[0036] For example, for the table structure shown in Table 1, after enabling the multi-version function for object data, for object data identified as Aa, access node 1 and access node 2 need to concurrently write two versions of object data, which can result in the table structure shown in Table 2.

[0037] Table 2

[0038] Referring to Table 2, the timestamps generated by access node 1 and access node 2 may be exactly the same (e.g., both are 1), but the Uuids of the two versions of object data are different. For example, the Uuids of the two versions of object data are 1 and 2 respectively. In the table structure sorted by byte order, the order of the different versions of object data is determined by the version number (versionid) composed of the timestamp and the Uuid.

[0039] In practical applications, steps 101 to 102 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor.

[0040] As can be seen, the embodiments of this application can concurrently write multiple versions of object data to the database without applying a write lock to the object data to be written. Compared with the related technologies, which serialize the writing of different versions of data after locking the data, this can improve the writing speed and efficiency of object data.

[0041] This application embodiment supports lock-free concurrent writing of multiple access node instances to multiple versions of object data. If a single access node fails, it will not have any visible impact on the node cluster. No additional table locking is required, reducing the redundancy of the architecture for implementing multi-version object data functionality.

[0042] In related technologies, different access nodes need to interact through transactional IO interfaces to determine the writing order of different versions of data. However, in this embodiment, the multi-version writing logic of object data can be determined at the metadata service layer without interacting through transactional IO interfaces. The writing process of object data can be accelerated through non-transactional IO interfaces.

[0043] Regarding the method of reading object data, in some embodiments, after enabling or suspending the multi-version function for object data, a read command for the object data can be obtained. The read command carries identification information, and the object data is read based on the identification information in the read command. The identification information includes one or more of the following: a first identifier for the bucket used to store the object data to be read, and a second identifier for the object data to be read.

[0044] In this embodiment of the application, after suspending the multi-version function for object data, the multi-version object data can still be processed at the underlying level of the data processing structure. However, the multi-version object data is imperceptible to the user, that is, the user cannot perceive the multiple versions of the object data.

[0045] For example, the first identifier of the bucket can be a name, ID, or other identifier. The second identifier of the object data can consist of the bucket's ID and the object data's ID.

[0046] As can be seen, the embodiments of this application can read object data from the database relatively accurately based on the bucket identifier or object data identifier carried in the read command.

[0047] In the first example, the object data can be read from the first bucket through the metadata service layer based on the first identifier carried in the read command. The identifier of the first bucket is the first identifier.

[0048] For example, after enabling or suspending multi-version functionality for object data, the metadata information of the object data in the metadata service layer can be read through the following four interfaces: GetBucketVersions interface, GetBucket interface, Get (without version number) interface, and Get (with version number) interface. Among them, the GetBucketVersions interface is used to retrieve the metadata information (including deletion markers) of all versions of object data in the first bucket. At this time, the metadata service layer can list all metadata for bucket ID A and then return the listed information to the user. Taking Table 2 as an example, it can return three metadata records for object data with ID Aa to the user.

[0049] As can be seen, the embodiments of this application can determine the object data to be read more accurately through the metadata service layer based on the bucket identifier carried in the read command, thereby reading the object data from the database more accurately.

[0050] Regarding the implementation of reading object data based on the identifier information in the read command, in the second example, the latest version of the object data in the first bucket can be determined based on the first identifier carried in the read command, through the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the first bucket has not been marked for deletion, the latest version of the object data in the first bucket is read. The identifier of the first bucket is the first identifier.

[0051] In this embodiment, the GetBucket interface is used to enumerate all data in the current bucket. For multi-version object data, the GetBucket interface can be used to present the metadata information of the latest version of the object data in the first bucket. For example, for general-purpose key-value storage, the coordinating node first scans all metadata for bucket ID A. For object data with the same ID but different versions, only the data with the largest version number is selected. When different versions of object data have the same timestamp, the Uuid can be used to determine the relationship between the version numbers of the different versions. If there is a NULL version of object data, the object data with the largest version number among the non-NULL versions needs to be determined. Then, the object data with the largest version number among the non-NULL versions is compared with the NULL version object data, and the object data with the larger generation is selected as the latest version of the object data in the first bucket. After determining the latest version of the object data in the first bucket, it is also necessary to determine whether the object data has been marked for deletion based on the Flag. If the latest version of the object data in the first bucket has not been marked for deletion, it is determined that the object data is visible to the user, and the metadata information of the object data can be returned to the user. Taking Table 2 as an example, metadata information for object data with ID Aa and version number (1,2) can be returned to the user. If the latest version of object data in the first bucket has a deletion mark added, it is determined that the object data is not visible to the user, and there is no need to return the metadata information of the object data to the user.

[0052] As can be seen, the embodiments of this application can accurately select the latest version of object data in the first bucket by comparing information such as the version number and generation of object data, and take into account the impact of deletion markers on the reading operation, thereby helping to ensure the consistency and accuracy of data reading.

[0053] In the third example, for the implementation of reading object data based on the identification information in the read command, the latest version of the object data in the target object data can be determined based on the second identifier carried in the read command, through the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the target object data has not been marked for deletion, the latest version of the object data in the target object data is read. The identifier of the target object data is the second identifier.

[0054] In this embodiment, the Get (without version number) interface is a non-multi-version interface for users to retrieve object data by its ID. This interface only returns the metadata information of the latest version of a specific object data. For example, if the user wants to read object data with ID Aa, the coordinating node first needs to scan the object data with ID Aa and select only the data with the largest version number. If a NULL version of the object data exists, the object data with the largest version number among the non-NULL versions needs to be determined. Then, the object data with the largest version number among the non-NULL versions is compared with the object data with the NULL version, and the object data with the larger generation is selected as the latest version of the target object data. After determining the latest version of the target object data, it is also necessary to determine whether the object data has been marked for deletion based on the Flag. If the latest version of the target object data has not been marked for deletion, it is determined that the object data is visible to the user, and the metadata information of the object data can be returned to the user. Taking Table 2 as an example, the metadata information of the object data with ID Aa and version number (1,2) can be returned to the user. If the latest version of the object data in the target object data has been marked for deletion, then the object data is determined to be invisible to the user, and there is no need to return the metadata information of the object data to the user.

[0055] As can be seen, the embodiments of this application can accurately select the latest version of object data in the target object data by comparing information such as the version number and generation of the object data, and take into account the impact of deletion markers on the reading operation, thereby helping to ensure the consistency and accuracy of data reading.

[0056] In the fourth example, regarding the implementation of reading object data based on the identification information in the read command, if the read command also carries a version number, the object data corresponding to the read command can be read through the metadata service layer based on the second identifier and version number carried in the read command.

[0057] In this embodiment of the application, the Get (with version number) interface is an interface for users to obtain data through the ID and specific version number of the object data. In practical applications, the metadata of the object data can be obtained based on the ID and specific version number of the object data in the Key data of the table structure. It is not necessary to determine whether the object data has been marked for deletion or addition based on the Flag. The metadata of the obtained object data can be returned directly.

[0058] As can be seen, the embodiments of this application can determine the object data to be read more accurately through the metadata service layer based on the identifier and version number of the object data carried in the read command, thereby reading the object data from the database more accurately.

[0059] Regarding the deletion method of object data, in some embodiments, after enabling the multi-version function for object data, a deletion request for the object data can be obtained, and the deletion request carries a third identifier of the object data; if the deletion request does not carry the version number of the object data, a deletion mark is added to the object data identified by the third identifier in the metadata service layer by adding new metadata; if the deletion request also carries the version number of the object data, the data to be deleted is determined, and the data to be deleted is deleted in the database, where the identifier of the data to be deleted is the third identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

[0060] For example, the third identifier of the object data can consist of the bucket ID and the object data ID. After enabling multi-version functionality for object data, when the access gateway receives a deletion request that does not carry the version number of the object data, it can generate a version number (including timestamp and Uuid) for a metadata record corresponding to the deletion request, and call the multi-version deletion interface provided by the metadata service layer. Finally, a metadata record is inserted into the table structure of the metadata service layer. In the inserted metadata, the object data ID is the same as the object data ID carried in the deletion request, the version number is the version number of the metadata corresponding to the deletion request, and the Flag field is used to indicate that a deletion flag has been added. For example, for the table structure shown in Table 2, after inserting a metadata record, the table structure shown in Table 3 can be obtained. In the inserted metadata of Table 3, the object data ID is Aa, the timestamp is 2, the Uuid is 3, and the value of the Flag field is b00000001, where b00000001 indicates that a deletion flag has been added.

[0061] Table 3

[0062] After enabling multi-version functionality for object data, if the obtained deletion request also carries the version number of the object data, the object data can be deleted at the database level based on the object data ID and version number carried in the deletion request, instead of inserting a metadata record into the table structure of the metadata service layer.

[0063] As can be seen, when the multi-version feature is enabled for object data, if the deletion request does not carry the version number of the object data, a deletion flag can be added to the object data to facilitate accurate determination of whether to return the object data's metadata information when reading data later. If the deletion request also carries the version number of the object data, the data to be deleted can be deleted from the database to meet the object data deletion requirements.

[0064] Regarding the method of updating object data, in some embodiments, after enabling or suspending the multi-version function for object data, an update request for object data can be obtained, and the update request carries the fifth identifier of the object data.

[0065] If the update request does not carry the version number of the object data, the object data to be updated is determined by the metadata service layer, and the latest version of the object data to be updated is determined by the metadata service layer. The latest version of the object data to be updated is then updated. The identifier of the object data to be updated is the fifth identifier. When the update request carries a version number of the object data, the object data corresponding to the update request is updated based on the fifth identifier and version number carried in the update request.

[0066] For example, the fifth identifier of the object data can consist of the bucket ID and the object data ID. If the update request does not carry the object data version number, it can carry an if-exist conditional statement. The access gateway can directly call the update interface and query the metadata service layer for the latest version of the object data to be updated via the Get (without version number) interface. If there is no data to be updated or the latest version of the object data to be updated has been marked for deletion, a response indicating no update is needed can be returned directly. If there is data to be updated and the latest version of the object data to be updated has not been marked for deletion, the retrieved data can be updated directly in the database.

[0067] When an update request carries a version number of the object data, the update operation can be performed directly on the object data corresponding to the update request in the database. This update operation is the same as the update operation in a normal database.

[0068] It can be seen that when the update request does not carry the version number of the object data, the latest version of the object data to be updated can be accurately determined at the metadata service layer, thus reliably updating the latest version of the object data to be updated. When the update request carries the version number of the object data, the object data corresponding to the update request can be reliably updated based on the fifth identifier and version number carried in the update request.

[0069] Regarding the writing method of object data, in some embodiments, the metadata of the object data in the NULL version can be constructed in the metadata service layer. The NULL version of the object data represents the object data stored in the database when the multi-version function is not enabled for the object data. After suspending the multi-version function for object data and obtaining the write command for the object data, the metadata of the NULL version of the object data is replaced in the metadata service layer.

[0070] In real-world scenarios, once the multi-version feature is enabled for object data, it cannot be disabled, but it can be suspended for object data.

[0071] For example, after enabling the multi-version function for object data, if different access gateways receive write commands for object data, they will all overwrite and replace the metadata of the original NULL version of the object data in the table structure.

[0072] It can be seen that by suspending the multi-version function for object data and obtaining the write command for the object data, replacing the metadata of the NULL version of the object data in the metadata service layer can ensure that there is at most one NULL version of metadata information at the same time. This can, to a certain extent, avoid the impact of multiple NULL versions of metadata on the processing of subsequent object data.

[0073] Regarding the deletion method of object data, in some embodiments, after suspending the multi-version function for object data, a deletion request for the object data can be obtained, and the deletion request carries the fourth identifier of the object data.

[0074] If the deletion request does not carry the version number of the object data, the metadata service layer adds a deletion mark to the first NULL version of the object data by data overwriting, and updates the first NULL version of the object data to the latest version of the object data. The first NULL version of the object data is the NULL version of the object data, and the identifier of the first NULL version of the object data is the fourth identifier.

[0075] If the deletion request also carries the version number of the object data, the data to be deleted is determined, and the data to be deleted is deleted from the database. The identifier of the data to be deleted is the fourth identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

[0076] For example, the fourth identifier of the object data can consist of the bucket ID and the object data ID. After suspending the multi-version function for the object data, if the deletion request does not carry the version number of the object data, the access node at the gateway layer can determine the NULL version of the object data identified by the fourth identifier as the first NULL version of the object data. At the metadata service layer, a deletion flag is added to the first NULL version of the object data, and the object data is updated to the latest version of the object data by modifying the generation of the metadata of the first NULL version of the object data (by incrementing the generation value).

[0077] For example, in the table structure shown in Table 4, there are two metadata entries. One metadata entry is the object data in the NULL version, and the other metadata entry is the data added based on the deletion request that does not carry the version number of the object data after the multi-version function is enabled for the object data.

[0078] Table 4

[0079] For the table structure shown in Table 4, after suspending the multi-version function of object data by executing the suspend command, the generation value of the NULL version of object data can be incremented to 2; after 5 minutes, based on the deletion request that does not carry the version number of object data, the gateway layer adds a deletion mark to the NULL version of object data through the metadata service, resulting in the table structure shown in Table 5.

[0080] Table 5

[0081] As can be seen, when suspending the multi-version functionality for object data and receiving a deletion request for the object data, if the deletion request does not carry the object data's version number, replacing the metadata of the NULL version of the object data at the metadata service layer ensures that at most one NULL version of the object data exists within the same timeframe. This can, to some extent, prevent multiple NULL versions of metadata from affecting subsequent object data processing. If the deletion request also carries the object data's version number, the data to be deleted can be removed from the database, thus satisfying the object data deletion requirement.

[0082] In some embodiments, object data processing can be performed at the metadata service layer by executing multi-version operation logic for the object data.

[0083] Here, the multi-version operation logic for object data is used to determine the processing method for multiple versions of object data. The processing method includes one or more of the following: read, write, update, and delete. After determining the processing method for multiple versions of object data, the access node can process the object data according to the processing method for multiple versions of object data.

[0084] In this embodiment, the multi-version operation logic can be decentralized to the metadata service layer. Access nodes do not need to concern themselves with excessive multi-version operation logic; they only need to call the interfaces provided by the metadata service layer to implement the relevant features. This decouples the feature from the access node at the business logic level and improves the performance of multi-version data read / write. In this embodiment, the complex internal implementation logic of multi-version functionality can be completely decentralized to the metadata service layer. The gateway layer only needs to focus on the business logic, perceive the bucket-related metadata information (e.g., generation), and reasonably call the relevant interfaces provided by the metadata service to implement complex multi-version functionality. This eliminates the need for redundant work at the gateway layer, such as handling object version priority judgments and locking when allocating object version IDs, which significantly impacts performance.

[0085] In this embodiment, generational synchronization can also be achieved across various access nodes. In some embodiments, after the generational update of the object data, the updated generation can be notified to each access node, where an access node represents a node used to process the object data.

[0086] For example, when multi-versioning functionality is enabled or suspended for object data, it can be determined that when performing related operations (read, write, update, delete, etc.) on the object data, the generation of the object data is incremented by 1 (the initial generation value can be 0), and this is notified to each access node in the gateway layer through a watch mechanism. In practical applications, object storage systems require that business I / O operations on object data be performed only 5 minutes after multi-versioning functionality is enabled or suspended (i.e., no business I / O operations on object data are required within 5 minutes of enabling or suspending multi-versioning functionality), so that each access node in the gateway layer can correctly perceive the latest generation before processing subsequent object data.

[0087] In summary, this application embodiment addresses object storage systems by using a version number composed of timestamps and Uuids, combined with generational relationships, to achieve concurrent, lock-free writing of multi-version object data. This allows business logic to be fully implemented at the metadata service layer, reducing bandwidth consumption and accelerating multi-version related I / O requests. Compared with related technical solutions, this application embodiment has at least the following advantages: 1) High-performance concurrent write. By using a version number composed of timestamps and Uuids, and combining generational implementation to achieve a concurrent lock-free multi-version object write process, the serialization problem during multi-version object writes is effectively avoided, greatly improving write performance. This enables the object storage system to easily cope with high-concurrency write scenarios, representing a significant improvement in write performance compared to related technologies.

[0088] 2) Architecture Decoupling and Simplification. The multi-version operation logic is decentralized to the metadata service layer. Access nodes do not need to concern themselves with excessive multi-version operation logic; they only need to call the interfaces provided by the metadata service layer to implement the relevant features. This embodiment achieves decoupling at the business logic level, reducing the redundancy of the architecture for multi-version object data processing, lowering the complexity and coupling of the object storage system, and facilitating the maintenance and expansion of the object storage system. This overcomes the drawbacks of related technologies, such as scattered multi-version processing logic and complex processes.

[0089] 3) Enhanced data consistency assurance. When performing read operations on object data across multiple versions, the latest valid version is accurately selected and returned to the user by comparing version numbers and generational information. The impact of deletion markers is also considered, ensuring the consistency and accuracy of data reading. Compared to related technologies where data consistency issues arise from not using read locks when reading version IDs, this application provides more reliable data consistency assurance.

[0090] 4) Efficient data reading, writing, and deletion. For reading and deleting multi-version object data, the embodiments of this application, through optimized processes and mechanisms, can quickly and accurately locate and process target data, reducing unnecessary RPC processes and bandwidth consumption, and improving the efficiency of data reading and writing.

[0091] 5) Easy to expand and maintain. Since the multi-version operation logic is concentrated in the metadata service layer, when it is necessary to upgrade, optimize or fix the multi-version function, only the corresponding adjustments need to be made in the metadata service layer. There is no need to modify the code in other layers such as access nodes, which greatly improves the scalability and maintainability of the object storage system and reduces the maintenance cost of the object storage system.

[0092] This application's embodiments can improve user experience specifically because: by supporting lock-free concurrent writing of multiple access node instances to multiple versions of data, and improving the performance of multi-version data read and write, the waiting time for users when performing object storage operations is shortened, the system's response speed is improved, and a faster and smoother object storage service experience is provided to users, which helps to improve user satisfaction. Furthermore, this application's embodiments can improve business continuity specifically because: it effectively prevents accidental overwriting or deletion of objects, and allows users to query and restore historical versions of object data, reducing the risk of data loss, ensuring the integrity and availability of business data, thereby improving business continuity and reducing business interruption time and losses caused by data loss or corruption. Furthermore, this application's embodiments can reduce operating costs specifically because: it reduces the redundancy of the architecture for processing multi-version object data and unnecessary RPC processes and bandwidth consumption, optimizes the system's resource utilization efficiency, and reduces operating costs such as hardware procurement costs and network bandwidth costs. At the same time, due to the simplification and decoupling of the system architecture, it also reduces the system's maintenance difficulty and labor costs.

[0093] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0094] Based on the object data processing method proposed in the foregoing embodiments, this application also proposes an object data processing apparatus.

[0095] Figure 2 This is a schematic diagram of the structure of the object data processing apparatus according to an embodiment of this application, as shown below. Figure 2 As shown, the device includes: The determination module 201 is used to determine the version number and generation of the object data after the multi-version function is enabled for the object data, wherein the generation is used to represent the operation sequence number of the object data; The processing module 202 is used to, after determining multiple object data of different versions to be written, concurrently write multiple object data of the same generation to the database based on the version number of the multiple object data without applying write locks to the multiple object data of different versions to be written.

[0096] In some embodiments, the processing module 202 is further configured to execute multi-version operation logic of the object data in the metadata service layer to process the object data.

[0097] In some embodiments, the processing module 202 is further configured to construct metadata for an invalid NULL version of the object data in the metadata service layer, and after suspending the multi-version function for the object data and obtaining a write command for the object data, replace the metadata of the NULL version of the object data in the metadata service layer; the NULL version of the object data represents the object data stored in the database when the multi-version function for the object data is not enabled.

[0098] In some embodiments, the processing module 202 is further configured to obtain a read command for object data after enabling or suspending the multi-version function for object data. The read command carries identification information, which includes one or more of the following: a first identifier for the bucket used to store the object data to be read, and a second identifier for the object data to be read. The processing module 202 is also used to read object data based on the identification information in the read command.

[0099] In some embodiments, the processing module 202 reads object data based on the identification information in the read command, including any one of the following: Based on the first identifier carried in the read command, the object data in the first bucket is read through the metadata service layer, and the identifier of the first bucket is the first identifier; Based on the first identifier carried in the read command, the latest version of the object data in the first bucket is determined by the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the first bucket has not been marked for deletion, the latest version of the object data in the first bucket is read. The identifier of the first bucket is the first identifier. Based on the second identifier carried in the read command, the latest version of the object data in the target object data is determined through the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the target object data has not been marked for deletion, the latest version of the object data in the target object data is read. The identifier of the target object data is the second identifier. If the read command also carries a version number, the object data corresponding to the read command is read through the metadata service layer based on the second identifier carried in the read command and the version number.

[0100] In some embodiments, the processing module 202 is further configured to obtain a deletion request for the object data after enabling the multi-version function for the object data, wherein the deletion request carries a third identifier of the object data. The processing module 202 is further configured to, when the deletion request does not carry the version number of the object data, add a deletion mark to the object data identified by the third identifier in the metadata service layer by adding new metadata; when the deletion request also carries the version number of the object data, determine the data to be deleted, delete the data to be deleted in the database, wherein the identifier of the data to be deleted is the third identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

[0101] In some embodiments, the processing module 202 is further configured to obtain a deletion request for the object data after suspending the multi-version function for the object data, the deletion request carrying a fourth identifier of the object data; The processing module 202 is further configured to, in the case that the deletion request does not carry the version number of the object data, add a deletion mark to the first NULL version of the object data in the metadata service layer by data overwriting, and update the first NULL version of the object data to the latest version of the object data, wherein the identifier of the first NULL version of the object data is the fourth identifier; The processing module 202 is further configured to determine the data to be deleted in the database when the deletion request also carries the version number of the object data, and delete the data to be deleted in the database. The identifier of the data to be deleted is the fourth identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

[0102] In some embodiments, the processing module 202 is further configured to obtain an update request for the object data after enabling or suspending the multi-version function for the object data, wherein the update request carries a fifth identifier of the object data. The processing module 202 is further configured to, when the update request does not carry the version number of the object data, determine the object data to be updated through the metadata service layer, determine the latest version of the object data to be updated through the metadata service layer, update the latest version of the object data to be updated, and the identifier of the object data to be updated is the fifth identifier; The processing module 202 is further configured to update the object data corresponding to the update request based on the fifth identifier carried in the update request and the version number when the update request carries a version number of the object data.

[0103] In some embodiments, the processing module 202 is further configured to notify each access node of the updated generation after the generational update of the object data, wherein the access node represents the node used to process the object data.

[0104] In practical applications, the determination module 201 and the processing module 202 can be implemented based on a processor and a communication device.

[0105] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0106] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0107] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the object data processing methods provided in this application embodiment.

[0108] Accordingly, this application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the object data processing methods provided in the above embodiments.

[0109] This application also provides an electronic device. Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 3 As shown, the electronic device 30 may include: Memory 301 is used to store executable instructions; The processor 302 is used to implement any of the above-described object data processing methods when executing executable instructions stored in the memory 301.

[0110] The processor 302 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0111] The aforementioned computer-readable storage medium and memory 301 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; or it may be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0112] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0113] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0114] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0115] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0116] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0118] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A method for processing object data, characterized in that, The method includes: After enabling the multi-version function for object data, the version number and generation of the object data are determined. The generation is used to represent the operation sequence number of the object data. The version number consists of a timestamp and a universally unique identifier (Uuid). After determining multiple object data of different versions to be written, without applying write locks to the multiple object data of different versions to be written, the multiple object data of the same generation are concurrently written to the database based on the version number of the multiple object data; The method further includes: In the metadata service layer, multi-version operation logic for the object data is executed to process the object data; The method further includes: In the metadata service layer, construct metadata for invalid NULL versions of object data. The NULL version of object data represents the object data stored in the database when the multi-version function is not enabled for object data. After suspending the multi-version function for object data and obtaining the write command for the object data, the metadata of the NULL version of the object data is replaced in the metadata service layer.

2. The method according to claim 1, characterized in that, The method further includes: After enabling or suspending the multi-version function for object data, a read command for the object data is obtained. The read command carries identification information, which includes one or more of the following: a first identifier for the bucket used to store the object data to be read, and a second identifier for the object data to be read. Based on the identification information in the read command, the object data is read.

3. The method according to claim 2, characterized in that, The process of reading object data based on the identification information in the read command includes any one of the following: Based on the first identifier carried in the read command, the object data in the first bucket is read through the metadata service layer, and the identifier of the first bucket is the first identifier; Based on the first identifier carried in the read command, the latest version of the object data in the first bucket is determined by the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the first bucket has not been marked for deletion, the latest version of the object data in the first bucket is read. The identifier of the first bucket is the first identifier. Based on the second identifier carried in the read command, the latest version of the object data in the target object data is determined through the version number and generation of each object data in the metadata service layer. When the latest version of the object data in the target object data has not been marked for deletion, the latest version of the object data in the target object data is read. The identifier of the target object data is the second identifier. If the read command also carries a version number, the object data corresponding to the read command is read through the metadata service layer based on the second identifier carried in the read command and the version number.

4. The method according to claim 1, characterized in that, The method further includes: After enabling the multi-version function for object data, obtain the deletion request of the object data, the deletion request carrying the third identifier of the object data; If the deletion request does not carry the version number of the object data, a deletion mark is added to the object data identified by the third identifier in the metadata service layer by adding new metadata; if the deletion request also carries the version number of the object data, the data to be deleted is determined and deleted from the database. The identifier of the data to be deleted is the third identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

5. The method according to claim 1, characterized in that, The method further includes: After suspending the multi-version function for object data, obtain the deletion request for the object data, which carries the fourth identifier of the object data; In the case where the deletion request does not carry the version number of the object data, the first NULL version of the object data is marked for deletion by data overwriting in the metadata service layer, and the first NULL version of the object data is updated to the latest version of the object data. The identifier of the first NULL version of the object data is the fourth identifier. If the deletion request also carries a version number of the object data, the data to be deleted is determined, and the data to be deleted is deleted from the database. The identifier of the data to be deleted is the fourth identifier, and the version number of the data to be deleted is the version number carried in the deletion request.

6. The method according to claim 1, characterized in that, The method further includes: After enabling or suspending the multi-version function for object data, an update request for the object data is obtained, and the update request carries the fifth identifier of the object data. In the case where the update request does not carry the version number of the object data, the object data to be updated is determined by the metadata service layer, and the latest version of the object data to be updated is determined by the metadata service layer. The latest version of the object data to be updated is then updated. The identifier of the object data to be updated is the fifth identifier. If the update request carries a version number of the object data, the object data corresponding to the update request is updated based on the fifth identifier carried in the update request and the version number.

7. The method according to claim 1, characterized in that, The method further includes: After the generational update of the object data, the updated generation is notified to each access node, where the access node represents the node used to process the object data.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing computer programs capable of running on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images