Methods and apparatus for storing metadata log entries

Abstract

This specification provides a method and apparatus for storing metadata log entries. In the process of storing log data for metadata, it breaks away from the traditional transaction commit process and uses a storage interface provided by the database for data storage. During storage, after receiving at least one log entry from the current storage batch, the database encapsulates the corresponding log cache according to the storage batch, stores the log entries of the current batch as a whole, and returns the storage address of each log entry. This solution enables on-demand reading of metadata log data, simplifies the storage process, and improves storage efficiency.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of secure computing technology, and more particularly to methods and apparatus for storing metadata log entries. Background Technology

[0002] Metadata typically describes data and records information such as data attributes; logs record operations performed on data / metadata, such as creation, modification, and deletion. Logs can also be used to restore the data state in memory by reading the logs during a computer restart. In complex storage systems, it is often necessary to persist metadata (copying data from volatile memory to non-volatile hard disk) to ensure normal operation after an operating system restart. The conventional method for persisting metadata is to write the modification log to disk, so that the memory state can be restored according to the log during the next restart. Conventional techniques often use a begin-write-commit process to record metadata logs in batches. begin represents the start of the current batch of log recording, i.e., the start of the transaction, and commit represents that the current batch of logs has been completely written, i.e., the transaction has been committed. This approach ensures the atomicity of a batch of log entries through commit (either all logs in the batch are written to disk, or none are written to disk). However, this approach requires maintaining two lists, redo and undo, when replaying the logs, and typically cannot return the position of each log entry on disk. Summary of the Invention

[0003] This specification describes one or more embodiments of a method and apparatus for storing multi-dimensional data log entries, which addresses one or more problems mentioned in the background art.

[0004] According to a first aspect, a method for storing metadata log entries is provided, executed via a database, comprising: receiving at least one log entry of the current storage batch, wherein the at least one log entry is determined based on metadata modification; encapsulating each log entry by generating a log header to obtain each entry encapsulation block, wherein a single log header is used to describe the attribute information of the corresponding log entry; generating a batch header for the current storage batch based on each entry encapsulation block to encapsulate each entry encapsulation block as a whole to obtain a batch encapsulation block, wherein the batch header records at least the number of the at least one log entry and a checksum; writing the batch encapsulation block to a hard disk and obtaining the corresponding storage address; and providing feedback on the storage address of each log entry of the current storage batch on the hard disk.

[0005] In one embodiment, the modification of the metadata occurs when a data table or a file shard in a data table is created, modified, or deleted.

[0006] In one embodiment, the attribute information of a log entry includes a data hierarchy; during log replay, the method further includes replaying each log entry in descending order of data hierarchy.

[0007] In a further embodiment, the data hierarchy, from highest to lowest, includes: tenant hierarchy, business module hierarchy, and data shard hierarchy. The tenant hierarchy includes at least one tenant, a single tenant corresponds to at least one business module, and a single business module corresponds to at least one data shard. The step of replaying each log entry in descending order of data hierarchy includes: replaying the log entries of the tenant hierarchy to restore the tenant hierarchy data corresponding to each tenant; and sequentially reading the log entries of each storage batch to restore each log entry to the corresponding tenant.

[0008] In one embodiment, the step of sequentially reading log entries of each storage batch to restore each log entry to the corresponding tenant includes: for log entries of a single storage batch, reading the corresponding batch header to obtain the number of log entries in that single storage batch; and sequentially reading each log entry according to the number of log entries.

[0009] In one embodiment, the attribute information of a log entry also includes the operation type for the data; when reading a single log entry, the corresponding log entry type is first obtained from the log header, and then the log entry is parsed according to the operation type recorded in the log header.

[0010] In one embodiment, the method further includes: receiving a log entry read request, wherein the log entry read request includes the storage address of the log entry to be read on the hard disk; reading the corresponding log entry from the hard disk according to the storage address of the log entry to be read on the hard disk, and loading the corresponding metadata.

[0011] In one embodiment, feeding back the storage address of each log entry in the current storage batch on the hard disk includes: obtaining the storage address of at least one log entry in the current batch as a whole on the hard disk; and determining and feeding back the storage address of each log entry and log header based on the number of log entries in the current storage batch.

[0012] According to the second aspect, a storage device for metadata log entries is provided, located at the database end, comprising:

[0013] A receiving unit is configured to receive at least one log entry of the current storage batch, wherein the at least one log entry is determined based on metadata modifications;

[0014] The first encapsulation unit is configured to encapsulate each log entry by generating a log header to obtain each entry encapsulation block, wherein a single log header is used to describe the attribute information of the corresponding log entry;

[0015] The second encapsulation unit is configured to generate a batch header for the current storage batch, so as to encapsulate each entry encapsulation block as a whole to obtain a batch encapsulation block. The batch header records at least the number of the at least one log entry and the checksum.

[0016] The storage unit is configured to write the batch packaged blocks to a hard disk and obtain the corresponding storage address;

[0017] The feedback unit is configured to provide the storage address of each log entry on the hard disk.

[0018] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0019] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.

[0020] The methods and apparatus provided in the embodiments of this specification can record metadata log entries in a format of batch header, log header, and log entries on a hard disk. The batch header stores the number of log entries, the total length of the current batch file, and a checksum, ensuring the atomicity of log entries within a batch. After the log entries are stored, the storage address is used as an output parameter to provide the storage address of the log entries in the current batch on the hard disk, facilitating on-demand loading. In summary, the technical solution of this specification simplifies the metadata log storage process and improves the availability of log entries. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This diagram illustrates the implementation architecture of a storage scheme for log entries of metadata based on the technical concept described in this specification.

[0023] Figure 2 This diagram illustrates the logical structure of log entries encapsulated based on a specific example.

[0024] Figure 3 A flowchart illustrating a method for storing metadata log entries according to one embodiment is shown;

[0025] Figure 4 This diagram illustrates the encapsulation logic for storing log entries under different scenarios based on a specific example of a first storage interface.

[0026] Figure 5 A flowchart illustrating a method for storing metadata log entries according to another embodiment is shown;

[0027] Figure 6 A schematic block diagram of a storage device for metadata log entries located at a database end, according to one embodiment;

[0028] Figure 7 A schematic block diagram of a storage device for metadata log entries located on an application server according to another embodiment is shown. Detailed Implementation

[0029] The technical solutions provided in this specification are described below with reference to the accompanying drawings.

[0030] To better understand the technical solutions in this specification, we will first describe the technical background of the solutions presented in this specification in conjunction with a specific implementation scenario.

[0031] Figure 1 This specification illustrates a specific implementation architecture. This architecture involves at least one business server and a database. The business server provides corresponding business support for various users' related business operations (such as search, query, payment, and navigation) on their respective terminals. In providing this support, the business server can write and read data from the database. For example, the business server might be a server providing support for shopping, lending, and navigation services on the terminal. Accordingly, the business server can write or read shopping record data (such as product category, amount, and time), lending record data (such as loan amount, loan type, and repayment time), and navigation record data (such as navigation time, time to destination, navigation route, and actual route). The database can provide data storage services for one type of business server or multiple business servers. Typically, the platform set up by the business server to provide the corresponding business can be referred to as the upper-layer application storing data in the database, such as a search application, payment application, or navigation application.

[0032] Data generated by the business server can be recorded through various data tables. Information about these data tables can be recorded through metadata. Metadata is data that describes the data. Thus, metadata can record at least one piece of information about a data table, such as its creation, deletion, update, and data length. This information can be recorded in the form of metadata logs. Metadata logging can be performed by the business server calling the corresponding interface on the computing platform within the database, which can then be handled by the database's computing platform. This computing platform can be a data management platform within the database, etc. A common logging method is to use a begin-write-commit process to record metadata logs in batches. begin represents the start of this batch of logs, i.e., the start of the transaction, and commit represents the completion of the transaction after all logs in this batch have been written. This scheme uses a transaction commit approach, calling the interface three times, corresponding to begin, write, and commit respectively, to complete the storage of metadata logs. It uses commit to ensure the atomicity of this batch of logs (either all logs in this batch are written to disk, or none are written to disk).

[0033] In conventional log storage methods, metadata logs can only be read sequentially during playback to restore metadata one by one. However, in practice, there may be a need to load metadata log entries on demand (loading only the necessary metadata log entries). Therefore, while ensuring the atomicity of log entries, this specification provides a new metadata logging protocol to simplify the process for business servers (such as...). Figure 1 The process of the business server in the database recording metadata to the database, and the on-demand loading of metadata based on the corresponding protocol.

[0034] The metadata logging protocol proposed in this specification allows a business server or a storage platform connected to the business server to call the batch storage interface provided by the database to send log entries to be stored to the database. The database then records the data in the format of batch header + (log header + log entry). Figure 2 As shown. The batch header can record the number of log entries, such as 1 or n (where n is an integer greater than 1), as well as the length of the current batch's log entries, checksum, etc. A batch of log entries can include one or more log entries. A single log entry can be stored in the database in the format of "log header (entry) + log entry (log)".

[0035] Under this technical concept, the database can provide one or two storage interfaces. Using storage interfaces avoids the three interface calls in the transaction commit process, simplifying the storage process. Simultaneously, a batch of log entries to be stored can be submitted via the input parameters of the storage interface, and the output parameters return the storage address of each log entry. Thus, when loading metadata logs, the corresponding log entries can be loaded via the storage address of the target log entry, achieving on-demand loading.

[0036] As an example, suppose the database can provide two batch storage interfaces. One batch storage interface is a single-entry storage interface, used to write a single log entry. A batch containing one log entry can call the single-entry storage interface, denoted as the `write` interface. The input parameter for this single-entry storage interface can be the identifier data corresponding to the single log entry (denoted as `param`). The business server or the aforementioned storage platform can use the identifier data corresponding to the single log entry as the input parameter to call the storage interface for writing the single log entry (such as the `write` interface) to store the metadata log. After completing the storage of the single log entry, the database can then return the corresponding single storage address to the business server or the aforementioned storage platform. The other batch storage interface can be a multi-entry storage interface, used to write multiple log entries. For example, if the current batch contains n log entries, this interface could be denoted as the `batch_write` interface. The input parameter for this multi-entry storage interface can be an array, denoted as an identifier array (`param_arr`). The business server or the aforementioned intermediate platform can use the data corresponding to multiple consecutive log entries as input parameters to call the storage interface (such as the batch_write interface) that writes multiple log entries to store metadata logs. The identifier data corresponding to each log entry can be the cache identifier (such as the buffer name or logical address of the buffer) of a pre-processed log buffer. Multiple log entries can correspond to multiple serialized buffers, and the identifiers of these buffers constitute the input parameter array (or input parameter group) of the storage interface. After storing each log entry, the database can then return the storage address corresponding to each log entry to the business server or the aforementioned storage platform.

[0037] The technical concept of this specification is described in detail below.

[0038] Figure 3 This specification illustrates a process for storing metadata log entries according to one embodiment. The entity executing this process can be a device, computer, server, or similar device with computing power. More specifically, for example... Figure 1The computing platform shown, etc. This process is based on logging changes to the metadata of data stored on the business data server.

[0039] Metadata can be used to record information such as the creation, modification, or deletion of database tables or shards within database tables. A shard (also called a partition) in a database table is a smaller storage unit (table or database) created from larger data within the same table. Sharding can be vertical or horizontal. Vertical sharding typically divides data by columns (fields), for example, dividing a table into one shard containing "ID," "Name," and "Occupation" fields, and another shard containing "ID," "Item of Purchase," and "Amount of Purchase." Horizontal sharding divides data by rows, with each shard containing all fields; for example, rows with IDs from 0 to 100 are divided into one shard, and rows with IDs from 101 to 300 are divided into another shard.

[0040] When metadata is modified, corresponding metadata log records can be generated. These metadata log records can be written to the hard disk for subsequent loading of metadata and reading of data table / data table shard modification records as needed, or for replay during system restarts.

[0041] like Figure 3 As shown, the process of storing metadata log entries may include the following steps: Step 301, receiving at least one log entry of the current storage batch, wherein the at least one log entry is determined based on the modification of metadata; Step 302, encapsulating each log entry by generating a log header to obtain each entry encapsulation block, wherein a single log header is used to describe the attribute information of the corresponding log entry; Step 303, generating a batch header for the current storage batch based on each entry encapsulation block to encapsulate each entry encapsulation block as a whole to obtain a batch encapsulation block, wherein the batch header records at least the number of log entries in the current storage batch and the checksum; Step 304, writing the above batch encapsulation block to the hard disk and obtaining the corresponding storage address; Step 305, providing feedback on the storage address of each log entry in the current storage batch on the hard disk.

[0042] First, in step 301, at least one log entry of the current storage batch is received.

[0043] Under the technical concept of this specification, metadata log entries are stored in batches. A single batch can store one or more log entries. Metadata log entries are generated based on changes to metadata. Changes to metadata are determined based on changes to data tables or sharded data within data tables. Metadata log entries can be determined and transmitted by the application server based on the data storage of the relevant platform, or they can be generated by the storage platform based on changes in database information detected when the application server writes data to the database, or they can be generated through other reasonable methods, which are not limited in this specification. For example, when the application platform corresponding to the business server (also known as the upper-layer application) writes data through a log structure merging tree (LSM-tree), and the in-memory table is transferred to an SSTable file, operations such as the creation of SSTable files are triggered, resulting in changes to the corresponding metadata. In this case, the metadata change information corresponding to one or more newly created SSTable files can be stored as a log entry for a storage batch.

[0044] The storage of metadata log entries can be initiated by the application server or the storage platform mentioned earlier. For example, the application server can call the batch storage interface described above to provide the database with at least one log entry of the current storage batch. The batch storage interface can be a single-entry storage interface or a multi-entry storage interface. Correspondingly, the database can receive a log cache identifier for a log entry, or a parameter array consisting of multiple log cache identifiers corresponding to multiple log entries, through the corresponding storage interface. The log cache identifier corresponding to a single log entry can be the name or logical address of the log cache (buffer) obtained by preprocessing the single log entry.

[0045] Next, in step 302, each log entry is encapsulated by generating a log header to obtain each entry encapsulation block.

[0046] A single log header file can be used to record the relevant attribute information of a single log entry. As a concrete example, a single shard of a data table can contain multiple SSTable files. A single log entry describes information such as the number, length, and version number of the SSTable files in that single shard, while the log header file can describe attribute information such as the length, checksum, and data type of the log entry. The database can generate corresponding log headers for each log entry based on its attribute information.

[0047] like Figure 2 As shown, a single log entry is appended to the corresponding log header, serving as the entry encapsulation block for that single log entry. Furthermore, in the case of multiple log entries, the entry encapsulation blocks corresponding to each log entry can be arranged sequentially to form an entry encapsulation block sequence (e.g., ...). Figure 2(As shown). This entry encapsulation block sequence can be generated in memory and can serve as the data that the current storage batch needs to be written to disk.

[0048] Then, via step 303, a batch header for the current storage batch is generated based on each entry encapsulation block, and each entry encapsulation block is encapsulated to obtain a batch encapsulation block.

[0049] When storing the corresponding log cache according to the storage protocol described in this specification, the log cache of a single storage batch can be encapsulated using a corresponding batch header. The batch header of the current storage batch is generated based on at least one log entry and its respective log header. The batch header can be used to describe information such as the number of log entries (e.g., the number of buffers), length (number of bytes), and checksum. The checksum is used to ensure data integrity. Figure 2 This diagram illustrates a batch packaging block based on the technical concept of this specification. Specifically, it consists of a "batch header + several entry packaging blocks". More specifically, as shown... Figure 4 As shown, when there are multiple log entries in the current storage batch, the batch encapsulation block can be, for example, [the following]. Figure 4 The logical form of dashed box 401 is batch header + (log header + log entry) + ... + (log header + log entry). On the other hand, when the log entry of the current storage batch is a single entry, the batch encapsulation block can be, for example, in the logical form of dashed box 402, which is "batch header + (log header + log entry)".

[0050] Then, in step 304, the batch packaging block is written to the hard disk and the corresponding storage address is obtained.

[0051] By calling the corresponding hard disk storage interface, the database can write at least one log entry from the current batch to the hard disk. This can be understood as the process of writing data from memory to hard disk, which is the process of data persistence. The data written to the hard disk can have a corresponding storage address. This storage address can correspond to a physical address on the hard disk, allowing direct access via the physical address when the relevant data is needed. This physical address can serve as feedback information from the corresponding storage interface. Figure 3 The execution entity of the illustrated process can obtain the storage address of the current storage batch from the output parameters of the storage interface that writes data to the hard disk.

[0052] Furthermore, the process includes step 305, which returns the storage address of each log entry on the hard drive. This is understandable. Figure 3 After obtaining the storage address of the current storage batch from the output parameters of the second storage interface, the execution entity of the illustrated process can also feed back the corresponding storage address to the application server or the storage platform mentioned above.

[0053] The database can provide the storage address of each log entry. Typically, the length of a single, encapsulated log entry is fixed. The storage address of each log entry can be determined using the storage address of the current storage batch, as well as information such as the length of the current storage batch and the number of log entries contained in the batch header. After obtaining the storage address of at least one log entry in the current batch on the hard drive, the storage address of each log entry can be calculated based on the batch header length and the number of log entries. For example, if the storage address of the batch encapsulation block corresponding to the current storage batch is 10225 to 10283, occupying 59 storage locations, and assuming the batch header occupies 5 storage locations (e.g., 10225 to 10230), and the number of log entries is 6, then a single log entry occupies 9 storage locations. The storage addresses of each log entry would be, for example, 10231 to 10239, 10240 to 10248…10275 to 10283. The database can provide these storage addresses.

[0054] Under the technical concept described in this specification, the application server or the aforementioned storage platform can perform corresponding operations and cooperate with the database to complete the storage process of metadata log entries. Taking the application server as an example, Figure 5 This is a metadata log entry storage process executed by the application server according to one embodiment. Wherein, Figure 5 The execution entity in the process can also be replaced by the aforementioned storage platform. For example... Figure 5 As shown, the metadata log entry storage process executed by the application server may include: Step 501, obtaining at least one log entry in the current storage batch, wherein at least one log entry is determined based on the modification of metadata; Step 502, serializing each log entry to each log cache; Step 503, providing each log entry to the database in the form of a log cache, so that the database can store each log entry according to the storage batch after performing entry encapsulation and batch encapsulation respectively; Step 504, receiving the storage address of each log entry on the hard disk from the database.

[0055] In step 501, at least one log entry for the current storage batch is obtained. This at least one log entry can be determined based on metadata modifications. For example, when an upper-layer application writes a purchase record, it might write the corresponding information to two shards of a shopping information table. This changes the length information of both shards, i.e., the metadata changes, potentially generating two log entries for the two shards' metadata respectively. These two metadata log entries can then be stored as two log entries in a single storage batch.

[0056] In step 502, each log entry is serialized to its respective log cache.

[0057] Buffers are frequently used to store data output to external devices. In the technical concept of this specification, individual log entries can be stored using buffers, or in other words, log entries can be encapsulated into log caches one by one using buffers. When the current storage batch contains only a single log entry, that single log entry can be encapsulated into a single log cache. When the current storage batch contains multiple log entries, to better store log metadata according to the batch, the individual log entries can be serialized, i.e., arranged sequentially.

[0058] In step 503, each log entry is provided to the database in the form of a log cache.

[0059] Individual log entries can be provided to the database via the logical address of the corresponding log cache. Specifically, this can be achieved by calling the batch storage interface provided by the database, and passing the cache identifier (such as buffer name or logical address) of each log cache corresponding to each log entry as the input parameter of the batch storage interface.

[0060] The database can further encapsulate and process each log entry in the current storage batch, obtaining the encapsulated result of the entire storage batch and storing it on disk. Specifically, the database receives the log cache for each log entry, encapsulates each log entry individually, and then performs batch encapsulation on the encapsulated log entries. The encapsulation result is, for example,... Figure 4 The multi-entry encapsulation result shown is "batch header + (log header + log entry) + ... + (log header + log entry)", or the single-entry encapsulation result is "batch header + (log header + log entry)". These multi-entry or single-entry encapsulation results can also be referred to as batch encapsulation blocks. Subsequently, the database can store the log entries of the current storage batch according to the corresponding encapsulation blocks, and obtain the storage results of each log entry on the hard disk through the output parameters of the storage interface. Correspondingly, the application server can receive the storage addresses of each log entry on the hard disk from the database through step 504.

[0061] Log data stored in this way provides the storage address of the current batch of log entries on the hard drive. Therefore, under the requirement of on-demand loading, the corresponding log data can be directly loaded according to the storage address, thus realizing the requirement of on-demand loading.

[0062] Furthermore, in situations such as system restarts, log data can be read sequentially to replay the logs and reload metadata into memory, eliminating the need to maintain additional lists such as undo and redo tables. During log replay, log entries from each storage batch can be read sequentially. Specifically, when reading log entries from a single storage batch, the batch header corresponding to the log entry is read first, followed by the log entry header and then the log entry itself.

[0063] In some optional embodiments, the database can also hierarchically divide the log entries of the metadata according to the data described by the metadata, and record the hierarchical information of the data corresponding to the log entries in the log header. Here, the data hierarchy can be the level to which the data belongs. For example, in a specific implementation, the data can be divided into three levels: tenant level, tenant business module level, and data sharding level. The tenant level can be a data classification at the tenant level, such as the shopping cart service provider on a shopping platform being a tenant, the payment service provider on a payment platform being a tenant, and so on. The tenant business module can be a data classification at the specific business module of the tenant, such as the add / delete scheduling function in the shopping cart being a business module, the collection and payment functions on the payment platform being a business module, and so on. Data sharding can be a data classification of the data storage structure of a data table. Specifically, the stored data table can be divided into multiple parts, each part being a data shard, such as a shard of the shopping cart add-to-cart list.

[0064] In this way, when the database encapsulates log entries for a single storage batch, it can describe the data level of the corresponding log entry in the log header. Since higher-level data is created before lower-level data, it ensures that the metadata corresponding to the user's business unit is written at the user level, before the corresponding data shard. For example, before the metadata log record corresponding to the metadata of a shard in the shopping cart add-to-cart list, the shopping cart tenant must have been established first, and the "add to cart" (add to cart) functional module must have been built before a data shard of the "architecture list" can be created.

[0065] Furthermore, during the log replay process based on operations such as system restart at the database end, log entries at each level can be replayed in descending order of data hierarchy. For example, first, log entries at the tenant level are replayed to restore the corresponding tenant-level data for each tenant. Then, log entries from each storage batch are read sequentially to restore each log entry to its corresponding tenant. During this process, because log entries from different storage batches are intertwined with data at the business module and data shard levels, the replay process, performed batch by batch, involves the intertwined replay of data from business modules and data shard levels, ultimately restoring the data to the corresponding tenant. For log data in a single storage batch, the batch header (batch_header) can be read first to determine the number of log entries (log) in the current storage batch, and then each log entry in the current storage batch can be read individually. When reading log entries, the log header can be read first to determine the data hierarchy to which the log entry belongs, and then the corresponding data can be replayed to the corresponding tenant and hierarchy.

[0066] In some alternative implementations, the log entry attribute information recorded in the log header also includes the operation type for the data. The data operation type of the metadata log entry can describe different operations on the corresponding data level, such as creation (e.g., creating a shard), deletion (e.g., deleting a shard), modification (e.g., adding or deleting at least one piece of data in a shard), etc. Under different operation types, the metadata can record different information, and the format of the log entry may also differ. For example, under the creation type, information such as the ID, length, and identifier of the corresponding data structure can be recorded, while under the deletion type, only the ID information of the corresponding data structure needs to be recorded. Different operation types may also have different data encapsulation and parsing processes. For example, each operation type corresponds to a data storage and parsing format. The database can record the corresponding type information through the log header (e.g., entry) when encapsulating entries. When reading each log entry, the operation type information recorded in the corresponding log header (entry) can be read first, and then the corresponding parsing operation can be performed according to the corresponding operation type to parse the content of the log entry and reconstruct the data in the corresponding log entry in memory.

[0067] In summary, the technical concept presented in this specification, when storing metadata log data, can break away from the traditional data storage model based on transaction commit processes through the storage interface provided by the database. After receiving the log entries of the current storage batch, the database first encapsulates the log entries (e.g., obtaining an entry encapsulation block), then encapsulates the entire batch of log entries (e.g., obtaining a batch encapsulation block), and stores the overall encapsulation result (e.g., the batch encapsulation block) on the hard disk. After obtaining the storage address of the log entries in the current storage batch on the hard disk, it returns the individual storage address of each log entry. In this way, on-demand loading or reading of metadata log data can be achieved, simplifying the storage process and improving storage efficiency.

[0068] According to another embodiment, a storage device for metadata log entries is also provided. This device can be located at the database end, such as at... Figure 1 The computing platform within it is used to control how data is stored.

[0069] Figure 6 This is a block diagram of a storage device for metadata log entries according to one embodiment of this specification. Figure 6 As shown, device 600 may include:

[0070] The receiving unit 601 is configured to receive at least one log entry of the current storage batch, wherein the at least one log entry is determined based on the modification of metadata;

[0071] The first encapsulation unit 602 is configured to encapsulate each log entry by generating a log header, thereby obtaining each entry encapsulation block. In this block, a single log header is used to describe the attribute information of the corresponding log entry.

[0072] The second encapsulation unit 603 is configured to generate a batch header for the current storage batch, so as to encapsulate each entry encapsulation block to obtain a batch encapsulation block. The batch header records at least the number of at least one log entry and the checksum.

[0073] Storage unit 604 is configured to write batch packaged blocks to the hard disk and obtain the corresponding storage address;

[0074] Feedback unit 605 is configured to provide feedback on the storage address of each log entry on the hard disk.

[0075] Figure 7 A block diagram of a storage device for metadata log entries according to one embodiment of this specification is also shown. This device can be located on an application server or the storage platform described above, and... Figure 6 The illustrated devices 600 work together to complete the storage process of metadata log data. For example... Figure 7 As shown, the device 700 may include:

[0076] The acquisition unit 701 is configured to acquire at least one log entry of the current storage batch, wherein the at least one log entry is determined based on the modification of the metadata corresponding to the application supported by the application server;

[0077] Preprocessing unit 702 is configured to serialize each log entry to each log cache;

[0078] Unit 703 is configured to provide each log entry to the database in the form of a log cache, so that the database can store each log entry in a storage batch after it has been encapsulated into an entry and a batch respectively.

[0079] The receiving unit 704 is configured to receive the storage addresses of each log entry on the hard disk from the database.

[0080] It is worth noting that, Figure 6 , Figure 7 The devices 600, 700 and shown are Figure 3 , Figure 5 Corresponding to the described method, Figure 3 , Figure 5 The corresponding descriptions in the method embodiments also apply to devices 600 and 700, and will not be repeated here.

[0081] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 , Figure 5 The methods described above.

[0082] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 , Figure 5 The methods described above.

[0083] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0084] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. A method for storing metadata log entries, executed via a database, comprising: Receive at least one log entry from the current storage batch, wherein the at least one log entry is determined based on metadata modifications and uploaded through a storage interface provided by the database. Each log entry is encapsulated by generating a log header to obtain an entry encapsulation block. Each log header is used to describe the attribute information of the corresponding log entry. A batch header for the current storage batch is generated based on each entry encapsulation block, and the batch encapsulation block is obtained by encapsulating each entry encapsulation block as a whole. The batch header records at least the number of the at least one log entry and the checksum. Write the batch packaged blocks to the hard disk and obtain the corresponding storage address; It provides the storage address on the hard drive for each log entry in the current storage batch.

2. The method according to claim 1, wherein, The modification of the metadata occurs when a data table or a file fragment in a data table is created, modified, or deleted.

3. The method according to claim 1, wherein, The attribute information of log entries includes data hierarchy; During log replay, the method further includes: Each log entry is replayed in descending order of data hierarchy.

4. The method according to claim 3, wherein, The data hierarchy, from highest to lowest, includes: tenant hierarchy, business module hierarchy, and data shard hierarchy. The tenant hierarchy includes at least one tenant, each tenant corresponds to at least one business module, and each business module corresponds to at least one data shard. The process of replaying log entries in descending order of data hierarchy includes: Replay the log entries at the tenant level to restore the tenant level data corresponding to each tenant. Log entries from each storage batch are read sequentially to restore each log entry to the corresponding tenant.

5. The method according to claim 4, wherein, The step of sequentially reading log entries from each storage batch to restore each log entry to the corresponding tenant includes: For log entries in a single storage batch, read the corresponding batch header to obtain the number of log entries in that single storage batch; Read each log entry sequentially according to the stated number of log entries.

6. The method according to any one of claims 3-5, wherein, The attribute information of a log entry also includes the operation type for the data; when reading a single log entry, the corresponding log entry type is first obtained from the log header, and then the log entry is parsed according to the operation type recorded in the log header.

7. The method according to claim 1, wherein, The method further includes: Receive a log entry read request, wherein the log entry read request includes the storage address of the log entry to be read on the hard disk; The corresponding log entry is read from the hard drive according to its storage address on the hard drive, and the corresponding metadata is loaded.

8. The method according to claim 1, wherein, The feedback includes the storage addresses on the hard drive for each log entry in the current storage batch, including: Obtain the storage address on the hard disk of at least one log entry in the current batch; Based on the number of log entries in the current storage batch, determine and return the storage address of each log entry and log header.

9. A storage device for metadata log entries, located at a database, comprising: The receiving unit is configured to receive at least one log entry of the current storage batch, wherein the at least one log entry is determined based on the modification of metadata and is uploaded through the storage interface provided by the database. The first encapsulation unit is configured to encapsulate each log entry by generating a log header to obtain each entry encapsulation block, wherein a single log header is used to describe the attribute information of the corresponding log entry; The second encapsulation unit is configured to generate a batch header for the current storage batch based on each entry encapsulation block, so as to encapsulate each entry encapsulation block as a whole to obtain a batch encapsulation block. The batch header records at least the number of the at least one log entry and the checksum. The storage unit is configured to write the batch packaged blocks to a hard disk and obtain the corresponding storage address; The feedback unit is configured to provide the storage address of each log entry on the hard disk.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-8.

Images