Log persistence method and device based on asynchronous pipeline, equipment and medium

Abstract

The present specification provides a log persistence method and device based on an asynchronous pipeline, equipment and medium. The application sets a dedicated persistence engine and FIFO linked list buffer for each bucket (i.e. log object), so that the log objects are physically and logically isolated, the persistence operations of multiple log objects are executed in parallel and do not interfere with each other, and each log object accurately delivers its own log data to a dedicated processing channel, i.e. when writing to disk, the log object is written to the corresponding storage area. This avoids the problem of existing "memory buffer + batch storage" that causes the logs of a target bucket to be stored in different places, and even causes serious disk IO competition and lock conflicts in high concurrency. The present embodiment completely breaks the global IO bottleneck by physically and logically isolating each log object, greatly improving the resource utilization efficiency in a high concurrency scenario.

Description

Technical Field

[0001] This specification relates to the field of storage technology, and in particular to a method, apparatus, device and medium for log persistence based on asynchronous pipelines. Background Technology

[0002] In a distributed object storage system, to meet the needs of auditing and troubleshooting, every operation a user performs on the source bucket generates a log record, which is then persisted to the target bucket's storage space.

[0003] Common log persistence methods typically employ a "memory buffer + batch write" strategy: the system first temporarily stores the generated logs in a memory buffer, and then writes this batch of logs to persistent storage all at once after the buffer has accumulated a certain number. However, since the target buckets corresponding to these logs are different, logs belonging to the same target bucket will be scattered and stored in different locations of persistent storage, which can lead to severe disk I / O contention and lock conflicts under high concurrency. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this specification provides a log persistence method, apparatus, device and medium based on asynchronous pipeline.

[0005] According to a first aspect of the embodiments of this specification, an asynchronous pipelined log persistence method is provided, the method comprising: when the persistence layer receives log data, routing the log data to the persistence engine associated with the log object to which the log data belongs, so that the persistence engine first records the log data into the first-in-first-out (FIFO) linked list buffer corresponding to the persistence engine; different log objects are associated with different persistence engines; The persistence engine retrieves log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner. The starting offset of this write operation is determined by the amount of written data to the log object to which the log data belongs. Based on the starting offset, the log data is persisted to the log object, and the amount of written data to the log object is updated.

[0006] According to a second aspect of the embodiments of this specification, an asynchronous pipeline-based log persistence device is provided, the device including a routing module and a persistence module; wherein different persistence modules are associated with different log objects; This routing module is used to route log data to the persistence module associated with the log object to which the log data belongs when the persistence layer receives log data. The persistence module is used to record the log data to its corresponding first-in-first-out (FIFO) linked list buffer. The persistence module is also used to retrieve a log data from its corresponding FIFO linked list buffer in a first-in-first-out manner, determine the starting offset of this write operation based on the amount of written data of the log object to which the recorded log data belongs, persist the log data to the log object based on the starting offset, and update the amount of written data of the log object.

[0007] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: Processor; memory used to store processor-executable instructions; The processor is configured to implement the asynchronous pipeline-based log persistence method of this application when executing the program.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the asynchronous pipeline-based log persistence method of this application.

[0009] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: This application sets up a dedicated persistence engine and FIFO linked list buffer for each bucket (i.e., log object) to physically and logically isolate each log object, allowing the persistence operations of multiple log objects to be executed in parallel without interference. Each log object accurately delivers its own log data to its dedicated processing channel, i.e., when writing to disk, it is written to the storage area corresponding to the log object in a concentrated manner. This avoids the problem of logs from the same target bucket being stored in a scattered manner, and even serious disk I / O competition and lock conflicts caused by the existing "memory buffer + batch transfer". This embodiment completely breaks the global I / O bottleneck by physically and logically isolating each log object, and greatly improves the resource utilization efficiency in high-concurrency scenarios.

[0010] Furthermore, in this embodiment, the persistence engine of each log object continuously extracts log data belonging to each log object from the head of the FIFO linked list like a pipeline and initiates asynchronous writing. The CPU does not need to wait for disk I / O to return before processing the next piece of data, ensuring that this "direct disk write" mode can still maintain extremely high throughput under extremely high concurrency.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0013] Figure 1 This is a flowchart illustrating an asynchronous pipeline-based log persistence method according to an exemplary embodiment of this specification; Figure 2 This is an exemplary flowchart of an asynchronous pipeline log persistence method according to this application; Figure 3 This is an interactive diagram illustrating an asynchronous pipeline log persistence method according to this application; Figure 4 This is a schematic diagram illustrating the structure of an asynchronous pipeline-based log persistence device according to an exemplary embodiment of this application; Figure 5 This is a hardware structure diagram of an electronic device containing an asynchronous pipelined log persistence device, as described in the embodiments of this specification. Detailed Implementation

[0014] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0015] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0017] The embodiments described in this specification will now be described in detail.

[0018] like Figure 1 As shown, Figure 1 This specification illustrates a flowchart of an asynchronous pipeline-based log persistence method according to an exemplary embodiment, including the following steps: In step 102, when the persistence layer receives log data, it routes the log data to the persistence engine associated with the log object to which the log data belongs, so that the persistence engine first records the log data into the first-in-first-out (FIFO) linked list buffer corresponding to the persistence engine; different log objects are associated with different persistence engines.

[0019] The persistence layer, acting as a data relay layer, receives log data generated to provide services to upper-layer applications and routes the log data to the persistence engine associated with its respective log object. Multiple log data belonging to the same log object share commonalities, such as being generated from operations on the same source bucket, or having similar characteristics in the operation process or result, for example, multiple log data recording operation results as system errors.

[0020] Each log object is pre-associated with a persistence engine, and each persistence engine maintains a first-in-first-out (FIFO) linked list buffer responsible for recording all log data belonging to its corresponding log object.

[0021] The characteristics of a FIFO linked list buffer are that it allows log data to be placed into the tail of the FIFO linked list in the order it arrives at the persistence engine; and it requires that when reading log data, it should be read from the head of the FIFO linked list.

[0022] In some embodiments, log data carries an identifier associated with a log object. Based on the identifier of the log data and a routing mapping table, the log data can be determined and routed to the persistence engine associated with the log object to which the log data belongs. The routing mapping table records the mapping relationship between log objects and persistence engines.

[0023] Through step 102, in the memory space of the persistence layer, log data can be routed to the persistence engine associated with its log object and stored in the corresponding FIFO linked list buffer, thus realizing the classification caching of log objects.

[0024] In step 104, the persistence engine retrieves log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner, and determines the starting offset of this write operation based on the amount of written data of the log object to which the log data belongs. Based on the starting offset, the log data is persisted to the log object, and the amount of written data of the log object is updated.

[0025] When persisting log data in the FIFO linked list buffer, a log data is retrieved from the head of the FIFO linked list corresponding to the persistence engine and persisted in a first-in-first-out manner. That is, the log data that first entered the FIFO linked list buffer among all the log data stored in the current FIFO linked list buffer is selected for persistent storage.

[0026] The amount of data written to the log object to which the recorded log data belongs is the amount of data that has been persistently written to the log object. This amount of data written can be maintained by the persistence engine. Using this initial amount of data as the starting offset for this write operation, after the log data is persistently written to the log object, it can ensure that the log data is stored continuously with the previously persistently written log data.

[0027] To ensure the accuracy of the written data volume, after the log data is persisted, the written data volume of the log object can be updated immediately based on the current written data volume and the current persistent written data volume. For example, the updated written data volume is the sum of the current written data volume and the log data volume.

[0028] pass Figure 1 As shown in the process diagram, this embodiment sets up a dedicated persistence engine and FIFO linked list buffer for each bucket (i.e., log object) to physically and logically isolate each log object, allowing the persistence operations of multiple log objects to be executed in parallel without interference. Each log object accurately delivers its own log data to its dedicated processing channel, i.e., it is written to the storage area corresponding to the log object when it is written to disk. This avoids the problem of logs from the same target bucket being stored in a scattered manner, and even serious disk I / O competition and lock conflicts caused by the existing "memory buffer + batch transfer". This embodiment completely breaks the global I / O bottleneck by physically and logically isolating each log object, and greatly improves the resource utilization efficiency in high-concurrency scenarios.

[0029] Furthermore, in this embodiment, the persistence engine of each log object continuously extracts log data belonging to each log object from the head of the FIFO linked list like a pipeline and initiates asynchronous writing. The CPU does not need to wait for disk I / O to return before processing the next piece of data, ensuring that this "direct disk write" mode can still maintain extremely high throughput under extremely high concurrency.

[0030] In some embodiments, step 106 may be performed after step 104: If there is still log data in the FIFO linked list buffer corresponding to the persistence engine that has not been persisted to the log object, then return to step 104, that is, return to the step of obtaining a log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner.

[0031] The process involves determining whether the FIFO linked list buffer corresponding to the persistence engine is empty. If it is not empty, the next log data is retrieved from the FIFO linked list buffer in a first-in-first-out manner. The starting offset for the write operation of the next log data is determined based on the amount of data written to the updated log object, and the next log data is then persistently written. This process is repeated, retrieving log data from the head of the FIFO linked list and persistently writing it, until the FIFO linked list buffer is empty.

[0032] In some embodiments, the next log data can be written to the log object asynchronously with the previously persisted log data, based on the starting offset. Specifically, after retrieving the log data from the FIFO linked list buffer, the amount of log data can be determined. After initiating a write operation on the log data, the persistence engine can update the write amount based on the log data amount, and then return to execute the operation of retrieving log data from the FIFO linked list buffer. For the retrieved next log data, the starting offset of the next log data can be determined based on the updated write amount, and the write operation of the next log data can be initiated; then the write amount of the log object is updated again. Through this asynchronous writing method, the persistence engine does not need to wait for the log data persistence write to complete before initiating the writing of the next log data. The persistence writes of multiple log data can be executed in parallel, significantly improving the throughput of persistence writes.

[0033] In some embodiments, based on asynchronous writing, multiple consecutive log data can be packaged into a data block starting from the head of the FIFO linked list according to the first-in-first-out rule, and the log data can be persistently written in units of data blocks, further reducing the number of IO operations.

[0034] In some embodiments, to prevent the loss of log data, when an instruction to restart or shut down the persistence engine is detected, if there is still log data in the FIFO linked list buffer corresponding to the persistence engine that has not been persisted to the log object, the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner is returned until the FIFO linked list buffer is cleared.

[0035] In some embodiments, step 101 may be included before step 102: The object storage gateway service process is started, at least one thread is created, and an independent cache is allocated to each thread. When the object storage gateway service process receives a read / write request, it assigns the read / write request to a thread for processing. The thread creates a worker coroutine for the read / write request, completes the read / write request, generates the corresponding log data, and terminates the worker coroutine after storing the log data in the thread's cache. When the flush condition is met, each thread sends the log data in its own cache to the persistence layer.

[0036] During the initial configuration of the object storage gateway service process, a preset number of threads are created, and each thread is allocated an independent cache. The object storage gateway service process translates upper-layer access to the object storage system into read and write requests, and then assigns these requests to one of its pre-created threads for processing. For example, when the object storage gateway service process receives multiple read and write requests, it distributes these requests relatively evenly across the multiple threads based on the number of read and write requests to be processed by each thread.

[0037] For a thread assigned a read / write request, upon receiving the request, it creates a worker coroutine to handle it. When a thread has multiple read / write requests to process, it can create multiple worker coroutines to handle one of the requests, and after processing, generate log data corresponding to the processed request and store the log record in the thread's buffer. Compared to threads, coroutines have lower creation and switching overhead, making them more suitable for handling a large number of read / write requests. Furthermore, since each thread has an independent buffer, only one worker coroutine is in the writing state at any given time. Therefore, serial writing of log data to the buffer on a worker coroutine basis enables exclusive access to the buffer, preventing contention among multiple worker coroutines writing to the same buffer simultaneously, and eliminating the need for buffer locking. The above configuration in this embodiment is particularly suitable for high-concurrency scenarios where a large number of read / write requests arrive at the object storage gateway service process in a short period of time, resulting in the generation of a large amount of log data.

[0038] Currently, the lifecycle of a coroutine is typically bound to the processing cycle of a single read / write request. To ensure timely feedback of read / write request results to the upper layer, the worker coroutine, after storing log data in the cache, must return the read / write request results to the upper layer to promptly meet the needs of upper-layer users and then terminate its lifecycle. This mechanism also prevents worker coroutines from occupying thread resources for extended periods, thus improving the efficiency of thread processing read / write requests.

[0039] When the flush condition is met, the log data in the buffer of each thread is sent to the persistence layer for step 102. By sending the log data in the buffers of all threads in a batch, the log data can be sent to the persistence layer in batches. The log data written to the log object in multiple batches can maintain a relative time relationship, which is convenient for users to search and retrieve. The sending order of log data in the buffers of multiple threads can be determined by the wake-up order of each thread or by the order in which the log data in the buffers of each thread enters the message queue. This embodiment does not limit this.

[0040] In some embodiments, when the total amount of log data in the buffers of all threads meets the persistence threshold, the flush condition is met, and each thread sends the log data in its own buffer to the persistence layer.

[0041] As mentioned earlier, when distributing multiple read / write requests across multiple threads, the distribution is relatively even. Therefore, after the object storage gateway service process has been running for a period of time, the number of read / write requests handled by each of its threads is also relatively even, resulting in a similar amount of log data in each thread. In some examples, the decision to send log data from each thread's cache to the persistence layer can be made based on whether the amount of log data in any thread's cache meets a single-thread threshold. For example, the maximum amount of log data in the caches of all threads can be determined, and if it exceeds the single-thread threshold, the log data can be sent to the persistence layer.

[0042] In some embodiments, when the object storage gateway service process determines that the time for the next flush has arrived, and the flush condition is met, each thread sends the log data in its own cache to the persistence layer.

[0043] For example, the object storage gateway service process uses a timer to periodically wake up threads so that the threads can send log data from the cache to the persistence layer. In some cases, the timer can be a thread created by the object storage gateway service process.

[0044] In some embodiments, when an instruction instructing the object storage gateway service process to restart or shut down is detected, and the flush condition is met, each thread sends the log data in its own cache to the persistence layer to prevent the object storage gateway service process from causing the loss of log data stored in the thread's cache.

[0045] In some embodiments, when there are multiple object storage gateway service processes, each object storage gateway service process needs to send the log data of the cache of each thread to the persistence layer.

[0046] In some embodiments, a push coroutine is created by a thread, which sends log data from the thread's cache to the persistence layer.

[0047] Since each thread's buffer is independent, and each thread's push coroutine is only responsible for sending log data from its own thread's buffer, cross-thread access to the same buffer's log data will not occur. Therefore, there is no need to lock the buffer. Furthermore, multiple threads can send log data concurrently, significantly improving the log data sending speed in high-concurrency environments.

[0048] The process of the above method will be described in detail below through a specific embodiment.

[0049] like Figure 2 and Figure 3 As shown, Figure 2 This is an exemplary flowchart of an asynchronous pipelined log persistence method according to this application. Figure 3 This is an interactive diagram illustrating an asynchronous pipeline log persistence method according to this application, including the following steps: In step 202, the object storage gateway service process is started, threads are created, and independent cache areas are allocated.

[0050] Start the object storage gateway service process, creating a preset number of threads during initialization (e.g., 3 threads, such as...). Figure 3 As shown in the figure, and allocate an independent cache area for each thread.

[0051] In step 204, read / write requests are received and assigned to threads for processing.

[0052] The object storage gateway service process translates upper-layer user accesses to the object storage system into read and write requests. When multiple read and write requests are received (e.g., user A uploads a file to source bucket 1, user B downloads a file to source bucket 2, and user C deletes a file from source bucket 3), the multiple read and write requests are relatively evenly distributed among the threads based on the number of read and write requests currently pending in each thread (e.g., in...). Figure 3 In this process, user A's request is assigned to thread 1, user B's request is assigned to thread 2, and user C's request is assigned to thread 3.

[0053] Step 206: The thread creates a worker coroutine, processes requests and generates log data, and terminates after writing the log data into the thread's independent buffer.

[0054] For a thread assigned a read / write request, a worker coroutine is created to handle the request upon receipt. Taking thread 1 as an example, within a short timeframe, in addition to receiving a read / write request from user A for uploading a file, thread 1 also receives a read / write request from user F for deleting a file. Thread 1 will then create worker coroutine 11 to handle user A's read / write request and worker coroutine 12 to handle user F's read / write request. Worker coroutines 11 and 12 share the resources of thread 1 and execute concurrently within thread 1. After completing user A's read / write request and generating the corresponding log data 1A, worker coroutine 11 stores log data 1A in the buffer and terminates its lifecycle. After completing user F's read / write request and generating the corresponding log data 1F, worker coroutine 12 stores log data 1F in the buffer and terminates its lifecycle.

[0055] Log data can record information such as the type of the log object file to which it belongs, and the time of the operation.

[0056] In step 208, it is determined whether the conditions for brushing down are met.

[0057] The object storage gateway service process continuously monitors whether the flushing conditions are met; for example, it determines whether the total amount of log data in the caches of all threads has reached a preset persistence threshold; another example is that the object storage gateway service process uses a timer to determine the current time when the preset flushing time has arrived (e.g., ...). Figure 3 (For example, when the timer on the left side of the middle meets the flush condition, thread 1 is woken up to push the log data to the persistence layer). Another example is detecting an instruction to restart or shut down the object storage gateway service process.

[0058] If the flush condition is not met, the object storage gateway service process continues to process new read and write requests; if any flush condition is met, step 210 is executed.

[0059] In step 210, each thread sends the log data in the cache to the persistence layer.

[0060] Each thread creates its own push coroutine, which sends log data from its own thread's buffer to the persistence layer. Each thread's push coroutine is only responsible for its own thread's buffer and does not access it across threads, thus eliminating the need for locking. Multiple thread push coroutines can send log data concurrently, significantly improving sending speed.

[0061] In step 212, the persistence layer receives log data, routes it to the corresponding persistence engine, and stores it in a FIFO linked list buffer.

[0062] After receiving log data from a thread, the persistence layer routes the log data to the persistence engine associated with its log object.

[0063] For example, based on the type of the log object file recorded in the log data and the routing table, it can be determined which persistence engine the log data should be routed to. Then, the corresponding persistence engine stores the log data at the tail of the FIFO linked list buffer. Through this step, log data of different log objects is classified and cached in memory.

[0064] In step 214, the persistence engine retrieves log data from the FIFO linked list buffer, persists the log data to the log object, and updates the amount of data written.

[0065] For a persistence engine, log data is retrieved from the head of its maintained FIFO linked list in a first-in, first-out (FIFO) manner. The persistence engine determines the starting offset for this log data write based on the amount of data written to the log object file, and then persists the log data to the log object (e.g., ...). Figure 3 As shown, the persistence engine writes log data to the log object in the database, and updates the write offset of the log object based on the amount of data currently being written to the log object and the starting offset of this write.

[0066] In step 216, determine whether the FIFO linked list buffer is empty. If it is not empty, repeat step 214.

[0067] If there is still next log data in the FIFO linked list buffer, return to step 214, retrieve the next log data from the head of the FIFO linked list. Based on the amount of data written to the updated log object, determine the starting offset of the next log data, write the next log data to the log file, and update the amount of data written to the log object again. Continue until the FIFO linked list buffer is empty.

[0068] Corresponding to the embodiments of the foregoing methods, this specification also provides an embodiment of a log persistence device based on asynchronous pipelines. See [link to documentation]. Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of an asynchronous pipeline-based log persistence device according to an exemplary embodiment of this application. The device may include a routing module and a persistence module; wherein different persistence modules are associated with different log objects. This routing module is used to route log data to the persistence module associated with the log object to which the log data belongs when the persistence layer receives log data. The persistence module is used to record the log data to its corresponding first-in-first-out (FIFO) linked list buffer. The persistence module is also used to retrieve a log data from its corresponding FIFO linked list buffer in a first-in-first-out manner, determine the starting offset of this write operation based on the amount of written data of the log object to which the recorded log data belongs, persist the log data to the log object based on the starting offset, and update the amount of written data of the log object.

[0069] The persistence module is also used to, after updating the amount of data written to the log object, if there is still log data in the FIFO linked list buffer corresponding to the persistence module that has not been persisted and written to the log object, return to the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence module in a first-in-first-out manner.

[0070] Optionally, if there is still log data in the FIFO linked list buffer corresponding to the persistence module that has not been persisted to the log object, then the persistence module is specifically used to: retrieve the next log data from the FIFO linked list buffer corresponding to the persistence module in a first-in-first-out manner; determine the starting offset of the write operation of the next log data based on the updated write data volume of the log object; write the next log data to the log object asynchronously with the log data that has been persisted earlier based on the starting offset; and update the write data volume of the log object.

[0071] Optionally, the persistence module is further configured to, when detecting an instruction to restart or shut down the persistence module, if there is still log data in the FIFO linked list buffer corresponding to the persistence module that has not been persisted to the log object, return to the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence module in a first-in-first-out manner.

[0072] Optionally, the device may further include a request processing module, which is configured to: before the routing module routes log data to the persistence module associated with the log object to which the log data belongs, start an object storage gateway service process, create at least one thread and allocate an independent cache for each thread; when the object storage gateway service process receives a read / write request, assign the read / write request to a thread for processing, and have the thread create a worker coroutine for the read / write request, complete the read / write request and generate the log data corresponding to the read / write request through the worker coroutine, and terminate the worker coroutine after storing the log data in the cache of the thread; when the flush condition is met, each thread sends the log data in its own cache to the persistence layer.

[0073] Optionally, the flushing condition can be any one of the following: the total amount of log data in the buffers of all threads meets the persistence threshold, the object storage gateway service process determines that the flushing time has arrived, or it detects an instruction that instructs the object storage gateway service process to restart or shut down.

[0074] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0075] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0076] Based on the same concept as the methods described above, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, Figure 5 This is a hardware structure diagram of an electronic device containing an asynchronous pipelined log persistence device according to an embodiment of this specification. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method disclosed in the above example of this application.

[0077] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods disclosed in the above examples of this application.

[0078] For example, the aforementioned readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, messages, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0079] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0080] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0081] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0082] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A log persistence method based on asynchronous pipelines, characterized in that, The method includes: When the persistence layer receives log data, it routes the log data to the persistence engine associated with the log object to which the log data belongs. The persistence engine then records the log data into the first-in-first-out (FIFO) linked list buffer corresponding to the persistence engine. Different log objects are associated with different persistence engines. The persistence engine retrieves log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner. The starting offset of this write operation is determined by the amount of written data to the log object to which the log data belongs. Based on the starting offset, the log data is persisted to the log object, and the amount of written data to the log object is updated.

2. The method according to claim 1, characterized in that, After updating the write data volume of the log object, the method also includes: If there is still log data in the FIFO linked list buffer corresponding to the persistence engine that has not been persisted to the log object, then return to the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner.

3. The method according to claim 1, characterized in that, This method also includes: Start the object storage gateway service process, create at least one thread and allocate an independent cache area for each thread; When the object storage gateway service process receives a read / write request, it assigns the read / write request to a thread for processing. The thread creates a worker coroutine for the read / write request, completes the read / write request and generates the corresponding log data, and terminates the worker coroutine after storing the log data in the thread's buffer. When the flush condition is met, each thread sends the log data in its own buffer to the persistence layer.

4. The method according to claim 3, characterized in that, The conditions for brushing down are: When the total amount of log data in the buffers of all threads meets the persistence threshold, or When the object storage gateway service process determines that the next refresh time has arrived, or When a command is detected instructing the object storage gateway service process to restart or shut down, A push coroutine is created by a thread, and the push coroutine sends the log data in the buffer of its own thread to the persistence layer.

5. The method according to claim 1, characterized in that, When an instruction to restart or shut down the persistence engine is detected, if there is still log data in the FIFO linked list buffer corresponding to the persistence engine that has not been persisted to the log object, then the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner is returned.

6. The method according to claim 2, characterized in that, The step of retrieving log data from the FIFO linked list buffer corresponding to the persistence engine in a first-in-first-out manner includes: Following a first-in-first-out (FIFO) approach, the next log data is retrieved from the FIFO linked list buffer corresponding to the persistence engine. The starting offset of the write operation for the next log data is determined based on the updated write data volume of the log object. Based on this starting offset, the next log data is written to the log object asynchronously with the log data that has been persisted earlier, and the write data volume of the log object is updated.

7. A log persistence device based on asynchronous pipelines, characterized in that, The device includes a routing module and a persistence module; wherein different persistence modules are associated with different log objects; This routing module is used to route log data to the persistence module associated with the log object to which the log data belongs when the persistence layer receives log data. The persistence module is used to record the log data to its corresponding first-in-first-out (FIFO) linked list buffer. The persistence module is also used to retrieve a log data from its corresponding FIFO linked list buffer in a first-in-first-out manner, determine the starting offset of this write operation based on the amount of written data of the log object to which the recorded log data belongs, persist the log data to the log object based on the starting offset, and update the amount of written data of the log object.

8. The apparatus according to claim 7, characterized in that, The persistence module is also used to, after updating the amount of data written to the log object, if there is still log data in the FIFO linked list buffer corresponding to the persistence module that has not been persisted and written to the log object, return to the step of retrieving a log data from the FIFO linked list buffer corresponding to the persistence module in a first-in-first-out manner.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images