Log object storage method and device, equipment and storage medium
By pre-applying the storage space of log objects in a distributed storage system and writing them directly to disk, the problem of large system resource overhead when writing logs is solved, which improves processing efficiency and reduces resource consumption.
Patent Information
- Application Number
- CN202510497981.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-21
- Publication Date
- 2025-07-22
AI Technical Summary
In distributed storage systems, the system resource overhead is too large when writing logs, resulting in unfriendly writing delays for small IOs, which becomes a bottleneck for performance improvement.
By applying for the storage space of the log object on the target disk in advance, determining the storage location of the log data to be written, and writing the data directly to the disk, avoiding the metadata of the update log object being written every time.
It improves the processing efficiency of log writing, reduces resource consumption, shortens IO processes, and reduces write delay.
Smart Images
Figure CN120353400A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data storage, and in particular to a method, device, equipment and storage medium for storing log objects. Background Art
[0002] Distributed storage systems generally exist in the form of a cluster of storage servers. According to different storage media, distributed storage servers can be divided into mechanical disk storage, hybrid flash storage, and all-flash storage. Generally, a trade-off solution is made according to the customer's business model and storage purchase cost. To fully utilize the performance of all-flash storage media, it is necessary to optimize small I / O (Input / Output) scenarios. I / O aggregation is one of the optimization methods. Through techniques such as write-time redirection, multiple random small I / Os are aggregated into a sequential large I / O to improve write bandwidth and reduce latency. This aggregation is usually achieved by means of write logs and write caches, so that after the small I / O write operation finishes writing the log and is completed in the cache, it can return to the client, while the background performs processing processes such as I / O aggregation. In distributed storage, the logs in the cache pool exist in the form of objects, and writing a log corresponds to writing 3 replica objects, that is, for one log write, it is necessary to write to the local and two remote ends simultaneously. Usually, the IO path for writing objects in distributed storage is too long, so the overhead of system resources for small IO writes is too large, which is not friendly to the latency of small IO writes. After the performance of small IO is improved to a certain level, it will become a bottleneck for performance improvement.
[0003] It can be seen that how to improve the efficiency of writing logs and save resource overhead is a problem to be solved in this field. Summary of the Invention
[0004] The purpose of the embodiments of the present invention is to provide a method, device, equipment and storage medium for storing log objects, which can solve the problem of large system resource overhead when writing logs.
[0005] To solve the above technical problems, in a first aspect, the embodiments of the present invention provide a method for storing log objects, including:
[0006] Obtaining a log data write request and corresponding log data to be written;
[0007] Determining a storage location corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is a storage space range corresponding to a number of log objects pre-applied by the storage engine on the target disk;
[0008] According to the storage location, encapsulating the log data to be written, and sending the encapsulated message to the target disk, so that the target disk directly writes the log data to be written to the storage location to complete the corresponding log object storage operation.
[0009] Optionally, obtaining the log data writing request and the corresponding log data to be written includes:
[0010] Obtaining the log data writing request from the client;
[0011] Judging whether the available space of the pre-applied log object meets the preset write operation condition according to the log data writing request;
[0012] If the available space of the log object meets the preset write operation condition, obtaining the data to be written corresponding to the log data writing request;
[0013] If the available space of the log object does not meet the preset write operation condition, storing the log data writing request in a preset cache queue so that when the available space of the log object meets the preset write operation condition, the corresponding data to be written can be obtained.
[0014] Optionally, judging whether the available space of the pre-applied log object meets the preset write operation condition according to the log data writing request includes:
[0015] Judging whether there is a target log object corresponding to the log data writing request among the log objects corresponding to the available space of the pre-applied log object;
[0016] If the target log object exists in the available space of the log object and the size of the target storage space corresponding to the target log object is not less than the size of the log data to be written corresponding to the log data writing request, it is determined that the available space of the log object meets the preset write operation condition.
[0017] Optionally, after storing the log data writing request in the preset cache queue, it further includes:
[0018] If the target log object does not exist in the available space of the log object, triggering a log object application operation for the target log object;
[0019] After performing the log object application operation, recording the storage space range corresponding to the target log object in the available space of the log object to obtain a new available space of the log object, so as to judge whether the new available space of the log object meets the preset write operation condition according to the log data writing request.
[0020] Optionally, after storing the log data writing request in the preset cache queue, it further includes:
[0021] If the target log object exists in the available space of the log object, and the size of the target storage space corresponding to the target log object is smaller than the size of the log data to be written corresponding to the log data write request, then trigger a storage space application operation for the target log object;
[0022] After performing the storage space application operation, record the storage space range corresponding to the target log object into the available space of the log object to obtain a new available space of the log object, so as to determine whether the new available space of the log object meets the preset write operation condition according to the log data write request.
[0023] Optionally, determining the storage location corresponding to the log data to be written based on the pre-applied available space of the log object includes:
[0024] According to the target log object corresponding to the log data to be written, determine the main disk, the first secondary disk, and the second secondary disk corresponding to the target log object from the pre-applied available space of the log object;
[0025] Based on the storage space ranges corresponding to the main disk, the first secondary disk, and the second secondary disk, calculate the write location of the log data to be written to obtain the storage location corresponding to the log data to be written;
[0026] Wherein, the storage location includes a first storage location corresponding to the main disk, a second storage location and a third storage location corresponding to the first secondary disk and the second secondary disk respectively;
[0027] Correspondingly, sending the encapsulated message to the target disk so that the target disk directly writes the log data to be written to the storage location includes:
[0028] Send the encapsulated message to the main disk, the first secondary disk, and the second secondary disk respectively; the encapsulated message includes a first encapsulated message corresponding to the main disk, a second encapsulated message corresponding to the first secondary disk, and a third encapsulated message corresponding to the second secondary disk;
[0029] Through the main disk, the first secondary disk, and the second secondary disk respectively call a preset data write function to directly write the log data to be written in the corresponding first encapsulated message, second encapsulated message, and third encapsulated message to the corresponding first storage location, second storage location, and third storage location respectively.
[0030] Optionally, completing the corresponding log object storage operation includes
[0031] Record the data write operation corresponding to the log data to be written into the storage engine, and return a message indicating that the log object storage operation has been completed.
[0032] In a second aspect, the present invention provides a log object storage device, including:
[0033] A data acquisition module, configured to acquire a log data write request and corresponding log data to be written;
[0034] A storage location determination module, configured to determine the storage location corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is the storage space range corresponding to a plurality of log objects pre-applied by the storage engine on the target disk;
[0035] A storage module, configured to encapsulate the log data to be written according to the storage location, and send the encapsulated message to the target disk, so that the target disk directly writes the log data to be written to the storage location to complete the corresponding log object storage operation.
[0036] In a third aspect, the present invention provides an electronic device, including:
[0037] A memory, configured to store a computer program;
[0038] A processor, configured to execute the computer program to implement the steps of the log object storage method as described above.
[0039] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the log object storage method as described above are implemented.
[0040] As can be seen from the above technical solution, the present application first obtains a log data writing request and corresponding log data to be written; then determines the storage location corresponding to the log data to be written based on the available space of pre-applied log objects; the available space of the log objects is the storage space range corresponding to a number of log objects pre-applied by a storage engine on a target disk; thereafter, according to the storage location, the log data to be written is encapsulated, and the encapsulated message is sent to the target disk so that the target disk directly writes the log data to be written to the storage location, completing the corresponding storage operation of the log object. In this way, before writing logs, the present application pre-applies the storage space of each log object on the disk through the storage engine, and the metadata of the log object is written to the disk; when subsequent log objects need to be written, the position of the log data to be written on the disk can be calculated in combination with the pre-applied storage space; and a write transaction is reconstructed according to the calculated storage location, and the data is directly written to the disk without additionally updating the metadata of the log object. In this way, the storage engine can be bypassed when writing logs, avoiding updating the metadata of the log object each time, which can improve the processing efficiency and reduce resource consumption. Description of the Drawings
[0041] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0042] Figure 1 It is a flowchart of a method for storing log objects disclosed in the present application;
[0043] Figure 2 It is a flowchart of a specific method for storing log objects disclosed in the present application;
[0044] Figure 3 It is a schematic structural diagram of a device for storing log objects disclosed in the present application;
[0045] Figure 4 It is a structural diagram of an electronic device disclosed in the present application. Detailed Embodiments
[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present invention.
[0047] As used in the specification of the present invention and the above-mentioned drawings, the terms "comprising" and "having", and any variations related to "comprising" and "having", are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.
[0048] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments.
[0049] As Figure 1 shown, this embodiment discloses a method for storing log objects, including:
[0050] Step S11, obtaining a log data write request and corresponding log data to be written.
[0051] First, it is necessary to obtain the write request and the corresponding data to be written. In a specific embodiment, the obtaining of the log data write request and the corresponding log data to be written may include: obtaining the log data write request from a client; judging whether the available space of the pre-applied log object meets the preset write operation condition according to the log data write request; if the available space of the log object meets the preset write operation condition, obtaining the data to be written corresponding to the log data write request; if the available space of the log object does not meet the preset write operation condition, storing the log data write request in a preset cache queue so as to obtain the corresponding data to be written when the available space of the log object meets the preset write operation condition. Specifically, the log data write request can be initiated by the client to the disk, and the log data to be written is sent together; the disk obtains the log data write request and the corresponding log data to be written from the client; it can be understood that if the available space of the pre-applied log object meets the write operation condition of the log data write request, the corresponding data to be written can be directly obtained to process the log data write request; if the available space of the log object does not meet the write operation condition of the log data write request, it means that the log data write request cannot be processed immediately, and the log data write request can be temporarily saved in the preset cache queue and processed when the write operation condition is met.
[0052] In another specific embodiment, determining whether the available space of the pre-applied log object meets the preset write operation condition according to the log data write request may include: determining whether there is a target log object corresponding to the log data write request among the log objects corresponding to the available space of the pre-applied log object; if there is the target log object in the available space of the log object, and the size of the target storage space corresponding to the target log object is not less than the size of the log data to be written corresponding to the log data write request, it is determined that the available space of the log object meets the preset write operation condition. Specifically, in the process of determining whether the available space of the log object meets the preset write operation condition, it is specifically determined whether there is a target log object among the log objects corresponding to the available space of the pre-applied log object, and the target log object is the log object corresponding to the current log data write request; and it is necessary to determine whether the target storage space of the target log object in the available space of the log object is sufficient to store the log data to be written corresponding to the current log data write request; it can be understood that when there is a target log object corresponding to the current log data write request in the available space of the log object, and the target storage space corresponding to the target log object is sufficient to store the log data to be written corresponding to the current log data write request, it can be determined that the available space of the log object and the current log data write request meet the preset write operation condition; correspondingly, if there is no target log object corresponding to the current log data write request in the available space of the log object, or there is the target log object but the corresponding target storage space is not sufficient to store the log data to be written corresponding to the current log data write request, it can be determined that the available space of the log object and the current log data write request do not meet the preset write operation condition.
[0053] In yet another specific embodiment, after storing the log data write request into the preset cache queue, it may further include: if there is no such target log object in the available space of the log object, triggering a log object application operation for the target log object; after performing the log object application operation, recording the storage space range corresponding to the target log object into the available space of the log object to obtain a new available space of the log object, so as to determine whether the new available space of the log object meets the preset write operation condition according to the log data write request. Specifically, for the judgment result of the write operation condition, if there is no target log object corresponding to the current log data write request in the available space of the log object, the target log object needs to be applied for; and after performing the corresponding log object application operation, applying for the storage information of the target log object, recording the storage space range of the target log object into the current available space of the log object, and updating to obtain a new available space of the log object; subsequently, it can be determined whether the new available space of the log object and the current log data write request meet the preset write operation condition.
[0054] Correspondingly, after storing the log data writing request into a preset cache queue, it may further include: if the target log object exists in the available space of the log object, and the size of the target storage space corresponding to the target log object is smaller than the size of the log data to be written corresponding to the log data writing request, then trigger a storage space application operation for the target log object; after performing the storage space application operation, record the storage space range corresponding to the target log object into the available space of the log object to obtain a new available space of the log object, so as to determine whether the new available space of the log object meets the preset write operation condition according to the log data writing request. Specifically, for the judgment result of the write operation condition, if there is a target log object corresponding to the current log data writing request in the available space of the log object, however, the storage space range corresponding to the target log object is not sufficient to store the log data to be written corresponding to the current log data writing request; at this time, the storage space application operation for the target log object can be triggered, and the newly applied storage space is updated to the available space of the log object to obtain a new available space of the log object. Then, the write operation condition between the new available space of the log object and the current log data writing request can be judged.
[0055] Step S12: Determine the storage location corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is the storage space range corresponding to a number of log objects pre-applied by the storage engine on the target disk.
[0056] In this application, the current log data writing request and the corresponding log data to be written can be obtained through the above steps, and furthermore, the storage space range of a number of log objects is pre-applied by the storage engine on the target disk; when the pre-applied available space of the log object meets the corresponding write operation condition, the storage location corresponding to the log data to be written can be further determined based on the available space of the log object. It can be understood that during the process of pre-applying the storage space of each log object, the storage engine can complete the metadata disk writing operation of the relevant log objects; at the same time, record the storage space range of each log object. Then, the storage location of the log data to be written on the disk can be calculated according to the storage space ranges of each log object applied and recorded.
[0057] Step S13: Encapsulate the log data to be written according to the storage location, and send the encapsulated message to the target disk, so that the target disk directly writes the log data to be written to the storage location to complete the corresponding log object storage operation.
[0058] In this application, the log data to be written and the corresponding storage location on the disk can be obtained through the above steps. Then, the log data to be written and the corresponding storage location are encapsulated to obtain an encapsulated message. It can be understood that the encapsulated message here is different from the log data write request. The log data write request may include information about the corresponding log object and information about the log object to be written, while the encapsulated message includes the log object to be written and the corresponding storage location on the disk. Further, after the encapsulated message is sent to the corresponding target disk, the disk can directly perform a write operation on the corresponding log data to be written using the storage location therein.
[0059] In a specific embodiment, determining the storage location corresponding to the log data to be written based on the pre-allocated available space of the log object may include: determining the main disk, the first secondary disk, and the second secondary disk corresponding to the target log object from the pre-allocated available space of the log object according to the target log object corresponding to the log data to be written; calculating the write location of the log data to be written based on the storage space ranges corresponding to the main disk, the first secondary disk, and the second secondary disk to obtain the storage location corresponding to the log data to be written; wherein the storage location includes a first storage location corresponding to the main disk, a second storage location and a third storage location corresponding to the first secondary disk and the second secondary disk respectively; correspondingly, sending the encapsulated message to the target disk so that the target disk directly writes the log data to be written to the storage location includes: sending the encapsulated message to the main disk, the first secondary disk, and the second secondary disk respectively; the encapsulated message includes a first encapsulated message corresponding to the main disk, a second encapsulated message corresponding to the first secondary disk, and a third encapsulated message corresponding to the second secondary disk; by respectively invoking a preset data writing function through the main disk, the first secondary disk, and the second secondary disk, directly writing the log data to be written in the corresponding first encapsulated message, the second encapsulated message, and the third encapsulated message to the corresponding first storage location, second storage location, and third storage location respectively. Specifically, writing a log corresponds to writing three copy objects, including writing to the local main disk and two remote secondary disks; the target log object corresponding to the current log data write request can be determined from the pre-allocated available space of the log object, and the target log object corresponds to a main disk, a first secondary disk, and a second secondary disk; it can be understood that the storage space ranges corresponding to these three disks are different, and the storage locations of the log data to be written in these three disks need to be calculated separately; that is, calculating the first storage location of the log data to be written in the main disk, the second storage location in the first secondary disk, and the third storage location in the second secondary disk. Further, based on the calculated storage location, it needs to be sent to the corresponding disk together with the corresponding log data to be written. Specifically, the encapsulated message containing the storage location and the log data to be written is sent to the relevant disk; according to the correspondence between the storage location and the main disk, the first secondary disk, and the second secondary disk, a first encapsulated message corresponding to the main disk, a second encapsulated message corresponding to the first secondary disk, and a third encapsulated message corresponding to the second secondary disk can be generated; after sending the corresponding encapsulated message to the corresponding disk, the preset data writing function can be invoked through each disk to directly write the log data to be written in the corresponding encapsulated message to the corresponding storage location.It can be understood that since each disk has previously applied for available space of the log object through the storage engine, when writing log data using the encapsulated message, the write operation interface can be called to directly complete the disk writing bypassing the storage engine, avoiding updating the metadata of the log object for each write, and reducing resource overhead.
[0060] In another specific embodiment, the completion of the corresponding log object storage operation may include recording the data write operation corresponding to the log data to be written into the storage engine and returning a message indicating that the log object storage operation has been completed. Specifically, after the data write operation of the log data to be written is completed, the corresponding data write operation may be recorded in the memory of the storage engine to facilitate processing subsequent possible read requests. And, in a specific embodiment, after recording the data write operation in the memory, an attempt may be made to trim (optimize) the log object. After the data in the memory is flushed to the data pool, the log object needs to be aged to avoid occupying too much space in the cache pool; the log object is stored in the cache pool, which is composed of high-speed storage media, and the data pool can be a high-speed storage media or a mechanical disk.
[0061] It can be seen that before writing the log in this application, the storage space of each log object on the disk is pre-applied through the storage engine, and the metadata of the log object is completed and written to the disk; when writing the log object subsequently, the position of the log data to be written on the disk can be calculated in combination with the pre-applied storage space; and a write transaction is reconstructed according to the calculated storage position, and the data is directly written to the disk without the need to additionally update the metadata of the log object. In this way, the storage engine can be bypassed when writing the log, avoiding updating the metadata of the log object for each write, improving processing efficiency, and reducing resource consumption.
[0062] As Figure 2 shown, an embodiment of this application discloses a log object storage method, which specifically includes:
[0063] In this embodiment, internal log objects are pre-applied, and the storage space of the log objects in three replicas (one primary replica and two other replicas, a total of three disks) is applied through the storage engine (bluestore), and the metadata corresponding to the log objects is completed and written to the disk in this step; the primary replica stores the address on the physical disk of the log object in the three replicas; specifically, a data structure valid_wal_list can be added to store the information of the log objects that have been applied, to implement the management of the available space of the log objects; that is, the primary replica caches the available space information of the pre-applied log objects on the three replicas, and allocates writable space positions for each data write of the log objects.
[0064] Further, after receiving the log data write request from the client, the primary replica triggers an action to write a log object. Before writing the log object, it checks whether there is an available log object or sufficient space for writing in the valid_wal_list. If not, it puts the current write request into the cache queue and simultaneously triggers the pre-application action for log objects. After the pre-application of log objects is completed, the primary replica collects the available space information of log objects from two replicas, aggregates them into the valid_wal_list, and simultaneously triggers an action to write a log object. Correspondingly, if there is available space in the valid_wal_list, it calculates the positions where the log data corresponding to the current write request should be written to the three replicas, constructs and sends a log object write message to the three replicas. After receiving the corresponding write message, the three replicas bypass the storage engine according to the position information recorded in the message and directly complete the data disk write by calling the aio_write (used to request asynchronous write) interface. After completing the disk write, it returns a message to notify the primary replica. After receiving the disk write completion messages sent by all replicas, the primary replica merges the data written this time (the client's write data) into the memory cache of the storage engine, checks whether the available space of the current log object is 0. If the available space is 0, it trims the information of the currently cached log object; finally, the primary replica can send a message indicating successful writing to the client, and the write process ends. In a specific embodiment, during the process of the three replicas writing to the raw disk, it can be selected to optimize the performance of writing data through NoF (NVMe over Fabric, a data transmission protocol).
[0065] It can be seen that before writing the log in this application, the storage space of each log object on the disk is pre-applied through the storage engine, and the metadata of the log object is written to the disk; when writing the log object subsequently, the position of the log data to be written on the disk can be calculated in combination with the pre-applied storage space; and a write transaction is reconstructed according to the calculated storage position, and the data is directly written to the disk without the need to update the metadata of the log object additionally. In this way, the storage engine can be bypassed when writing the log, avoiding updating the metadata of the log object every time, shortening the IO process, reducing the write latency, improving the processing efficiency, and reducing resource consumption.
[0066] As Figure 3 shown, an embodiment of this application discloses a log object storage device, including:
[0067] A data acquisition module 11, configured to acquire a log data write request and the corresponding log data to be written;
[0068] A storage position determination module 12, configured to determine the storage position corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is the storage space range corresponding to a plurality of log objects pre-applied based on the storage engine on the target disk;
[0069] A storage module 13, configured to encapsulate the log data to be written according to the storage location, and send the encapsulated message to the target disk, so that the target disk directly writes the log data to be written to the storage location to complete the corresponding log object storage operation.
[0070] It can be seen that before writing the log in this application, the storage space of each log object of the disk is pre-applied through the storage engine, and the metadata of the log object is written to the disk; when writing the log object later, the position of the log data to be written on the disk can be calculated in combination with the pre-applied storage space; and a write transaction is reconstructed according to the calculated storage position, and the data is directly written to the disk without additionally updating the metadata of the log object. In this way, the storage engine can be bypassed when writing the log, avoiding updating the metadata of the log object each time, improving the processing efficiency and reducing resource consumption.
[0071] In a specific embodiment, the data acquisition module 11 may include:
[0072] A log data write request acquisition unit, configured to acquire a log data write request from a client;
[0073] A write operation condition judgment sub-module, configured to judge whether the available space of the pre-applied log object meets a preset write operation condition according to the log data write request;
[0074] A data acquisition sub-module, configured to acquire the data to be written corresponding to the log data write request when the available space of the log object meets the preset write operation condition;
[0075] A log data write request storage unit, configured to store the log data write request in a preset cache queue when the available space of the log object does not meet the preset write operation condition, so as to acquire the corresponding data to be written when the available space of the log object meets the preset write operation condition.
[0076] In another specific embodiment, the write operation condition judgment sub-module may include:
[0077] A log object judgment unit, configured to judge whether there is a target log object corresponding to the log data write request in each log object corresponding to the available space of the pre-applied log object;
[0078] A first determination unit, configured to determine that the available space of the log object meets the preset write operation condition when there is the target log object in the available space of the log object and the size of the target storage space corresponding to the target log object is not less than the size of the log data to be written corresponding to the log data write request.
[0079] In a specific embodiment, the apparatus may further include:
[0080] A log object application module, configured to trigger a log object application operation for the target log object if the target log object does not exist in the available space of the log object;
[0081] A first space update module, configured to record the storage space range corresponding to the target log object into the available space of the log object after performing the log object application operation, to obtain a new available space of the log object, so as to determine whether the new available space of the log object meets a preset write operation condition according to the log data write request.
[0082] In a specific embodiment, the apparatus may further include:
[0083] A storage space application module, configured to trigger a storage space application operation for the target log object when the target log object exists in the available space of the log object and the size of the target storage space corresponding to the target log object is smaller than the size of the to-be-written log data corresponding to the log data write request;
[0084] A first space update module, configured to record the storage space range corresponding to the target log object into the available space of the log object after performing the storage space application operation, to obtain a new available space of the log object, so as to determine whether the new available space of the log object meets a preset write operation condition according to the log data write request.
[0085] In a specific embodiment, the storage location determination module 12 may include:
[0086] A disk determination unit, configured to determine a main disk, a first secondary disk, and a second secondary disk corresponding to the target log object from the pre-applied available space of the log object according to the target log object corresponding to the to-be-written log data;
[0087] A location calculation unit, configured to calculate a write location of the to-be-written log data based on the storage space ranges corresponding to the main disk, the first secondary disk, and the second secondary disk, to obtain a storage location corresponding to the to-be-written log data; wherein the storage location includes a first storage location corresponding to the main disk, and second and third storage locations corresponding to the first secondary disk and the second secondary disk respectively;
[0088] Correspondingly, the storage module 13 may include:
[0089] A message sending unit, configured to send the encapsulated messages to the main disk, the first secondary disk, and the second secondary disk respectively; the encapsulated messages include a first encapsulated message corresponding to the main disk, a second encapsulated message corresponding to the first secondary disk, and a third encapsulated message corresponding to the second secondary disk;
[0090] A storage unit, configured to respectively call a preset data writing function through the main disk, the first secondary disk, and the second secondary disk, and directly write the to-be-written log data in the corresponding first encapsulated message, second encapsulated message, and third encapsulated message to the corresponding first storage location, second storage location, and third storage location respectively.
[0091] In a specific embodiment, the storage module 13 may include
[0092] An operation recording unit, configured to record the data writing operation corresponding to the to-be-written log data to the storage engine, and return a message indicating that the storage operation of the log object has been completed.
[0093] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 4 which is a structural diagram of an electronic device shown according to an exemplary embodiment. The content in the figure cannot be considered as any limitation to the usage scope of the present application. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the log object storage method disclosed in any of the foregoing embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.
[0094] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.
[0095] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.
[0096] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the log object storage method executed by the electronic device disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks.
[0097] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the foregoing disclosed log object storage method is implemented. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.
[0098] Furthermore, the present application also discloses a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the foregoing disclosed log object storage method is implemented.
[0099] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the description in the method section.
[0100] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.
[0101] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.
[0102] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.
[0103] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this text to elaborate on the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.
Claims
1. A method for storing log objects, characterized in that, Including: Obtaining a log data writing request and corresponding log data to be written; Determining a storage location corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is a storage space range corresponding to a plurality of log objects pre-applied by the storage engine on the target disk; According to the storage location, encapsulating the log data to be written and sending the encapsulated message to the target disk, so that the target disk directly writes the log data to be written to the storage location to complete the corresponding log object storage operation.
2. The log object storage method according to claim 1, wherein The obtaining the log data writing request and corresponding log data to be written includes: Obtaining a log data writing request from a client; Judging whether the pre-applied available space of the log object meets a preset write operation condition according to the log data writing request; If the available space of the log object meets the preset write operation condition, obtaining the log data to be written corresponding to the log data writing request; If the available space of the log object does not meet the preset write operation condition, storing the log data writing request in a preset cache queue, so as to obtain corresponding log data to be written when the available space of the log object meets the preset write operation condition.
3. The log object storage method according to claim 2, wherein The judging whether the pre-applied available space of the log object meets a preset write operation condition according to the log data writing request includes: Judging whether there is a target log object corresponding to the log data writing request among the log objects corresponding to the pre-applied available space of the log object; If the target log object exists in the available space of the log object and the size of the target storage space corresponding to the target log object is not less than the size of the log data to be written corresponding to the log data writing request, it is determined that the available space of the log object meets the preset write operation condition.
4. The log object storage method according to claim 3, wherein After storing the log data writing request in the preset cache queue, it further includes: If the target log object does not exist in the available space of the log object, triggering a log object application operation for the target log object; After executing the log object application operation, recording the storage space range corresponding to the target log object into the available space of the log object to obtain a new available space of the log object, so as to judge whether the new available space of the log object meets the preset write operation condition according to the log data writing request.
5. The log object storage method according to claim 3, characterized in that After storing the log data writing request in the preset cache queue, it further includes: If the target log object exists in the available space of the log object and the size of the target storage space corresponding to the target log object is less than the size of the log data to be written corresponding to the log data writing request, triggering a storage space application operation for the target log object; After executing the storage space application operation, recording the storage space range corresponding to the target log object into the available space of the log object to obtain a new available space of the log object, so as to judge whether the new available space of the log object meets the preset write operation condition according to the log data writing request.
6. The log object storage method according to claim 1, characterized in that Determining the storage location corresponding to the log data to be written based on the pre-applied available space of the log object includes: Determining the main disk, the first secondary disk, and the second secondary disk corresponding to the target log object from the pre-applied available space of the log object according to the target log object corresponding to the log data to be written; Calculating the write location of the log data to be written based on the storage space ranges corresponding to the main disk, the first secondary disk, and the second secondary disk, to obtain the storage location corresponding to the log data to be written; Wherein, the storage location includes a first storage location corresponding to the main disk, a second storage location and a third storage location respectively corresponding to the first secondary disk and the second secondary disk; Correspondingly, sending the encapsulated message to the target disk so that the target disk directly writes the log data to be written into the storage location includes: Sending the encapsulated message to the main disk, the first secondary disk, and the second secondary disk respectively; the encapsulated message includes a first encapsulated message corresponding to the main disk, a second encapsulated message corresponding to the first secondary disk, and a third encapsulated message corresponding to the second secondary disk; Respectively calling a preset data write function through the main disk, the first secondary disk, and the second secondary disk, and directly writing the log data to be written in the corresponding first encapsulated message, second encapsulated message, and third encapsulated message into the corresponding first storage location, second storage location, and third storage location respectively.
7. The log object storage method according to any one of claims 1 to 6, characterized in that Completing the corresponding log object storage operation includes Recording the data write operation corresponding to the log data to be written into the storage engine, and returning a message indicating that the log object storage operation has been completed.
8. A log object storage device, characterized in that, Includes: A data acquisition module, configured to acquire a log data write request and the corresponding log data to be written; A storage location determination module, configured to determine the storage location corresponding to the log data to be written based on the pre-applied available space of the log object; the available space of the log object is the storage space range corresponding to a plurality of log objects pre-applied by the storage engine on the target disk; A storage module, configured to encapsulate the log data to be written according to the storage location, and send the encapsulated message to the target disk so that the target disk directly writes the log data to be written into the storage location to complete the corresponding log object storage operation.
9. An electronic device, characterized in that, Includes: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the steps of the log object storage method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the log object storage method according to any one of claims 1 to 7 are implemented.