Log data processing method and device, terminal equipment and program product

Abstract

The invention is suitable for the technical field of computers, and provides a log data processing method and device, terminal equipment and a program product.The method comprises the steps that in response to a storage instruction for target log data, historical initial offset and historical length data of stored log data in block equipment are determined; determining a first initial offset of the target log data to be stored to the block device according to the historical initial offset and the historical length data; processing the first length data and the first initial offset of the target log data into index data of the target log data in a key value pair form, wherein the index data is used for providing index information for reading the target log data; and storing the target log data into the block device according to the first initial offset and the index data. According to the method, by storing the target log data to the block device, extra file overhead is reduced, meanwhile, by means of the independent reading function of the block device, convenience can be provided for subsequent log data reading, and the reading operation of the log data is simplified.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a method, apparatus, terminal equipment and program product for processing log data. Background Art

[0002] Write-Ahead Logging (WAL) can be a log management mechanism used to ensure data consistency and reliability. The core idea is to record the operation in a log (such as a WAL log) before modifying the data in the database or file system. Only when the log is safely written to the disk will the actual data modification be performed. In this way, even if the operating system fails in the middle, the data can be restored to a consistent state by reading the log to ensure data integrity.

[0003] Existing log data processing methods usually use the operating system's file system to store log data, which brings additional file indexing overhead. At the same time, when reading log data, it is necessary to traverse the file in sequence from the starting position to access the log data at a specific location, making the reading operation more cumbersome. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, terminal device, and program product for processing log data, which can reduce additional file indexing overhead and simplify log data reading operations.

[0005] In a first aspect, an embodiment of the present application provides a method for processing log data, including:

[0006] In response to a storage instruction for target log data, determining a history start offset and history length data of the log data stored in the block device;

[0007] Determine the first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data;

[0008] Processing the first length data and the first starting offset of the target log data in the form of a key-value pair into index data of the target log data, where the index data is used to provide index information for reading the target log data from the block device;

[0009] The target log data is stored in the block device according to the first starting offset and the index data.

[0010] In a second aspect, an embodiment of the present application provides a log data processing device, including:

[0011] A first determination module, configured to determine, in response to a storage instruction for target log data, a history start offset and history length data of log data stored in a block device;

[0012] A second determination module is used to determine a first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data;

[0013] A processing module, used for processing the first length data and the first starting offset of the target log data into index data of the target log data in the form of a key-value pair, where the index data is used for providing index information for reading the target log data from the block device;

[0014] The storage module is used to store the target log data in the block device according to the first starting offset and the index data.

[0015] In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the first aspect above.

[0018] The embodiment of the present application provides a method, apparatus, terminal device and program product for processing log data, the method comprising: in response to a storage instruction for target log data, determining the historical starting offset and historical length data of the log data stored in the block device; determining the first starting offset of the target log data to be stored in the block device according to the historical starting offset and historical length data; processing the first length data and the first starting offset of the target log data in the form of a key-value pair as index data of the target log data, the index data is used to provide index information for reading the target log data from the block device; storing the target log data in the block device according to the first starting offset and index data. By using the above technical solution, by storing the target log data in the block device, the additional file index overhead is reduced, and at the same time, by using the independent reading function of the block device, it can provide convenience for subsequent reading of log data, further simplifying the reading operation of log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a schematic diagram of storing log data provided by the prior art;

[0021] Figure 2 It is a flowchart of a method for processing log data provided by an embodiment of the present application;

[0022] Figure 3 is a flow chart of a method for processing log data provided by another embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a structure for storing target log data and index data provided by an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a structure for obtaining target log data provided by an embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of a structure for recycling storage resources provided by an embodiment of the present application;

[0026] Figure 7 It is a structural block diagram of a log data processing device provided by an embodiment of the present application;

[0027] Figure 8 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0030] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0031] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] It can be considered that, taking a key-value database such as LevelDB as an example, each time the data is updated, the operating system will first append the changed key-value pairs sequentially to the WAL log file, and then temporarily store this data in the data structure in the memory. When the data in the memory reaches a certain amount, the operating system will persist this data in batches to the actual storage space. Since the WAL log uses a sequential write method, compared to directly writing data to the actual storage system, the above log data processing method usually has a higher write efficiency. At the same time, in each data change process, most of the overhead is often concentrated on the writing of the WAL log.

[0035] In some scenarios with high performance requirements, improving the reading and writing efficiency of WAL logs can significantly improve the user experience. Therefore, how to further optimize the storage performance of logs is an urgent problem to be solved at this stage.

[0036] The existing method of processing log data usually adopts log segment storage and log rotation strategy. Figure 1 It is a schematic diagram of storing log data provided by the prior art, such as Figure 1 As shown, the WAL log file can be divided into multiple fixed-size segments (such as 20MB). Each time the change data is stored, it can be written into different segments in sequence according to the pre-designed coding layout (such as the format of [length][data]), such as storing log data 1 in the form of [len1][data1] in segment file _{1}. The WAL log file can usually be named according to the sequence number of the change, such as the sequence number of the first log data stored in the log file can be used as the name of the WAL log file, which can facilitate the sorting, retrieval and scanning of the log file; when the WAL log file reaches the specified size, the operating system will create a new WAL log file.

[0037] When reading log data, the operating system can read the file sequentially from left to right, first reading the length field to determine the size of the data, and then reading the data of the corresponding length. If necessary, the entire segment file can be read at once and parsed item by item in memory.

[0038] However, the existing log data processing method brings additional file indexing overhead, which affects system performance. At the same time, when reading the middle part of the WAL log file, it is necessary to traverse the file in order from the beginning to access the data at the specific location, which has an adverse effect on random read performance. In addition, the data reading operation in a single WAL log file must be performed in order, and concurrent reading is not possible, which limits the performance of this method in high concurrent access scenarios.

[0039] Based on this, the embodiment of the present application provides a method for processing log data, designs a high-performance storage component for WAL write-ahead logs, further optimizes the storage performance of logs, and improves file indexing and concurrent reading and writing capabilities.

[0040] Figure 2 It is a flowchart of a method for processing log data provided in an embodiment of the present application. As an example but not a limitation, the method can be applied to a terminal device.

[0041] S101 . In response to a storage instruction for target log data, determine a history start offset and history length data of log data stored in a block device.

[0042] The storage instruction can be used to store the target log data, which is any log data to be stored; the block device can be a storage device in the operating system that reads and writes data in fixed-size blocks. The data block size is usually between 512 bytes and 32,768 bytes. Each block of the block device has independent read and write functions. For example, the block device can include a hard disk. Under normal circumstances, the block device will be formatted as a file system (such as ext4) for organization and management. The file system is responsible for maintaining the mapping relationship between files and physical blocks, and using metadata (such as indexes) to record the association between files and blocks. Although files are stored continuously at the abstract level, they can actually be stored in a dispersed manner on the block device, and the file system is responsible for managing the above mapping index data.

[0043] In this embodiment, a block device can be used to store log data. Each log data has a different offset in the block device. The historical starting offset can be understood as the starting offset of the log data currently stored in the block device, such as including the starting offset of all the log data stored in the block device, or the starting offset of one or several stored log data in the block device; optionally, the historical starting offset can be the maximum starting offset. For example, when storing log data, the block device also assigns an incremental log identifier to each log data to identify the order of the log data. The maximum starting offset is the starting offset of the log data corresponding to the current maximum log identifier in the block device.

[0044] Correspondingly, the historical length data can be understood as the length data of the log data currently stored in the block device, such as the length data of all the log data stored in the block device, or the length data of a certain log data stored in the block device, such as the length data of the maximum starting offset.

[0045] S102: Determine a first starting offset of target log data to be stored in a block device according to the historical starting offset and the historical length data.

[0046] The first starting offset may refer to the starting offset of the target log data to be stored in the block device.

[0047] After determining the historical starting offset and historical length data of the log data stored in the block device through the above steps, the first starting offset of the target log data can be determined according to the historical starting offset and historical length data. The specific determination method is not limited, and different determinations can be made according to the current actual storage situation and actual free space of the block device. For example, when the log data currently stored in the block device are stored in sequence according to the log identifiers, and the free space after the maximum log identifier in the block device can carry the storage of the target log data, the first starting offset of the target log data to be stored in the block device can be determined according to the maximum starting offset and historical length data of the maximum log identifier. For example, the sum of the maximum starting offset and the historical length data of the maximum log identifier can be calculated, and the calculation result is used as the first starting offset of the target log data.

[0048] It is also possible to perform an error processing when the log data currently stored in the block device are stored in sequence according to the log identifiers, but the free space after the maximum log identifier in the block device cannot accommodate the storage of the target log data. Alternatively, the first starting offset of the target log data can be further determined according to the recovery status of the current storage space in the block device. This embodiment will not be further elaborated on this and will be expanded in subsequent parts.

[0049] S103: Process the first length data and the first starting offset of the target log data in the form of a key-value pair into index data of the target log data, where the index data is used to provide index information for reading the target log data from the block device.

[0050] S104. Store the target log data in a block device according to the first starting offset and the index data.

[0051] The index data may be used to provide index information for reading target log data from a block device, and specific content of the index data may be a key-value pair formed by length data and a first starting offset.

[0052] In order to achieve efficient data management, this step can process the first length data and the first starting offset of the target log data in the form of a key-value pair as index data of the target log data, so as to be used for subsequent reading of the target log data.

[0053] Specifically, the target log data can be stored in the block device according to the first starting offset and the index data. The specific storage process is not limited. For example, the target log data can be directly stored in the block device, or the index data of the target log data can be stored in the same or different location while storing the target log data. For example, the target log data and the index data can be associated and stored in different locations, etc., as long as the storage of the target log data can be achieved.

[0054] The present embodiment provides a method for processing log data, which, in response to a storage instruction for target log data, determines the historical starting offset and historical length data of the log data stored in the block device; determines the first starting offset of the target log data to be stored in the block device according to the historical starting offset and historical length data; processes the first length data and the first starting offset of the target log data in the form of a key-value pair as index data of the target log data, the index data being used to provide index information for reading the target log data from the block device; stores the target log data in the block device according to the first starting offset and the index data. By using this method, by storing the target log data in the block device, the additional file index overhead is reduced, and by using the independent reading function of the block device, it can provide convenience for subsequent reading of log data, further simplifying the reading operation of log data.

[0055] Figure 3 This is a flow chart of a log data processing method provided by another embodiment of the present application. This embodiment further optimizes storing the target log data in the block device according to the first starting offset and index data as follows: determining the first log identifier of the target log data based on the maximum log identifier of the log data stored in the block device; storing the target log data in the block device according to the first starting offset and the first log identifier, and storing the index data in the first index file in the block device, wherein the data length of the index data is a preset fixed length, and the first index file corresponds to the first log identifier. Figure 3 As shown, the method includes:

[0056] S201 . In response to a storage instruction for target log data, determine a history start offset and history length data of log data stored in a block device.

[0057] S202: Determine a first starting offset of target log data to be stored in a block device according to the historical starting offset and the historical length data.

[0058] S203: Process the first length data and the first starting offset of the target log data in the form of a key-value pair into index data of the target log data, where the index data is used to provide index information for reading the target log data from the block device.

[0059] S204: Determine a first log identifier of the target log data based on the maximum log identifier of the log data stored in the block device.

[0060] S205. Store the target log data in the block device according to the first starting offset and the first log identifier, and store the index data in a first index file in the block device, wherein the data length of the index data is a preset fixed length, and the first index file corresponds to the first log identifier.

[0061] Among them, the maximum log identifier can refer to the largest log identifier among the log data stored in the block device, the first log identifier is the log identifier assigned to the target log data, and the specific value of the first log identifier can be equal to the maximum log identifier plus 1; in this embodiment, the index file can be used to store a specified number of multiple index data, such as multiple index data can be stored in the index file in sequence, and when an index file completes storing a specified number of index data, a new index file can be constructed to store the new index data, wherein the file name of each index file can be the log identifier of the first index data stored in the index file, and the first index file is the index file that stores the index data of the target log data.

[0062] Furthermore, the data length of the index data stored in the index file is not limited. It can be configured according to the actual situation of the first length data and the first starting offset of the target log data, or the data length of the index data can be limited to a preset fixed length. The value of the preset fixed length can be determined based on empirical values.

[0063] In this embodiment, the target log data can be stored according to the first starting offset and the log identifier, and the index data can be stored in the first index file in the block device. The specific storage means can be to call a corresponding program to complete the storage of the target log data and the index data.

[0064] Figure 4 is a schematic diagram of a structure for storing target log data and index data provided by an embodiment of the present application, such as Figure 4 As shown, a WAL log item can be represented by an entry, which represents a log record of a data change. The WAL log item can be stored in a block device in a sequential append manner. Accordingly, each entry is assigned an increasing sequence number (i.e., a log identifier) ​​to identify the append order of the log item.

[0065] At the same time, in order to achieve efficient data management, the index data of each entry is recorded in a separate index file. The data length of the index data of each entry can be 12 bytes. The index data can include two parts: the starting offset in the block device and the data length of the entry. While storing each entry, the index data of each entry can be concurrently written to the corresponding index file. For example, the index data {offset1|len1} of entry1 can be stored in the first storage position in the index file _{1}, and the index data of entry2, entry3, ... can be stored in the index file _{1} in a sequentially appended order. On this basis, the overall writing efficiency can be improved, and the position of each entry in the block device can be quickly located through the index file when the log data needs to be read.

[0066] In addition, for the convenience of management and reading, a new index file can be created after processing several serial numbers (for example, after 10,000 entries). That is, after index file _{1} stores 10,000 index data, a new index file can be constructed and named index file _{10001}. On this basis, since each index data is only 12 bytes, the index data of 10,000 entries only takes up about 120KB of storage space. Therefore, when reading WAL logs, the entire index file can be read at one time, and the corresponding log data can be read from the block device in batches and concurrently according to the offset and length information.

[0067] In some embodiments, after storing the target log data in the block device according to the first starting offset and the index data, the method further includes:

[0068] In response to the log acquisition instruction carrying the first log identifier, index data corresponding to the first log identifier is acquired from the first index file according to the first log identifier and a preset fixed length;

[0069] Parse the index data to obtain the first length data and the first starting offset in the index data;

[0070] According to the first starting offset, data having a length of the first length is read from the block device as target log data corresponding to the log acquisition instruction.

[0071] In a specific implementation, after the log data is stored, there may be a need to obtain the log data. At this time, the user may generate a log acquisition instruction through device operation, wherein the log acquisition instruction may carry an identifier of the log data to be obtained, such as a first log identifier. At this time, the present embodiment may obtain the index data corresponding to the first log identifier from the first index file according to the carried first log identifier and a preset fixed length. The method for obtaining the index data is not limited. For example, the index data corresponding to the first log identifier may be obtained by directly calling the first index file, or the corresponding index data may be determined by the actual value of the first log identifier to determine whether it is the first or last index data stored in the first index file, thereby directly obtaining the index data corresponding to the first log identifier. For example, when the first log identifier is the file name of the first index file, it means that the index data corresponding to the first log identifier is the first index data stored in the first index file. Then, the index data corresponding to the first log identifier may be directly obtained, or the corresponding index data may be obtained through certain calculations. The present embodiment does not limit this.

[0072] After obtaining the index data, the first length data and the first starting offset in the index data can be obtained by parsing the index data. The parsing method can be determined according to the specific configuration of the index data, for example, the first length data and the first starting offset in the index data can be obtained respectively according to the type of data stored in the index data, or can be determined according to the length of the stored data, such as the data length of the first length data can be pre-configured as the first fixed length, the data length of the first starting offset can be the second fixed length, and the sum of the first fixed length and the second fixed length can be the preset fixed length, and then the index data can be parsed according to the pre-configured first fixed length and the second fixed length to obtain the corresponding first length data and the first starting offset. Therefore, according to the obtained first starting offset, the data with the length of the first length data can be read in the block device, so that the read data can be used as the target log data corresponding to the log acquisition instruction.

[0073] In some embodiments, obtaining index data corresponding to the first log identifier from the first index file according to the first log identifier and a preset fixed length includes:

[0074] Calculate the second starting offset of the index data corresponding to the first log identifier at the first index file according to the first log identifier and the preset fixed length;

[0075] Based on the second starting offset, data having a preset fixed length is read from the first index file as index data corresponding to the first log identifier.

[0076] The second starting offset may be considered as the starting offset of the index data corresponding to the first log identifier stored in the first index file.

[0077] In a specific implementation, if the index data corresponding to the first log identifier is located in the middle of the first index file, the second starting offset of the index data corresponding to the first log identifier in the first index file can be calculated based on the premise that the index data are all of a preset fixed length. Assuming that the length of each index data is 12 bytes, the second starting offset of the index data with the first log identifier of 3 in the first index file can be calculated by the formula = (3-1) × 12 = 24; and then based on the second starting offset, data with a preset fixed length can be read from the first index file as the index data corresponding to the first log identifier, such as reading 12 bytes of data at offset 24 from the first index file as the index data corresponding to the first log identifier. On this basis, by utilizing the fixed length of the index data, when accessing log data of any sequence number, the physical location of the data can be directly located by a simple offset calculation, avoiding the operation of traversing the log from the beginning, thereby improving the reading efficiency of the log data and greatly optimizing the random reading performance of the operating system.

[0078] Figure 5 is a schematic diagram of a structure for obtaining target log data provided by an embodiment of the present application, such as Figure 5 As shown, for the scenario of reading a certain serial number entry in the middle, since the length of each index data is fixed, the data position corresponding to the log entry can be quickly located according to the serial number. Specifically, assuming that the log entry with serial number 3 (i.e., the first log identifier is 3) is to be read, it is first necessary to determine its position in the index file _{1}. For example, the starting offset of the index entry with serial number 3 in the index file can be calculated by the formula = (serial number - 1) × 12 = (3-1) × 12 = 24, which means that the index information of serial number 3 starts from the 24th byte of the index file. Then 12 bytes are read from offset 24, of which the first 8 bytes are the starting offset of the log data corresponding to serial number 3 in the block device (i.e., offset3), and the last 4 bytes are the length of the log data (i.e., len3); finally, according to the acquired data, the log data of serial number 3 can be directly read from the block device. The reading process can be to read data with a length of len3 from the offset3 position of the block device, and the complete log entry with serial number 3 can be obtained.

[0079] The present embodiment provides a method for processing log data, which can facilitate the subsequent reading of target log data, realize efficient reading of log data, and further improve the efficiency of log data reading by storing target log data in a block device according to a first starting offset and a first log identifier, and storing index data in a first index file in the block device, and configuring the data length of the index data to be a preset fixed length.

[0080] In some embodiments, the historical starting offset includes: a maximum starting offset and a minimum starting offset, and determining the first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data includes:

[0081] According to the maximum starting offset, the minimum starting offset and the historical length data, the first starting offset of the target log data to be stored in the block device is determined. The maximum starting offset is the starting offset of the log data corresponding to the maximum log identifier in the block device, and the minimum starting offset is the starting offset of the log data corresponding to the minimum log identifier in the block device.

[0082] It can be considered that, in practical applications, after the log data in the memory is persisted to the block device, the old and no longer needed data can be cleaned up regularly to reclaim the corresponding storage resources.

[0083] For example, if the block device records the maximum starting offset and the minimum starting offset, it means that the storage space before the minimum starting offset has been recycled and can be used to store new log data, that is, the new log data can be stored in the storage space after the maximum starting offset of the block device. When the storage space after the maximum starting offset of the block device is zero or insufficient to store the new log data, the new log data can be stored in the storage space between offset 0 and the minimum starting offset in the block device. Exemplarily, the process of determining the first starting offset of the target log data to be stored in the block device can be to determine the first starting offset of the target log data to be stored in the block device according to the recorded maximum starting offset, minimum starting offset and historical length data. For example, it can be determined according to the maximum starting offset whether the log data of the maximum log identifier can be continued to be appended. If the data length of the log data corresponding to the maximum starting offset plus the maximum log identifier is less than the maximum data length that the block device can accommodate, and the remaining storage capacity is greater than or equal to the data capacity of the target log data, then it means that the storage space after the log data corresponding to the maximum log identifier can continue to append new log data, and the first starting offset can be determined to be the maximum starting offset plus the data length of the log data corresponding to the maximum log identifier.

[0084] If the data length of the log data corresponding to the maximum starting offset plus the maximum log identifier is equal to the maximum data length that the block device can accommodate, or the data length of the log data corresponding to the maximum starting offset plus the maximum log identifier is less than the maximum data length that the block device can accommodate, but the remaining storage capacity is less than the data capacity of the target log data, indicating that the storage space after the log data corresponding to the maximum log identifier cannot continue to append new log data, then the first starting offset is determined according to the minimum starting offset, and the target log data is stored in the storage location area before the log data corresponding to the minimum log identifier. The minimum starting offset is the starting offset of the log data corresponding to the minimum log identifier in the block device, and the minimum starting offset can be used to limit the total length of the log data to be stored in the block device.

[0085] Furthermore, this embodiment can also continuously update the maximum starting offset and the minimum starting offset of the block device after storing the target log data, so as to more accurately implement the storage of subsequent new log data.

[0086] In some embodiments, before determining the first starting offset of the target log data to be stored in the block device according to the maximum starting offset, the minimum starting offset and the history length data, the method further includes:

[0087] In response to the storage resource recycling instruction carrying the second log identifier, determining whether the second log identifier satisfies a preset recycling condition;

[0088] If the second log identifier meets the preset recycling condition, deleting the second index file corresponding to the second log identifier;

[0089] The maximum starting offset and the minimum starting offset corresponding to the block device are determined based on the second log identifier.

[0090] The storage resource recovery instruction can be used to recover all storage resources before the second log identifier, such as cleaning up index files and recovering storage space for log data in block devices. The second log identifier can be the first log identifier, or a log identifier other than the first log identifier; the preset recovery condition can be that the index identifiers of all index data in the index file are smaller than a certain log identifier, and the second index file is an index file in which the index identifiers corresponding to the stored index data are smaller than the second log identifier.

[0091] In a specific implementation, the minimum unit for cleaning the index file can be a whole file. This embodiment can respond to the storage resource recovery instruction carrying the second log identifier, judge whether the second log identifier meets the preset recovery condition, and perform different operations according to different judgment results. For example, if the second log identifier meets the preset recovery condition, it means that the index identifiers corresponding to the currently stored index data are all smaller than the index file of the second log identifier, then the second index file corresponding to the second log identifier can be directly deleted, and the maximum starting offset and the minimum starting offset corresponding to the block device can be determined, such as the starting offset of the log data corresponding to the current maximum log identifier in the block device can be determined as the maximum starting offset, and the starting offset of the log data corresponding to the second log identifier in the block device can be determined as the minimum starting offset. On this basis, it is possible to effectively reclaim space without physically deleting the old log data in the block device, and only the above two key offsets need to be managed and updated. Different from the traditional file system cleaning, the cleaning overhead of this embodiment is very small, avoiding the cumbersome operations in the operating system, especially when processing large-scale log data, which can significantly improve the processing efficiency of log data.

[0092] Figure 6 is a schematic diagram of a structure for recycling storage resources provided by an embodiment of the present application, such as Figure 6 As shown, when the WAL log before a certain sequence number 10001 (i.e., log identifier 10001) is no longer needed, all index files _{1} with index sequence numbers less than the sequence number 10001 can be directly deleted, wherein the log sequence numbers corresponding to the index data stored in the index file _{1} are all less than 10001.

[0093] At the same time, for the block device storing WAL logs, the space that is no longer needed can be managed through two offsets, including the effective offset of the minimum sequence number and the additional offset of the maximum sequence number, wherein it can be considered that the space before the block device offset corresponding to the minimum sequence number (e.g., 10001) corresponding to the existing minimum index file (i.e., the additional offset offset10001 of the minimum sequence number) can be reused, that is, after cleaning up the old index file, these spaces in the block device can be used for subsequent data appending. At the same time, the offset position currently available for writing new data is determined based on the offset of the maximum sequence number (e.g., 10004) (i.e., the additional offset offset10004 of the maximum sequence number), and the new log data can continue to be appended and written from the position after the log data storing the maximum sequence number.

[0094] This embodiment can save the effective offset of the minimum sequence number and the additional offset of the maximum sequence number in the memory during the operation of the operating system, and dynamically update them. Even if the operating system crashes, these offsets can be recalculated by scanning the existing index file list when restarting, so there is no need to store this information additionally.

[0095] From the above description, it can be found that in order to obtain higher performance, this embodiment uses block devices to store log data in sequence instead of using the file system to store log data, which improves the reading and writing efficiency and reduces the storage space of files. There is no need to rely on the metadata management of the file system, so the reading and writing performance of log data can be significantly improved, especially in the scenarios of large-scale data storage and high concurrent access, reducing the index overhead of the file system and improving the processing efficiency of the overall system.

[0096] At the same time, since the index file storing index data has a simple structure and fixed length, this embodiment only needs to maintain the offset of each log data in the block device to achieve efficient reading of random sequence number logs. The position of the log data can be directly located during reading, which greatly reduces the overhead of random reading.

[0097] In addition, the file size of the index file is relatively small. This embodiment can load the entire index file into the memory at one time, and then read the original log data in batches concurrently according to the index data, further improving the reading efficiency; the log data and index information of this embodiment can also be written concurrently to fully utilize hardware resources and improve write throughput.

[0098] Therefore, the log data processing method provided in this embodiment can directly access the block device by bypassing the index retrieval process of the file system, thereby avoiding the file system overhead, and realizing the cleaning of invalid data and the recovery of storage resources, thereby reducing the processing overhead.

[0099] Corresponding to the method for processing log data in the above embodiment, Figure 7 This is a structural block diagram of a log data processing device provided in an embodiment of the present application. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0100] Reference Figure 7 , the device comprises:

[0101] A first determination module 301 is used to determine the history start offset and history length data of the log data stored in the block device in response to a storage instruction for the target log data;

[0102] A second determination module 302 is used to determine a first starting offset of target log data to be stored in a block device according to the historical starting offset and the historical length data;

[0103] A processing module 303 is used to process the first length data and the first starting offset of the target log data into index data of the target log data in the form of a key-value pair, where the index data is used to provide index information for reading the target log data from the block device;

[0104] The storage module 304 is used to store the target log data in the block device according to the first starting offset and the index data.

[0105] The present embodiment provides a log data processing device, which determines the historical starting offset and historical length data of the log data stored in the block device in response to the storage instruction for the target log data through the first determination module; determines the first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data through the second determination module; processes the first length data and the first starting offset of the target log data in the form of a key-value pair into index data of the target log data through the processing module, and the index data is used to provide index information for reading the target log data from the block device; and stores the target log data in the block device according to the first starting offset and the index data through the storage module. By using this device, by storing the target log data in the block device, the additional file index overhead is reduced, and by using the independent reading function of the block device, it can provide convenience for subsequent reading of log data, and further simplify the reading operation of log data.

[0106] Optionally, the storage module includes:

[0107] Determine a first log identifier of the target log data based on a maximum log identifier of the log data stored in the block device;

[0108] The target log data is stored in the block device according to the first starting offset and the first log identifier, and the index data is stored in the first index file in the block device, wherein the data length of the index data is a preset fixed length, and the first index file corresponds to the first log identifier.

[0109] Optionally, the log data processing device provided by this embodiment further includes:

[0110] An acquisition module is used for, after storing the target log data in the block device according to the first starting offset and the index data, responding to a log acquisition instruction carrying the first log identifier, acquiring index data corresponding to the first log identifier from the first index file according to the first log identifier and a preset fixed length;

[0111] A parsing module, used to parse the index data to obtain the first length data and the first starting offset in the index data;

[0112] The reading module is used to read data with a length of the first length in the block device according to the first starting offset as target log data corresponding to the log acquisition instruction.

[0113] Optionally, the acquisition module is specifically used to:

[0114] Calculate the second starting offset of the index data corresponding to the first log identifier at the first index file according to the first log identifier and the preset fixed length;

[0115] Based on the second starting offset, data having a preset fixed length is read from the first index file as index data corresponding to the first log identifier.

[0116] Optionally, the data length of the first length data is a first fixed length, the data length of the first starting offset is a second fixed length, and the sum of the first fixed length and the second fixed length is a preset fixed length.

[0117] The parsing module is specifically used for:

[0118] The index data is parsed according to the first fixed length and the second fixed length respectively to obtain the first length data and the first starting offset in the index data.

[0119] Optionally, the historical starting offset includes: a maximum starting offset and a minimum starting offset, and the second determining module is specifically used to:

[0120] According to the maximum starting offset, the minimum starting offset and the historical length data, the first starting offset of the target log data to be stored in the block device is determined. The maximum starting offset is the starting offset of the log data corresponding to the maximum log identifier in the block device, and the minimum starting offset is the starting offset of the log data corresponding to the minimum log identifier in the block device.

[0121] Optionally, the log data processing device provided by this embodiment further includes:

[0122] a judgment module, configured to judge whether the second log identifier satisfies a preset recycling condition in response to a storage resource recycling instruction carrying the second log identifier before determining the first starting offset of the target log data to be stored in the block device according to the maximum starting offset, the minimum starting offset and the historical length data;

[0123] A deletion module, used for deleting the second index file corresponding to the second log identifier if the second log identifier meets a preset recycling condition;

[0124] The third determination module is used to determine the maximum starting offset and the minimum starting offset corresponding to the block device.

[0125] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0126] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0127] The present application also provides a terminal device, Figure 8 is a schematic diagram of the structure of a terminal device provided by an embodiment of the present application, such as Figure 8 As shown, the terminal device includes: at least one processor 401, a memory 402, an input device 403, an output device 404, and a computer program stored in the memory 402 and executable on at least one processor 401. When the processor 401 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.

[0128] The input device 403 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the terminal device. The output device 404 may include a display device such as a display screen.

[0129] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 401, the steps in the above-mentioned method embodiments can be implemented.

[0130] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 401, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0132] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0133] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing log data, characterized in that: include: In response to a storage instruction for target log data, determining a history start offset and history length data of the log data stored in the block device; Determine, according to the historical starting offset and the historical length data, a first starting offset of the target log data to be stored in the block device; Processing the first length data and the first starting offset of the target log data in the form of a key-value pair into index data of the target log data, wherein the index data is used to provide index information for reading the target log data from the block device; The target log data is stored in the block device according to the first starting offset and the index data.

2. The method for processing log data according to claim 1, characterized in that: Storing the target log data in the block device according to the first starting offset and the index data includes: Determine a first log identifier of the target log data based on a maximum log identifier of the log data stored in the block device; The target log data is stored in the block device according to the first starting offset and the first log identifier, and the index data is stored in a first index file in the block device, wherein the data length of the index data is a preset fixed length, and the first index file corresponds to the first log identifier.

3. The method for processing log data according to claim 2, characterized in that: After storing the target log data in the block device according to the first starting offset and the index data, the method further includes: In response to a log acquisition instruction carrying a first log identifier, obtaining index data corresponding to the first log identifier from the first index file according to the first log identifier and the preset fixed length; Parsing the index data to obtain first length data and a first starting offset in the index data; According to the first starting offset, data having a length of the first length is read from the block device as the target log data corresponding to the log acquisition instruction.

4. The method for processing log data according to claim 3, characterized in that: The acquiring, from the first index file, index data corresponding to the first log identifier according to the first log identifier and the preset fixed length includes: Calculate, according to the first log identifier and the preset fixed length, a second starting offset of the index data corresponding to the first log identifier in the first index file; Based on the second starting offset, data having a length of the preset fixed length is read from the first index file as index data corresponding to the first log identifier.

5. The method for processing log data according to claim 3, characterized in that: The data length of the first length data is a first fixed length, the data length of the first starting offset is a second fixed length, and the sum of the first fixed length and the second fixed length is the preset fixed length. The parsing of the index data to obtain the first length data and the first starting offset in the index data includes: The index data is parsed according to the first fixed length and the second fixed length to obtain first length data and a first starting offset in the index data.

6. The method for processing log data according to claim 1, characterized in that: The historical starting offset includes: a maximum starting offset and a minimum starting offset, and determining the first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data includes: According to the maximum starting offset, the minimum starting offset and the historical length data, a first starting offset of the target log data to be stored in the block device is determined, wherein the maximum starting offset is the starting offset of the log data corresponding to the maximum log identifier in the block device, and the minimum starting offset is the starting offset of the log data corresponding to the minimum log identifier in the block device.

7. The method for processing log data according to claim 6, characterized in that: Before determining the first starting offset of the target log data to be stored in the block device according to the maximum starting offset, the minimum starting offset and the historical length data, the method further includes: In response to a storage resource recycling instruction carrying a second log identifier, determining whether the second log identifier satisfies a preset recycling condition; If the second log identifier satisfies the preset recycling condition, deleting the second index file corresponding to the second log identifier; Determine the maximum starting offset and the minimum starting offset corresponding to the block device.

8. A log data processing device, characterized in that: include: A first determination module, configured to determine, in response to a storage instruction for target log data, a history start offset and history length data of log data stored in a block device; A second determination module, configured to determine a first starting offset of the target log data to be stored in the block device according to the historical starting offset and the historical length data; a processing module, configured to process the first length data and the first starting offset of the target log data into index data of the target log data in the form of a key-value pair, wherein the index data is used to provide index information for reading the target log data from the block device; A storage module is used to store the target log data in the block device according to the first starting offset and the index data.

9. A terminal device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the terminal device implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, enables the method according to any one of claims 1 to 7 to be performed.

Images