Software method, device and equipment for log retrieval and storage medium

Abstract

The invention discloses a log retrieval software method, device and equipment and a storage medium. A linked list data structure is adopted to store and retrieve log data. The log data is firstly stored in an initial sector of an initial Block, when the storage space of the initial sector is insufficient, the log data is stored from the next sector, when all the sectors are fully stored, the log data is stored from the sector of the next Block, when all the Blocks are fully stored, the log data is circularly stored from an initial address, and the Block represents a storage unit in a Flash memory; and searching the stored log data, and determining a reading address by taking whether the log data overflows or not as a judgment basis. According to the method, the linked list is used as a storage structure of the log data, and a circulating storage mechanism is designed, so that the retrieval efficiency is improved, the storage overhead is reduced, and the service life of the Flash memory is remarkably prolonged.

Description

Technical Field

[0001] The present application relates to the field of log retrieval of Flash memory, and in particular to a software method, apparatus, device and storage medium for log retrieval. Background Art

[0002] In Flash memory, log data is usually stored in the following ways:

[0003] In the linear storage method, log data is stored sequentially in the continuous address space of the Flash memory, with each log entry occupying a fixed amount of storage space. This method is simple to implement, but has low storage efficiency and easily wastes storage space. Frequent erase and rewrite operations are concentrated in the same Flash memory area, causing severe wear and shortening the storage medium life. Retrieval efficiency is also low, especially when the log volume is large, and linear scanning speed is slow.

[0004] The file system storage method uses an embedded file system (such as FAT32 or EXT4) to manage log files, with log data stored as files within the file system. This method offers high flexibility and can easily manage different types of files and data, but it also has high storage overhead, as the file system itself consumes a significant amount of storage space. Frequent read and write operations on the file system increase the number of flash erases, shortening the flash lifespan. Retrieval speed is limited by the file system's index address mechanism, resulting in low retrieval efficiency when log volumes are large.

[0005] The database storage method uses an embedded database (such as SQLite) to store and manage log data, storing log data in a structured manner. This provides structured storage and complex query capabilities, resulting in high retrieval efficiency. However, the database system itself consumes a lot of storage space and system resources, making it unsuitable for resource-constrained embedded systems. Frequent database read and write operations and index address updates cause severe wear on the Flash memory, shortening its lifespan.

[0006] Based on the above, current storage methods cannot balance storage overhead, retrieval efficiency, data flexibility, and Flash lifespan. Summary of the Invention

[0007] In view of this, the present invention provides a software method for log retrieval, which uses a linked list data structure to store and retrieve log data;

[0008] The log data is first stored in the initial sector of the initial block. When the storage space of the initial sector is insufficient, storage starts from the next sector. When all sectors are full, storage starts from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. Block represents a storage unit in the Flash memory.

[0009] The newly stored log data is then retrieved, and the reading address is determined based on whether the log data overflows and the number of forward log data.

[0010] Furthermore, the entry of the log data serves as a linked list node, including log content and a pointer to the next node.

[0011] Furthermore, the latest log entry is located by maintaining the header pointer of the linked list node.

[0012] Furthermore, the single storage process is as follows:

[0013] a1: Get the storage index address;

[0014] a2: Determine whether all blocks are not fully stored. If so, directly obtain the starting address for writing this log; if not, initialize the storage index address and restart storage from the starting address;

[0015] a3: Determine whether the log data overflows. If not, execute step a6. If so, determine whether TolalNum is greater than the number of log entries stored in a single block.

[0016] a4: If TolalNum is not greater than the number of log entries stored in a single block, execute step a6. If it is greater, determine whether the current sector is full.

[0017] a5: If the current sector is not full, execute step a6. If it is full, erase the next sector and move the storage index address to the next sector, and execute step a6 again.

[0018] a6: Write the current log, then write the new storage index address, and end.

[0019] Furthermore, in step a2, determining whether all blocks are not fully stored is specifically to determine whether TotalNum is less than the storage log size of all blocks, wherein TotalNum represents the current number of forward log data items.

[0020] Furthermore, in step a3, whether the log data overflows is determined by judging whether Overflow is equal to 0. Overflow is a data overflow flag bit, and when it is equal to 0, it indicates no overflow.

[0021] Furthermore, the newly stored log data is retrieved again, and the reading address is determined based on whether the log data overflows and the number of forward log data, which is specifically the following steps:

[0022] Assume that the number of new log data to be queried is n. First, the overflow flag is used to determine whether the data overflows.

[0023] If there is no overflow, determine whether TolalNum is less than or equal to n. If so, directly read 0 to TolalNum log data; if not, read (TolalNum-n) to TolalNum log data;

[0024] If overflow occurs, determine whether TolalNum is less than n. If so, read 0 to TolalNum pieces of log data and (n-TolalNum) to the last piece of log data; if not, read (TolalNum-n) to TolalNum pieces of log data.

[0025] Accordingly, the present invention also provides a log retrieval device, comprising:

[0026] A determination module, configured to determine a user's request type, wherein the request type includes a storage request and a retrieval request;

[0027] A storage module, configured to store log data in a storage medium. The log data is first stored in an initial sector of an initial block. When the storage space of the initial sector is insufficient, storage is started from the next sector. When all sectors are full, storage is started from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. The block represents a storage unit in the Flash memory.

[0028] The retrieval module is used to retrieve the newly stored log data and determine the reading address based on whether the log data overflows and the number of forward log data.

[0029] Correspondingly, the present invention also provides a computer device, including a memory and a processor, wherein the memory is used to store a program stored in the processor to implement the software method for log retrieval.

[0030] Correspondingly, the present invention also provides a storable medium for storing a program that can implement the software method for log retrieval, preferably using W25q series SPIFlash as the storage medium.

[0031] The present invention adopts a linked list as the storage structure of log data, making the writing and reading operations of log data more efficient, and eliminating the need for a large number of data moves or search operations as in traditional linear storage. A circular storage mechanism is also designed, and when all blocks are fully stored, the circular storage starts again from the beginning. This strategy avoids frequent erasure of the same block, thereby significantly reducing the number of erasures of the Flash memory, and significantly extending the service life of the Flash memory. The storage location can be dynamically switched according to the size and location of the storage space. When a sector or block is fully stored, it automatically switches to the next available space, avoiding the waste of storage space. There is no fixed storage size limit, ensuring the consistency and integrity of the data. The W25q series SPIFlash is used as the storage medium. This series of Flash has high reliability and durability and is suitable for embedded system applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of a software method for log retrieval in the present application;

[0033] Figure 2 yes Figure 1 Software method storage process diagram for log retrieval;

[0034] Figure 3 It is the logical process of storing operation logs in a single time;

[0035] Figure 4 It is the logical process of querying the 100 latest operation logs at a time;

[0036] Figure 5 This is a composition diagram of a log retrieval device of the present application. DETAILED DESCRIPTION

[0037] In Flash memory, log data is usually stored in the following ways:

[0038] In the linear storage method, log data is stored sequentially in the continuous address space of the Flash memory, with each log entry occupying a fixed amount of storage space. This method is simple to implement, but has low storage efficiency and easily wastes storage space. Frequent erase and rewrite operations are concentrated in the same Flash memory area, causing severe wear and shortening the storage medium life. Retrieval efficiency is also low, especially when the log volume is large, and linear scanning speed is slow.

[0039] The file system storage method uses an embedded file system (such as FAT32 or EXT4) to manage log files, with log data stored as files within the file system. This method offers high flexibility and can easily manage different types of files and data, but it also has high storage overhead, as the file system itself consumes a significant amount of storage space. Frequent read and write operations on the file system increase the number of flash erases, shortening the flash lifespan. Retrieval speed is limited by the file system's index address mechanism, resulting in low retrieval efficiency when log volumes are large.

[0040] The database storage method uses an embedded database (such as SQLite) to store and manage log data, storing log data in a structured manner. This provides structured storage and complex query capabilities, resulting in high retrieval efficiency. However, the database system itself consumes a lot of storage space and system resources, making it unsuitable for resource-constrained embedded systems. Frequent database read and write operations and index address updates cause severe wear on the Flash memory, shortening its lifespan.

[0041] Based on the above, current storage methods cannot balance storage overhead, retrieval efficiency, data flexibility, and Flash lifespan.

[0042] In view of this, the present invention proposes a software method, apparatus, device and storage medium for log retrieval to solve the above technical problems to a certain extent.

[0043] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below with reference to the accompanying drawings. However, it will be understood by those skilled in the art that many technical details are provided in each embodiment of the present application to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0044] In the following description, for the purpose of illustrating the various disclosed embodiments, certain specific details are set forth in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with this application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0045] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, that is, should be interpreted to mean "including, but not limited to."

[0046] The following will describe in detail the various embodiments of the present application in conjunction with the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present application. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present application, but are only intended to illustrate the essential spirit of the technical solution of the present application.

[0047] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0048] First, the present invention proposes a software method for log retrieval, such as Figure 1 As shown,

[0049] Use linked list data structure to store and retrieve log data;

[0050] The log data is first stored in the initial sector of the initial block. When the storage space of the initial sector is insufficient, storage starts from the next sector. When all sectors are full, storage starts from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. Block represents a storage unit in the Flash memory.

[0051] The stored data is retrieved, and a retrieval address is determined by judging whether the log data overflows.

[0052] Furthermore, the entry of the log data serves as a linked list node, including log content and a pointer to the next node.

[0053] The log content includes timestamp, event description, device status, etc. The pointer to the next node records the physical address of the next log entry, which is specifically the sector offset in Flash.

[0054] Furthermore, the latest log entry is located by maintaining the header pointer of the linked list node.

[0055] Combine Figure 2 , the following describes the stored procedure in detail:

[0056] In one embodiment, the data table records the sector address range (addr_st to addr_en), the search pointer (P), and the operation sequence.

[0057] Phase A: When writing for the first time or the sector is empty: read the storage unit value. If the agreed empty mark is 0xFFFF, it means that the address is an idle starting position and can be used to write data.

[0058] When all blocks in phase B are not fully stored: space is continuously allocated based on the length of the linked list node (len). The format of each data segment is [len+data(0)~data(n)]. After writing, the search pointer (P→P+n→P+n+n) is updated. The writable area is confirmed by checking whether the first address of the next segment is 0xFFFF through the offset.

[0059] When the space in stage C is insufficient, the contents of the starting address sector are erased and restored.

[0060] Combine Figure 3 Next, we will explain the logic of single storage, which is the following steps:

[0061] a1: Enter the operation command, organize the operation data, and obtain the storage index address;

[0062] a2: Determine whether all blocks are not fully stored. If so, directly obtain the starting address for writing this log; if not, initialize the storage index address and restart storage from the starting address;

[0063] To determine whether all blocks are not fully stored, specifically, determine whether TotalNum is less than the storage log size of all blocks, where TotalNum represents the current number of forward log data items. In the figure, 16384 is the threshold derived from the physical structure of the Flash.

[0064] a3: Determine whether the log data overflows. If not, execute step a6. If so, determine whether TolalNum is greater than the number of log entries stored in a single block.

[0065] Check whether the log data overflows by checking whether Overflow is equal to 0. Overflow is the data overflow flag. When it is equal to 0, it indicates no overflow, and when it is non-zero, it indicates overflow.

[0066] It is understandable that if Overflow == 0, it means that the remaining space in the current sector or block is sufficient to accommodate the new log;

[0067] If Overflow! = 0: You need to first combine the sectors to determine and refine the problem of insufficient space.

[0068] a4: If TotalNum is not greater than the number of log entries stored in a single block (64 in this case), execute step a6. If it is greater, determine whether the current sector is full.

[0069] a5: If the current sector is not full, execute step a6. If it is full, erase the next sector and move the storage index address to the next sector, and execute step a6 again.

[0070] a6: Write the current log, then write the new storage index address, and end.

[0071] That is, in the above steps, use TotalNum to determine whether the Block cycle and sector are full; use Overflow to determine whether the written space overflows and predict in advance whether the remaining space is sufficient for writing; and convert the Flash physical space into a log number threshold.

[0072] It can be known that the above steps a1 to a6 are divided to a certain extent only for ease of understanding. Any adjustments, additions and deletions made by those skilled in the art on this basis, including but not limited to the order of steps, should also be included in the scope of protection of the present invention.

[0073] Further, such as Figure 4 As shown, the newly stored log data is retrieved again, and the reading address is determined based on whether the log data overflows and the number of forward log data, which is specifically the following steps:

[0074] Assume that the number of new log data to be queried is n ( Figure 4 is a specific embodiment, n is 100), firstly, the data overflow flag is used to determine whether overflow occurs;

[0075] If there is no overflow, determine whether TolalNum is less than or equal to n. If so, directly read 0 to TolalNum log data, as shown in Figure 1; if not, read (TolalNum-n) to TolalNum log data, as shown in Figure 2;

[0076] If overflow occurs, determine whether TolalNum is less than n. If so, read 0 to TolalNum pieces of log data and (n-TolalNum) to the last piece of log data, as shown in Figure ③; if not, read (TolalNum-n) to TolalNum pieces of log data, as shown in Figure ④.

[0077] In this way, the reading address is determined based on whether the log data overflows and the number of forward log data, so as to ensure that the latest n log data are read.

[0078] Furthermore, the present invention also provides a log retrieval device, comprising:

[0079] A determination module, configured to determine a user's request type, wherein the request type includes a storage request and a retrieval request;

[0080] A storage module, configured to store log data in a storage medium. The log data is first stored in an initial sector of an initial block. When the storage space of the initial sector is insufficient, storage is started from the next sector. When all sectors are full, storage is started from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. The block represents a storage unit in the Flash memory.

[0081] The retrieval module is used to retrieve the newly stored log data and determine the reading address based on whether the log data overflows and the number of forward log data.

[0082] A computer device includes a memory and a processor, wherein the memory is used to store a program stored in the processor to implement the above-mentioned software method for log retrieval.

[0083] A storable medium is characterized in that it is used to store a program that can implement the software method of log retrieval as described above, and the storable medium uses the SPIFlash of the W25q series.

[0084] The present invention adopts a linked list as the storage structure of log data, making the writing and reading operations of log data more efficient, and eliminating the need for a large number of data moves or search operations as in traditional linear storage. A circular storage mechanism is also designed, and when all blocks are fully stored, the circular storage starts again from the beginning. This strategy avoids frequent erasure of the same block, thereby significantly reducing the number of erasures of the Flash memory, and significantly extending the service life of the Flash memory. The storage location can be dynamically switched according to the size and location of the storage space. When a sector or block is fully stored, it automatically switches to the next available space, avoiding the waste of storage space. There is no fixed storage size limit, ensuring the consistency and integrity of the data. The W25q series SPIFlash is used as the storage medium. This series of Flash has high reliability and durability and is suitable for embedded system applications.

[0085] The above is a detailed introduction to a software method, device, equipment and storage medium for log retrieval provided by the embodiments of the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

[0086] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two.

Claims

1. A software method for log retrieval, characterized by: Use linked list data structure to store and retrieve log data; The log data is first stored in the initial sector of the initial block. When the storage space of the initial sector is insufficient, storage starts from the next sector. When all sectors are full, storage starts from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. Block represents a storage unit in the Flash memory. The newly stored log data is then retrieved, and the reading address is determined based on whether the log data overflows and the number of forward log data.

2. The software method for log retrieval according to claim 1, characterized in that: The entry of the log data serves as a linked list node, including log content and a pointer to the next node.

3. The software method for log retrieval according to claim 2, characterized in that: The latest log entry is located by maintaining the header pointer of the linked list node.

4. The software method for log retrieval according to claim 1, characterized in that: The single storage process is as follows: a1: Get the storage index address; a2: Determine whether all blocks are not fully stored. If so, directly obtain the starting address for writing this log; if not, initialize the storage index address and restart storage from the starting address; a3: Determine whether the log data overflows. If not, execute step a6. If so, determine whether TotalNum is greater than the number of log entries stored in a single block. TotalNum indicates the current number of forward log data entries. a4: If TolalNum is not greater than the number of log entries stored in a single block, execute step a6. If it is greater, determine whether the current sector is full. a5: If the current sector is not full, execute step a6. If it is full, erase the next sector and move the storage index address to the next sector, and execute step a6 again. a6: Write the current log, then write the new storage index address, and end.

5. The software method for log retrieval according to claim 4, characterized in that: In step a2, determining whether all blocks are not fully stored is specifically to determine whether TolalNum is less than the storage log size of all blocks.

6. The software method for log retrieval according to claim 5, characterized in that: In step a3, whether the log data overflows is determined by judging whether Overflow is equal to 0. Overflow is a data overflow flag bit, and when it is equal to 0, it indicates no overflow.

7. The software method for log retrieval according to claim 6, characterized in that: The newly stored log data is then retrieved, and the reading address is determined based on whether the log data overflows and the number of forward log data entries, specifically the following steps: Assume that the number of new log data to be queried is n. First, the overflow flag is used to determine whether the data overflows. If there is no overflow, determine whether TolalNum is less than or equal to n. If so, directly read 0 to TolalNum log data; if not, read (TolalNum-n) to TolalNum log data; If overflow occurs, determine whether TolalNum is less than n. If so, read 0 to TolalNum pieces of log data and (n-TolalNum) to the last piece of log data; if not, read (TolalNum-n) to TolalNum pieces of log data.

8. A log retrieval device, characterized in that: include: A determination module, configured to determine a user's request type, wherein the request type includes a storage request and a retrieval request; A storage module, configured to store log data in a storage medium. The log data is first stored in an initial sector of an initial block. When the storage space of the initial sector is insufficient, storage is started from the next sector. When all sectors are full, storage is started from the sector of the next block. When all blocks are full, storage is cyclically started from the starting address. The block represents a storage unit in the Flash memory. The retrieval module is used to retrieve the newly stored log data and determine the reading address based on whether the log data overflows and the number of forward log data.

9. A computer device, characterized in that: The system comprises a memory and a processor, wherein the memory is used to store a program stored in the processor to implement the software method for log retrieval according to any one of claims 1 to 7.

10. A storable medium, characterized in that: A program for storing a software method for implementing log retrieval according to any one of claims 1 to 7.

Images