Lightweight storage system and method suitable for embedded system

Abstract

The invention discloses a lightweight storage system and method suitable for an embedded system. The system comprises a class allocation table area, a block write-in position storage area and a data storage area. The method comprises the steps of initialization, block writing, block number reading, block reading and erasing of all blocks corresponding to classes. According to the invention, a uniform interface is provided, high-frequency small-data efficient writing, circular management and power-off protection are realized, multi-medium uniform management is supported, the occupied resources are extremely low, and the method is particularly suitable for being applied to embedded equipment with limited resources.

Description

Technical Field

[0001] This invention relates to the field of data storage technology for embedded systems, and specifically to a lightweight storage system and method for efficient, unified, and persistent management of large amounts of structured small data in embedded devices. Background Technology

[0002] In embedded systems, such as battery management systems, industrial control, smart meters, automotive devices, and smart homes, large amounts of structured small data are continuously generated and need to be stored. Examples include device operation logs, alarm event records, user operation behavior, system configuration information, and sensor sampling data. This data has the following characteristics: small data volume (each record typically ranges from a few bytes to several hundred bytes), structured format, relatively consistent length, high-frequency writing (potentially multiple writes per second), more recent data being more important, and the ability to be cyclically overwritten.

[0003] Common handling methods currently include: Using the file system for recording: writing to text files line by line has the following disadvantages: relatively high resource consumption, no index, no formatting standards, poor portability, and difficulty in querying.

[0004] Complete database systems, such as SQLite, have the following drawbacks: high resource consumption, slow startup, and unsuitability for frequent writing of small amounts of data.

[0005] Custom structure array + file mapping has the following drawbacks: complex maintenance, inconsistent interfaces, and lack of extensibility.

[0006] A custom single-loop read / write mechanism has the following drawbacks: it can only manage one type of data, resulting in scattered management of multiple types of data and poor consistency.

[0007] Therefore, there is an urgent need for a lightweight, structured, highly portable, and small embedded storage system that supports efficient writing and querying. Summary of the Invention

[0008] The main objective of this invention is to provide a lightweight storage system and method suitable for embedded systems, in order to solve the problems of high resource consumption, complex maintenance, inconsistent interfaces, and poor scalability in the existing technology.

[0009] To achieve the above object, the present invention adopts the following technical solutions: A first aspect of the present invention is to provide a lightweight storage system suitable for embedded systems, comprising: The class allocation table area describes the version of the allocation table and information about each class. Each class's information includes at least the media location, class start page, number of pages, and block size. A page is the smallest erase unit of the storage medium. The media location indicates the hardware medium where the storage area is located. The class start page indicates the starting page of the class in the storage area. The number of pages is the total number of pages allocated by the user for this class. The block size is the maximum size of this type of structured data. The block write location storage area is used to record the next block write location for each class. The block write location is stored in two locations: one location is RAM and the other location is flash. The data storage area is used to record actual data. The data storage area can be composed of different media, and the data of each class is stored in the same medium or in a storage area with the same operation and contiguous addresses.

[0010] A second aspect of the present invention is to provide a lightweight storage method suitable for embedded systems, comprising: Initialization, which includes the initialization of the storage system itself and the initialization of classes; Block write: The block write function receives the class handle and the data to be written, reads the class allocation table to obtain the basic information of the class, writes the data to the block write position according to the allocation table information, and updates the next block write position after writing. If the next block write position wraps or has been filled with old data, the corresponding page is erased to prepare for the next write. The function reads the number of blocks. It receives a class handle, reads the class allocation table to obtain basic information about the class and the block write position, calculates and returns the number of valid blocks based on whether wraparound has occurred. Block read: The block read function receives the class handle and the read offset, reads the class allocation table to obtain the basic information of the class and the block write position, calculates the position of the block corresponding to the offset in the storage area, and reads the data and returns it. The function erases all blocks corresponding to a class. It receives the class handle, reads the class allocation table to obtain the media location, the class's starting page, and the number of pages, erases the corresponding storage area data, and clears the block write position to 0.

[0011] Furthermore, the initialization of the storage system itself includes the initialization of each storage medium, the initialization of basic information, including the page size of the medium, the initialized class count, the unallocated starting page of each medium, and the copying of the block write location from the flash area to the RAM area. The initialization of the class requires passing three variables to the initialization function: media location, block size, and number of pages. After the initialization function completes the initialization, it returns the initialized class count decremented by 1 as a handle for operating the class. The initialized class count is initialized to 0 and incremented by 1 after each class is initialized.

[0012] Furthermore, during the first initialization of the class, if the allocation table version is not written, the storage system deletes the allocation table content due to version inconsistency and clears the block write location area. Then, according to the passed parameters, it retrieves the unallocated start pages of each medium as the class start pages, writes them together with the three passed variables to the class corresponding to the initialized class count in the allocation table, and erases all allocated pages. Then, it adds the number of unallocated start pages to the page count, increments the initialized class count by 1, and finally returns the initialized class count decremented by 1. If the allocation table version remains unchanged and the corresponding classes have been initialized during non-first initialization, only the number of unallocated starting pages for each medium is added, the count of initialized classes is incremented by 1, and finally the count of initialized classes is decremented by 1. The allocation table is not adjusted.

[0013] Furthermore, the method for obtaining the number of blocks can be divided into the following two cases: When no wraparound occurs: Number of blocks = number of blocks written to; When a loop occurs: Number of blocks = number of blocks written + ((number of pages - 1) * (page size / block size)).

[0014] Furthermore, when the block to be read has an offset of N, the location of the block at offset N in the memory area must first be calculated as follows: When the block write position is greater than the offset position: The location of the block at offset N in the memory area = block write location - offset location - 1; If the block write position is less than or equal to the offset position, the read is invalid if no wraparound has occurred; otherwise, the read is invalid. The location of the block at offset N in the memory area = number of pages * (page size / block size) + block write location - offset location - 1; The location of block N in the storage area is calculated, the data is read from the storage area, and the data is returned to the user.

[0015] Furthermore, the class allocation table area and the block write location storage area are preferably placed in a medium with fast read / write speed and high security.

[0016] Furthermore, in some devices with a power backup area, the block write location storage area is defined in the backup area, and the block write location variable is combined with the block write location storage area.

[0017] A third aspect of the present invention is to provide a terminal comprising: a memory, a processor, and a lightweight storage program stored in the memory and executable on the processor, wherein the lightweight storage program, when executed by the processor, implements the steps of the lightweight storage method applicable to embedded systems described above.

[0018] A fourth aspect of the present invention is to provide a computer-readable storage medium storing a lightweight storage program that, when executed by a processor, implements the steps of the lightweight storage method applicable to embedded systems described above.

[0019] This storage system provides a unified data access interface, including standard initialization, block write, block count read, block read, and all block interfaces corresponding to the erase class. The block count read interface retrieves the number of valid records, supporting record-by-record reading. The block read interface allows passing in an offset to read a block at a specific location, supporting reading based on relative record positions.

[0020] During writing, you only need to write directly according to the write position, and update the write position after writing. The write operation only involves two main actions: writing the actual data and updating the write position, achieving high write performance. This system also supports a circular write mechanism. When updating the write position after writing a block, if wraparound occurs or the data has been filled with old data, the corresponding page is erased to prepare for the next write, and a wraparound mark is set. Block reads and block count reads are calculated accordingly based on the wraparound mark.

[0021] This system supports power failure protection. The storage system consists of three areas. After initialization, if no data is written, none of the three areas will change. Since the class allocation table area does not change after initialization, power failure during writing has no impact on the class allocation table. The block write location storage area is written after the block write. If power fails, only the latest written data will be lost, and the older data can still be read back.

[0022] Advantages and beneficial effects of the present invention: 1. This invention consumes very few resources, the entire storage system design is extremely simple, each class in the class allocation table occupies only a dozen bytes, and the system operation is simple, making it very suitable for resource-constrained embedded devices, and can effectively reduce hardware costs and system complexity.

[0023] 2. The storage medium of this invention is flexible and efficient, and different combinations of storage media can be used, such as internal flash of MCU, external SPI flash, EEPROM, etc., and the storage strategy can be optimized according to the characteristics of the media to improve storage efficiency.

[0024] 3. This invention offers superior write performance. The write operation consists only of actual data writing and block write position updates, making the process simple. Combined with a cyclic write mechanism and timely erasure of old data or wrap-around preparation, it ensures efficient and smooth high-frequency small data writing, meeting the real-time requirements of embedded systems.

[0025] 4. This invention provides convenient and accurate reading. By using information such as the class allocation table and block write position, it can quickly and accurately calculate and read block data at a specified offset. Regardless of whether the data wraps around, it can achieve efficient querying and improve system response speed.

[0026] 5. This invention is fully functional and practical, providing a unified data access interface that covers a full range of functions such as initialization, block writing, block count reading, block reading, and erasing corresponding blocks, meeting the various management needs of embedded systems for structured small data.

[0027] 6. The present invention has reliable power-off protection. The class allocation table remains unchanged after initialization, and the block write location storage area can be preserved even if power is lost. Even if power is lost, only the latest written data will be lost, and the rest of the data can still be safely read back, effectively ensuring data integrity.

[0028] 7. This invention achieves wear equalization, with the class allocation table remaining essentially unchanged. Both the block write area and the storage area class storage employ a cyclic write method, resulting in even wear of the storage medium, extending its service life, and reducing long-term maintenance costs.

[0029] 8. The present invention provides structured data management, which refers to a set of structured data as a class and each piece of data as a block. This clear data organization method facilitates data classification, storage and management, and improves data maintainability and scalability.

[0030] 9. This invention supports cyclic overwrite writing. For embedded systems that frequently generate data, it can automatically overwrite old data with new data, eliminating the need for users to frequently and manually clean up storage space and ensuring that the storage area always contains the latest valid data. Attached Figure Description

[0031] Figure 1 This is a diagram showing the configuration of the storage system.

[0032] Figure 2 A diagram illustrating the contents of the class allocation table.

[0033] Figure 3 A flowchart for use in storage systems.

[0034] Figure 4 This is a diagram illustrating the class initialization process. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] This storage system calls a collection of structured data a class, and each piece of data a block. The storage system can store multiple classes, and each class can record many blocks.

[0037] like Figure 1 As shown, the storage system consists of three parts: the class allocation table area, the block write location storage area, and the data storage area.

[0038] The class allocation table is used to describe the version of the allocation table and information about each class, such as... Figure 2 As shown, the table includes: Media Location, ClassStartPage, PageCount, and BlockSize. In this storage system, a page is the smallest erase unit of the storage medium. Media Location refers to the hardware medium on which this type of storage area is located, such as the internal flash of an MCU or external SPI flash. The ClassStartPage is the starting page of the class in the storage area, used for storage location. PageCount is the total number of pages allocated to this class by the user. BlockSize is the maximum size of this type of structured data; both writing and reading are performed using this size as the smallest unit. Each class in the class allocation table occupies only a dozen or so bytes, resulting in a very small actual space usage. The class allocation table is stored in a convenient location.

[0039] Block Write Position: This records the next block write position for each class. The block write position is stored in two locations: RAM and flash. The RAM location is for convenient calculation, while the flash location is for power-off retention. For class N, the block write position is incremented by 1 when a block is written. When the maximum value for this class is exceeded, the block write position is cleared to 0, and the wraparound flag for this class is set to 1. The block write position only changes during initialization and block writes, updating the flash memory's block write position each time. The block write position storage area uses a circular write method and is only read during initialization, where the latest recorded block write position is initialized to a system variable.

[0040] Data storage area: Used to record actual data. This area can consist of different media, such as the MCU's internal flash, external flash, EEPROM, etc., with no limit on the size or quantity of each medium. Data of each class needs to be stored on the same medium or, from the system's perspective, operate on the same way, and have contiguous addresses.

[0041] The initialization of a storage system includes the initialization of the storage system itself and the initialization of the classes.

[0042] The initialization of the storage system itself includes the initialization of each storage medium and the initialization of basic information, such as the page size of the medium, the initialized class count, the available pages of each medium, and the copying of block write locations from the flash area to the RAM area.

[0043] Figure 4 This diagram illustrates the class initialization process, which is divided into initialization and non-initial initialization. During initialization, three variables need to be passed to the initialization function: MediaLocation, BlockSize, and PageCount. After initialization, the function returns ClassInitCount-1 as the handle (Name) for future operations on this class. ClassInitCount is initialized to 0 and incremented by 1 after each class initialization. Therefore, the order of initialization for each class in the function must be consistent and cannot be reduced, but can be increased. During the first initialization, because the allocation table version has not been written, the storage system will delete the allocation table content due to version inconsistency and clear the block write location area. Then, based on the passed parameters, it retrieves AvailablePage from the corresponding medium as ClassStartPage, and writes it along with the three passed variables to the class corresponding to ClassInitCount in the allocation table. Then, all allocated pages are erased. Then, AvailablePage is incremented by PageCount, ClassInitCount is incremented by 1, and finally, (ClassInitCount-1) is returned. If the allocation table version has not changed and the corresponding class has already been initialized, then only `AvailablePage` is incremented by `PageCount`, `ClassInitCount` is incremented by 1, and finally `(ClassInitCount-1)` is returned without adjusting the allocation table.

[0044] As seen in the class initialization, modifications to the allocation table are avoided as much as possible. After the user provides the necessary information, the storage system performs a very simple allocation operation, laying the foundation for lightweight design. To achieve this lightweight design, the class allocation table only supports adding new classes and complete erasure and reinitialization; it does not support modification of individual classes. The storage area corresponding to a class is erased during the initial initialization of that class.

[0045] Figure 3 The flowchart illustrates the operations that can be performed after the storage system is initialized, including block writing, block count reading, block reading, and erasing all blocks corresponding to the erase class.

[0046] The block write function requires two parameters: a class handle (Name) and the data to be written. Before writing the block, the class allocation table is read to obtain the basic information of the class. Then, based on the allocation table information, the passed information is directly written to the block write position. After writing, the next block write position is updated. If the next block write position has already wrapped around or been filled with old data, the corresponding page is erased to prepare for the next write. If only one page of storage space is prepared for this class, then after wrapping around and erasing the page, the data just written must be written to the new position. Generally, two pages or more of space are allocated to the storage area. However, for scenarios such as storage configuration information that are not frequently changed and only require the latest data, allocating one page of data is supported, which is still necessary for small-capacity systems. The system updates the block write position immediately after writing, allowing for a fast response to the next write.

[0047] Block count reading typically involves obtaining the valid number of storage blocks before actually reading the data. The block count reading function requires one parameter: a class handle (Name), and returns the valid number of blocks for that class. During reading, the class allocation table is first read to obtain basic class information, and then the block count is obtained based on the class allocation table information and the block write position. The block count acquisition method falls into two categories: When no wraparound occurs: Number of blocks = BlockWritePosition; When a loop occurs: Number of blocks = BlockWritePosition + ((PageCount - 1) * (PageSize / BlockSize)); BlockWritePosition: Block write position PageCount: Number of pages PageSize: Page size BlockSize: Block size The block read function requires two parameters: a class handle (Name) and a read offset (Offset). The offset for the newest data is 0, the offset for the second newest data is 1, and the offset for earlier written blocks is larger. During reading, the class allocation table is first read to obtain basic class information, and then the block is read based on the class allocation table information and the block's write position. When we need to read a block with offset N, we first need to calculate the location of the block with offset N in the memory area, which can be done as follows: When the block write position is greater than the offset position (BlockWritePosition > Offset) The location of the block at offset N in the memory area = BlockWritePosition - Offset - 1; If the block write position is less than or equal to the offset position, the read is invalid if no wraparound has occurred; otherwise, the read is invalid. The location of the block at offset N in the memory area = PageCount * (PageSize / BlockSize) + BlockWritePosition - Offset - 1; BlockWritePosition: Block write position PageCount: Number of pages PageSize: Page size BlockSize: Block size Once the location of block N in the storage area is calculated, the data can be read from the storage area and returned to the user.

[0048] The function to erase all blocks corresponding to a class requires passing one parameter: the class handle (Name). The function will read the class allocation table to obtain the media location (MediaLocation), the class start page (ClassStartPage), and the number of pages (PageCount). Based on these three pieces of information, it will erase the data in the corresponding storage area and clear the block write position to 0.

[0049] The class allocation table area and the block write location storage area are preferentially placed on media with fast read and write speeds and high security. As the management unit of the entire system, the class allocation table area and the block write location storage area require very little storage space. The class allocation table generally does not change during operation, while the block write location storage area needs to be updated after each block is written, resulting in frequent writes.

[0050] The storage system consists of three areas. After initialization, if no data is written, none of the three areas will change. Since the class allocation table area does not change after initialization, a power outage during writing will not affect the class allocation table. The block write location storage area is written after the block write. If a power outage occurs, only the latest written data will be lost, and the older data can still be read back.

[0051] The write operation only updates the block write location storage area in addition to writing the actual data, and the read operation only reads the class allocation table and the actual read data. In all operations, the computational tasks are very light, and the management-related overhead added by these two main actions is very low.

[0052] The class allocation table remains basically unchanged. Block writes are written in a circular manner in the write area, and the class storage in the storage area is also written in a circular manner. Therefore, the system experiences very little wear on the storage. For large amounts of data that are written in a circular manner, a larger space can be allocated to reduce wear.

[0053] In some devices with a power backup area, the memory backup area is never powered off. The block write location storage area can also be defined in the backup area, and the block write location variable and the block write location storage area can be combined into one, which makes the storage system simpler.

Claims

1. A lightweight storage system suitable for embedded systems, characterized in that, include: The class allocation table area describes the version of the allocation table and information about each class. Each class's information includes at least the media location, class start page, number of pages, and block size. A page is the smallest erase unit of the storage medium. The media location indicates the hardware medium where the storage area is located. The class start page indicates the starting page of the class in the storage area. The number of pages is the total number of pages allocated by the user for this class. The block size is the maximum size of this type of structured data. The block write location storage area is used to record the next block write location for each class. The block write location is stored in two locations: one location is RAM and the other location is flash. The data storage area is used to record actual data. The data storage area can be composed of different media, and the data of each class is stored in the same medium or in a storage area with the same operation and contiguous addresses.

2. A storage method for a lightweight storage system suitable for embedded systems as described in claim 1, characterized in that, include: Initialization, which includes the initialization of the storage system itself and the initialization of classes; Block write: The block write function receives the class handle and the data to be written, reads the class allocation table to obtain the basic information of the class, writes the data to the block write position according to the allocation table information, and updates the next block write position after writing. If the next block write position wraps or has been filled with old data, the corresponding page is erased to prepare for the next write. The function reads the number of blocks. It receives a class handle, reads the class allocation table to obtain basic information about the class and the block write position, calculates and returns the number of valid blocks based on whether wraparound has occurred. Block read: The block read function receives the class handle and the read offset, reads the class allocation table to obtain the basic information of the class and the block write position, calculates the position of the block corresponding to the offset in the storage area, and reads the data and returns it. The function erases all blocks corresponding to a class. It receives the class handle, reads the class allocation table to obtain the media location, the class's starting page, and the number of pages, erases the corresponding storage area data, and clears the block write position to 0.

3. The storage method according to claim 2, characterized in that, The initialization of the storage system itself includes the initialization of each storage medium and the initialization of basic information, including the page size of the medium, the initialized class count, the unallocated starting page of each medium, and the copying of the block write location from the flash area to the RAM area. The initialization of the class requires passing three variables to the initialization function: media location, block size, and number of pages. After the initialization function completes the initialization, it returns the initialized class count decremented by 1 as a handle for operating the class. The initialized class count is initialized to 0 and incremented by 1 after each class is initialized.

4. The storage method according to claim 3, characterized in that, During the first initialization of the class, if the allocation table version is not written, the storage system deletes the allocation table content due to version inconsistency and clears the block write location area. Then, according to the passed parameters, it retrieves the unallocated start pages of each medium as the class start pages, writes them together with the three passed variables to the class corresponding to the initialized class count in the allocation table, and erases all allocated pages. Then, it adds the number of unallocated start pages to the page count, increments the initialized class count by 1, and finally returns the initialized class count decremented by 1. If the allocation table version remains unchanged and the corresponding classes have been initialized during non-first initialization, only the number of unallocated starting pages for each medium is added, the count of initialized classes is incremented by 1, and finally the count of initialized classes is decremented by 1. The allocation table is not adjusted.

5. The storage method according to claim 2, characterized in that, There are two methods for obtaining the number of blocks: When no wraparound occurs: Number of blocks = number of blocks written to; When a loop occurs: Number of blocks = number of blocks written + ((number of pages - 1) * (page size / block size)).

6. The storage method according to claim 2, characterized in that, When the block to be read has an offset of N, the location of the block at offset N in the memory area must first be calculated as follows: When the block write position is greater than the offset position: The location of the block at offset N in the memory area = block write location - offset location - 1; If the block write position is less than or equal to the offset position, the read is invalid if no wraparound has occurred; otherwise, the read is invalid. The location of the block at offset N in the memory area = number of pages * (page size / block size) + block write location - offset location - 1; The location of block N in the storage area is calculated, the data is read from the storage area, and the data is returned to the user.

7. The storage method according to claim 1, characterized in that, The class allocation table area and block write location storage area are preferably placed in media with fast read / write speeds and high security.

8. The storage method according to claim 1, characterized in that, In some devices with a power backup area, the block write location storage area is defined in the backup area, and the block write location variable is combined with the block write location storage area.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a lightweight storage program stored in the memory and executable on the processor, wherein the lightweight storage program, when executed by the processor, implements the steps of the storage method as described in any one of claims 2-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a lightweight storage program, which, when executed by a processor, implements the steps of the storage method as described in any one of claims 2-8.

Images