Instrument data lightweight embedded database system
By designing a lightweight embedded database system, the storage and retrieval problems of traditional databases in systems with limited memory space are solved, achieving fast, secure, durable and efficient instrument data storage.
Patent Information
- Application Number
- CN202510211143.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2045-02-25
AI Technical Summary
Traditional embedded databases such as SQLite3 support large data storage but not data anomaly recovery. FlashDB supports large data storage and wear leveling but its large size makes it unsuitable for systems with limited memory.
A lightweight embedded database system for instrument data was designed, including a header module, a mapping table module, and a real-time data module. The system manages data using a specific management method, supports data redundancy, security, and durability, and uses AES-CBC encryption, CRC16 verification, and MQTT protocol filling. The data is dynamically written to flash memory in turn to balance wear.
It enables rapid storage of instrument data in systems with limited memory space, and features data redundancy, security, and durability, thereby improving system reliability and storage efficiency.
Smart Images
Figure CN120179646B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of verification and calibration systems, in particular to a lightweight embedded database system for instrument data. BACKGROUND
[0002] A verification and calibration system is a system for verifying and calibrating measuring instruments and equipment, aiming to ensure the accuracy and reliability of measurement results. Verification and calibration are two different activities, but they both involve the evaluation of the accuracy and reliability of measuring equipment. Verification and calibration systems are widely used in various measuring instruments and equipment, such as temperature sensors, pressure gauges, electronic scales, etc., and are generally used as complete solutions for laboratories and industrial applications. With the complete solution of the verification and calibration system, there is a large amount of instrument data processing demand, and a database system for instrument data.
[0003] In the current verification and calibration system, the verification and calibration system and the database system communicate through a serial port or a TCP port to interact with instrument data. Therefore, a large amount of instrument data is stored in the database system, and at the same time, the stored instrument data needs to be checked for redundancy and have an abnormal recovery mechanism. If the instrument data is stored in a Flash space, the database system also needs to support wear leveling functions. In the event of a system anomaly or power anomaly during data operation, the database system needs to have a write-protect mechanism. Among them, the traditional embedded database such as Sqlite3 database supports large data storage, but does not support recovery mechanism when data is abnormal; the flashDB database supports large data storage, wear leveling and data abnormal backup recovery, but the database package reaches 10MB or more, which cannot be transplanted and developed in a database with small memory space, and the applicable verification and calibration system is limited. SUMMARY
[0004] In view of the above analysis, the embodiments of the present application aim to provide a lightweight embedded database system for instrument data to solve the problem in the prior art that the traditional embedded database such as Sqlite3 database supports large data storage, but does not support recovery mechanism when data is abnormal. The flashDB database supports large data storage, wear leveling and data abnormal backup recovery, but the database package reaches 10MB or more, which cannot be transplanted and developed in a database with small memory space, and the applicable verification and calibration system is limited.
[0005] The lightweight embedded database system for instrument data of the embodiments of the present application comprises:
[0006] A table header module configured to store a database table header of instrument data;
[0007] A mapping table module configured to store a database mapping table of the instrument data;
[0008] a real-time data module configured to store a database real-time data block of the instrument data;
[0009] a data management module configured to manage the database table header in a first management form, manage the database mapping table in a second management form, and manage the database real-time data block in a third management form.
[0010] The instrument data lightweight embedded database system of the embodiment of the present application comprises a table header module, a mapping table module, a real-time data module, and a data management module. The table header module is configured to store a database table header of instrument data. The mapping table module is configured to store a database mapping table of instrument data. The real-time data module is configured to store a database real-time data block of instrument data. The data management module is configured to manage the database table header in a first management form, manage the database mapping table in a second management form, and manage the database real-time data block in a third management form. The instrument data lightweight embedded database system with the above architecture has a lightweight storage mode and does not occupy too much space, is convenient for application to various calibration systems, and has the function management design basis of data redundancy, security, and durability while facilitating the rapid storage of instrument data of various calibration systems.
[0011] As one of the optional embodiments, the database table header is stored in a first set number of bytes. The database mapping table is stored in a second set number of bytes. The database real-time data block is stored in a third set number of bytes.
[0012] As one of the optional embodiments, the database table header comprises a database version number, a table header storage space, a total number of data tables, and each data table.
[0013] Each of the data tables comprises a table name, a table real-time data length, a real-time number, and an offset of the database mapping table in a flash space.
[0014] As one of the optional embodiments, the mapping table module stores a database mapping table in the form of a master mapping table and a database mapping table in the form of a slave mapping table.
[0015] The database mapping table comprises an insertion state of the database real-time data block, an insertion index number of the database real-time data block, a data length of the database real-time data block, and an offset of the database real-time data block in a flash space.
[0016] As one of the optional embodiments, the database real-time data block comprises a serial number of the instrument data, a time of the instrument data, a type of the instrument data, a network signal of the instrument data, content of the instrument data, a check code of the instrument data, and a data reporting state of the instrument data.
[0017] As one of the optional embodiments, the first, second and third set byte numbers are all 4kb.
[0018] As one of the optional embodiments, the process of managing the database table header in the first management form includes:
[0019] The database table header is checked for damage by the unique identifier of the database version number, the threshold of the total number of data tables, and the table header storage space.
[0020] The table names of different data tables are named, and different information record tables and table real-time data lengths are constructed according to the types of instrument data.
[0021] As one of the optional embodiments, the database mapping table is managed in the second management form, including:
[0022] Each of the inserted database real-time data blocks is managed by a red-black tree, facilitating the upper layer application to add, delete, update and search the database.
[0023] As one of the optional embodiments, the process of managing the database real-time data block in the third management form includes:
[0024] The database real-time data block is encrypted by using the AES-CBC algorithm, and the encryption and decryption operations in the AES-ECB mode are realized by using the API provided by the mbed TLS library.
[0025] The content of the database real-time data block is checked by using the CRC16 algorithm.
[0026] The data content of the database real-time data block is filled according to the mqtt protocol; wherein the data format of the data content supports json and hex formats.
[0027] As one of the optional embodiments, the process of managing the database real-time data block in the third management form further includes:
[0028] The database real-time data block is written into flash, and each sector is written dynamically and alternately according to the offset of the database real-time data block in the flash space, so that the wear of each sector is balanced.
[0029] When the database real-time data block is about to be erased and written, the flash operation is prohibited when soft restart is needed by using the system soft restart judgment mechanism. BRIEF DESCRIPTION OF DRAWINGS
[0030] Figure 1A module structure diagram of the instrument data lightweight embedded database system of an application embodiment is shown in FIG. 1;
[0031] Figure 2 A database space layout diagram is shown in FIG. 2;
[0032] Figure 3 A workflow diagram of the data management module of an application embodiment is shown in FIG. 3;
[0033] Figure 4 A database operation flowchart is shown in FIG. 4. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort fall within the scope of protection of the present application.
[0035] Unless otherwise defined, technical or scientific terms used in the present application should be understood as having the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms "first", "second" and similar terms used in the present application do not denote any order, quantity or importance, but are used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right" and the like are used only to represent relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.
[0036] In order to keep the following description of the embodiments of the present application clear and brief, the present application omits the detailed description of some known functions and known components.
[0037] The embodiments of the present application provide an instrument data lightweight embedded database system.
[0038] Figure 1 A module structure diagram of the instrument data lightweight embedded database system of an application embodiment is shown in FIG. 1; Figure 1 As shown in FIG. 1, the instrument data lightweight embedded database system of an application embodiment includes:
[0039] The table header module 100 is configured to store the database table header of the instrument data;
[0040] The mapping table module 101 is configured to store a database mapping table of the instrument data;
[0041] The real-time data module 102 is configured to store a database real-time data block of the instrument data;
[0042] The data management module 103 is configured to manage the database table header in a first management form, the database mapping table in a second management form, and the database real-time data block in a third management form.
[0043] In the embodiment, the instrument data lightweight embedded database system includes a base database base configured to store instrument data. The table header module 100, the mapping table module 101, and the real-time data module 102 are sub-units of the base database. The data management module 103 is independent of the base database base and is configured to manage the base database base.
[0044] In the embodiment, the base database base is built based on a flash memory, the flash space is planned and laid out, and is divided into a database table header, a database mapping table, and a database real-time data block. Compared with a traditional database storage mode, the structure has very fast read and write speed and higher performance. Moreover, the storage mode is lightweight and does not occupy too much space.
[0045] As one of the optional embodiments, the database table header is stored in a first set number of bytes; the database mapping table is stored in a second set number of bytes; and the database real-time data block is stored in a third set number of bytes.
[0046] Preferably, the first set number of bytes, the second set number of bytes, and the third set number of bytes are all 4096 bytes, so as to support 4096-byte data storage and facilitate storage.
[0047] As one of the optional embodiments, Figure 2 The database space layout diagram is shown in FIG. 1. Figure 2 As shown in the figure, the database table header includes a database version number table header storage space, a total number of data tables, and each data table.
[0048] Each of the data tables includes a table name, a table real-time data length, a real-time number of entries, and an offset of the database mapping table in a flash space. The mapping table module stores a database mapping table in a form of a master mapping table and a database mapping table in a form of a slave mapping table. The database mapping table includes an insertion state of the database real-time data block, an insertion index number of the database real-time data block, a data length of the database real-time data block, and an offset of the database real-time data block in the flash space. The database real-time data block includes a serial number of metering data, a time of the metering data, and a type of the metering data, a network signal of the metering data, content of the metering data, a check code of the metering data, and a data reporting state of the metering data.
[0049] As shown in a database space layout Figure 2 , taking a flash database space capacity of 8 MB as an example, the database is divided into a database table header, a table N mapping table, and a table N real-time data block, where 1<=N<=7. The database supports creation of multiple tables, each of which is independent of each other, and each of which has a master-slave mapping table and a real-time data block.
[0050] Based on the database space layout Figure 2 , Figure 3 , a data management module workflow diagram of an application embodiment is shown in FIG. 2. As shown in FIG. 2, a process of managing the database table header in a first management form includes steps S10 and S11: Figure 3
[0051] S10, checking whether the database table header is damaged through a unique identification code of the database version number, a threshold of a total number of data tables, and a table header storage space;
[0052] S11, naming a table name of a different data table, and constructing different information record tables and table real-time data lengths according to a type of metering data.
[0053] A second management form manages the database mapping table, including step S12:
[0054] S12, managing information of each of the inserted database real-time data blocks through a red-black tree, facilitating upper-layer applications to add, delete, update, and search the database.
[0055] A third management form manages the database real-time data block, including steps S13 to S17:
[0056] S13, encrypting the database real-time data block by using an AES-CBC algorithm, and realizing encryption and decryption operations in an AES-ECB mode through an API provided by an mbed TLS library;
[0057] S14, the database real-time data block is checked by using CRC16 algorithm;
[0058] S15, the data content of the database real-time data block is filled according to mqtt protocol; wherein, the data format of the data content supports json and hex format;
[0059] S16, the database real-time data block is written into flash, and each sector is written dynamically and alternately according to the offset of the database real-time data block in flash space, so that the wear of each sector is balanced;
[0060] S17, when the database real-time data block is about to be erased and written, the flash operation is prohibited by system soft restart judgment mechanism when soft restart is needed.
[0061] In order to explain steps S10 to S17, taking the first set byte number, the second set byte number and the third set byte number of the preferred embodiment as 4kb for example, referring to the database operation flowchart shown in Figure 4 The first management form of the database table header specifically includes:
[0062] 1, the storage space is 4KB, including database version number, table header storage space, total number of data tables and each data table.
[0063] 2, the table header is checked whether it is damaged by database version number unique identification code, data table total quantity threshold and table header storage space.
[0064] 3, the table header supports multiple data tables, and different data table names are named, and each data table constructs different information record table according to different calibration systems. The real-time data storage size RealLength of different calibration systems is different, and the number of real-time data storage RealRecord is different, wherein the calculation formula (1) of RealRecord is as follows:
[0065]
[0066]
[0067] The information content items of each table in the table header mainly include the table real-time data length RealLength of the calibration system, the maximum number of table real-time data storage RealRecord, and the offset tableOffset of the database mapping table in the flash space, and tableBackupOffset.
[0068] Calculation formula (2): tableOffset=0x1000+20*i (0<=i<=RealRecord)
[0069] The calculation formula (3) is obtained by calculation formula (1) (2):
[0070] tableBackupOffset = (tableOffset + 4096 - 1) & (~ (4096 - 1)) + 20 * j (0 <= j <= RealRecord, i = RealRecord).
[0071] Based on Figure 4 The second management form of the database mapping table is as follows according to the flow shown in
[0072] 1. The mapping table manages the information of each real-time data inserted by the red-black tree, which facilitates the upper layer application to add, delete, update and find the database.
[0073] 2. Each table has a main mapping table and a slave mapping table, which facilitates the automatic recovery of the data table when one of the mapping tables is damaged or abnormal during the insertion and erasure operation of the database.
[0074] 3. The single information content item in the mapping table mainly includes the real-time data insertion state, the real-time data length and the real-time data offset itemOffset in the flash space.
[0075] The calculation formula (4) is obtained by formula (1) (2) (3):
[0076] itemOffset = (tableBackupOffset + 4096 - 1) & (~ (4096 - 1)) + RealLength * k (0 <= k <= RealRecord, i = j = RealRecord).
[0077] Based on Figure 4 The third management form of the database real-time data block is as follows according to the flow shown in
[0078] 1. The real-time data adopts AES-CBC algorithm to encrypt the large data block, and the encryption and decryption operation of AES-ECB mode can be realized through the API provided by the mbed TLS library.
[0079] 2. The real-time data adopts CRC16 algorithm to check the real-time data content.
[0080] 3. The real-time data supports a maximum of 4KB, the data content is filled according to the mqtt protocol, and the data format supports json and hex formats.
[0081] 4. Real-time data is written to flash, and according to the write offset address itemOffset, each sector is written dynamically and alternately, so that each sector is evenly worn.
[0082] 5. When real-time data is about to be erased and written, a system soft restart judgment mechanism is added, and once soft restart is required, flash operation is prohibited.
[0083] Figure 4 Under the database operation process, the advantages that can be achieved include:
[0084] 1. Fast. The read and write speed is very fast, and the performance is higher than that of the traditional database storage method.
[0085] 2. Lightweight. The storage method is light and does not occupy too much space.
[0086] 3. Convenience. The storage method supports 4096 byte data storage, and storage is convenient.
[0087] 4. Reliability. It has data redundancy function, and can automatically recover when data fails.
[0088] 5. Security. It has data encryption function, which can avoid the risk of data leakage.
[0089] 6. Durability. It has wear leveling, which prolongs the life of flash.
[0090] The instrument data lightweight embedded database system of the embodiment of the application comprises a table header module, a mapping table module, a real-time data module and a data management module. The table header module is configured to store the database table header of the instrument data. The mapping table module is configured to store the database mapping table of the instrument data. The real-time data module is configured to store the database real-time data block of the instrument data. The data management module is configured to manage the database table header in a first management form, manage the database mapping table in a second management form, and manage the database real-time data block in a third management form. The instrument data lightweight embedded database system with the above architecture is light in storage method and does not occupy too much space, is convenient for application in various calibration systems, and has the function management design basis of data redundancy, security and durability while facilitating fast storage of instrument data of various calibration systems.
[0091] For the present application, the following points need to be explained:
[0092] (1) The embodiment drawings of the present application only involve the structures involved in the embodiment of the present application, and other structures can be referred to the usual design.
[0093] (2) For clarity, in the drawings used to describe the embodiments of the present application, the thickness and size of layers or structures are exaggerated. It can be understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, it can be "directly" on or under the other element, or an intervening element can also be present.
[0094] (3) The embodiments of the present application and the features in the embodiments can be combined with each other to obtain new embodiments without conflict, and the above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and the protection scope of the present application should be subject to the protection scope of the claims.
[0095] The technical features of the above embodiments can be combined arbitrarily, and for the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not conflict, they should be considered as the scope of the present disclosure.
[0096] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.
Claims
1. A lightweight embedded database system for instrument data, characterized in that, include: The header module is configured as the database header for storing instrument data; The mapping table module is configured as a database mapping table to store the instrument data; The real-time data module is configured as a database real-time data block to store the instrument data; The data management module is configured to manage the database table header in a first management mode, manage the database mapping table in a second management mode, and manage the database real-time data blocks in a third management mode. This also includes: The basic database is used to store instrument data; the header module, mapping table module, and real-time data module are sub-units of the basic database; the data management module is independent of the basic database and is configured to manage the basic database. The basic database is built on flash memory, and the flash space is planned and laid out, divided into database header, database mapping table and database real-time data block; The database header includes the database version number, header storage space, total number of data tables, and each data table; Each of the data tables includes a table name, real-time data length, number of real-time rows, and the offset of the database mapping table in the flash space. Each data table includes a table name, real-time data length, number of real-time rows, and the offset of the database mapping table in flash space; the mapping table module stores database mapping tables in the form of master mapping tables and slave mapping tables; the database mapping table includes the insertion status of the real-time data block, the insertion index number of the real-time data block, the data length of the real-time data block, and the offset of the real-time data block in flash space; the real-time data block includes the serial number of the instrument data, the time and type of the instrument data, the network signal of the instrument data, the content of the instrument data, the checksum of the instrument data, and the data reporting status of the instrument data; The database supports the creation of multiple independent data tables, each with a master-slave mapping table and real-time data blocks.
2. The lightweight embedded database system for instrument data according to claim 1, characterized in that... The database table header is stored with a first set number of bytes; the database mapping table is stored with a second set number of bytes; and the database real-time data block is stored with a third set number of bytes.
3. The lightweight embedded database system for instrument data according to claim 1, characterized in that... The database header includes the database version number, header storage space, total number of data tables, and each data table; Each of the data tables includes a table name, the real-time data length of the table, the number of real-time rows, and the offset of the database mapping table in the flash space.
4. The lightweight embedded database system for instrument data according to claim 1, characterized in that... The mapping table module stores database mapping tables in the form of master mapping tables and database mapping tables in the form of slave mapping tables; The database mapping table includes the insertion status of the real-time data block, the insertion index number of the real-time data block, the data length of the real-time data block, and the offset of the real-time data block in the flash space.
5. The lightweight embedded database system for instrument data according to claim 4, characterized in that, The real-time data block of the database includes the serial number of the instrument data, the time of the instrument data, the type of the instrument data, the network signal of the instrument data, the content of the instrument data, the check code of the instrument data, and the data reporting status of the instrument data.
6. The lightweight embedded database system for instrument data according to claim 2, characterized in that, The first set number of bytes, the second set number of bytes, and the third set number of bytes are all 4kb.
7. The lightweight embedded database system for instrument data according to claim 3, characterized in that... The process of managing the database table headers in the first management form includes: The database header is checked for corruption by using the unique identifier of the database version number, the threshold of the total number of data tables, and the storage space of the table header. Name the different data tables and construct different information record tables and real-time data lengths for each table according to the type of instrument data.
8. The lightweight embedded database system for instrument data according to claim 4, characterized in that, Managing the database mapping table in a second management manner includes: The red-black tree manages the information of each real-time data block inserted into the database, making it easier for upper-layer applications to add, delete, update, and search the database.
9. The lightweight embedded database system for instrument data according to claim 5, characterized in that, The process of managing the real-time data blocks of the database in a third management manner includes: The database uses the AES-CBC algorithm to encrypt real-time data blocks, and implements AES-ECB mode encryption and decryption operations through the API provided by the mbed TLS library; The CRC16 algorithm is used to verify the content of the real-time data blocks in the database; The data content of the real-time data block in the database is populated according to the MQTT protocol; wherein, the data content supports JSON and Hex formats.
10. The lightweight embedded database system for instrument data according to claim 5, characterized in that, The process of managing the real-time data blocks of the database in a third management manner also includes: Write the real-time data blocks of the database into the flash memory. Based on the offset of the real-time data blocks of the database in the flash memory space, write each sector in turn to ensure that the wear of each sector is even. When the real-time data block of the database is about to be erased or written, the system soft restart judgment mechanism will prohibit flash operation when a soft restart is required.
Citation Information
Patent Citations
Embedded lightweight database system
CN116303242A