An on-board embedded data reading and writing method and application

Abstract

The application relates to an on-board embedded data reading and writing method and application, which adopts a full-amount writing mode to load and save data in a flight environment of an airplane, and is divided into data file reading and data file writing. The data file reading loads data in a data file from an external storage into an internal memory at the start of a database, and does not include picture and video data; the data file writing saves data in the internal memory into the external storage at the exit of the database. The application can maximally improve database operation performance, provide efficient access and query data capacity, effectively reduce storage space, improve space utilization, and effectively avoid the problem of low efficiency of database interface access to large objects by adopting the design idea of storing large object data outside system files and managing the large object data by the database.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data storage scheme, and relates to an airborne embedded data reading and writing method and application. BACKGROUND

[0002] In recent years, the systematization, integration and complexity of the aviation equipment in China are increasing with the rapid development of aviation technology, the airborne data is dramatically expanding and tends to be numerous and complex, and the traditional file management has been unable to meet the needs of the current airborne data management. On the other hand, as a well-organized, shareable and manageable data collection, the database is involved in more and more extensive fields, and the theory and technology are quite mature. The application range of the database is also more in-depth and specific with the rapid development of data storage. The database technology is introduced into the avionics system for data management, which is to enhance the data management capability, and generally is an airborne embedded database management system.

[0003] The database technology is introduced into the airborne system to realize data management, so as to further enhance the data management capability. Limited by the database technology which is only applicable to PC, bulky and long delay, the airborne database generally adopts an embedded database to meet the requirements of the airborne environment [1]. The airborne embedded database is one of the important links to ensure flight and operation safety, is the main information source and important basis for the flight control of the modern aircraft flight management system and automatic flight control system, and is an important part of the flight data management system.

[0004] The storage management of data is the most basic function of the database management system, and is the most frequent operation with the highest reliability requirement in the database management system. Therefore, the design idea, processing strategy and implementation algorithm of the data storage in the database management system not only directly determine the performance and reliability of the storage management subsystem, but also have a direct relationship with the performance and robustness of the entire database management system. Therefore, the storage technology in the database management system is worth paying attention to.

[0005] At present, there are the following problems about the storage of the airborne data:

[0006] 1. The airborne data is generally structured data. The structured data refers to the data which strictly follows the data format and length specification, is logically expressed and implemented through a two-dimensional table structure, is also called behavior data, and has the following characteristics: the data is expressed in units of behaviors, one row of data represents the information of one entity, the attributes of each row of data are the same, the data is expressed in the form of behavior units, and the total amount of data in a single data table is not very large, and the maximum number of records is defined in advance, which is convenient for preprocessing and applies for the corresponding storage space in advance. In order to efficiently store the structured data of the airborne data, a suitable and appropriate database data storage model needs to be selected.

[0007] 2. The data type of the on-board data not only includes integer, string and floating point data, but also sometimes uses large object data such as satellite pictures. Different from the conventional integer and string data, the large object data itself occupies a large space, and it is difficult to utilize the storage space and efficiently read the data. In addition, the two forms (Char type and Varchar type) of the string in the conventional database lead to the question of whether the record is fixed-length, and how to realize the fixed-length record in the on-board embedded database may have a focus.

[0008] 3. The main operation of the on-board data is addition, deletion, modification and query, and the operation of the on-board data basically does not involve statistical analysis, which is a typical OLTP use scenario, and this puts certain requirements on the database storage model and the organization form of the data. SUMMARY

[0009] Technical problems to be solved

[0010] In order to avoid the shortcomings of the prior art, the present application provides an on-board embedded data reading and writing method and application, which solves the problem that the database technology which is only suitable for PC, large in size and long in delay cannot meet the needs of on-board system development. The present application starts from the storage scheme of the database, and designs and constructs the storage medium, the data storage model and the data file organization.

[0011] Technical scheme

[0012] An on-board embedded data reading and writing method, characterized in that: a full-amount writing mode is adopted to load and save data in the aircraft flight environment, and is divided into data file reading and data file writing.

[0013] An on-board embedded data reading and writing method, characterized in that: a full-amount writing mode is adopted to load and save data in the aircraft flight environment, and is divided into data file reading and data file writing.

[0014] Data file reading: when the database starts, the data in the data file is loaded into the memory from the external storage, and the picture and video data are not included, and the process is as follows:

[0015] Firstly, the data file is opened and the Header Section file is read, and the basic information contained in the file is the number of user tables, the number of user-defined indexes and the size of the Root Tuple in the root information;

[0016] The Header Section data is processed to extract TableNum, IndexNum, RootTupleSize and Offset information;

[0017] The range of RootTable is calculated:

[0018] (TableNum+IndexNum)*RootTupleSize};

[0019] All the Tuple in RootTable is read from the database file, and TableList and IndexList are constructed according to the Tuple;

[0020] Finally, the index structure of each table in TableList is constructed in the memory according to the index information in IndexList;

[0021] Data file writing: when the database exits, the data in the memory is saved to the external storage, and the process is as follows:

[0022] Regular Data Offset is calculated according to the information in the memory. Regular Data Offset refers to the offset of Regular Data Section in the file:

[0023] Regular Data Offset = FileHeadSize + RootOInfoSize + SchemaInfoSize

[0024] The database file is opened and the Header is written to the database file again, which contains the basic information of the database and the offset information of each data block;

[0025] TableList and IndexList in the memory are written to the end of the Header in the file.

[0026] The TablNum, IndexNum, RootTupleSize, Offset attributes include the number of tables and indexes in the database, the size of Root Tuple, and the offset of each data block.

[0027] The building of TableList and IndexList according to Tuple is that each Tuple contains the meta information of a table or index, the meta information of the index includes table name, column name, attribute, column number and object type, i.e., table or index; each Tuple is parsed into corresponding table or index information and added to TableList or IndexList; data of each table is read from Regular Data Section in the order of the tables and loaded into memory; Regular Data Section stores actual data records in the tables and the data records are read and loaded into memory in the order of the tables.

[0028] The writing of TableList and IndexList in memory to the end of Header in file is that Table information is written first and then Index information is written according to the writing order of the information of each Table.

[0029] An on-board computer for implementing the on-board embedded data reading and writing method, characterized in that it comprises a processor and a memory, the processor is used to implement the steps of the data reading and writing method when executing the computer program stored in the memory.

[0030] A readable storage medium for the on-board embedded data reading and writing method, characterized in that the readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps of the data reading and writing method.

[0031] An application of the on-board embedded data reading and writing method, characterized in that it is used for timely acquisition and transmission of data, the computer program is executed by a processor to implement the steps of the data reading and writing method.

[0032] Advantages

[0033] The on-board embedded data reading and writing method and application provided by the application adopt a full-amount writing mode to load and save data in a flight environment of an airplane, and are divided into data file reading and data file writing. The data file reading loads data in a data file from an external storage into memory at the start of a database, and does not include picture and video data. The data file writing saves data in the memory into the external storage at the exit of the database.

[0034] Advantages of the application

[0035] 1. The storage scheme designed in the application takes memory as a main storage medium, and a non-volatile device such as a disk is used as persistent storage, which can maximize the database operation performance and provide efficient access and query data capability.

[0036] 2、The storage scheme designed by the application has simple structure, stores relevant table information in the form of a system table, avoids saving logical fields in a row record as in SQLlite, can effectively reduce storage space, and improves space utilization.

[0037] 3、The storage scheme designed by the application adopts the design idea of storing large object data outside a system file and managing the large object data by a database, and can effectively avoid the problem of low efficiency of database interface access to large objects. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 a schematic diagram of a database storage file composition;

[0039] Figure 2 a schematic diagram of an organization form in a database runtime memory;

[0040] Figure 3 a schematic diagram of a data table header file;

[0041] Figure 4 a schematic diagram of a Data Tuple composition;

[0042] Figure 5 a schematic diagram of a data table in a free linked list organization form; DETAILED DESCRIPTION

[0043] The application will be further described in combination with embodiments and drawings:

[0044] The problem to be solved by the application is to alleviate the problem of space resource utilization of a database, so that the database does not consume too much space resource in storage and runtime of an embedded system, thereby affecting the normal operation of other programs.

[0045] The technical scheme adopted by the application to solve the technical problem comprises the following steps:

[0046] 1. Data type. According to the application requirement, the database supports the following three data types, that is, integer, floating point and string, which can meet the basic requirement. As for the data type of pictures and videos, the storage form is similar to that of the string data type, and details are described below.

[0047] 2. Storage file. Considering the characteristics of the on-board embedded database, the data of the flight process often needs to be checked and sorted after each flight. If the database files are scattered in various parts of the embedded system, it is often difficult to collect the database files, and the portability of the database files will be poor, so the on-board embedded database is often a single file system. Taking the embedded database SQLite as an example, it saves all objects in the database, such as tables, indexes, views, triggers, and patterns, in the same independent system file. The SQLite database file is a normal operating system file, and the byte order and binary are consistent on the operating system it supports, and all hardware architectures are consistent, which also reflects the portability of the embedded database.

[0048] Considering that the amount of on-board data is not very large, the size of the occupied space is about several hundred MB, and all data in a database instance is persistently stored in an independent physical file, which can be allowed. At the same time, the file should be self-descriptive (Self Descriptive), in addition to containing user data (User Data), it also contains the meta information (Meta Data) of the data, according to which the database file can be explained without providing additional information to analyze the database.

[0049] 3. Data file writing method. During the operation of the database, data is loaded into memory (except for Blob large objects such as pictures and videos), and operations on data are performed in memory, without immediately writing data files (except for log files for fault recovery, which are not considered here). The operation of the database data file only occurs during system startup and exit. When the system starts, initialize the database module, read the MetaData and User Data from the data file and load them into memory, then perform various operations such as adding, deleting, modifying and querying in memory, and the operation results are also saved in memory. When the system exits (or the application needs to actively initiate file writing at the right time), all data in memory is written to the data file at one time.

[0050] Considering that the overall data volume will not be too large, and the real-time requirement for the file writing scenario is not very high, the file writing scenario generally occurs before takeoff or after landing, not during flight. If data loading and saving are performed in the aircraft flight environment, it will pose a considerable challenge to safety. The full data writing algorithm is simple, easy to implement and maintain, so the full data writing (Full Data Writing) mode is adopted instead of the incremental data writing (Incremental Data Writing) mode.

[0051] 4. Storage Model. The on-board data is structured data, which has a relatively strict limit on its data format, so document storage, Key-Value storage and graph storage are not suitable for the storage of on-board data. Row storage and column storage can better support such structured data. In addition, considering that the use scenario of on-board data is often an insert, delete, modify and query operation, which does not involve statistical analysis of big data, it is a typical OLTP use scenario. Row storage model is generally used in OLTP scenario, and column storage model is generally used in OLAP scenario. Therefore, the row storage model based on traditional relational database is the best choice.

[0052] 5. Table Structure. The data file is organized in the form of a heap table rather than an index-organized table or a hash table. When the user defines a table, the user can specify a primary key (Primary Key) whose data type can be integer, floating-point and string.

[0053] The database system adds a hidden column RowId to the table defined by the user, which is used to uniquely identify a record in the table. When data is inserted into the table, the database automatically increments the current maximum RowId as the RowId of the newly inserted data. RowId can be implemented as a 4-byte unsigned integer. At the same time, in order to facilitate maintenance and management, the table records are set to fixed-length records.

[0054] RowId is monotonically increasing each time it is added. If a RowId is deleted, it can be reused. If it exceeds the range, an error will be reported.

[0055] 6. Index Storage. Considering that the amount of on-board data is not large and the cost of maintaining the index, only the definition of the index is stored in the database file, and the specific index structure is not persistently stored in the file. When the system starts, the database initializes and obtains the index information created by the user from the data file, and then loads the data from the data file. According to the user's index information, the index structure is constructed in the memory at the same time. During the database operation, the data in the memory also changes, and the index structure may also be modified. When the system exits, only the index structure in the memory is destroyed, and it does not need to be persisted to the file.

[0056] Non-persistent storage of index is considered because the main purpose of index is to speed up data retrieval. In the database, the more indexes built, the better the query efficiency, but it is necessary to build appropriate indexes according to the data characteristics and actual use environment. Moreover, it is unrealistic for users to master the theoretical basis of the database, so the index created by the user may be a redundant index, which will waste the storage space of the database, negatively affect the running performance of the database, and is not conducive to the effective improvement of the query efficiency of the database.

[0057] 7. Large object data type. The storage of such picture data is placed in the file system outside the database, the attribute of such record in the database is usually stored as a file name [3], the actual picture file is stored in a fixed system path under the unified numbering naming, often in the same system path as the database system file. In this way, the storage mode of large object attribute is similar to the string, and there is no need to pay attention to the storage space occupied by the large object data. When the large object data needs to be accessed, the file path name in the record is queried, and the object can be quickly accessed through the underlying operating system. In addition, the picture file name is stored in the form similar to the string, and the corresponding index structure can be established in the large object column (the file name string is used as the index key), which is also beneficial to the query of the picture data through the file name. The design idea has another advantage, if the pictures stored in multiple records are the same, since only the file name of the picture file is stored, the picture file names stored in the several records can be consistent, and only one picture file needs to be stored in the actual picture file rather than decoding the picture into binary and repeatedly storing it as in SQLite, which can save storage space.

[0058] The specific process is as follows:

[0059] All data in the database is stored in a separate file, referred to as a data file. The data file is a physical file composed of bytes stored on an external persistent storage device (disk or flash memory).

[0060] The data file is divided into three parts in structure: File Header Section, Catalog Section, and Regular Data. The structure is shown in the accompanying Figure 1

[0061] File Header Section: File header information area, used to store some basic information of the database as a whole.

[0062] Catalog Section: Catalog area, storing database metadata, including Root Info and Schema Info. Root Info is used to record which tables and indexes are in the database, and Schema Info is used to record the schema of each table or index, that is, the column name of all columns in the table, the column data type, and the data range of the column. If it is an index, the table where the index is located, the index type, and the column of the index are recorded. For indexes, only the definition of the index is saved in the Catalog Section, and the structure of the index is not physically stored.

[0063] ​Regular Data: Regular Data Section, used to store all the data in the database except the large data such as pictures and videos, the data in the same table is stored continuously, and the data such as pictures and videos is stored in the form of file path name only.

[0064] The file header information area occupies 64 bytes in total, and is used to store the basic information of the whole database. The components are shown in Table 1.

[0065] Table 1 File Header Logical Structure

[0066]

[0067] Because the size of the File Header Section is fixed, the starting position of the Catalog Section is fixed. In the Catalog Section, the size of the Root Info can be calculated according to the number of tables and indexes in the Header, so that the starting position of the Schema Info can be known. The size of the Schema Info and the size of the Regular Data Section are uncertain, which determine the starting positions of the two data areas, so the starting offset of the Regular Data Section is recorded in the File Header, and the starting position of the Regular Data Section is the end of the Schema Info.

[0068] When initializing and loading the File Header Section into the memory space, it is parsed into a structure body structure, and the member attributes in the structure body are the database basic information introduced in the above table.

[0069] 2. The Catalog Section records the meta information of the database. It can be divided into Root Info and Schema Info.

[0070] (1) Root Info

[0071] Which tables or indexes in the database are recorded by the Root Info. Logically, it is a table called Root Table with the following structure. The contents of the table are shown in Table 2.

[0072] Table 2 Root Table Logical Structure

[0073] Id Name MaxRec RecNum Nature ColNum MaxRowId Type 1 table1 2000 1452 1 6 1500 0 2 table2 1000 546 0 10 546 0 3 index1 1 -- 0x10001 2 -- 1 ... ... ... ... ... ... ... ... n tablen 5000 3476 0 8 3476 0

[0074] Each record in the Root Table corresponds to a Root Tuple stored in the data file. The fields of the Root Table and their meanings are shown in Table 3 below.

[0075] Table 3 Field description of Root Table

[0076]

[0077] Since the table data is stored contiguously and read only once at startup, there is no need to store the starting position (offset in the file) of each table's regular data area in the Root Table.

[0078] (2) Schema Info

[0079] The Schema Info area records the Schema information of the table or index. Logically, it is a table with the following structure, called the Schema Table. The details are shown in Table 4 below:

[0080] Table 4 Logical structure of Schema Table

[0081] Id ColNo ColName DataType Min Max Nature 1 1 ID 1 (int) -- -- 0x02 1 2 Ident 3 (varchar) 1 5 0x01 1 3 Lon 2 (real) -180 180 0x02 1 4 Lat 2 (real) -90 90 0x02 2 1 Lon -- -- -- -- 2 2 Lat -- -- -- --

[0082] The fields of the Schema Table and their meanings are shown in Table 5 below:

[0083] Table 5 Field description of Schema Table

[0084]

[0085] When reading the directory area from the disk into memory, each record in the Root info is parsed into the respective structure type Table info and Index info according to the table record or index record, and the corresponding attributes in the structure are the values of the column attributes in the Root info table related to the table or index. Similarly, the records in the schema table are parsed into the structure type Schema Info.

[0086] For the database structure loaded into memory, all Table Info is composed into a hash table with table name name as the key, and Index Info and Schema Info belonging to the same table are also composed into hash table structures with name as the key. In the Table Info, pointers to the corresponding Index Info hash table and Schema Info hash table are added, as well as a pointer to Regular Data. The key of the hash table is name, which is mainly considered for the user's usage habits, and the internal access logic uses id in Root Info as the unique identifier. The specific structure is shown in the accompanying Figure 2

[0087] 3. The Regular Data area stores data in all data tables, and the data of each table is stored continuously. All tables are stored in the order of Id in the root information table. The Regular Data area of each table is divided into two parts: Regular Data Header and Regular Data Segment.

[0088] The length of Regular Data Header is fixed at 16 bytes, and the first byte stores Table Id. The remaining 15 bytes are reserved for future expansion, and the physical structure is shown in the accompanying Figure 3

[0089] Regular Data Segment is used to store data in the table. Each record in the Table corresponds to a Data Tuple in Regular Data Segment. The physical structure of the Data Tuple is shown in the accompanying Figure 4

[0090] The Data Tuple stores the values of each field in a record in the Table in sequence. For a certain Table, the Data Tuple is fixed-length. First, there is an implicit RowId field, which is the unique identifier of the row and occupies 4 bytes. Then, the fields are stored in the order defined in the Table. If the column where the field is located is of Int type, 4 bytes are used for storage, if it is of Real type, 8 bytes are used for storage, if it is of Varchar type, the character array of the attribute is stored, and regardless of the actual length of the string, the length of the character array is the maximum length, i.e., the maximum length of the column specified by the Table when it is defined. If it is a large object data type such as picture or video, it is similar to Varchar, and the field only stores the file name of the large object file. The character array is temporarily set to 256 in length. ​​​

[0091] We add the address pointer of the first deleted record to the Regular Data Header loaded into memory after parsing and loading. This is because when initializing data from disk, there will be no blank records (blank records are not written to disk during persistence, and this address is not persisted with the Regular Data Header), so the address of this record is empty after the initial loading and parsing of the Regular Data Header. Similarly, the address of the second deleted record is stored in the first deleted record. This clearly and intuitively treats the addresses storing a record as pointers. By continuing this process, a free linked list is formed. Figure 5 The given data table is a free linked list formed after deleting a few records.

[0092] With the structure described above, when inserting a record, we first check the address of the deleted record in the Regular Data Header. If the address exists, we use that address to store the newly inserted record and simultaneously update the address of the deleted record in the Regular Data Header, recording the address of the next deleted record. If no space is available, the new record is appended directly to the end of the data table.

[0093] 4. Load the database file

[0094] When the database starts, it needs to load all data except for data types such as images and videos from the data files into memory. Subsequent data operations are performed in memory. Data reading follows the order in the file: Header, Catalog, and Regular Data, and is scanned, read, and parsed accordingly. The process is as follows:

[0095] (1) Open the data file and read the Header Section;

[0096] (2) Parse the Header to obtain the TablNum, IndexNum, RootTupleSize, and Offset attributes;

[0097] (3) Calculate the range of RootTable based on TableNum, IndexNum, and RootTupleSize. Then, read the entire contents of RootTable from the file and construct TableList and IndexList in memory.

[0098] (4) Read the data from each table in the Regular Data Section sequentially and parse and load it into memory;

[0099] (5) According to the index information in IndexList, build the index structure of each table in TableList in memory.

[0100] 5. Save the database file

[0101] When exiting, the database needs to save the data in memory to a persistent device such as a disk. Saving will rebuild a file and write all the Header, Catalog and Regular Data in memory to the newly built file. The process of saving data to a file is as follows:

[0102] (1) Calculate the Regular Data Offset according to the information in memory;

[0103] (2) Open the file and first write the Header;

[0104] (3) According to the TableList and IndexList in memory, write the RootTable in sequence (first write TableList, then write IndexList) immediately after the end of the Header in the file;

[0105] (4) Write the Regular Data Section of each Table in sequence (according to the writing order of the Table information in TableList).

Claims

1. An on-board embedded data reading and writing method, characterized by: The data is loaded and saved in the airplane flight environment in the full amount writing mode, and is divided into data file reading and data file writing. The data file reading: when the database is started, the data in the data file is loaded into the memory from the external storage, and the picture and video data are not included, and the process is as follows: First, open the data file and read the Header Section file, and the basic information contained in the file is the number of user tables, the number of user-defined indexes, and the size of the Root Tuple in the root information. The Header Section data is processed, and the TableNum, IndexNum, RootTupleSize and Offset information are extracted. The range of the RootTable is calculated: (TableNum+IndexNum)*RootTupleSize}; All the Tuple in the RootTable is read from the database file, and the TableList and the IndexList are constructed according to the Tuple. Finally, according to the index information in the IndexList, the corresponding index structure of each table in the TableList is constructed in the memory. The data file writing: when the database is exited, the data in the memory is saved to the external storage, and the process is as follows: The Regular Data Offset is calculated according to the information in the memory, and the Regular Data Offset is the offset of the RegularData Section in the file: Regular Data Offset=FileHeadSize+RootInfoSize+SchemaInfoSize The database file is opened and the Header is written into the database file again, and the Header contains the basic information of the database and the offset information of each data block. The TableList and the IndexList in the memory are written into the end of the Header in the file.

2. The method of claim 1, wherein: The TablNum, IndexNum, RootTupleSize, Offset attributes include the number of tables and indexes in the database, the size of the Root Tuple, and the offset of each data block.

3. The method of claim 1, wherein: The TableList and the IndexList are constructed according to the Tuple: each Tuple contains table or index meta information, and the index meta information includes table name, column name, attribute, column number and object type, i.e. table or index; each Tuple is parsed into corresponding table or index information, and is added to the TableList or the IndexList; the data of each table is read from the Regular DataSection in the order of the table and is loaded into the memory; the Regular DataSection stores the actual data records in the table, and the data is read and loaded into the memory in the order of the table.

4. The method of claim 1, wherein: The writing of the TableList and the IndexList in the memory to the end of the Header in the file is: writing the Table information first, and then writing the Index information according to the writing order of the information of each Table.

5. An on-board computer implementing the on-board embedded data reading and writing method according to any one of claims 1 to 4, characterized in that: The data reading and writing method comprises the steps of: reading data from a memory; and writing the data into a file.

6. A readable storage medium for the on-board embedded data reading and writing method according to any one of claims 1 to 4, characterized in that: The computer program is stored on the readable storage medium and is executed by the processor to realize the steps of the data reading and writing method.

7. Use of the method for reading and writing data in- flight according to any one of claims 1 to 4, characterized in that: The computer program is stored on the readable storage medium and is executed by the processor to realize the steps of the data reading and writing method.

Images