A write method and a read method of a data storage file structure

Abstract

The application discloses a data storage file structure writing method and reading method, and the data storage file structure comprises a file header, a table area, a column area and a data packet area; wherein the file header comprises information such as the number of table structures in the table area, the maximum column ID in the column area, the first column pointer and the column structure length; the table structure comprises information such as the number of table structure rows, the number of associated column structures, the first associated column ID and the first associated column pointer; the column structure comprises information such as the first associated data packet pointer and the data type; and the data packet comprises information such as the logical pointers of the front and rear data packets in the same column structure, the compressed data and the data length before compression. The application realizes the writing and reading of the stored sampling data through the storage structure, achieves the effect of high-performance reading and writing of multi-channel sampling data, and supports the column compression of the content, so that the capacity occupation problem of file storage and migration can be effectively solved, and the hardware cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a data storage file structure writing method and reading method. BACKGROUND

[0002] In the prior art, in the current lithium battery formation and capacity production process, the technical personnel in the field need to sample the key production data such as current, voltage, temperature and capacity of each channel at a millisecond or second interval (most use 1 second), and persistently save the sampling data on the host computer. In order to ensure the smooth progress of the production process, the data needs to be viewed during and after the production process, and the host computer can also view the data after the data is migrated to other computers away from the production environment.

[0003] However, with the rapid development of modern industrial technology and the continuous increase of data volume, the conventional data storage method, such as sql database or excel document, can only normally process data within 200 channels if the data is stored on a mechanical hard disk at a sampling interval of 1 second. If the hardware cost has no advantage on a conventional 5000-channel production line, the existing sql database or Excel table storage mode has low concurrent read-write performance, can drive a small number of production channels, and occupies a large space for storing data, which seriously affects the hardware cost. SUMMARY

[0004] The embodiments of the present application provide a data storage file structure writing method and reading method, which can at least ensure that the present application scheme generates a data storage file structure including a file header, a table area, a column area and a data packet area, writes and reads through the data storage file structure, realizes the writing and reading of the stored sampling data, and further realizes the high-performance reading and writing of multi-channel sampling data. Through the design of the file structure, the content is compressed by column, the capacity occupation problem of file storage and migration is solved, and the hardware cost is reduced.

[0005] In a first aspect, the embodiments of the present application provide a data storage file structure writing method, which comprises:

[0006] generating a data storage file structure, the data storage file structure comprising a file header, a table area, a column area and a data packet area; wherein the file header comprises the following information: the number of table structures in the table area, the maximum column ID in the column area, the first column pointer and the column structure length; the table structure comprises the following information: the number of table structure rows, the number of associated column structures, the first associated column ID and the first associated column pointer; the column structure comprises the following information: the first associated data packet pointer and the data type;

[0007] fixing the occupied length of each of the file header, the table region and the column region;

[0008] compressing the to-be-stored sampling data according to a preset compression method to obtain compressed data, and writing the compressed data into the data packet region to obtain a plurality of data packets, the data packet including the following information: logical pointers of data packets before and after each other in the same column structure, the compressed data and the length of data before compression.

[0009] In some embodiments, the table region is arranged after the file header and adjacent to the file header; the column region is arranged after the table region and adjacent to the table region; and the data packet region is arranged after the column region and adjacent to the column region.

[0010] In some embodiments, the plurality of table structures in the table region are sorted according to respective table IDs, and each table structure has a same length; and the plurality of column structures in the column region are sorted according to respective column IDs, and each column structure has a same length.

[0011] In some embodiments, the compressing the to-be-stored sampling data according to a preset compression method to obtain compressed data, and writing the compressed data into the data packet region includes:

[0012] compressing the data packets in the data packet region according to a preset compression method to obtain compressed data packets;

[0013] writing the compressed data packets into the data packet region, and adjusting the following information corresponding to the compressed data packets: the number of table structure rows, the number of column data and the logical pointers of data packets before and after each other in the same column structure.

[0014] In some embodiments, the preset compression method includes compressing repeated data by column and compressing repeated data by column.

[0015] In a second aspect, the embodiments of the present application provide a reading method of a data storage file structure, the data storage file structure including a file header, a table region, a column region and a data packet region; wherein the file header includes the following information: the number of table structures in the table region, the maximum column ID in the column region, a first column pointer and a column structure length; the table structure includes the following information: the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer; and the column structure includes the following information: a first associated data packet pointer and a data type, and the reading method includes:

[0016] reading whole-column data from the data storage file structure according to a column ID;

[0017] Alternatively, the whole table data is read from the data storage file structure according to the table ID.

[0018] In some embodiments, the reading of the whole column data from the data storage file structure according to the column ID comprises:

[0019] obtaining a first column pointer and a column structure length in the file header;

[0020] obtaining a column object offset address according to the column ID, the first column pointer and the column structure length;

[0021] performing a data packet reading action according to the column object offset address to obtain the whole column data.

[0022] In some embodiments, the reading of the whole table data from the data storage file structure according to the table ID comprises:

[0023] obtaining a table region address according to the file header, and obtaining a table structure length in the file header;

[0024] obtaining a table object offset address according to the table ID, the table region address and the column structure length;

[0025] obtaining a plurality of column objects according to the table object offset address and the first associated column ID;

[0026] generating a corresponding two-dimensional array according to the plurality of column objects and the table structure row number, and performing a data packet reading action to obtain the whole table data.

[0027] In a third aspect, an embodiment of the present application provides a controller, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the writing method of the data storage file structure according to any one of the embodiments of the first aspect, or the reading method of the data storage file structure according to any one of the embodiments of the second aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute the writing method of the data storage file structure according to any one of the embodiments of the first aspect, or the reading method of the data storage file structure according to any one of the embodiments of the second aspect.

[0029] The application has at least the following beneficial effects: the application generates a data storage file structure, which includes a file header, a table area, a column area and a data packet area; the file header includes the following information: the number of table structures in the table area, the maximum column ID in the column area, a first column pointer and a column structure length; the table structure includes the following information: the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer; the column structure includes the following information: a first associated data packet pointer and a data type; and the respective occupied lengths of the file header, the table area and the column area are fixed; the sampling data to be stored is compressed according to a preset compression method to obtain compressed data, and the compressed data is written into the data packet area to obtain a plurality of data packets, which include the following information: the logical pointers of the front and rear data packets in the same column structure, the compressed data and the data length before compression, and the whole column data is read from the data storage file structure according to the column ID, or the whole table data is read from the data storage file structure according to the table ID; the application realizes the writing and reading of the sampling data through the writing and reading of the data storage file structure, and realizes the high-performance reading and writing of the multi-channel sampling data, and the data storage file structure of the application supports column-by-column compression of the content, which can effectively solve the capacity occupation problem of file storage and migration, thereby reducing the hardware cost. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 Flow chart of the writing method of the data storage file structure according to an embodiment of the application;

[0031] Figure 2 Flow chart of the compression data in the writing method of the data storage file structure according to another embodiment of the application;

[0032] Figure 3 Flow chart of the reading method of the data storage file structure according to another embodiment of the application;

[0033] Figure 4 Flow chart of the reading of the whole column data in the reading method of the data storage file structure according to another embodiment of the application;

[0034] Figure 5 Flow chart of the reading of the whole table data in the reading method of the data storage file structure according to another embodiment of the application;

[0035] Figure 6 Schematic diagram of the data storage file structure according to an embodiment of the application;

[0036] Figure 7 Schematic diagram of the controller according to another embodiment of the application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0038] In some embodiments, although the functional modules are divided in the system schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the sequence in the flowchart. The terms first, second, etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0039] In the prior art, in the formation and capacity production process of lithium batteries, the key production data (such as current, voltage, temperature, capacity, etc.) of each channel need to be sampled at millisecond or second intervals (most use 1 second), and saved persistently in the host computer. The data can be viewed during and after production; and the host computer can also view the data after the data is migrated to other computers. However, using conventional data storage methods such as sql database or excel document, at a sampling interval of 1 second, the system can only normally process data within 200 channels if the data is stored on a mechanical hard disk. If the production line has 5000 channels, the hardware cost will have no advantage.

[0040] That is, the main existing production mode in the prior art is to store data through sql database or excel table, so there are the following problems in the prior art:

[0041] 1) Low concurrent read-write performance, and few production channels can be driven;

[0042] 2) Data migration in sql database needs to be exported to common formats such as excel and csv, and the export process consumes system performance;

[0043] 3) Sql database and excel format generally do not have compression storage, and occupy a huge space.

[0044] To at least solve the above problems, the application discloses a data storage file structure writing method and a reading method. The data storage file structure comprises a file header, a table area, a column area and a data packet area. The file header comprises the number of table structures in the table area, the maximum column ID in the column area, a first column pointer and column structure length, etc. The table structure comprises the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer, etc. The column structure comprises a first associated data packet pointer and a data type, etc. The data packet comprises the logical pointers of the front and rear data packets in the same column structure, compressed data and the length of data before compression, etc. The application realizes the writing and reading of sampling data through the data storage file structure, so as to realize the high-performance reading and writing of multi-channel sampling data. The data storage file structure of the application supports column compression, effectively solves the capacity occupation problem of file storage and migration, and further reduces the hardware cost.

[0045] The embodiments of the application are further described below with reference to the drawings.

[0046] Reference Figure 1 , Figure 1 The flowchart of the data storage file structure writing method according to an embodiment of the application comprises the following steps S110, S120 and S130, but is not limited to the following steps.

[0047] In step S110, a data storage file structure is generated. The data storage file structure comprises a file header, a table area, a column area and a data packet area. The file header comprises the following information: the number of table structures in the table area, the maximum column ID in the column area, a first column pointer and column structure length. The table structure comprises the following information: the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer. The column structure comprises the following information: a first associated data packet pointer and a data type.

[0048] In step S120, the occupied lengths of the file header, the table area and the column area are fixed.

[0049] In step S130, the preset compression method is used to compress the to-be-stored sampling data, to obtain compressed data. The compressed data is written into the data packet area, to obtain a plurality of data packets. The data packet comprises the following information: the logical pointers of the front and rear data packets in the same column structure, compressed data and the length of data before compression.

[0050] In some embodiments, the data storage file structure in the application is a binary file structure, which can store 200 channels of data within 30 milliseconds, uses multiple files, and normally drives 5000 channels of data reading and writing on a common office computer; and the file can be copied to other computers for viewing by the upper computer software, effectively solving the hardware cost problem caused by the sql database or excel document technology in the prior art, which can normally process only 200 channels of data within 1 second sampling interval, and effectively realize the high-performance reading and writing function of multi-channel sampling data.

[0051] In some embodiments, the sampling data to be stored is compressed according to a preset compression method to obtain compressed data, and the compressed data is written into the data packet area, including the file header, the table area, and the column area. After the fixed length of the column data is accumulated in the memory or temporary file, the compression content is compressed according to the compression design, and a data packet is generated and written. After writing, the corresponding table row number, the corresponding column data number, and the previous data packet pointer are adjusted, so that the column compression data function is realized while the file is written, and the file space occupation is reduced.

[0052] In some embodiments, the table area is arranged after the file header and adjacent to the file header; the column area is arranged after the table area and adjacent to the table area; and the data packet area is arranged after the column area and adjacent to the column area. Therefore, when reading the data storage file structure in the subsequent step, the table area address can be obtained according to the fixed length of the file header, and the column area address can be obtained according to the first column pointer in the file header, so as to know the column ID, read the entire column data, or know the table ID, read the entire table data, realize the writing data reading, so that the application can realize the data structure of multiple tables and multiple columns through a small number of contents, and only create once. The data amount is reduced after compression, and the compressed data is only appended to the end of the file (data packet area), without deleting the file content (file header, table area and column area), which can effectively improve the file writing performance.

[0053] In some embodiments, the plurality of table structures in the table area are sorted according to the respective table IDs, and the lengths of the table structures are consistent; and the plurality of column areas in the column area are sorted according to the respective column IDs, and the lengths of the column structures are consistent. Specifically, the table ID and the column ID in the table area and the column area are sequentially increased from 1, so that the offset address can be obtained according to the known table ID or column ID in the subsequent step, and the corresponding data reading is performed.

[0054] Reference Figure 2 , Figure 2The flowchart of the method for writing compressed data in the data storage file structure of another embodiment of the present application includes the following steps S210 and S220 in some embodiments: the sampling data to be stored is compressed according to a preset compression method to obtain compressed data, and the compressed data is written into the data packet area.

[0055] In step S210, the data packets in the data packet area are compressed according to a preset compression method to obtain compressed data packets.

[0056] In step S220, the compressed data packets are written into the data packet area, and the following information corresponding to the compressed data packets is adjusted: the number of table structure rows, the number of column data, and the logical pointers of the front and rear data packets in the same column structure.

[0057] In some embodiments, the preset compression method in the present application includes compressing repeated data by column and compressing incremental data by column, and the specific process is as follows: compressing repeated data by column: data value: repetition number; if it is uint32 type, N values can be represented by only 8 bytes; compressing incremental data by column: fixed value: repetition number, incremental value 1, incremental value 2, incremental value 3…; if the incremental voltage is represented by uint32, such as 3.241, 3.251, 3.266, 3.301, 3.308, 3.316, and 3.0, part of which is fixed, and the incremental part is 241, 251, 266, 301, 308, and 316, which can be represented by two bytes, so the total length after compression is less than half of the original data size; wherein, the present application designs a file structure to support column compression of stored data, thereby reducing the capacity occupied by file storage and migration.

[0058] In some embodiments, the present application supports other existing compression algorithms, and the contents of the entire data packet can also be compressed by a general compression algorithm before being written.

[0059] Reference Figure 3 , Figure 3 The flowchart of the reading method of the data storage file structure of another embodiment of the present application includes the following steps S310 and S320 in some embodiments: the present application proposes a reading method of a data storage file structure, which includes a file header, a table area, a column area, and a data packet area; wherein, the file header includes the following information: the number of table structures in the table area, the maximum column ID in the column area, the first column pointer, and the column structure length; the table structure includes the following information: the number of table structure rows, the number of associated column structures, the first associated column ID, and the first associated column pointer; the column structure includes the following information: the first associated data packet pointer and the data type; and the reading method includes the following steps S310 and S320.

[0060] Step S310, reading the whole column data from the data storage file structure according to the column ID;

[0061] Step S320, or, reading the whole table data from the data storage file structure according to the table ID.

[0062] Wherein, the column ID and the table ID are known column ID and known table ID.

[0063] Reference Figure 4 , Figure 4 The flow chart of reading the whole column data in the reading method of the data storage file structure proposed in another embodiment of the present application, in some embodiments, reading the whole column data from the data storage file structure according to the column ID, including but not limited to the following steps S410, S420 and S430;

[0064] Step S410, obtaining the first column pointer and column structure length in the file header;

[0065] Step S420, obtaining the column object offset address according to the column ID, the first column pointer and the column structure length;

[0066] Step S430, performing data packet reading action according to the column object offset address to obtain the whole column data.

[0067] Specifically, steps S410 to S430 correspond to the known column ID, reading the whole column, obtaining the column region address from the file header (according to the first column pointer in the file header), the column structure length, and the information of the column object according to the offset address (column region address + column structure length * (column ID-1)); from the first data packet pointed by the column object, reading and continuing to read the next packet until the last packet; finally, decompressing to obtain the whole column data, realizing example data reading, and after obtaining the column structure object, the whole file offset reading corresponding data without other complex structure processing, which can effectively improve the file reading performance.

[0068] Reference Figure 5 , Figure 5 The flow chart of reading the whole table data in the reading method of the data storage file structure proposed in another embodiment of the present application, in some embodiments, reading the whole table data from the data storage file structure according to the table ID, including but not limited to the following steps S510, S520, S530 and S540;

[0069] Step S510, obtaining the table region address according to the file header, and obtaining the table structure length in the file header;

[0070] Step S520, obtaining the table object offset address according to the table ID, the table region address and the column structure length;

[0071] Step S530, obtaining a plurality of column objects according to the table object offset address and the first associated column ID;

[0072] Step S540, generating a corresponding two-dimensional array according to the plurality of column objects and the table structure row number, and performing a data packet reading action to obtain the entire table data.

[0073] Specifically, steps S510 to S540 correspond to a known table ID, and the entire table data is read: the table region address is obtained from the file header (since the file header and the table region are adjacent, the table region address can be obtained according to the fixed length of the file header), the table structure length, the table object is obtained according to the offset address (table region address + table structure length * (table ID-1)); the ID of the first column is obtained from the table object, and then all column objects are obtained; a corresponding two-dimensional array is generated according to the number of records, and each value is written in each row as an array subscript in the order of the columns, and finally the data of a table is obtained, realizing the reading of table data.

[0074] Reference Figure 6 , Figure 6 The schematic diagram of the data storage file structure of an embodiment of the present application, in some embodiments, as shown in Figure 6 the data storage file structure in the present application includes a file header, a table region, a column region and a data packet region, and each part is specifically designed as follows:

[0075] 1) File header: The file header has a fixed length, defines the related information of the file, defines the number of tables, the length of the table structure; records the maximum column ID, the pointer of the first column, and the column structure length.

[0076] 2) Table region: The entire table region has a fixed length and is located behind the file header position; from the first table to the last table, they are sorted by ID; each table structure has the same length, each table records the number of associated columns, the ID and pointer of the first column; and the data row number of the entire table.

[0077] 3) Column region: The entire column region has a fixed length and is located behind the table region position; from the first column to the last column, they are sorted by ID; each column structure has the same length, each column records the pointer of the first data packet of the column and the data type of the column.

[0078] 4) Data packet region: The entire data packet is located behind the column region, each packet needs to record the logical pointer association of the previous and next packets in the same column; each packet needs to record the length before data compression and store the compressed content.

[0079] Based on the above data storage file structure, the writing method and the reading method of the data storage file structure disclosed in the present application have the following beneficial effects:

[0080] 1) The file structure is concise, using very little content to represent the data structure of multiple tables and columns, and it is created only once;

[0081] 2) High file writing performance, small data size after compression, only appended to the end of the file, without deleting file content;

[0082] 3) High file reading performance: After obtaining the column structure object, the corresponding data is read from the entire file offset without any other complex structure processing;

[0083] 4) Compress data by column, resulting in smaller file content, suitable for file download or long-term storage migration.

[0084] refer to Figure 7 , Figure 7 This is a schematic diagram of the controller provided in the embodiments of this application.

[0085] Some embodiments of this application provide a controller, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for writing or reading a data storage file structure as described in any of the above embodiments. For example, it executes the method described above. Figure 1 Method steps S110 to S130, Figure 2 Method steps S210 to S220, Figure 3 Method steps S310 to S320, Figure 4 Method steps S410 to S430, Figure 5 Method steps S510 to S540.

[0086] The controller 700 in this embodiment includes one or more processors 710 and a memory 720. Figure 7 The example uses a processor 710 and a memory 720.

[0087] The processor 710 and memory 720 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0088] The memory 720, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the memory 720 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device.

[0089] In some embodiments, the memory 720 can optionally include a memory 720 that is remotely located with respect to the processor 710, which can be connected to the controller 700 through a network, examples of which include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0090] In some embodiments, the processor executes the computer program according to a preset interval of time to perform the writing method or reading method of the data storage file structure of any one of the above embodiments.

[0091] Those skilled in the art can understand that, Figure 7 The system structure shown in the above figure does not constitute a limitation on the controller 700, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0092] In Figure 7 In the controller 700 shown, the processor 710 can be used to call the writing method or reading method of the data storage file structure stored in the memory 720, thereby implementing the writing method or reading method of the data storage file structure.

[0093] Based on the hardware structure of the above controller 700, various embodiments of the writing method or reading system of the data storage file structure of the present application are proposed, and the non-transitory software programs and instructions required to implement the writing method or reading method of the data storage file structure of the above embodiments are stored in the memory, and when executed by the processor, the writing method or reading method of the data storage file structure of the above embodiments is executed.

[0094] In addition, the writing method or reading system of the data storage file structure of the embodiments of the present application also provides a writing method or reading system of a data storage file structure, which includes the above-mentioned controller.

[0095] In some embodiments, since the writing method or reading system of the data storage file structure of the embodiments of the present application has the controller of the above embodiments, and the controller of the above embodiments can execute the writing method or reading method of the data storage file structure of the above embodiments, the specific embodiments and technical effects of the writing method or reading system of the data storage file structure of the embodiments of the present application can refer to the specific embodiments and technical effects of the writing method or reading method of the data storage file structure of any one of the above embodiments.

[0096] This application also provides a computer-readable storage medium storing computer-executable instructions. These instructions are used to execute the aforementioned method for writing or reading the data storage file structure. For example, they can cause one or more processors to execute the method for writing or reading the data storage file structure described above, such as executing the above-described... Figure 1 Method steps S110 to S130, Figure 2 Method steps S210 to S220, Figure 3 Method steps S310 to S320, Figure 4 Method steps S410 to S430 Figure 5 Method steps S510 to S540.

[0097] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network nodes. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage systems, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0099] The above describes the preferred embodiments of the present application, but the present application is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method of writing a data storage file structure, characterized by, The writing method comprises: generating a data storage file structure, the data storage file structure comprising a file header, a table region, a column region and a data packet region; wherein the file header comprises the following information: the number of table structures in the table region, the maximum column ID in the column region, a first column pointer and a column structure length; the table structure comprises the following information: the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer; the column structure comprises the following information: a first associated data packet pointer and a data type; fixing the occupied length of each of the file header, the table region and the column region; compressing the to-be-stored sampling data according to a preset compression method to obtain compressed data, and writing the compressed data into the data packet region to obtain a plurality of data packets, the data packet comprising the following information: the logical pointers of the front and rear data packets in the same column structure, the compressed data and the data length before compression.

2. The method of claim 1, wherein, The table region is arranged after the file header and adjacent to the file header; the column region is arranged after the table region and adjacent to the table region; and the data packet region is arranged after the column region and adjacent to the column region.

3. The method of claim 1, wherein the data storage file structure is written in a manner such that, The plurality of table structures in the table region are sorted according to the respective table IDs, and the lengths of the respective table structures are consistent; and the plurality of column structures in the column region are sorted according to the respective column IDs, and the lengths of the respective column structures are consistent.

4. The method of claim 1, wherein the data storage file structure is written in a manner such that, The compression processing of the to-be-stored sampling data according to the preset compression method to obtain compressed data, and the writing of the compressed data into the data packet region, comprises: compressing the data packets in the data packet region according to the preset compression method to obtain compressed data packets; writing the compressed data packets into the data packet region, and adjusting the following information corresponding to the compressed data packets: the number of table structure rows, the number of column data and the logical pointers of the front and rear data packets in the same column structure.

5. The method of claim 1, wherein the data storage file structure is written in a manner such that, The preset compression method comprises compressing repeated data by column and compressing repeated data by column.

6. A method of reading a data storage file structure, characterized by, The data storage file structure comprises a file header, a table region, a column region and a data packet region; wherein the file header comprises the following information: the number of table structures in the table region, the maximum column ID in the column region, a first column pointer and a column structure length; the table structure comprises the following information: the number of table structure rows, the number of associated column structures, a first associated column ID and a first associated column pointer; the column structure comprises the following information: a first associated data packet pointer and a data type, and the reading method comprises: reading the whole column data from the data storage file structure according to the column ID; or, reading the whole table data from the data storage file structure according to the table ID.

7. The method of reading a data storage file structure of claim 6, wherein, The reading of the whole column data from the data storage file structure according to the column ID comprises: obtaining the first column pointer and the column structure length in the file header; obtaining the column object offset address according to the column ID, the first column pointer and the column structure length; performing a data packet reading action according to the column object offset address to obtain the whole column data.

8. The method of reading a data storage file structure of claim 6, wherein, The reading of the whole table data from the data storage file structure according to the table ID comprises: obtaining a table region address according to the file header, and obtaining a table structure length in the file header; obtaining a table object offset address according to the table ID, the table region address and the column structure length; obtaining a plurality of column objects according to the table object offset address and the first associated column ID; generating a corresponding two-dimensional array according to the plurality of column objects and the table structure row number, and performing a data packet reading action to obtain whole table data.

9. A controller characterized by comprising: The processor executes the computer program to implement the data storage file structure writing method in any one of claims 1 to 5, and the data storage file structure reading method in any one of claims 6 to 8. 10.A computer readable storage medium storing computer executable instructions for executing the data storage file structure writing method in any one of claims 1 to 5, and the data storage file structure reading method in any one of claims 6 to 8.

Images