A pan-matrix data file structure and its indexing method

Abstract

The present invention discloses a pan-matrix data file structure, which consists of 4 main parts: metadata segment, configuration data segment, input data segment and result data segment. The metadata segment stores the metadata of the file, including the indexes of the other three data segments; the configuration data segment includes the configuration segment index and configuration data, and the configuration segment index is used to store the index information of each configuration data; the input data segment includes the input segment index and input data, and the input segment index is used to store the index information of each input data; the result data segment includes the result segment index and result data, and the result segment index is used to store the index information of each result data. The present invention also discloses its indexing method, including the steps of reading and writing SDK to identify the metadata segment, read the configuration data, read the input data, and write the result data. The present invention has the advantages of unified data format, good data storage diversity, random data reading, high reading efficiency, dynamic storage, etc.

Description

Technical Field

[0002] The present invention relates to a data file structure and an indexing method, and in particular to a pan-matrix data file structure and an indexing method thereof. Background Art

[0003] In large-scale parallel computing scenarios, traditional data files are usually in the following formats: CSV or Excel, Feather, Parquet, and database storage.

[0004] CSV or Excel format: CSV is a simple, universal text format that is easy for humans to read and write, and is widely supported and compatible. The storage and reading speed of both is relatively slow, especially when processing large-scale data. It does not support complex query and analysis operations, and has no built-in data integrity check mechanism. Excel is similar.

[0005] Feather format: Feather is an efficient binary column storage format used for storing and exchanging data frames. It supports fast reading and writing and improves the reading speed of data sets, but lacks support for multiple data segments and complex data structures.

[0006] Parquet format: Parquet is an efficient columnar storage format widely used in the Hadoop ecosystem that supports compression and encoding schemes to improve storage efficiency, but lacks support for storing diverse data in a single data file.

[0007] Database storage: Database storage can conveniently support the storage and reading of various types of data, but on the one hand, it is not as efficient as column storage in terms of matrix data storage and reading, and it does not have the advantages of data files in terms of data exchange and distribution.

[0008] The engine health status monitoring and analysis scenario is a typical large-scale parallel computing scenario, and its sensor sampling data is mainly CSV files or Excel files. The analysis engine faces great challenges when processing traditional data files such as the above CSV files or Excel files. Taking a certain model as an example, an engine has more than 300 channels of concurrent data, that is, more than 300 parallel data. The sampling frequency is 40 Hz per second on average, which means that 40 data with 300 columns per second are generated. The entire test cycle is about two to four hours, and the generated data files range from 2-3G to more than 10G, and they are complete CSV files. In the processing of the analysis engine, the problem of insufficient computer memory is often faced. Because data processing is processed channel by channel, that is, column by column. However, CSV itself is a table type of data. To process this kind of data, all the data must be read into the computer's memory first, and then the columns to be processed must be found, the data must be read out, and then the algorithm must be executed. In this case, when the analysis engine processes engine data, the computer configuration requirements are very high, and the processing process is very unstable. Other software occupies memory, and it is often easy to have insufficient memory and cause application crashes.

[0009] In summary, the traditional data file storage method has the following problems when storing data: 1. Inconsistent data formats: Traditional data storage files (such as Excel, CSV, etc.) mostly store data in an unstructured manner, and both writing and reading require encoding and decoding and data format conversion. This patent provides data storage standards for different types of data through standard structured storage specifications to meet the unified exchange of data between various systems; 2. Data storage diversity: Traditional data storage files can generally only store single-structure data. For data of multiple data types, multiple files with different storage formats are often required, which brings complexity to file management. This patent supports storing multiple data formats such as arrays, strings, JSON, XML, matrices, etc. in one file at the same time, so that all data required for the entire business can be managed in one data file; 3. Unable to read data randomly: Traditional data storage files need to be read into computer memory once before the specified data can be further read. It is impossible to directly extract data fragments from the specified location of the file. This patent introduces a multi-level index structure to provide data files with the ability to search layer by layer, which can accurately locate the location of the data to be read, thereby achieving the purpose of random reading of stored data. 4. Low reading efficiency: For a two-dimensional matrix data, traditional data storage files are mostly stored sequentially in rows. When reading by column or by row and column at the same time, it is necessary to traverse the data file, which leads to low reading efficiency. This patent optimizes the multi-level indexing. 5. Non-dynamic storage: Traditional data storage files are generally static when storing data. It is impossible to modify only part of the data content without changing the rest of the file. The data must be read into the memory, modified, and then written back to the file at one time, resulting in low efficiency. Summary of the invention

[0010] The present invention aims to provide a pan-matrix data file structure and an indexing method thereof with unified data format, good data storage diversity, random data reading, high reading efficiency, dynamic storage, and better security and privacy.

[0011] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a pan-matrix data file structure, consisting of 4 main parts: a metadata segment, a configuration data segment, an input data segment and a result data segment; the metadata segment stores the metadata of the file, including the indexes of the other three data segments; the configuration data segment includes a configuration segment index and configuration data, and the configuration segment index is used to store the index information of each configuration data in the configuration data information; the input data segment includes an input segment index and input data, and the input segment index is used to store the index information of each input data in the input data information; the result data segment includes a result segment index and result data, and the result segment index is used to store the index information of each result data in the result data information.

[0012] Furthermore, the data information of the three data segments are all stored based on the column storage method, and data of different granularities such as one item, one column, one row, multiple columns, and multiple rows can be accessed through indexes.

[0013] Furthermore, the metadata segment also includes a version number, a creation time, and an update time.

[0014] Furthermore, the index information of the three data segments includes index type, data name, data type, starting position and data length.

[0015] Furthermore, the result data segment has dynamic characteristics.

[0016] Furthermore, the input data and the result data both support multiple data formats, including string, JSON, XML, array and matrix.

[0017] Furthermore, the string storage format is an offset index plus data content, which supports the storage of long and short strings.

[0018] Furthermore, the array storage structure supports single-type data storage of string or numeric type, wherein the string array is stored separately for index and content.

[0019] Furthermore, the matrix storage structure includes an index and a number of data columns, wherein the index item includes the column name, column type and location information of the data column.

[0020] The above-mentioned indexing method of the pan-matrix data file structure comprises the following steps: S1. The software system issues an index instruction to the dedicated read-write SDK supporting the pan-matrix data file structure; the dedicated read-write SDK identifies whether the version number in the metadata segment is the target version. If the version number in the data segment is higher than the target version (the version that the SDK can process), an error is reported and processing is stopped; the SDK can obtain the creation time and last modification time of the file from the data segment, which is used by the software system to confirm whether the file content is within the validity period; S2. The software system sends a command to read configuration data to the SDK. The SDK finds the configuration data segment according to the index of the metadata segment, finds the corresponding configuration item index according to the configuration name, reads the specified configuration data information according to the index information, and returns it to the software system. The software finds the data block name corresponding to the input data and the required data column name (if necessary) according to the configuration information. S3. The software system sends a command to read input data to the SDK. The SDK finds the input data segment according to the index of the metadata segment, finds the corresponding data block according to the data block name, and performs the following operations according to the data block type: S3-1. If the data block is a string or an array, the SDK returns the entire string or array content to the software; for strings, they can be further converted into XML, JSON and other structured objects according to the instruction parameter requirements of the software system; S3-2. When the SDK finds that the stored string is in JSON or XML format, it will decode the string and directly return the structured data object it represents for the caller to use; S3-3. If the data block is a matrix structure (CSV), the SDK returns the entire content of the matrix structure to the software, or returns a specified number of columns or rows according to the instruction parameters of the software system; S4. After the software completes the data calculation, it sends a save data instruction to the SDK. The SDK finds the result data segment according to the index of the metadata segment, adds a data block containing the result data at the end of the data segment, and writes the name and position of the data block as an index to the reserved position of the result data index to complete the data storage; S5. When the software needs to read the result data, the SDK will search for the result data segment index from the metadata segment index and complete the data reading according to the process described in S3.

[0021] Compared with the prior art, the present invention has the following advantages due to the adoption of the above technical solution: 1. Unified data format to improve data consistency and credibility: The data file structure and indexing mechanism in the patent help ensure the consistency and credibility of the data, which is crucial to improving the accuracy and reliability of decision-making.

[0022] 2. Good data storage diversity and support for data asset value-added: This patent helps enterprises increase the value of data as assets by optimizing data storage and management, realizing the entry and management of data assets, promoting the circulation and value-added of data, and thus providing operational needs for the business.

[0023] 3. Data can be read randomly with high reading efficiency, which improves data retrieval efficiency: Through efficient data file structure and indexing methods, data can be read randomly, which significantly improves the speed of data retrieval and reduces the IO cost of the database, thereby improving the operational efficiency of the enterprise.

[0024] 4. Dynamic storage to support real-time data analysis and decision-making: This patent supports real-time data analysis, enabling enterprises to make timely decisions based on the latest available data, thereby improving the enterprise's response speed and market competitiveness.

[0025] 5. Promote data operability and interoperability: Through standardized data standards and index structures, seamless data connection and sharing can be achieved between different systems, reducing the complexity of data conversion and integration.

[0026] 6. Promote cooperation and innovation among systems: By adopting unified data standards and efficient indexing methods, systems can share and exchange data more conveniently, promote data-driven business model innovation, and bring more business opportunities and competitive advantages.

[0027] 7. Optimize the use of storage resources: Through efficient data organization and indexing, the waste of storage resources can be reduced, efficient data storage and compression can be achieved, and the storage costs of the enterprise can be reduced.

[0028] 8. Improve data security and privacy protection: The data storage rules in the patent help protect data from unauthorized access or leakage, while supporting data privacy protection, which is very important for complying with data protection regulations and maintaining customer trust.

[0029] 9. Support data-driven product innovation: Through the patented data file structure and indexing method, enterprises can better use data to develop new products and services, thereby achieving business growth and innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the structure of the pan-matrix data file of the present invention; Figure 2Schematic diagram of the index information structure of each data segment of the present invention; Figure 3 It is a numerical array storage structure of input data and result data of the present invention; Figure 4 It is a string array storage structure of input data and result data of the present invention; Figure 5 CSV storage structure for input data and result data of the present invention; Figure 6 It is a schematic diagram of the reading and writing process of the present invention; Figure 7 This is the actual test chart of Linux + CSV format reading performance for the comparison group; Figure 8 This is a measured graph of the reading performance of the present invention under the Linux system; Fig. 9 This is the measured performance chart of Windows + CSV format reading for the comparison group; Fig.10 This is a measured graph of the reading performance of the present invention under Windows system; DETAILED DESCRIPTION Example 1

[0031] A pan-matrix data file structure, as shown in the attached Figure 1 As shown, it consists of four main parts: metadata segment, configuration data segment, input data segment, and result data segment; each part is organized through an index structure to support fast data access and retrieval.

[0032] The metadata segment stores the metadata of the file, including the version number, creation time and update time, as well as indexes of the other three data segments, pointing to the storage locations of the three data segments respectively.

[0033] The configuration data segment is used to store data configuration information, which serves as the basis and description of data consumption by the data consumer, including a configuration segment index and configuration data. The configuration segment index is used to store the index information of each configuration data in the configuration data information, that is, the name and location of each configuration; the configuration content is stored in the configuration data in a string format.

[0034] The input data segment is used to store input parameters of the algorithm, including an input segment index and input data. The input segment index is used to store index information of each input data in the input data information.

[0035] The result data segment is used to store the result of the algorithm execution, including the result segment index and the result data. The result segment index is used to store the index information of each result data in the result data information. The result data segment has dynamic characteristics and its content can be modified without changing other parts of the data file.

[0036] The index information of the other three data segments except the metadata segment includes index type, data name, data type, starting position and data length to support fast data positioning. Figure 2 As shown in the figure, all data information is stored in column storage mode, and data of different granularities such as one item, one column, one row, multiple columns, and multiple rows can be accessed through indexes.

[0037] The input data in the input data segment and the result data in the result data segment support multiple data formats, including string, JSON, XML, array and matrix. The string storage format is length plus content, that is, offset index plus data content, supporting the storage of long and short strings, where the content length of short strings is not more than 65535 bytes, and the content length of long strings is not more than 2147483647 bytes. The array storage structure supports the storage of single-type data of numeric or string type, where: numeric type, such as the attached Figure 3 As shown in the figure, it is divided into integers and floating-point numbers, and the storage follows the Big-Endian format. Integers are stored in the specified byte order, and floating-point numbers follow the IEEE-754 standard. String type is a separate storage for index and content, as shown in the attached Figure 4 As shown in the figure, in order to be able to perform random queries on the string array, the storage of the string array is also divided into two parts: index and content. The index is an int64 array, which represents the index of the subsequent string content. The matrix storage structure is shown in the attached figure. Figure 5 As shown, the matrix is ​​stored in a data block (DATA), and the storage content includes an index and several data columns, where: each index item corresponds to a data column, which contains the column name + column type + the starting position of the data column in the file + the length of the data column; each data column is an array storage structure, which can be any single type among int32, int64, float, double, string, and datetime.

[0038] This patent is applicable to large-scale parallel computing, especially engine data analysis, intelligent production scheduling and other fields. The more parallel data and tasks there are, the greater the advantage of this patent.

[0039] The data file format described in this patent, namely the pan-matrix data file structure, will be equipped with a dedicated data reading and writing SDK and provide support for Linux, Windows and other systems. The SDK is written in Rust and provides support for Javascript, Python, C++, Java and other languages. Figure 6 As shown, the application sends read and write instructions, interacts with the pan-matrix data file structure through the data read and write SDK to achieve input and output.

[0040] The indexing method of the pan-matrix data file structure comprises the following steps: S1. The software system issues an index instruction to the dedicated read-write SDK supporting the pan-matrix data file structure; the dedicated read-write SDK identifies whether the version number in the metadata segment is the target version. If the version number in the data segment is higher than the target version (the version that the SDK can process), an error is reported and processing is stopped; the SDK can obtain the creation time and last modification time of the file from the data segment, which is used by the software system to confirm whether the file content is within the validity period; S2. The software system sends a command to read configuration data to the SDK. The SDK finds the configuration data segment according to the index of the metadata segment, finds the corresponding configuration item index according to the configuration name, reads the specified configuration data information according to the index information, and returns it to the software system. The software finds the data block name corresponding to the input data and the required data column name (if necessary) according to the configuration information. S3. The software system sends a command to read input data to the SDK. The SDK finds the input data segment according to the index of the metadata segment, finds the corresponding data block according to the data block name, and performs the following operations according to the data block type: S3-1. If the data block is a string or an array, the SDK returns the entire string or array content to the software; for strings, they can be further converted into XML, JSON and other structured objects according to the instruction parameter requirements of the software system; S3-2. When the SDK finds that the stored string is in JSON or XML format, it will decode the string and directly return the structured data object it represents for the caller to use; S3-3. If the data block is a matrix structure (CSV), the SDK returns the entire content of the matrix structure to the software, or returns a specified number of columns or rows according to the instruction parameters of the software system; S4. After the software completes the data calculation, it sends a save data instruction to the SDK. The SDK finds the result data segment according to the index of the metadata segment, adds a data block containing the result data at the end of the data segment, and writes the name and position of the data block as an index to the reserved position of the result data index to complete the data storage; S5. When the software needs to read the result data, the SDK will search for the result data segment index from the metadata segment index and complete the data reading according to the process described in S3.

[0041] In order to test the speed difference between the pan-matrix data file structure of the present invention and the traditional CSV format, this embodiment performs the following reading tests of the two formats under Linux.

[0042] 1) Data scale: Number of data rows: 1300503 rows Number of data columns: 298 columns, including 162 columns of 64-bit integers, 130 columns of double-precision floating-point numbers, and 6 columns of strings Total data: 1.43GB 2) Data storage format: CSV format This patent description format 3) Test method: Read the data file and read the data column by column, calculating the reading time of each column and the overall reading time Tested on Windows and Linux systems The tests were all performed using Python code 4) Test environment: Processor: 12th Gen Intel(R) Core(TM) i7-12700H 2.30 GHz Memory: 32.0 GB Disk: 512GB SSD Linux + CSV format test results are shown in Figure 7 As shown: It took 27248.04782 milliseconds, or about 27.24 seconds, to read the file; After the file is read, it takes an average of 0.04 milliseconds to read each column of data; The overall reading time is 27281.94866 milliseconds, about 27.28 milliseconds, and the average time per column is about 91.4 milliseconds; in conclusion: The CSV format requires that the data be read completely into memory first, which takes a lot of time. After the reading is completed, each column takes very little time to read; If only a few columns are read, this file format will consume more reading time in the early stage.

[0043] Linux + The file format of this patent is the pan-matrix data file structure test results are shown in Figure 8 As shown: It takes 1.98092 milliseconds to read the file, which is the "metadata segment" reading time; Segment index loading took 0.00228 milliseconds; The data block index loading took 0.94111 milliseconds; The overall index read takes about 3 milliseconds; Each column read takes an average of 48.458 milliseconds, a total of 14473.41619 milliseconds, about 14 seconds in conclusion: This patent format only needs to read the index in the early stage, which takes very little time; The data of the specified columns can be read on demand. This format has great advantages when only a few columns are to be read. For scenarios where all data is read completely, it is about 50% faster than the CSV format.

[0044] In summary, the present invention has a huge improvement in reading speed compared to the traditional CSV format. Example 2

[0045] The whole is the same as Example 1, except that the test is performed in a Windows environment.

[0046] Windows + CSV format test results are shown in Fig. 9 As shown: The reading time is 25000.1353 milliseconds, with an average of 83.947 milliseconds per column.

[0047] Windows + The file format of this patent is the pan-matrix data file structure test results are shown in Fig.10 As shown: It takes a total of 10164.7614 milliseconds, with an average of 29.162 milliseconds per column, which is 60% faster than the Windows + CSV format, still far ahead.

[0048] The above description of the disclosed embodiments is intended only to enable those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pan-matrix data file structure, characterized in that: It consists of 4 main parts: metadata segment, configuration data segment, input data segment and result data segment; The metadata segment stores metadata of the file, including indexes of the other three data segments; The configuration data segment includes a configuration segment index and configuration data, and the configuration segment index is used to store index information of each configuration data in the configuration data information; The input data segment includes an input segment index and input data, and the input segment index is used to store index information of each input data in the input data information; The result data segment includes a result segment index and result data, and the result segment index is used to store index information of each result data in the result data information.

2. A pan-matrix data file structure as claimed in claim 1, characterized in that: The data information of the three data segments are all stored based on the column storage method, and data of different granularities such as one item, one column, one row, multiple columns, and multiple rows can be accessed through indexes.

3. A pan-matrix data file structure as described in claim 2, wherein the metadata segment also includes a version number, a creation time, and an update time.

4. A pan-matrix data file structure as claimed in claim 3, characterized in that: The index information of the three data segments includes index type, data name, data type, starting position and data length.

5. A pan-matrix data file structure as claimed in claim 4, characterized in that: The result data segment has dynamic characteristics.

6. A pan-matrix data file structure according to any one of claims 1 to 5, characterized in that: Both the input data and the result data support multiple data formats, including string, JSON, XML, array, and matrix.

7. A pan-matrix data file structure as claimed in claim 6, characterized in that: The string storage format is offset index plus data content, supporting the storage of long and short strings.

8. A pan-matrix data file structure as claimed in claim 6, characterized in that: The array storage structure supports single-type data storage of string or numeric type, where string arrays are stored separately for index and content.

9. A pan-matrix data file structure as claimed in claim 6, characterized in that: The matrix storage structure includes an index and several data columns, where the index item contains the column name, column type and location information of the data column.

10. An indexing method for a pan-matrix data file structure as claimed in claim 6, characterized in that: The following steps are involved: S1. The software system issues an index instruction to the dedicated read-write SDK supporting the pan-matrix data file structure; the dedicated read-write SDK identifies whether the version number in the metadata segment is the target version. If the version number in the data segment is higher than the target version (the version that the SDK can process), an error is reported and processing is stopped; the SDK can obtain the creation time and last modification time of the file from the data segment, which is used by the software system to confirm whether the file content is within the validity period; S2. The software system sends a command to read configuration data to the SDK. The SDK finds the configuration data segment according to the index of the metadata segment, finds the corresponding configuration item index according to the configuration name, reads the specified configuration data information according to the index information, and returns it to the software system. The software finds the data block name corresponding to the input data and the required data column name (if necessary) according to the configuration information. S3. The software system sends a command to read input data to the SDK. The SDK finds the input data segment according to the index of the metadata segment, finds the corresponding data block according to the data block name, and performs the following operations according to the data block type: S3-1. If the data block is a string or an array, the SDK returns the entire string or array content to the software; for strings, they can be further converted into XML, JSON and other structured objects according to the instruction parameter requirements of the software system; S3-2. When the SDK finds that the stored string is in JSON or XML format, it will decode the string and directly return the structured data object it represents for the caller to use; S3-3. If the data block is a matrix structure (CSV), the SDK returns the entire content of the matrix structure to the software, or returns a specified number of columns or rows according to the instruction parameters of the software system; S4. After the software completes the data calculation, it sends a save data instruction to the SDK. The SDK finds the result data segment according to the index of the metadata segment, adds a data block containing the result data at the end of the data segment, and writes the name and position of the data block as an index to the reserved position of the result data index to complete the data storage; S5. When the software needs to read the result data, the SDK will search for the result data segment index from the metadata segment index and complete the data reading according to the process described in S3.

Images