Method for compressing and reading high-capacity file

Abstract

The invention belongs to the technical field of task planning data processing, and particularly relates to a high-capacity file compression reading implementation method, which comprises the following steps of: dividing groups of each row of data of a file according to design parameters; storing non-repeated data items in the file; establishing a mapping table of the non-duplicated data columns and storage positions thereof; establishing a file data storage position index table according to effective rows; establishing a file effective row index table according to groups; removing invalid data and compressing the data file; quickly indexing the compressed data file based on an index table; and automatically complementing invalid data and outputting the invalid data. According to the method, the large-capacity file can be compressed into the small file which is only about 3% of the source file, the compressed data can be directly read from the compressed file, and then the compressed data is analyzed into the original data through the data analysis process. The compression rate can be stabilized at about 97%, meanwhile, extremely high reading efficiency is guaranteed, and installation, deployment and use of large-capacity system data are facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of task planning data processing, and in particular relates to a method for implementing compression reading of large-capacity files. Background Art

[0002] With the rapid development of computer application technology, the complexity of aircraft mission planning algorithms has increased, and the amount of information processed has exploded. Mission planning algorithms often need to store a large amount of knowledge data as parameters for the execution of mission planning algorithms. A large amount of data ensures the accuracy of mission planning algorithm execution, but it brings huge challenges to algorithm execution efficiency, mission planning hardware platform, and information transmission efficiency.

[0003] When storing certain high-dimensional, large-capacity knowledge data, in order to improve the data indexing speed, an index structure will be added to the data file, which will cause a sharp increase in the size of the data file. The size of a single file will reach hundreds of GB. Since there are usually a large amount of invalid data occupying the structure in this type of knowledge data, the file can be compressed. However, the traditional compression and decompression method requires the entire file to be decompressed each time the file content is read, which cannot support the task planning algorithm to read the compressed file content in real time and quickly. Therefore, there is an urgent need to propose a file compression method that allows computers to read the contents of compressed files in real time and losslessly. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is: how to reduce the occupied space of data files while ensuring real-time reading efficiency.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention provides a method for implementing compression reading of large-capacity files, the method comprising the following steps:

[0008] Step 1: Divide each row of data in the file into groups according to design parameters;

[0009] Step 2: Store the non-repeated data items in the file;

[0010] Step 3: Create a mapping table between unique data columns and their storage locations;

[0011] Step 4: Create a file data storage location index table based on valid rows;

[0012] Step 5: Create a valid row index table for the file by group;

[0013] Step 6: Remove invalid data and compress the data file;

[0014] Step 7: Quickly index the compressed data file based on the index table;

[0015] Step 8: Automatically complete invalid data and output.

[0016] Among them, through the above steps 1 to 6, efficient compression of large-capacity files is achieved; through the above steps 7 to 8, real-time reading of large-capacity compressed files is achieved.

[0017] Among them, in step 1, for files that store data in fixed-length data groups, it is first necessary to compress the file into the smallest unit from the file line to the data group, and divide the file into equal intervals according to the size of the data group, and each data group corresponds to a group number; for files in which each line represents independent information, one line is a data group, and the group number is equivalent to the line number.

[0018] Among them, in step 2, it is monitored whether each row of data appears for the first time; if the data appears for the first time, the data is taken as an absolute value and stored in a binary file and the storage location is recorded, and then temporarily stored in the memory in the form of a key-value pair of "data: storage location"; if the data does not appear for the first time, it is not processed.

[0019] Wherein, in step 3: the data key value in step 2 is temporarily stored in the form of a map table "key+value" in C++, where key is non-repeated data and value is the location where the data is stored; since only non-repeated data columns are processed, a mapping table between non-repeated data columns and their storage locations is established in the memory;

[0020]

[0021]

[0022] Among them, in the step 4, a two-level index table is established, the first-level index table records the number of valid rows and the storage position of the second-level index, and the second-level index table records the storage position and data symbol of the last 5 data in each valid row; the current step is to establish the second-level index table: if the data row is valid, then the data storage position value is obtained from the map table stored in step 3 according to the data key, and the storage position of the data is stored in the 10-bit space; then according to the sign of the data, the value of the 1-bit sign bit is changed, if it is a positive sign, it is set to 0, if it is a negative sign, it is set to 1; the storage position and sign bit of the last 5 data of each row are stored in the index table, that is, 55 bits of data are stored in each row, and after each row is calculated, one bit is added to make 7 bytes to facilitate subsequent calculations; thus, a second-level index table of the file data storage position is built.

[0023] Among them, in the step 5, since the original data contains a lot of invalid data to keep the table structure intact, first check whether the last 5 data in each row are all invalid data. If they are all invalid data, the entire row is considered invalid; if there is one valid data, the entire row is considered valid; query and record the valid rows in each group, store the storage position of the first valid row in each group in the secondary index table with 4Byte of space, and store the number of valid rows with 1Byte of space; so each group occupies 5Byte of space to store the index content in the primary index file.

[0024] Among them, in step 6, a large number of invalid data rows will be encountered in the process of establishing indexes and saving data. If there are invalid data rows, they will be directly discarded without processing, and other valid data will be cyclically compressed.

[0025] Among them, in the step 7, when querying the xth row of data, where x is a random number within the total number of rows in the file, first calculate the group number G where x belongs, and the algorithm formula of G is as follows:

[0026] G = "(x / g)

[0027] That is, x belongs to group G;

[0028] According to G, find the starting position P1 of the primary index storage. The algorithm formula of P1 is as follows:

[0029] P1=(G-1)*5

[0030] In the primary index file, read 5 bytes of data from P1, set the first 4 bytes to B, and set the 5th byte to V. When V is 0, it means that the current group is all invalid data and returns directly. When V is not 0, according to B, the index storage start position P2 of the current group of valid row data can be found from the secondary index file. The algorithm formula of P2 is as follows:

[0031] P2=B

[0032] In the secondary index file, read V times from P2, 7 bytes each time, a total of 56 bits, of which the 0th bit is the fill bit and is not processed. In the remaining 55 bits, each 11 bits represent the storage index DI of 1 data; the 0th bit in DI is the sign bit S, and the remaining 10 bits represent the data storage position DP. Therefore, you can directly read 4 bytes from DP to get the absolute value of the original data from the file storing the data, and then re-assign the data according to S to decompress the complete original valid data.

[0033] Among them, in step 8, after obtaining the original valid data, every 5 valid data arrays form a row of data. According to the sequence number of the first data in each row, the missing invalid data rows of this group can be obtained, and the invalid data rows of this group are filled with invalid data to obtain a group of original data.

[0034] (III) Beneficial effects

[0035] Compared with the prior art, the present invention proposes a method for real-time compression and reading of large-capacity files. By adopting the file compression and reading method of the present invention, large-capacity files can be compressed into small files that are only about 3% of the source files, and compressed data can be directly read from the compressed files, and then the compressed data can be parsed into original data through the data parsing process. The method for real-time compression and reading of large-capacity files proposed by the present invention has a significant compression effect, and the compression rate can be stabilized at about 97%, while ensuring extremely high reading efficiency. It is convenient for the installation, deployment and use of large-capacity system data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is an example diagram of the binary data storage format.

[0037] Figure 2 The following is an example diagram of the secondary index format.

[0038] Figure 3 This is an example diagram of the first-level index format.

[0039] Figure 4 This is a schematic diagram of the index and data storage structure after file compression.

[0040] Figure 5 This is a diagram of the large file compression process.

[0041] Figure 6 A process diagram for reading data directly from a compressed file.

[0042] Figure 7 Outline infographic for original document.

[0043] Figure 8 This is the content picture of XX.txt.

[0044] Fig. 9 The data location index table.

[0045] Fig.10 A table of valid row indexes.

[0046] Fig.11 The actual content map of the output. DETAILED DESCRIPTION

[0047] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0048] In order to solve the above technical problems, the present invention provides a method for implementing compression reading of large-capacity files, the method comprising the following steps:

[0049] Step 1: Divide each row of data in the file into groups according to design parameters;

[0050] Step 2: Store the non-repeated data items in the file;

[0051] Step 3: Create a mapping table between unique data columns and their storage locations;

[0052] Step 4: Create a file data storage location index table based on valid rows;

[0053] Step 5: Create a valid row index table for the file by group;

[0054] Step 6: Remove invalid data and compress the data file;

[0055] Step 7: Quickly index the compressed data file based on the index table;

[0056] Step 8: Automatically complete invalid data and output.

[0057] Among them, through the above steps 1 to 6, efficient compression of large-capacity files is achieved; through the above steps 7 to 8, real-time reading of large-capacity compressed files is achieved.

[0058] Among them, in step 1, for files that store data in fixed-length data groups, it is first necessary to compress the file into the smallest unit from the file line to the data group, and divide the file into equal intervals according to the size of the data group, and each data group corresponds to a group number; for files in which each line represents independent information, one line is a data group, and the group number is equivalent to the line number.

[0059] In step 2, it is monitored whether each row of data appears for the first time; if the data appears for the first time, the data is taken as an absolute value and stored in a binary file and the storage location is recorded, and then temporarily stored in the memory in the form of a key-value pair of "data: storage location"; if the data does not appear for the first time, it is not processed. This method allows each data to be stored only once, saving a lot of space; the binary data storage format example is as follows Figure 1 shown.

[0060] Wherein, in step 3: the data key value in step 2 is temporarily stored in the form of a map table "key+value" in C++, where key is non-repeated data and value is the location where the data is stored; since only non-repeated data columns are processed, a mapping table between non-repeated data columns and their storage locations is established in the memory;

[0061]

[0062] Among them, in step 4, a two-level index table is established. The first-level index table records the number of valid rows and the storage position of the second-level index, and the second-level index table records the storage position and data symbol of the last 5 data in each valid row; the current step is to establish a second-level index table: if the data row is valid, then the data storage location value is obtained from the map table stored in step 3 according to the data key, and the storage position of the data is stored in a 10-bit space; then according to the sign of the data, the value of the 1-bit sign bit is changed, if it is a positive sign, it is set to 0, and if it is a negative sign, it is set to 1; the storage position and sign bit of the last 5 data of each row are stored in the index table, that is, 55 bits of data are stored in each row. After each row is calculated, one bit is added to make 7 bytes to facilitate subsequent calculations; thus, a second-level index table of the file data storage location is built. The storage format of the second-level index table is as follows: Figure 2 shown.

[0063] Among them, in step 5, since the original data contains a lot of invalid data to keep the table structure intact, first check whether the last 5 data in each row are all invalid data. If they are all invalid data, the entire row is considered invalid; if there is one valid data, the entire row is considered valid; query and record the valid rows in each group, store the storage position of the first valid row in each group in the secondary index table with 4Byte space, and store the number of valid rows with 1Byte space; so each group occupies 5Byte space to store the index content in the primary index file. The storage format of the primary index table is as follows: Figure 3 shown.

[0064] Among them, in step 6, a large number of invalid data rows will be encountered in the process of establishing indexes and saving data. If there are invalid data rows, they will be directly discarded without processing, and other valid data will be cyclically compressed.

[0065] Among them, in the step 7, when querying the xth row of data, where x is a random number within the total number of rows in the file, first calculate the group number G where x belongs, and the algorithm formula of G is as follows:

[0066] G = "(x / g)

[0067] That is, x belongs to group G;

[0068] According to G, find the starting position P1 of the primary index storage. The algorithm formula of P1 is as follows:

[0069] P1=(G-1)*5

[0070] In the primary index file, read 5 bytes of data from P1, set the first 4 bytes to B, and set the 5th byte to V. When V is 0, it means that the current group is all invalid data and returns directly. When V is not 0, according to B, the index storage start position P2 of the current group of valid row data can be found from the secondary index file. The algorithm formula of P2 is as follows:

[0071] P2=B

[0072] In the secondary index file, read V times from P2, each time reading 7 bytes, a total of 56 bits, of which the 0th bit is a fill bit and is not processed. In the remaining 55 bits, each 11 bits represent the storage index DI of 1 data; the 0th bit in DI is the sign bit S, and the remaining 10 bits represent the data storage location DP, so you can directly read 4 bytes from DP to get the absolute value of the original data from the file storing the data, and then re-sign the data according to S, so that the complete original valid data can be decompressed. The entire reading process and index and data storage structure are shown in the figure. Figure 4 shown.

[0073] Among them, in step 8, after obtaining the original valid data, every 5 valid data arrays form a row of data. According to the sequence number of the first data in each row, the missing invalid data rows of this group can be obtained, and the invalid data rows of this group are filled with invalid data to obtain a group of original data.

[0074] Example 1

[0075] To compress a large file named XX, the file is XX.txt, follow the steps below:

[0076] Step 1: First, pre-process the large file and divide each row of data into groups according to the design parameters. File information such as Figure 7 shown.

[0077] Step 2: Store the non-repeated data columns in the large file XX.txt by line. The content of XX.txt is as follows Figure 8 shown.

[0078] Step 3: Create a mapping table between unique data columns and their storage locations.

[0079] Step 4: Create a file data storage location index table based on valid rows. The data location index table is as follows: Fig. 9 shown.

[0080] Step 5: Create a valid row index table for each group. Fig.10 shown.

[0081] Step 6: Remove invalid data and compress the data file.

[0082] Step 7: Quickly index the compressed data file based on the index table

[0083] Step 8: Automatically complete the invalid data and output it. For example, if you query the 15273226th group of data, the output content is as follows: Fig.11 shown.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing compression reading of large-capacity files, characterized in that: The method comprises the following steps: Step 1: Divide each row of data in the file into groups according to design parameters; Step 2: Store the non-repeated data items in the file; Step 3: Create a mapping table between unique data columns and their storage locations; Step 4: Create a file data storage location index table based on valid rows; Step 5: Create a valid row index table for the file by group; Step 6: Remove invalid data and compress the data file; Step 7: Quickly index the compressed data file based on the index table; Step 8: Automatically complete invalid data and output.

2. The method for implementing compression reading of a large-capacity file according to claim 1, characterized in that: Through the above steps 1 to 6, efficient compression of large-capacity files is achieved; through the above steps 7 to 8, real-time reading of large-capacity compressed files is achieved.

3. The method for implementing compression reading of large-capacity files as claimed in claim 1, characterized in that: In step 1, for files that store data in fixed-length data groups, firstly, the file is compressed into the smallest unit, i.e., the file line is processed into data groups, and the file is divided into equal intervals according to the size of the data groups, and each data group corresponds to a group number; for files in which each line represents independent information, one line is a data group, and the group number is equivalent to the line number.

4. The method for implementing compression reading of large-capacity files as claimed in claim 3, characterized in that: In step 2, it is monitored whether each row of data appears for the first time; if the data appears for the first time, the data is taken as an absolute value and stored in a binary file and the storage location is recorded, and then temporarily stored in the memory in the form of a key-value pair of "data: storage location"; if the data does not appear for the first time, it is not processed.

5. The method for implementing compression reading of large-capacity files as claimed in claim 4, characterized in that: In step 3: the data key value in step 2 is temporarily stored in the form of a map table "key+value" in C++, where key is non-repeated data and value is the location where the data is stored; since only non-repeated data columns are processed, a mapping table between non-repeated data columns and their storage locations is established in the memory; 6. The method for implementing compression reading of large-capacity files as claimed in claim 5, characterized in that: In step 4, a two-level index table is established, the first-level index table records the number of valid rows and the storage location of the second-level index, and the second-level index table records the storage location and data symbol of the last 5 data in each valid row; The current step is to build a secondary index table: if the data row is valid, then get the data storage location value from the map table stored in step 3 according to the data key, and store the data storage location in a 10-bit space; then change the value of the 1-bit sign bit according to the sign of the data, if it is a positive sign, set it to 0, if it is a negative sign, set it to 1; store the storage location and sign bit of the last 5 data of each row in the index table, that is, store 55 bits of data in each row. After each row is calculated, add one bit to make it 7 bytes to facilitate subsequent calculations; thus, a secondary index table of the file data storage location is built.

7. The method for implementing compression reading of a large-capacity file according to claim 6, characterized in that: In step 5, since the original data contains a lot of invalid data to keep the table structure intact, first check whether the last 5 data in each row are all invalid data. If they are all invalid data, the entire row is considered invalid; if there is one valid data, the entire row is considered valid; query and record the valid rows in each group, store the storage position of the first valid row in each group in the secondary index table with 4Byte of space, and store the number of valid rows with 1Byte of space; so each group occupies 5Byte of space to store the index content in the primary index file.

8. The method for implementing compression reading of large-capacity files as claimed in claim 7, characterized in that: In step 6, a large number of invalid data rows will be encountered during the process of indexing and saving data. If there are invalid data rows, they will be directly discarded without processing, and other valid data will be cyclically compressed.

9. The method for implementing compression reading of a large-capacity file according to claim 8, characterized in that: In step 7, when querying the xth row of data, where x is a random number within the total number of rows in the file, first calculate the group number G where x belongs, and the algorithm formula of G is as follows: G = "(x / g) That is, x belongs to group G; According to G, find the starting position P1 of the primary index storage. The algorithm formula of P1 is as follows: P1=(G-1)*5 In the primary index file, read 5 bytes of data from P1, set the first 4 bytes to B, and set the 5th byte to V. When V is 0, it means that the current group is all invalid data and returns directly. When V is not 0, according to B, the index storage start position P2 of the current group of valid row data can be found from the secondary index file. The algorithm formula of P2 is as follows: P2=B In the secondary index file, read V times from P2, 7 bytes each time, a total of 56 bits, of which the 0th bit is the fill bit and is not processed. In the remaining 55 bits, each 11 bits represent the storage index DI of 1 data; the 0th bit in DI is the sign bit S, and the remaining 10 bits represent the data storage position DP. Therefore, you can directly read 4 bytes from DP to get the absolute value of the original data from the file storing the data, and then re-assign the data according to S to decompress the complete original valid data.

10. The method for implementing compression reading of a large-capacity file according to claim 9, characterized in that: In step 8, after obtaining the original valid data, every 5 valid data arrays form a row of data. According to the sequence number of the first data in each row, the missing invalid data rows of this group can be obtained. The invalid data rows of this group are filled with invalid data to obtain a group of original data.

Images