A method and system for quickly parsing compressed file content

Abstract

A method and system for quickly parsing the contents of a compressed file, the method comprising: decompressing a compressed file using a decompression thread pool to obtain at least one piece of decompressed data and storing it in a memory block queue; obtaining a memory block from the memory block queue using a read thread pool and reading the data of the memory block into a database; determining whether all data in the memory block has been read in, and if so, placing the memory block in an empty memory block queue; otherwise, temporarily storing the unread memory block and updating the temporarily stored memory block; and placing the memory block in an empty queue after all data in the memory block has been read in; the decompression thread pool and the read thread pool each maintain multiple threads. The present invention, based on shared memory, message queues, and parallel threads, enables simultaneous decompression and reading of files, greatly improving the efficiency of parsing the contents of compressed files.

Description

Technical Field

[0001] The present invention relates to the technical field of decompression, and in particular to a method and system for quickly parsing compressed file contents. Background Art

[0002] In the computer field, to save storage space or improve data transmission efficiency, files are processed through the following process: generating files, compressing files, saving and transmitting files, receiving and decompressing files, and reading files. Decompressing files and reading files are essential steps. Currently, there are two ways to handle these two steps:

[0003] Divide into independent steps, and process decompression and reading files serially, that is, first decompress the complete file, and then read the decompressed file;

[0004] Decompress and read files in parallel. When decompressing a file, put the decompressed data into a queue, and when reading a file, take the data from the queue.

[0005] Both of the above methods have the problem of long file reading time and low efficiency. Especially in the field of EDA (electronic design automation), as the compressed files become larger and larger, the time taken to read the files also becomes longer and longer. Summary of the Invention

[0006] In order to solve the defects of the prior art, the purpose of the present invention is to provide a method and system for quickly parsing the content of compressed files. Based on memory sharing, multi-threaded decompression and multi-threaded reading are performed simultaneously to improve the efficiency of parsing the content of compressed files.

[0007] To achieve the above object, the present invention provides a method for quickly parsing the contents of a compressed file, comprising the following steps:

[0008] Decompress the compressed file through the decompression thread pool, obtain at least one decompressed data and store it in the memory block queue;

[0009] Obtaining a memory block from the memory block queue through a reading thread pool, and reading the data of the memory block into a database;

[0010] Determine whether all data of the memory block have been read in. If so, put the memory block into the empty memory block queue. Otherwise, temporarily store the unread memory block and update the temporarily stored memory block. After all data of the memory block have been read in, put the memory block into the empty queue.

[0011] The decompression thread pool and the reading thread pool respectively maintain multiple threads.

[0012] Furthermore, it also includes:

[0013] Marking the dependency or non-dependency of the decompressed data and placing it into a data queue;

[0014] The data in the data queue is extracted through a pre-processing thread pool, and after determining the dependency of the data and marking the range of the data block, the data is put into the memory block queue; the pre-processing thread pool maintains multiple threads.

[0015] Furthermore, the dependency relationship means that the later content in the compressed file uses the previously defined content, and the later content data has a dependency relationship with the previously defined content data. When reading data into the database through the reading thread pool, it needs to be read in serially; the non-dependency relationship means that the later content does not depend on the previously defined content. When reading data into the database through the reading thread pool, the two parts of the non-dependent content data are read in parallel.

[0016] Furthermore, the range of the data block includes:

[0017] Starting point: memory block number + address within the block;

[0018] Intermediate content: 0 or more complete memory blocks;

[0019] End point: memory block number + address within the block.

[0020] Furthermore, in the preprocessing thread pool, each thread puts the corresponding data block into the memory block queue according to the memory block number.

[0021] Furthermore, if a statement spans multiple memory blocks, when the preprocessing thread processes the last statement in the memory block and finds that the statement is incomplete, it reads the data of the next memory block until the statement is processed completely; when the preprocessing thread processes the first statement in the memory block and finds that the statement is incomplete, it ignores the incomplete statement until a new statement is found and then starts parsing.

[0022] On the other hand, the present invention also provides a system for quickly parsing compressed file content, comprising:

[0023] The decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one piece of decompressed data, mark the dependency of the decompressed data, and store it in the data queue;

[0024] A preprocessing module is used to obtain data from the data queue, determine the dependency of the data, mark the range of the data, obtain the data block and put it into the memory block queue;

[0025] The reading module is used to obtain a memory block from the memory block queue, and read the data of the memory block into the database in parallel or serially according to the dependency relationship of the data in the memory block to obtain a decompressed file; the reading module is used to determine whether all the data of the memory block has been read in, and if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and put the memory block into the empty queue after all the data of the memory block has been read in.

[0026] On the other hand, the present invention also provides a system for quickly parsing compressed file content, comprising:

[0027] A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one decompressed data and put it into a memory block queue;

[0028] The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory blocks into the database, and put the memory blocks with completely read data into the empty memory block queue; the reading module is used to determine whether all the data in the memory block has been read in, and if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and put the memory block into the empty queue after all the data in the memory block has been read in.

[0029] On the other hand, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the method for quickly parsing compressed file contents as described above.

[0030] On the other hand, the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the method for quickly parsing compressed file content as described above.

[0031] The method for quickly parsing compressed file contents provided by the present invention has the following beneficial effects compared with the prior art:

[0032] Based on shared memory, message queues and parallel threads, decompressing files and reading files can be performed simultaneously, greatly improving the efficiency of parsing the contents of compressed files.

[0033] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0035] Figure 1 Flowchart of a method for quickly parsing compressed file contents according to Embodiment 1 of the present invention;

[0036] Figure 2 Schematic diagram of the process of decompressing a compressed file and reading a file according to the first embodiment of the present invention;

[0037] Figure 3 Flowchart of a method for quickly parsing compressed file contents according to Embodiment 2 of the present invention;

[0038] Figure 4 Schematic diagram of the process of decompressing a compressed file and reading a file according to the second embodiment of the present invention;

[0039] Figure 5 Schematic diagram of the system structure for quickly parsing compressed file contents according to the third embodiment of the present invention;

[0040] Figure 6 Schematic diagram of an application scenario of a system for quickly parsing compressed file contents according to embodiment 3 of the present invention;

[0041] Figure 7 FIG. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0044] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0045] It should be noted that the concepts of "first" and "second" may be mentioned in the present invention only to distinguish different devices, components or parts, and are not used to limit the order or interdependence of the functions performed by these devices, components or parts.

[0046] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative and non-restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, they should be understood as "one or more". "Plurality" should be understood as two or more.

[0047] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] In an embodiment of the present invention, a method for quickly parsing the content of a compressed file is provided, comprising: decompressing the compressed file through a decompression thread pool to obtain at least one decompressed data and storing it in a memory block queue; obtaining a memory block from the queue through a reading thread pool, and reading the data of the memory block into a database; judging whether all the data of the memory block has been read in, and if so, placing the memory block in an empty memory block queue; otherwise, temporarily storing the unread memory block and updating the temporarily stored memory block; and after all the data of the memory block has been read in, placing the memory block in an empty queue.

[0050] Figure 1 This is a flow chart of a method for quickly parsing compressed file contents according to Embodiment 1 of the present invention. Figure 2 The following is a schematic diagram of the process of decompressing and reading compressed files according to the first embodiment of the present invention (the solid arrows in the figure indicate the direction of data flow, and the dotted arrows indicate the direction of empty memory block flow). Figure 1 and Figure 2 The method for quickly parsing compressed files of the present invention is further described in detail.

[0051] In step 101 , the compressed file 201 is decompressed by the decompression thread pool 202 to obtain at least one piece of decompressed data and store it in the memory block queue 203 .

[0052] In an embodiment of the present invention, a plurality of threads for performing decompression tasks are maintained through the decompression thread pool 201. After any idle thread decompresses the compressed file 201, the decompressed data is obtained and put into a memory block, and a memory block queue 203 is maintained.

[0053] In step 102 , a memory block is obtained from the memory block queue 203 through the reading thread pool 204 , and the data of the memory block is read into the database.

[0054] In the embodiment of the present invention, a plurality of threads for executing read and write tasks are maintained by the read thread pool 201 . The plurality of threads obtain memory blocks from the memory block queue 203 and read the data in the memory blocks into the database.

[0055] In step 103, it is determined whether all the data of the memory block has been read in. If so, the memory block is placed in the empty memory block queue. Otherwise, the unread memory block is temporarily stored and the temporarily stored memory block is updated. After all the data of the memory block has been read in, the memory block is placed in the empty queue.

[0056] The method for quickly parsing the contents of compressed files provided in this embodiment adopts a multi-threaded approach in both the decompression and reading stages, based on shared memory and queue transmission to reduce data processing time; with a small increase in memory, the performance of reading compressed files can be greatly improved, and as the number of threads increases, the efficiency of reading compressed files can be multiplied, saving program running time and greatly improving user experience.

[0057] Example 2

[0058] The difference between this embodiment and the above embodiment is that: the compressed file is decompressed by the decompression thread pool, and after obtaining the decompressed data, the dependencies of the decompressed data are marked and a data queue is maintained; a preprocessing thread pool is added to extract the data in the data queue, determine the dependencies of the data, mark the range of the data blocks, and put them into the memory block queue, and then process the data blocks in the memory block queue through the reading thread pool.

[0059] Figure 3 This is a flow chart of a method for quickly parsing compressed file contents according to embodiment 2 of the present invention. Figure 4 The following is a schematic diagram of the process of decompressing and reading compressed files according to the second embodiment of the present invention (the solid arrows in the figure indicate the direction of data flow, and the dotted arrows indicate the direction of empty memory block flow). Figure 3 and Figure 4 The method for quickly parsing compressed files of the present invention is further described in detail.

[0060] In step 301 , the compressed file 401 is decompressed by the decompression thread pool 402 to obtain at least one piece of decompressed data.

[0061] In step 302 , the dependency of the decompressed data is marked and placed in a data queue 403 .

[0062] In an embodiment of the present invention, the contents of compressed file 401 may be dependent or independent. A dependent relationship means that the subsequent content in compressed file 401 uses the previously defined content. When reading these contents into the database through read-in thread pool 404, if the previously defined content is not read first, the subsequent content cannot be correctly parsed. Therefore, the two parts of the dependent content need to be read in serial. An independent relationship means that the subsequent content does not depend on the previously defined content and can be read first. That is, the two parts of the non-dependent content can be read in parallel. Therefore, the data placed in memory block queue 407 needs to be marked with a dependency relationship with the previous data.

[0063] In step 303 , the data in the data queue 403 is extracted through the pre-processing thread pool 406 , and after determining the dependency of the data and marking the range of the data block, the data is put into the memory block queue 407 .

[0064] In the embodiment of the present invention, in order to identify the dependencies between file contents, a pre-processing thread pool 406 is added. The pre-processing thread pool 406 obtains different data blocks by determining the dependencies between data and marking the range of data blocks through the pre-processing module. That is, the pre-processing module not only determines the dependencies between file contents, but also marks the range of each data block. The range of the data block includes:

[0065] Starting point: memory block number + address within the block;

[0066] Middle content: complete memory block (0 or more);

[0067] End point: memory block number + address within the block;

[0068] That is, the preprocessing module outputs a data block with the following structure:

[0069] {

[0070] bool is_depend_previous_;

[0071] int data_type_

[0072] std::vector <std::pair<memory_block_number,address> >data_content_;

[0073] }

[0074] In the embodiment of the present invention, the pre-processing thread pool 406 also uses multi-threaded parallel technology. After each thread processes a memory block, it puts the corresponding data block into the memory block queue 407 according to the memory block number. It should be noted that this link has the following special scenarios:

[0075] The statement spans multiple memory blocks, meaning the last statement in a memory block is incomplete and some of the data is in the next memory block. When the preprocessing thread processes the last statement, if it finds the statement incomplete, it reads the data from the next memory block until the statement is completely processed. When the preprocessing thread processes the first statement in a memory block, if it finds the statement incomplete, it ignores the incomplete statement and continues parsing until a new statement is found. If the file format is special and it is impossible to determine whether there is a new statement in the middle of the file, the preprocessing module cannot use the multi-threaded preprocessing thread pool 406.

[0076] In step 304 , the data blocks in the memory block queue 407 are processed by the read thread pool 404 , and the contents of the data blocks are read into the database.

[0077] In an embodiment of the present invention, a multi-threaded read thread pool 404 retrieves data blocks from a memory block queue 407 and, based on the dependencies within the data blocks, decides whether to process the data blocks serially or in parallel. If the current data block is dependent on the previous data block, the current data block is processed after the previous data block is processed, meaning that the current data block needs to be processed serially. If the current data block is not dependent on the previous data block, the read thread pool 404 processes the current data block in parallel. After data block processing is complete, if all data in the memory block has been read, the memory block is placed in an empty queue 405. If processing is incomplete, the unread memory block is temporarily stored, or a temporarily stored memory block is updated. When all data in the memory block has been read, the memory block is placed in an empty queue.

[0078] This embodiment adopts multiple technologies to achieve fast parsing of compressed files, such as using memory sharing and queue methods to transfer data, which greatly saves the time used for data copying; the parsing process of compressed files is divided into three stages, namely decompression, preprocessing and reading, and a multi-threaded approach is adopted in each stage to reduce data processing time; the present invention also takes into account data dependencies and introduces a data block queue that can express dependencies, solves the problem of direct data dependencies, and better enriches the application scenarios of multi-threaded parallelism.

[0079] The method for quickly parsing the contents of compressed files provided by the present invention can significantly improve the performance of reading compressed files with a small increase in memory. As the number of threads increases, the efficiency of reading compressed files can be multiplied, saving program running time and greatly improving user experience.

[0080] Example 3

[0081] In an embodiment of the present invention, a system for quickly parsing the contents of a compressed file is also provided. Figure 5 Schematic diagram of the system structure for quickly parsing compressed file contents according to the third embodiment of the present invention. Figure 5 As shown, the system 500 of the present invention for quickly parsing compressed file content includes:

[0082] The decompression module 501 is used to decompress the compressed file through the decompression thread pool to obtain at least one piece of decompressed data, mark the dependency relationship of the decompressed data, and store it in the data queue;

[0083] The pre-processing module 502 is used to obtain data from the data queue, determine the data dependency, mark the data range, obtain the data block and put it into the memory block queue;

[0084] The reading module 503 is used to obtain memory blocks from the memory block queue, and read the data of the memory blocks into the database in parallel or serially according to the dependency relationship of the data in the memory blocks to obtain a decompressed file.

[0085] In one embodiment, the range of the data block includes:

[0086] Starting point: memory block number + address within the block;

[0087] Middle content: complete memory block (0 or more);

[0088] End point: memory block number + address within the block;

[0089] The pre-processing module 502 outputs a data block with the following structure:

[0090] {

[0091] bool is_depend_previous_;

[0092] int data_type_

[0093] std::vector <std::pair<memory_block_number,address> >data_content_;

[0094] }

[0095] The reading module 503 uses a multi-threaded reading thread pool to process data blocks. When reading the contents of a data block into the database, if the current data block is dependent on the previous data block, the current data block will be processed after the previous data block is processed. That is, the current data block will be processed serially by the reading thread pool. If the current data block is not dependent on the previous data block, the current data block will be processed in parallel by the reading thread pool. After the data block processing is completed, if all data in the corresponding memory block has been read, the memory block will be placed in an empty queue.

[0096] The system for rapidly parsing compressed file content of the present invention can be applied to any data receiving scenario. Figure 6 Schematic diagram of a system application scenario for quickly parsing compressed file contents according to embodiment 3 of the present invention. Figure 6 As shown, the first terminal 601 sends a compressed file to the second terminal 602. The second terminal 602 receives the compressed file and decompresses, preprocesses and reads the compressed file through the system 500 that quickly parses the compressed file content, which greatly improves the efficiency of the second terminal 602 in decompressing and reading the compressed file.

[0097] Example 4

[0098] The difference between this embodiment and embodiment 3 is that there is no pre-processing module. The system for quickly parsing the contents of a compressed file in this embodiment includes: a decompression module for decompressing the compressed file through a multi-threaded decompression thread pool, obtaining at least one decompressed data and placing it into a memory block queue;

[0099] The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory block into the database, and put the memory block with completely read data into the empty memory block queue; it determines whether all the data in the memory block has been read in. If so, the memory block is put into the empty memory block queue; otherwise, the unread memory block is temporarily stored and the temporarily stored memory block is updated. After all the data in the memory block has been read in, the memory block is put into the empty queue.

[0100] Example 5

[0101] In an embodiment of the present invention, an electronic device is further provided. Figure 7 FIG. 1 is a structural diagram of an electronic device according to a fourth embodiment of the present invention. Figure 7 As shown, the electronic device of the present invention includes a processor 701 and a memory 702, wherein:

[0102] The memory 702 stores a computer program. When the computer program is read and executed by the processor 701 , the computer program executes the steps in the embodiment of the method for quickly parsing the contents of a compressed file as described above.

[0103] Example 6

[0104] In an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored. The computer program is configured to execute the steps of the embodiment of the method for quickly parsing compressed file content as described above when running.

[0105] In this embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0106] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will be able to modify the technical solutions described in the foregoing embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for quickly parsing the contents of a compressed file, characterized in that: The following steps are involved: Decompress the compressed file through the decompression thread pool, obtain at least one decompressed data and store it in the memory block queue; Obtaining a memory block from the memory block queue through a reading thread pool, and reading the data of the memory block into a database; Determine whether all data of the memory block have been read in. If so, put the memory block into the empty memory block queue. Otherwise, temporarily store the unread memory block and update the temporarily stored memory block. After all data of the memory block have been read in, put the memory block into the empty queue. The decompression thread pool and the reading thread pool respectively maintain multiple threads; Also includes: Marking the dependency or non-dependency of the decompressed data and placing it into a data queue; Extracting data from the data queue through a preprocessing thread pool, determining data dependencies, marking data block ranges, and then placing the data into a memory block queue; the preprocessing thread pool maintains multiple preprocessing threads; If a statement spans multiple memory blocks, when the preprocessing thread processes the last statement in the memory block and finds that the statement is incomplete, it reads the data of the next memory block until the statement is processed completely; when the preprocessing thread processes the first statement in the memory block and finds that the statement is incomplete, it ignores the incomplete statement until a new statement is found and then starts parsing.

2. The method for quickly parsing compressed file contents according to claim 1, wherein: The dependency relationship means that the later content in the compressed file uses the previously defined content, so the later content data has a dependency relationship with the previously defined content data. When reading data into the database through the reading thread pool, it needs to be read in serial; the non-dependency relationship means that the later content does not depend on the previously defined content. When reading data into the database through the reading thread pool, the two parts of the non-dependent content data are read in parallel.

3. The method for quickly parsing compressed file contents according to claim 1, wherein: The scope of the data block includes: Starting point: memory block number + address within the block; Intermediate content: 0 or more complete memory blocks; End point: memory block number + address within the block.

4. The method for quickly parsing compressed file contents according to claim 3, wherein: In the preprocessing thread pool, each thread puts the corresponding data block into the memory block queue according to the memory block number.

5. A system for quickly parsing compressed file contents, characterized in that: include: The decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one piece of decompressed data, mark the dependency of the decompressed data, and store it in the data queue; A preprocessing module is used to obtain data from the data queue, determine the dependency of the data, mark the range of the data, obtain the data block and put it into the memory block queue; The reading module is used to obtain a memory block from the memory block queue, and read the data of the memory block into the database in parallel or serially according to the dependency relationship of the data in the memory block to obtain a decompressed file; the reading module is used to determine whether all the data of the memory block has been read in, and if so, put the memory block into the empty memory block queue; otherwise, temporarily store the unread memory block and update the temporarily stored memory block, and put the memory block into the empty queue after all the data of the memory block has been read in; The preprocessing module uses multiple preprocessing threads to extract data from the data queue, and after determining the dependency relationship of the data and marking the range of the data block, puts the data into the memory block queue; During the process of processing a statement by each of the preprocessing threads, if the statement spans multiple memory blocks, when the preprocessing thread processes the last statement in the memory block and finds that the statement is incomplete, it reads data from the next memory block until the statement is completely processed; When the preprocessing thread processes the first statement in the memory block and finds that the statement is incomplete, it ignores the incomplete statement until a new statement is found and then starts parsing.

6. A system for rapidly parsing compressed file contents, using the method for rapidly parsing compressed file contents according to any one of claims 1 to 4, characterized in that: include: A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one decompressed data and put it into a memory block queue; The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory blocks into the database, and put the memory blocks with completely read data into the empty memory block queue; the reading module is used to determine whether all the data in the memory block has been read in, and if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and put the memory block into the empty queue after all the data in the memory block has been read in.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor is used to execute the computer program stored in the memory to implement the method steps of quickly parsing the content of a compressed file as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is loaded and executed by a processor to implement the method steps for quickly parsing the contents of a compressed file according to any one of claims 1 to 4.

Images