Method and system for rapidly analyzing compressed file content

Abstract

The invention discloses a method and a system for quickly analyzing contents of a compressed file, and the method comprises the following steps: decompressing the compressed file through a decompression thread pool to obtain at least one piece of decompression data, and storing the decompression data into a memory block queue; memory blocks are obtained from the memory block queue through a read-in thread pool, and data of the memory blocks are read into a database; judging whether all data of the memory block are read in, if so, putting the memory block into an empty memory block queue, otherwise, temporarily storing an unread memory block and updating the temporarily stored memory block, and after all data of the memory block are read in, putting the memory block into the empty queue; and a plurality of threads are respectively maintained in the decompression thread pool and the read-in thread pool. Based on the shared memory, the message queue and the parallel thread, the file decompression and the file reading are performed at the same time, and the efficiency of analyzing the compressed file content is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of decompression, and in particular to a method and system for quickly parsing compressed file contents. Background Art

[0002] In the computer field, in order to save storage space or improve data transmission efficiency, files are processed through the following process: generate files, compress files, save and transmit files, receive and decompress files, and read files. Decompressing files and reading files are essential steps. Currently, there are two processing methods for these two steps:

[0003] Divide into independent steps, and process decompression and reading files serially, that is, first decompress the complete file, and then read the decompressed file;

[0004] Decompress and read files in parallel. When decompressing files, put the decompressed data into a queue, and when reading files, take data from the queue.

[0005] Both of the above methods have the problem of long file reading time and low efficiency. Especially in the field of EDA (electronic design automation), as the compressed files become larger and larger, the time taken to read the files also becomes longer and longer. Summary of the invention

[0006] In order to solve the defects of the prior art, the purpose of the present invention is to provide a method and system for quickly parsing the content of compressed files. Based on memory sharing, multi-threaded decompression and multi-threaded reading are performed simultaneously to improve the efficiency of parsing the content of compressed files.

[0007] In order to achieve the above object, the present invention provides a method for quickly parsing the contents of a compressed file, comprising the following steps:

[0008] Decompress the compressed file through the decompression thread pool, obtain at least one decompressed data and store it in the memory block queue;

[0009] Obtaining a memory block from the memory block queue through a reading thread pool, and reading the data of the memory block into a database;

[0010] Determine whether all the data of the memory block has been read in, if so, put the memory block into an empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and then put the memory block into an empty queue after all the data of the memory block has been read in;

[0011] The decompression thread pool and the reading thread pool respectively maintain a plurality of threads.

[0012] Furthermore, it also includes:

[0013] Marking the dependency or non-dependency of the decompressed data and putting it into a data queue;

[0014] The data in the data queue is extracted through a preprocessing thread pool, and after determining the dependency of the data and marking the range of the data block, the data is put into a memory block queue; the preprocessing thread pool maintains a plurality of threads.

[0015] Furthermore, the dependency relationship means that the later content in the compressed file uses the previously defined content, and the later content data has a dependency relationship with the previously defined content data. When reading data into the database through the reading thread pool, serial reading is required; the non-dependency relationship means that the later content does not depend on the previously defined content. When reading data into the database through the reading thread pool, the two parts of the non-dependent content data are read in parallel.

[0016] Furthermore, the range of the data block includes:

[0017] Starting point: memory block number + address within the block;

[0018] Intermediate content: 0 or more complete memory blocks;

[0019] End point: memory block number + address within the block.

[0020] Furthermore, in the preprocessing thread pool, each thread puts the corresponding data block into the memory block queue according to the memory block number.

[0021] Furthermore, if a statement spans multiple memory blocks, when the preprocessing thread processes the last statement in the memory block and finds that the statement is incomplete, it reads the data of the next memory block until the statement is processed completely; when the preprocessing thread processes the first statement in the memory block and finds that the statement is incomplete, it ignores the incomplete statement until a new statement is found and then starts parsing.

[0022] On the other hand, the present invention also provides a system for quickly parsing compressed file content, comprising:

[0023] A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool to obtain at least one decompressed data, mark the dependency of the decompressed data, and store it in a data queue;

[0024] A preprocessing module, used to obtain data from the data queue, determine the dependency of the data, mark the range of the data, obtain the data block and put it into the memory block queue;

[0025] The reading module is used to obtain the memory block from the memory block queue, and read the data of the memory block into the database in parallel or serially according to the dependency relationship of the data in the memory block to obtain the decompressed file; the reading module is used to determine whether all the data of the memory block has been read in, and if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and when all the data of the memory block has been read in, put the memory block into the empty queue.

[0026] On the other hand, the present invention also provides a system for quickly parsing compressed file content, comprising:

[0027] A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one decompressed data and put it into a memory block queue;

[0028] The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory blocks into the database, and put the memory blocks with completely read data into the empty memory block queue; the reading module is used to determine whether all the data in the memory block has been read in, if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory blocks and update the temporarily stored memory blocks, and when all the data in the memory block has been read in, put the memory block into the empty queue.

[0029] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to execute the computer program stored in the memory to implement the method for quickly parsing compressed file contents as described above.

[0030] On the other hand, the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the method for quickly parsing compressed file content as described above.

[0031] The method for quickly parsing compressed file content provided by the present invention has the following beneficial effects compared with the prior art:

[0032] Based on shared memory, message queues and parallel threads, decompressing files and reading files can be performed simultaneously, which greatly improves the efficiency of parsing the compressed file content.

[0033] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0035] Figure 1 A flowchart of a method for quickly parsing compressed file contents according to Embodiment 1 of the present invention;

[0036] Figure 2 Schematic diagram of the process of decompressing a compressed file and reading a file according to the first embodiment of the present invention;

[0037] Figure 3 This is a flow chart of a method for quickly parsing compressed file contents according to Embodiment 2 of the present invention;

[0038] Figure 4 Schematic diagram of the process of decompressing a compressed file and reading a file according to the second embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of a system for quickly parsing compressed file contents according to Embodiment 3 of the present invention;

[0040] Figure 6 This is a schematic diagram of an application scenario of a system for quickly parsing compressed file contents according to Embodiment 3 of the present invention;

[0041] Figure 7 FIG. 4 is a schematic diagram of the structure of an electronic device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0044] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0045] It should be noted that the concepts of “first”, “second”, etc. may be mentioned in the present invention only to distinguish different devices, components or parts, and are not used to limit the order or interdependence of the functions performed by these devices, components or parts.

[0046] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". "Plurality" should be understood as two or more.

[0047] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0048] Example 1

[0049] In an embodiment of the present invention, a method for quickly parsing the content of a compressed file is provided, comprising: decompressing the compressed file through a decompression thread pool to obtain at least one decompressed data and storing it in a memory block queue; obtaining a memory block from the queue through a reading thread pool, and reading the data of the memory block into a database; judging whether all the data of the memory block has been read in, and if so, placing the memory block in an empty memory block queue, otherwise temporarily storing the unread memory block and updating the temporarily stored memory block, and after all the data of the memory block has been read in, placing the memory block in an empty queue.

[0050] Figure 1 This is a flow chart of a method for quickly parsing compressed file contents according to Embodiment 1 of the present invention. Figure 2 FIG. 1 is a schematic diagram of the process of decompressing a compressed file and reading a file according to Embodiment 1 of the present invention (the solid arrows in the figure indicate the direction of data flow, and the dotted arrows indicate the direction of empty memory block flow). Figure 1 and Figure 2 The method for quickly parsing compressed files of the present invention is further described in detail.

[0051] In step 101 , the compressed file 201 is decompressed by the decompression thread pool 202 to obtain at least one decompressed data and store it in the memory block queue 203 .

[0052] In an embodiment of the present invention, a plurality of threads for performing decompression tasks are maintained through a decompression thread pool 201. After decompressing the compressed file 201 through any idle thread, decompressed data is obtained and put into a memory block, and a memory block queue 203 is maintained.

[0053] In step 102, a memory block is obtained from the memory block queue 203 through the reading thread pool 204, and the data of the memory block is read into the database.

[0054] In the embodiment of the present invention, a plurality of threads for executing read and write tasks are maintained by the read thread pool 201 , and the plurality of threads obtain memory blocks from the memory block queue 203 and read the data in the memory blocks into the database.

[0055] In step 103, it is determined whether all the data of the memory block has been read in. If so, the memory block is placed in an empty memory block queue, otherwise the unread memory block is temporarily stored and the temporarily stored memory block is updated, and when all the data of the memory block has been read in, the memory block is placed in an empty queue.

[0056] The method for quickly parsing the contents of compressed files provided in this embodiment adopts a multi-threaded approach in both the decompression and reading stages, based on shared memory and queue transmission to reduce data processing time; with a small increase in memory, the performance of reading compressed files can be greatly improved, and as the number of threads increases, the efficiency of reading compressed files can be multiplied, saving program running time and greatly improving user experience.

[0057] Example 2

[0058] The difference between this embodiment and the above embodiment lies in that: the compressed file is decompressed by the decompression thread pool, and after obtaining the decompressed data, the dependency of the decompressed data is marked and a data queue is maintained; a preprocessing thread pool is added to extract the data in the data queue, determine the dependency of the data, mark the range of the data block, and put it into the memory block queue, and then process the data blocks in the memory block queue through the reading thread pool.

[0059] Figure 3 This is a flow chart of a method for quickly parsing compressed file contents according to Embodiment 2 of the present invention. Figure 4 FIG. 1 is a schematic diagram of the process of decompressing a compressed file and reading a file according to Embodiment 2 of the present invention (the solid arrows in the figure indicate the direction of data flow, and the dotted arrows indicate the direction of empty memory block flow). Figure 3 and Figure 4 The method for quickly parsing compressed files of the present invention is further described in detail.

[0060] In step 301, the compressed file 401 is decompressed by the decompression thread pool 402 to obtain at least one decompressed data.

[0061] In step 302 , the dependency of the decompressed data is marked and put into a data queue 403 .

[0062] In the embodiment of the present invention, there is a dependency relationship or a non-dependency relationship between the contents of the compressed file 401, wherein the dependency relationship means that the subsequent contents in the compressed file 401 use the previously defined contents. When these contents are read into the database through the read-in thread pool 404, if the previously defined contents are not read in first, the subsequent contents cannot be correctly parsed, so the two parts of the contents with a dependency relationship need to be read in serial; the non-dependency relationship means that the subsequent contents do not depend on the previously defined contents, and the subsequent contents can be read in first, that is, the two parts of the non-dependency relationship can be read in parallel. Therefore, the data put into the memory block queue 407 needs to be marked with a dependency relationship with the previous data.

[0063] In step 303 , the data in the data queue 403 is extracted through the pre-processing thread pool 406 , and after determining the dependency of the data and marking the range of the data block, the data is put into the memory block queue 407 .

[0064] In the embodiment of the present invention, in order to identify the dependency between file contents, a preprocessing thread pool 406 is added. The preprocessing thread pool 406 obtains different data blocks by judging the dependency of data and marking the range of data blocks through the preprocessing module, that is, the preprocessing module not only judges the dependency of file contents, but also marks the range of each data block. The range of the data block includes:

[0065] Starting point: memory block number + address within the block;

[0066] Intermediate content: complete memory block (0 or more);

[0067] End point: memory block number + address within the block;

[0068] That is, the preprocessing module outputs a data block with the following structure:

[0069] {

[0070] bool is_depend_previous_;

[0071] int data_type_

[0072] std::vector <std::pair<memory_block_number,address> >data_content_;

[0073] }

[0074] In the embodiment of the present invention, the preprocessing thread pool 406 also adopts multi-thread parallel technology. After each thread processes a memory block, it puts the corresponding data block into the memory block queue 407 according to the memory block number. It should be noted that there are the following special scenarios in this link:

[0075] The statement spans multiple memory blocks, that is, the last statement of a memory block is incomplete, and part of the data is in the next memory block. When the preprocessing thread processes the last statement, if it is found that the statement is incomplete, the data of the next memory block can be read until the statement is processed completely. When the preprocessing thread processes the first statement of a memory block, if it is found that the statement is incomplete, the incomplete statement is ignored, and the parsing is started again until a new statement is found. If the file format is special and it is impossible to determine whether there is a new statement in the middle of the file, the preprocessing module cannot use the multi-threaded preprocessing thread pool 406.

[0076] In step 304, the data blocks in the memory block queue 407 are processed by the read thread pool 404, and the contents of the data blocks are read into the database.

[0077] In the embodiment of the present invention, the multi-threaded read-in thread pool 404 obtains the data blocks in the memory block queue 407, and decides to process the data blocks in series or in parallel according to the dependency relationship in the data blocks: if the current data block is dependent on the previous data block, the current data block is processed after the previous data block is processed, that is, the current data block needs to be processed in series; if the current data block is not dependent on the previous data block, the read-in thread pool 404 processes the current data block in parallel. After the data block processing is completed, if all the data of the memory block has been read in, the memory block is placed in the empty queue 405. If it is not processed, the unread memory block is temporarily stored, or the temporarily stored memory block is updated. When all the data of the memory block has been read in, the memory block is placed in the empty queue.

[0078] This embodiment adopts multiple technologies to realize fast parsing of compressed files, such as adopting memory sharing and queue mode to transfer data, which greatly saves the time used for data copying; the parsing process of compressed files is divided into three stages, namely decompression, preprocessing and reading, and a multi-threading mode is adopted in each stage to reduce the time of data processing; the present invention also takes into account data dependencies, introduces a data block queue that can express dependencies, solves the problem of direct data dependencies, and better enriches the application scenarios of multi-threaded parallelism.

[0079] The method for quickly parsing compressed file contents provided by the present invention can greatly improve the performance of reading compressed files when the memory is slightly increased. As the number of threads increases, the efficiency of reading compressed files can be increased exponentially, saving program running time and greatly improving user experience.

[0080] Example 3

[0081] In an embodiment of the present invention, a system for quickly parsing the contents of a compressed file is also provided. Figure 5 FIG. 1 is a schematic diagram of the structure of a system for quickly parsing compressed file contents according to Embodiment 3 of the present invention. Figure 5 As shown, the system 500 for quickly parsing compressed file content of the present invention includes:

[0082] The decompression module 501 is used to decompress the compressed file through the decompression thread pool to obtain at least one decompressed data, mark the dependency relationship of the decompressed data, and store it in the data queue;

[0083] The preprocessing module 502 is used to obtain data from the data queue, determine the dependency of the data, mark the range of the data, obtain the data block and put it into the memory block queue;

[0084] The reading module 503 is used to obtain a memory block from the memory block queue, and read the data of the memory block into the database in parallel or serially according to the dependency relationship of the data in the memory block to obtain a decompressed file.

[0085] In one embodiment, the range of the data block includes:

[0086] Starting point: memory block number + address within the block;

[0087] Intermediate content: complete memory block (0 or more);

[0088] End point: memory block number + address within the block;

[0089] The pre-processing module 502 outputs a data block with the following structure:

[0090] {

[0091] bool is_depend_previous_;

[0092] int data_type_

[0093] std::vector <std::pair<memory_block_number,address> >data_content_;

[0094] }

[0095] The read-in module 503 uses a multi-threaded read-in thread pool to process data blocks. In the process of reading the content of the data block into the database, if the current data block is dependent on the previous data block, the current data block will be processed after the previous data block is processed, that is, the current data block will be processed serially by the read-in thread pool. If the current data block is not dependent on the previous data block, the current data block will be processed in parallel by the read-in thread pool. After the data block processing is completed, if all data in the corresponding memory block has been read in, the memory block is placed in an empty queue.

[0096] The system for quickly parsing compressed file content of the present invention can be applied to any data receiving scenario. Figure 6 FIG. 1 is a schematic diagram of an application scenario of a system for quickly parsing compressed file contents according to Embodiment 3 of the present invention. Figure 6 As shown, the first terminal 601 sends a compressed file to the second terminal 602. The second terminal 602 receives the compressed file and decompresses, preprocesses and reads the compressed file through the system 500 that quickly parses the compressed file content, which greatly improves the efficiency of the second terminal 602 in decompressing and reading compressed files.

[0097] Example 4

[0098] The difference between this embodiment and embodiment 3 is that there is no preprocessing module. The system for quickly parsing compressed file content in this embodiment includes: a decompression module, which is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one decompressed data and put it into a memory block queue;

[0099] The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory block into the database, and put the memory block with completely read data into the empty memory block queue; determine whether all the data in the memory block has been read, if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and put the memory block into the empty queue after all the data in the memory block has been read.

[0100] Example 5

[0101] In an embodiment of the present invention, an electronic device is also provided. Figure 7 FIG. 1 is a schematic diagram of the structure of an electronic device according to Embodiment 4 of the present invention. Figure 7 As shown, the electronic device of the present invention includes a processor 701 and a memory 702, wherein:

[0102] The memory 702 stores a computer program. When the computer program is read and executed by the processor 701, the computer program executes the steps in the embodiment of the method for quickly parsing compressed file content as described above.

[0103] Example 6

[0104] In an embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of the method embodiment for quickly parsing compressed file content as described above when running.

[0105] In this embodiment, the above-mentioned computer-readable storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0106] Those skilled in the art can understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for quickly parsing the contents of a compressed file, characterized in that: The following steps are involved: Decompress the compressed file through the decompression thread pool, obtain at least one decompressed data and store it in the memory block queue; Obtaining a memory block from the memory block queue through a reading thread pool, and reading the data of the memory block into a database; Determine whether all the data of the memory block has been read in, if so, put the memory block into an empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and then put the memory block into an empty queue after all the data of the memory block has been read in; The decompression thread pool and the reading thread pool respectively maintain a plurality of threads.

2. The method for quickly parsing compressed file contents according to claim 1, characterized in that: Also includes: Marking the dependency or non-dependency of the decompressed data and putting it into a data queue; The data in the data queue is extracted through a preprocessing thread pool, and after determining the dependency of the data and marking the range of the data block, the data is put into a memory block queue; the preprocessing thread pool maintains a plurality of threads.

3. The method for quickly parsing compressed file contents according to claim 2, characterized in that: The dependency relationship means that the later content in the compressed file uses the previously defined content, so the later content data has a dependency relationship with the previously defined content data, and when reading data into the database through the reading thread pool, it needs to be read in serial; the non-dependency relationship means that the later content does not depend on the previously defined content, and when reading data into the database through the reading thread pool, the two parts of the non-dependent content data are read in parallel.

4. The method for quickly parsing compressed file contents according to claim 2, characterized in that: The scope of the data block includes: Starting point: memory block number + address within the block; Intermediate content: 0 or more complete memory blocks; End point: memory block number + address within the block.

5. The method for quickly parsing compressed file contents according to claim 4, characterized in that: In the preprocessing thread pool, each thread puts the corresponding data block into the memory block queue according to the memory block number.

6. The method for quickly parsing compressed file contents according to claim 5, characterized in that: If a statement spans multiple memory blocks, when the preprocessing thread processes the last statement in a memory block and finds that the statement is incomplete, it reads the data of the next memory block until the statement is processed completely; When the preprocessing thread processes the first statement of the memory block and finds that the statement is incomplete, it ignores the incomplete statement until a new statement is found and then starts parsing.

7. A system for quickly parsing compressed file contents, characterized in that: include: A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool to obtain at least one decompressed data, mark the dependency of the decompressed data, and store it in a data queue; A preprocessing module, used to obtain data from the data queue, determine the dependency of the data, mark the range of the data, obtain the data block and put it into the memory block queue; The reading module is used to obtain the memory block from the memory block queue, and read the data of the memory block into the database in parallel or serially according to the dependency relationship of the data in the memory block to obtain the decompressed file; the reading module is used to determine whether all the data of the memory block has been read in, and if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory block and update the temporarily stored memory block, and when all the data of the memory block has been read in, put the memory block into the empty queue.

8. A system for quickly parsing compressed file contents, characterized in that: include: A decompression module is used to decompress the compressed file through a multi-threaded decompression thread pool, obtain at least one decompressed data and put it into a memory block queue; The reading module is used to obtain memory blocks from the memory block queue through a multi-threaded reading thread pool, read the data in the memory blocks into the database, and put the memory blocks with completely read data into the empty memory block queue; the reading module is used to determine whether all the data in the memory block has been read in, if so, put the memory block into the empty memory block queue, otherwise temporarily store the unread memory blocks and update the temporarily stored memory blocks, and when all the data in the memory block has been read in, put the memory block into the empty queue.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is used to execute the computer program stored in the memory to implement the method steps for quickly parsing the content of a compressed file as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the method steps for quickly parsing compressed file content as described in any one of claims 1 to 6.

Images