Compressed file processing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for processing compressed files, which are used to improve the processing efficiency of data decompressed from compressed files. The main technical scheme includes: dividing the compressed file into multiple virtual files, each virtual file corresponds to a starting position and an ending position; the virtual file corresponds to a starting position and an ending position; the compressed file is compressed from a data file whose data volume exceeds a preset value; for each thread corresponding to the virtual file, read the target compressed data from the starting position to the target position of the compressed file, and the target position is within the preset range of the ending position of the corresponding virtual file; decompress the target compressed data to obtain decompressed data, and according to the positional relationship of multiple virtual files in the compressed file, obtain the target decompressed data corresponding to the virtual file in the decompressed data, and process the target decompressed data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for processing compressed files. Background Art

[0002] Compressed files usually encode the original data into a specified data format through a specific compression algorithm, so when reading, they also need to be decoded through a specific decompression algorithm. For compression algorithms with good compression ratios, data can be compressed to one-tenth of its original size.

[0003] The decompression of compressed files is to extract data files of corresponding sizes from the compressed files. If the amount of data in the decompressed data files is large, the subsequent processing efficiency of the data files is low. Therefore, a method for processing compressed files is urgently needed to improve the processing efficiency of data decompressed from compressed files. Summary of the invention

[0004] In view of this, the present application provides a method, apparatus, computer device and storage medium for processing compressed files, so as to improve the processing efficiency of data decompressed from compressed files.

[0005] In a first aspect, an embodiment of the present application provides a method for processing a compressed file, the method comprising:

[0006] Dividing the compressed file into a plurality of virtual files, wherein the virtual files have corresponding starting positions and ending positions; the compressed files are formed by compressing data files whose data volume exceeds a preset value;

[0007] For each thread corresponding to the virtual file, read the target compressed data from the starting position to the target position of the compressed file, wherein the target position is within a preset range of the end position of the corresponding virtual file;

[0008] The target compressed data is decompressed to obtain decompressed data, and according to the position relationship of the multiple virtual files in the compressed file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained, and the target decompressed data is processed.

[0009] In an optional embodiment, for each thread corresponding to the virtual file, reading the target compressed data from the starting position to the target position of the compressed file includes:

[0010] For each thread corresponding to the virtual file, the compressed data is sequentially read from the starting position of the compressed file until the file pointer points to the ending position corresponding to the virtual file; the file pointer is used to indicate the current reading position of the virtual file;

[0011] Reading compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file;

[0012] The compressed data from the start position to the target position of the compressed file is determined as the target compressed data.

[0013] In an optional embodiment, for each thread corresponding to the virtual file, sequentially reading compressed data from the starting position of the compressed file until the file pointer points to the end position corresponding to the virtual file includes:

[0014] Read compressed data of a first preset length from a file pointer corresponding to the compressed file into a cache;

[0015] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a first distance length;

[0016] If it is determined that the difference is less than the first distance length, the compressed data of a second preset length is read from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read; the second preset length is less than the first preset length.

[0017] In an optional embodiment, before continuing to read compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache, the method further includes:

[0018] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a second distance length; the second distance length is less than the first distance length;

[0019] If it is determined that the difference is less than the second distance length, reading compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read;

[0020] If it is determined that the difference is greater than or equal to the second distance length, the compressed data of the third preset length is read from the file pointer corresponding to the compressed file into the cache, and the step of jumping to determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than the second distance length is continued; the third preset length is greater than or equal to the second preset length and less than the first preset length.

[0021] In an optional embodiment, the reading of compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file includes:

[0022] Read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes;

[0023] The compressed data from the starting position of the compressed file to the ending position of the virtual file and the regional compressed data are taken together as the intermediate compressed data;

[0024] If the intermediate compressed data cannot be decompressed successfully, then N=N+1, and the process jumps to read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file and having a length of N bytes, and continues to execute;

[0025] If the intermediate compressed data can be successfully decompressed, the position corresponding to N bytes after the end position of the virtual file is determined as the target position.

[0026] In an optional embodiment, the first preset length is M times the length of the first distance, and M is less than 1.

[0027] In an optional embodiment, acquiring target decompressed data corresponding to the virtual file in the decompressed data according to the positional relationship of the multiple virtual files in the compressed file includes:

[0028] Obtaining the target location of the previous virtual file corresponding to the virtual file;

[0029] According to the target position corresponding to the virtual file and the target position of the previous virtual file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained.

[0030] In an optional embodiment, acquiring target decompressed data corresponding to the virtual file in the decompressed data according to the positional relationship of the multiple virtual files in the compressed file includes:

[0031] Acquire a first position where the last preset delimiter in the decompressed data is located, and a second position where the last preset delimiter is located before the decompressed data of a previous virtual file corresponding to the virtual file;

[0032] The data between the second position and the first position in the decompressed data is determined as the target decompressed data corresponding to the virtual file.

[0033] In an optional embodiment, acquiring target decompressed data corresponding to the virtual file in the decompressed data according to the positional relationship of the multiple virtual files in the compressed file includes:

[0034] Acquire a third position where the first preset delimiter in the decompressed data is located, and a fourth position where the first preset delimiter after the target position of the virtual file is located;

[0035] The data between the third position and the fourth position in the decompressed data are determined as target decompressed data corresponding to the virtual file.

[0036] In an optional embodiment, the preset separator includes at least one of the following: a punctuation mark, a preset character string, and a preset paragraph mark.

[0037] In a second aspect, an embodiment of the present application further provides a compressed file processing device, the device comprising:

[0038] A division module, used for dividing a compressed file into a plurality of virtual files, each of which has a corresponding start position and an end position; the compressed file is formed by compressing a data file whose data volume exceeds a preset value;

[0039] A reading module, for reading target compressed data from a starting position to a target position of the compressed file for each thread corresponding to the virtual file, wherein the target position is within a preset range of an end position of the virtual file;

[0040] A decompression module, used for decompressing the target compressed data to obtain decompressed data;

[0041] An acquisition module, configured to acquire target decompressed data corresponding to the virtual file in the decompressed data according to the position relationship of the multiple virtual files in the compressed file;

[0042] A processing module is used to process the target decompressed data.

[0043] In an optional embodiment, the reading module is specifically used to:

[0044] For each thread corresponding to the virtual file, the compressed data is sequentially read from the starting position of the compressed file until the file pointer points to the ending position corresponding to the virtual file; the file pointer is used to indicate the current reading position of the virtual file;

[0045] Reading compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file;

[0046] The compressed data from the start position to the target position of the compressed file is determined as the target compressed data.

[0047] In an optional embodiment, the reading module is specifically used for:

[0048] Read compressed data of a first preset length from a file pointer corresponding to the compressed file into a cache;

[0049] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a first distance length;

[0050] If it is determined that the difference is less than the first distance length, the compressed data of a second preset length is read from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read; the second preset length is less than the first preset length.

[0051] In an optional embodiment, the reading module is specifically used to:

[0052] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a second distance length; the second distance length is less than the first distance length;

[0053] If it is determined that the difference is less than the second distance length, reading compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read;

[0054] If it is determined that the difference is greater than or equal to the second distance length, the compressed data of the third preset length is read from the file pointer corresponding to the compressed file into the cache, and the step of jumping to determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than the second distance length is continued; the third preset length is greater than or equal to the second preset length and less than the first preset length.

[0055] In an optional embodiment, the reading module is specifically used to:

[0056] Read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes;

[0057] The compressed data from the starting position of the compressed file to the ending position of the virtual file and the regional compressed data are taken together as the intermediate compressed data;

[0058] If the intermediate compressed data cannot be decompressed successfully, then N=N+1, and the process jumps to read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file and having a length of N bytes, and continues to execute;

[0059] If the intermediate compressed data can be successfully decompressed, the position corresponding to N bytes after the end position of the virtual file is determined as the target position.

[0060] In an optional embodiment, the first preset length is M times the length of the first distance, and M is less than 1.

[0061] In an optional embodiment, the acquisition module is specifically used to:

[0062] Obtaining the target location of the previous virtual file corresponding to the virtual file;

[0063] According to the target position corresponding to the virtual file and the target position of the previous virtual file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained.

[0064] In an optional embodiment, the acquisition module is specifically used to:

[0065] Acquire a first position where the last preset delimiter in the decompressed data is located, and a second position where the last preset delimiter is located before the decompressed data of a previous virtual file corresponding to the virtual file;

[0066] The data between the second position and the first position in the decompressed data is determined as the target decompressed data corresponding to the virtual file.

[0067] In an optional embodiment, the acquisition module is specifically used to:

[0068] Acquire a third position where the first preset delimiter in the decompressed data is located, and a fourth position where the first preset delimiter after the target position of the virtual file is located;

[0069] The data between the third position and the fourth position in the decompressed data are determined as target decompressed data corresponding to the virtual file.

[0070] In an optional embodiment, the preset separator includes at least one of the following: a punctuation mark, a preset character string, and a preset paragraph mark.

[0071] In a third aspect, an embodiment of the present application further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the compressed file processing method of the first aspect.

[0072] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the compressed file processing method in the first aspect are executed.

[0073] The present application provides a method, device, computer equipment and storage medium for processing compressed files. First, the compressed file is divided into multiple virtual files, each of which corresponds to a starting position and an ending position. Then, for each virtual file, different threads are used to read the target compressed data from the starting position to the target position of the compressed file, and the target position is within the preset range of the ending position of the virtual file. Finally, the target compressed data is decompressed to obtain decompressed data. According to the positional relationship of multiple virtual files in the compressed file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained, and the target decompressed data is processed. Compared with the current data files that can only decompress the original data volume from the compressed file, the present application splits the data files whose original data volume exceeds a certain value into multiple data files during the data decompression process, that is, the present application decompresses the compressed file in parallel through multiple threads to obtain multiple decompressed data, and then processes the decompressed data in parallel through threads, so that the present application can improve the processing efficiency of the data decompressed from the compressed file.

[0074] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A flowchart of a method for processing a compressed file provided in an embodiment of the present application is shown;

[0077] Figure 2 A flow chart of a method for reading target compressed data provided by an embodiment of the present application is shown;

[0078] Figure 3 A flowchart of reading compressed data provided by an embodiment of the present application is shown;

[0079] Figure 4 A structural block diagram of a compressed file processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0080] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects rather than to limit a specific order.

[0081] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0082] In the description of this application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0083] In the embodiments of the present application, at least one can also be described as one or more, and multiple can be two, three, four or more, which is not limited in the present application.

[0084] like Figure 1 As shown, an embodiment of the present application provides a method for processing a compressed file. The method for processing a compressed file provided by the present application may include:

[0085] S101: Divide a compressed file into a plurality of virtual files, each of the virtual files corresponding to a start position and an end position.

[0086] Among them, the compressed file is compressed from a data file, and the data volume of the data file exceeds the preset value. The preset value is set according to actual needs, and can be 1G, 5G, 10G, etc., which is not specifically limited in this application. In this embodiment, the starting position and the ending position are used to indicate the relative offset position of the virtual file in the compressed file, that is, the starting position and the ending position of the file corresponding to the virtual file in the compressed file, such as the starting position is the position of the file pointer at 10000; the ending position is the position of the file pointer at 20000. The virtual file in this embodiment does not really divide the compressed file into multiple files, but provides a basis for obtaining multiple target decompressed data later, that is, the corresponding target decompressed data is determined from the compressed file according to the starting position and the ending position of each virtual file. Specifically, this embodiment can determine the number of virtual files to be divided according to the data volume of the compressed file or the data file, or determine the number of virtual files to be divided according to the number of parallel processed data files, which is not specifically limited in this embodiment.

[0087] For example, if the data volume of a compressed file is 3G and there are 3 threads that can process the compressed file in parallel, the compressed file can be divided into 3 virtual files, where the starting position to the end position of the first virtual file is 0-1G, the starting position to the end position of the second virtual file is 1G-2G, and the starting position to the end position of the third virtual file is 2G-3G.

[0088] For another example, the data size of a compressed file obtained by compressing a data file with a data size of 4G is 2G. If the data size of the parallel processed data file is 1G, the compressed file is divided into 4 virtual files. Among them, the starting position to the end position of the first virtual file is 0-0.5G, the starting position to the end position of the second virtual file is 0.5G-1G, the starting position to the end position of the third virtual file is 1G-1.5G, and the starting position to the end position of the fourth virtual file is 1.5G-2G.

[0089] S102 : For each thread corresponding to the virtual file, read target compressed data from the starting position to the target position of the compressed file.

[0090] The starting address of the compressed file is used to indicate the starting position of the data of the compressed file (such as the starting position is 0), and the target position corresponds to the preset range of the end position of the virtual file, and the preset range is a certain number of bytes before or after the origin with the end position of the virtual file as the origin. Specifically, the preset range can be set to a fixed value, such as 4 bytes, 8 bytes, 32 bytes, 64 bytes, etc.; it can also be determined according to the compression algorithm corresponding to the compressed file. This embodiment does not specifically limit the preset range.

[0091] In this embodiment, the target compressed data is determined by multiple threads, and each thread can process one or more virtual files, that is, each thread can determine at least one target compressed data. For example, the compressed file is divided into three virtual files, wherein the starting position to the end position of the first virtual file is 0-1G, the starting position to the end position of the second virtual file is 1G-2G, and the starting position to the end position of the third virtual file is 2G-3G. If three threads are used to process the compressed file, each thread processes one virtual file. Specifically, thread 1 reads the target compressed data within the preset range (within the preset range of 0-1G) from the starting position of the compressed file to the end position of the first virtual file, thread 2 reads the target compressed data within the preset range (within the preset range of 0-2G) from the starting position of the compressed file to the end position of the second virtual file, and thread 3 reads the target compressed data within the preset range (within the preset range of 0-3G) from the starting position of the compressed file to the end position of the third virtual file.

[0092] S103, decompressing the target compressed data to obtain decompressed data, acquiring target decompressed data corresponding to the virtual file in the decompressed data according to the position relationship of the multiple virtual files in the compressed file, and processing the target decompressed data.

[0093] The target decompressed data is included in the data file, that is, the target decompressed data is a subset of the data file. Processing the target decompressed data includes but is not limited to the following processing: data query operation, image processing operation, video processing operation and other operations that require more time.

[0094] It should be noted that the contents corresponding to step S102 and step S103 in this embodiment are all executed for the thread corresponding to each virtual file. That is, for the thread corresponding to each virtual file, the target compressed data from the starting position to the target position of the compressed file is first read, and then the target compressed data is decompressed to obtain decompressed data, and according to the position relationship of the virtual file in the compressed file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained, and the target decompressed data is processed.

[0095] In an optional embodiment provided in the present application, obtaining the target decompressed data corresponding to the virtual file in the decompressed data includes: obtaining the target position of the previous virtual file corresponding to the virtual file; and obtaining the target decompressed data corresponding to the virtual file in the decompressed data based on the target position corresponding to the virtual file and the target position of the previous virtual file.

[0096] For example, the starting position to the end position of the first virtual file is 0-1G, the starting position to the end position of the second virtual file is 1G-2G, and the starting position to the end position of the third virtual file is 2G-3G. If thread 1 reads the target compressed data from the starting position of the compressed file to the target position 1.01G of the first virtual file (the target position 1.01G is within the preset range of the end position of the first virtual file); thread 2 reads the target compressed data from the starting position of the compressed file to the target position 2.03G of the second virtual file; thread 3 reads the target compressed data from the starting position of the compressed file to the end position 3G of the third virtual file. After each thread decompresses the acquired target compressed data to obtain decompressed data, the thread obtains the target decompressed data corresponding to the virtual file in the decompressed data according to the positional relationship of the virtual file in the compressed file. Since the target position of the first virtual file in this example is 1.01G and the target position of the second virtual file is 2.03G, according to the target position corresponding to the virtual file and the target position of the previous virtual file, the target decompression data of the first virtual file is (0-1.01G), the target decompression data of the second virtual file is (1.01G-2.03G), and the target decompression data of the third virtual file is (2.03G-3G).

[0097] It should be noted that, since the present embodiment adopts a multi-thread processing mechanism, and the amount of data read and decompressed by each thread is different, the target position thread is determined first, and the target position can be sent to another thread corresponding to it, so that the other thread can determine the target decompressed data according to the target position. For the above example, after thread 1 determines the target position of the first virtual file, it sends the target position of the first virtual file to thread 2, and after thread 2 determines the target position of the second virtual file, it sends the target position of the second virtual file to thread 3. In this way, the target decompressed data corresponding to each virtual file is determined.

[0098] In an optional embodiment provided by the present application, the step of obtaining the target decompressed data corresponding to the virtual file in the decompressed data includes: obtaining the first position where the last preset delimiter in the decompressed data is located, and the second position where the first preset delimiter is located before the target position of the previous virtual file corresponding to the virtual file is located; and determining the data between the second position and the first position in the decompressed data as the target decompressed data corresponding to the virtual file. The preset delimiter includes at least one of the following: punctuation marks, preset character strings, and preset paragraph marks.

[0099] For example, a compressed file is divided into three virtual files, namely the first virtual file, the second virtual file and the third virtual file. First, the position of the last preset delimiter in the decompressed data corresponding to the first virtual file is determined to be position a, and the position of the last preset delimiter in the decompressed data corresponding to the second virtual file is determined to be position b. Then, the target decompressed data corresponding to the first virtual file is from the starting position to position a in the decompressed data, the target decompressed data corresponding to the second virtual file is from position a to position b in the decompressed data, and the target decompressed data corresponding to the third virtual file is from position b in the decompressed data to the end position of the decompressed data.

[0100] In an optional embodiment provided by the present application, the step of obtaining the target decompressed data corresponding to the virtual file in the decompressed data includes: obtaining the third position where the first preset delimiter in the decompressed data is located, and the fourth position where the first preset delimiter after the target position of the virtual file is located; and determining the data between the third position and the fourth position in the decompressed data as the target decompressed data corresponding to the virtual file. It should be noted that if the virtual file is the first virtual file or the last virtual file corresponding to the compressed file, the target decompressed data corresponding to the first virtual file is from the starting position to position A in the decompressed data, and the target decompressed data corresponding to the last virtual file is from position B in the decompressed data to the end position of the decompressed data. Among them, position A is the position where the first preset delimiter in the decompressed data corresponding to the second virtual file is located, and position B is the position where the first preset delimiter after the target position of the second-to-last virtual file is located.

[0101] For example, a compressed file is divided into three virtual files, namely the first virtual file, the second virtual file and the third virtual file. First, determine the first position a where the first preset delimiter in the decompressed data corresponding to the second virtual file is located, and the second position b where the first preset delimiter after the target position of the second virtual file is located. Then, the target decompressed data corresponding to the first virtual file is from the starting position to position a in the decompressed data, the target decompressed data corresponding to the second virtual file is from position a to position b in the decompressed data, and the target decompressed data corresponding to the third virtual file is from position b in the decompressed data to the end position of the decompressed data.

[0102] After obtaining the target decompressed data corresponding to the virtual file, this embodiment needs to verify whether the target decompressed data is successfully decompressed data. Specifically, by matching the target decompressed data with the data file, it is determined whether the target decompressed data is a subset of the data file. If the target decompressed data is a subset of the data file, the target decompressed data can be processed; if the target decompressed data is not a subset of the data file, it means that there is an error in the decompressed target decompressed data. At this time, an alarm message can be output to prompt the user that the compressed file decompression failed.

[0103] The embodiment of the present application provides a method for processing a compressed file, firstly, the compressed file is divided into a plurality of virtual files, each of which corresponds to a starting position and an ending position, then a different thread is used for each virtual file to read the target compressed data from the starting position to the target position of the compressed file, the target position is within the preset range of the ending position of the virtual file, and finally the target compressed data is decompressed to obtain decompressed data, according to the positional relationship of the plurality of virtual files in the compressed file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained, and the target decompressed data is processed. Compared with the current data files that can only decompress the original data amount from the compressed file, the present application splits the data files whose original data amount exceeds a certain value into a plurality of data files during the decompression process, that is, the present application decompresses the compressed file in parallel through multiple threads to obtain a plurality of decompressed data, and then processes the decompressed data in parallel through threads, so that the present application can improve the processing efficiency of the data decompressed from the compressed file.

[0104] like Figure 2 As shown, in an optional embodiment provided by the present invention, the process of reading target compressed data is as follows:

[0105] S201 . For each thread corresponding to a virtual file, sequentially read compressed data from a starting position of the compressed file until a file pointer points to an end position corresponding to the virtual file.

[0106] The file pointer is used to indicate the current reading position of the virtual file, and the file pointer is updated after each pair of compressed data.

[0107] like Figure 3 As shown, for each thread corresponding to the virtual file, the compressed data is sequentially read from the starting position of the compressed file until the file pointer points to the end position corresponding to the virtual file, including:

[0108] S21. Read compressed data of a first preset length from a file pointer corresponding to the compressed file into a cache.

[0109] The first preset length may be the size of the data loaded from the disk each time, and the first preset length may be 1KB, 2KB, etc., which is not specifically limited in this embodiment. Further, the first preset length is determined according to the data length of the virtual file.

[0110] S22: Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a first distance length.

[0111] The difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is the distance that the file pointer will reach the end position corresponding to the virtual file. Specifically, the first preset length is M times the first distance length, where M is less than 1. Preferably, the first distance length is usually 1.1 times the first preset length.

[0112] S23: If it is determined that the difference is less than the first distance length, read compressed data of a third preset length from the file pointer corresponding to the compressed file into the cache.

[0113] The third preset length is greater than or equal to the second preset length and less than the first preset length.

[0114] In this embodiment, if it is determined that the difference is greater than or equal to the first distance length, the process jumps to step S21 to read compressed data of a first preset length from the compressed file into the cache and continue the execution.

[0115] S24: Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a second distance length.

[0116] The second distance is shorter than the first distance.

[0117] S25: If it is determined that the difference is less than the second distance length, read compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache.

[0118] The first preset length may specifically be 1 byte.

[0119] On the contrary, if it is determined that the difference is greater than or equal to the second distance length, the process jumps to step S23 to read compressed data of a third preset length from the file pointer corresponding to the compressed file into the cache.

[0120] S26. Determine whether the file pointer is at the end position corresponding to the virtual file.

[0121] If the file pointer is not at the end position corresponding to the virtual file, jump to execute S25 to read the compressed data of the second preset length from the file pointer corresponding to the compressed file into the cache; if the file pointer is at the end position corresponding to the virtual file, the reading process of the target compressed data is terminated.

[0122] S202: Read compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file.

[0123] In an optional embodiment provided by the present application, reading compressed data starting from the end position of the virtual file to the target position after the end position of the virtual file includes: successively reading compressed data starting from the end position of the virtual file to the target position after the end position of the virtual file, wherein the data length between the target position and the end position of the virtual file is successively increased by bytes.

[0124] The compressed data starting from the end position of the virtual file to the target position after the end position of the virtual file is read one by one. If the virtual file is the last file in the compressed file, the target position is the end position of the compressed file.

[0125] In an optional embodiment provided by the present invention, the reading of compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file includes:

[0126] Read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes, where the initial value of N is 1 byte;

[0127] The compressed data from the starting position of the compressed file to the ending position of the virtual file and the regional compressed data are taken together as the intermediate compressed data;

[0128] If the intermediate compressed data cannot be decompressed successfully, then N=N+1, and the process jumps to read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file and having a length of N bytes, and continues to execute;

[0129] If the intermediate compressed data can be successfully decompressed, the position corresponding to N bytes after the end position of the virtual file is determined as the target position.

[0130] S203: Determine the compressed data from the start position to the target position of the read compressed file as target compressed data.

[0131] For the embodiment of the present application, the compressed data is first read sequentially from the starting position of the compressed file until the file pointer points to the end position corresponding to the virtual file, and then the compressed data starting from the end position of the virtual file to the target position after the end position of the virtual file is read, or the compressed data starting from the end position of the virtual file to the target position before the end position of the virtual file is read, and finally the compressed data from the starting position to the target position of the compressed file is determined as the target compressed data. Thus, the present application realizes dividing a compressed file into multiple copies of target compressed data, so that the multiple copies of target compressed data are processed in parallel through multi-threading, thereby improving the processing efficiency of the data decompressed from the compressed file through the embodiment of the present application.

[0132] The embodiment of the present application can divide the function modules of the compressed file processing device according to the above method example. For example, each function module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0133] In the case of dividing each functional module into corresponding functional modules, Figure 4 A possible schematic diagram of the composition of the compressed file processing device involved in the above embodiments is shown. Figure 4 As shown, the compressed file processing device may include: a division module 41, a reading module 42, a decompression module 43, an acquisition module 44, and a processing module 45.

[0134] A division module 41 is used to divide the compressed file into a plurality of virtual files, each of which has a corresponding start position and an end position; the compressed file is formed by compressing a data file whose data volume exceeds a preset value;

[0135] A reading module 42, for reading target compressed data from a starting position to a target position of the compressed file for each thread corresponding to the virtual file, wherein the target position is within a preset range of an end position of the virtual file;

[0136] A decompression module 43, used for decompressing the target compressed data to obtain decompressed data;

[0137] An acquisition module 44, configured to acquire target decompressed data corresponding to the virtual file in the decompressed data according to the position relationship of the multiple virtual files in the compressed file;

[0138] The processing module 45 is used to process the target decompressed data.

[0139] In an optional embodiment, the reading module 42 is specifically configured to:

[0140] For each thread corresponding to the virtual file, the compressed data is sequentially read from the starting position of the compressed file until the file pointer points to the ending position corresponding to the virtual file; the file pointer is used to indicate the current reading position of the virtual file;

[0141] Reading compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file;

[0142] The compressed data from the start position to the target position of the compressed file is determined as the target compressed data.

[0143] In an optional embodiment, the reading module 42 is specifically configured to:

[0144] Read compressed data of a first preset length from a file pointer corresponding to the compressed file into a cache;

[0145] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a first distance length;

[0146] If it is determined that the difference is less than the first distance length, the compressed data of a second preset length is read from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read; the second preset length is less than the first preset length.

[0147] In an optional embodiment, the reading module 42 is specifically configured to:

[0148] Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a second distance length; the second distance length is less than the first distance length;

[0149] If it is determined that the difference is less than the second distance length, reading compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read;

[0150] If it is determined that the difference is greater than or equal to the second distance length, the compressed data of the third preset length is read from the file pointer corresponding to the compressed file into the cache, and the step of jumping to determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than the second distance length is continued; the third preset length is greater than or equal to the second preset length and less than the first preset length.

[0151] In an optional embodiment, the reading module 42 is specifically configured to:

[0152] Read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes;

[0153] The compressed data from the starting position of the compressed file to the ending position of the virtual file and the regional compressed data are taken together as the intermediate compressed data;

[0154] If the intermediate compressed data cannot be decompressed successfully, then N=N+1, and the process jumps to read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file and having a length of N bytes, and continues to execute;

[0155] If the intermediate compressed data can be successfully decompressed, the position corresponding to N bytes after the end position of the virtual file is determined as the target position.

[0156] In an optional embodiment, the first preset length is M times the length of the first distance, and M is less than 1.

[0157] In an optional embodiment, the acquisition module 44 is specifically configured to:

[0158] Obtaining the target location of the previous virtual file corresponding to the virtual file;

[0159] According to the target position corresponding to the virtual file and the target position of the previous virtual file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained.

[0160] In an optional embodiment, the acquisition module 44 is specifically configured to:

[0161] Acquire a first position where the last preset delimiter in the decompressed data is located, and a second position where the last preset delimiter is located before the decompressed data of a previous virtual file corresponding to the virtual file;

[0162] The data between the second position and the first position in the decompressed data is determined as the target decompressed data corresponding to the virtual file.

[0163] In an optional embodiment, the acquisition module 44 is specifically configured to:

[0164] Acquire a third position where the first preset delimiter in the decompressed data is located, and a fourth position where the first preset delimiter after the target position of the virtual file is located;

[0165] The data between the third position and the fourth position in the decompressed data are determined as target decompressed data corresponding to the virtual file.

[0166] In an optional embodiment, the preset separator includes at least one of the following: a punctuation mark, a preset character string, and a preset paragraph mark.

[0167] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the compressed file processing method provided in the above embodiment are executed.

[0168] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned compressed file processing method can be executed. Compared with the current data files that can only decompress the original data volume from the compressed file, the present application splits the data file whose original data volume exceeds a certain value into multiple data files during the decompression process, that is, the present application decompresses the compressed file through multi-threaded parallelism to obtain multiple decompressed data, and then processes the decompressed data in parallel through threads, so that the present application can improve the processing efficiency of the data decompressed from the compressed file.

[0169] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0172] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0173] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for processing a compressed file, characterized in that: The method comprises: According to the number of threads for parallel processing of compressed files and the data volume of compressed files or data files, the compressed files are divided into a plurality of virtual files, wherein the virtual files have corresponding starting positions and ending positions; the compressed files are formed by compressing data files whose data volume exceeds a preset value; For each thread corresponding to the virtual file, read the target compressed data from the starting position to the target position of the compressed file, wherein the target position is within a preset range of the end position of the corresponding virtual file; The target compressed data is decompressed to obtain decompressed data, and according to the position relationship of the multiple virtual files in the compressed file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained, and the target decompressed data is processed.

2. The method according to claim 1, characterized in that: The step of reading target compressed data from a starting position to a target position of the compressed file for each thread corresponding to the virtual file includes: For each thread corresponding to the virtual file, the compressed data is sequentially read from the starting position of the compressed file until the file pointer points to the ending position corresponding to the virtual file; the file pointer is used to indicate the current reading position of the virtual file; Reading compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file; The compressed data from the start position to the target position of the compressed file is determined as the target compressed data.

3. The method according to claim 2, characterized in that The thread corresponding to each virtual file sequentially reads compressed data from the starting position of the compressed file until the file pointer points to the ending position corresponding to the virtual file, including: Read compressed data of a first preset length from a file pointer corresponding to the compressed file into a cache; Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a first distance length; If it is determined that the difference is less than the first distance length, the compressed data of a second preset length is read from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read; the second preset length is less than the first preset length.

4. The method according to claim 3, characterized in that Before reading compressed data of a second preset length from a file pointer corresponding to the compressed file into a cache, the method further includes: Determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than a second distance length; the second distance length is less than the first distance length; If it is determined that the difference is less than the second distance length, reading compressed data of a second preset length from the file pointer corresponding to the compressed file into the cache until the end position of the virtual file is read; If it is determined that the difference is greater than or equal to the second distance length, the compressed data of the third preset length is read from the file pointer corresponding to the compressed file into the cache, and the step of jumping to determine whether the difference between the end position corresponding to the virtual file and the position corresponding to the file pointer is less than the second distance length is continued; the third preset length is greater than or equal to the second preset length and less than the first preset length.

5. The method according to claim 2, characterized in that: The reading of compressed data starting from the end position of the virtual file to a target position after the end position of the virtual file includes: Read compressed data of a region starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes; The compressed data from the starting position of the compressed file to the ending position of the virtual file and the regional compressed data are taken together as the intermediate compressed data; If the intermediate compressed data cannot be decompressed successfully, N=N+1, and jump to read the compressed data of the area starting from the end position of the virtual file to the end position of the virtual file with a length of N bytes to continue execution; If the intermediate compressed data can be successfully decompressed, the position corresponding to N bytes after the end position of the virtual file is determined as the target position.

6. The method according to claim 3, characterized in that The first preset length is M times the first distance length, and M is less than 1.

7. The method according to any one of claims 1 to 6, characterized in that: The acquiring, according to the positional relationship of the multiple virtual files in the compressed file, target decompressed data corresponding to the virtual files in the decompressed data comprises: Obtaining the target location of the previous virtual file corresponding to the virtual file; According to the target position corresponding to the virtual file and the target position of the previous virtual file, the target decompressed data corresponding to the virtual file in the decompressed data is obtained.

8. The method according to any one of claims 1 to 6, characterized in that: The acquiring, according to the positional relationship of the multiple virtual files in the compressed file, target decompressed data corresponding to the virtual files in the decompressed data comprises: Acquire a first position where the last preset delimiter in the decompressed data is located, and a second position where the last preset delimiter is located before the decompressed data of a previous virtual file corresponding to the virtual file; The data between the second position and the first position in the decompressed data is determined as the target decompressed data corresponding to the virtual file.

9. The method according to any one of claims 1 to 6, characterized in that: The acquiring, according to the positional relationship of the multiple virtual files in the compressed file, target decompressed data corresponding to the virtual files in the decompressed data comprises: Acquire a third position where the first preset delimiter in the decompressed data is located, and a fourth position where the first preset delimiter after the target position of the virtual file is located; The data between the third position and the fourth position in the decompressed data are determined as target decompressed data corresponding to the virtual file.

10. The method according to claim 8, characterized in that The preset separator includes at least one of the following: punctuation mark, preset character string, preset paragraph mark.

11. The method according to claim 9, characterized in that The preset separator includes at least one of the following: punctuation mark, preset character string, preset paragraph mark.

12. A compressed file processing device, characterized in that: The device comprises: A division module is used to divide a compressed file into a plurality of virtual files according to the number of threads for parallel processing of the compressed file and the data volume of the compressed file or the data file, wherein each of the virtual files has a corresponding start position and an end position; the compressed file is formed by compressing a data file whose data volume exceeds a preset value; A reading module, for reading target compressed data from a starting position to a target position of the compressed file for each thread corresponding to the virtual file, wherein the target position is within a preset range of an end position of the corresponding virtual file; A decompression module, used for decompressing the target compressed data to obtain decompressed data; An acquisition module, configured to acquire target decompressed data corresponding to the virtual file in the decompressed data according to the position relationship of the multiple virtual files in the compressed file; A processing module is used to process the target decompressed data.

13. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the compressed file processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for processing a compressed file according to any one of claims 1 to 11 are executed.

Images