Method and system for restoring HTTP (Hyper Text Transport Protocol) file in small file local storage scene

Abstract

The invention discloses a method and a system for restoring an HTTP (Hyper Text Transport Protocol) file in a small file local storage scene. The method comprises the following steps: acquiring a to-be-analyzed data packet; performing protocol identification on the to-be-analyzed data packet, and judging whether a transport layer protocol of the data packet is a TCP and an application layer protocol is an HTTP; if not, returning to obtain the to-be-analyzed data packet; if yes, recombining the TCP session in the data packet; and extracting the content of the HTTP file, checking whether compression or coding exists in the HTTP response or not according to the content of the HTTP file, and if the compression or coding exists, decoding or decompressing to obtain an original file. According to the method, the number of files restored and stored in a local disk by the HTTP file is reduced, the burden of a system disk is relieved, and the efficiency of IO operation is not influenced; in the continuous HTTP file restoration process, the overall performance of the system cannot be reduced along with sharp increase of the number of local disk files.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic data analysis, and in particular to an HTTP file restoration method, system, intelligent terminal and storage medium in a small file local storage scenario. Background Art

[0002] Hypertext Transfer Protocol (HTTP) is an application layer protocol for transferring hypertext data between clients and servers. It is the foundation of the Web, allowing users to communicate with Web servers through browsers. Users can send HTTP requests to WEB servers through browsers, receive HTML pages, CSS, JavaScript resources from servers, and download documents, pictures, audio and video files in other formats. With the continuous development of HTTP applications, HTTP has more and more application scenarios in the field of network communications, so it is of great significance to restore and store HTTP files in traffic.

[0003] The HTTP protocol has no limit on the size of the transmitted content. It can be a large file of hundreds of GB or a small file of a few bytes. In current application scenarios, the size of most transmitted content does not exceed 1K, or even only a few bytes, such as forms, pages, resource files, or a paragraph of text. Traditional HTTP file restoration restores the payload of each request and response into a file and stores it locally. This method will cause a sharp increase in the number of local files and will have the following effects:

[0004] 1. Memory usage. The operating system usually maintains some file system structures in memory. As the number of files in the directory increases, the size of these data structures may also increase, thus affecting the system's memory usage;

[0005] 2. Metadata overhead. Each file has certain metadata (such as permissions, timestamps, etc.). When the number of files increases, the overhead required to manage these metadata will also increase, which may lead to a decrease in overall performance;

[0006] 3. Degraded search performance. When there are a large number of files in a directory, it takes longer to find a specific file because more entries need to be traversed. This affects the efficiency of IO operations, especially in scenarios where file reading / writing is frequent;

[0007] 4. Fragmentation: As files are constantly added and deleted, the degree of fragmentation on the disk may increase, causing delays in IO operations.

[0008] In short, with HTTP restoration, more and more small files are stored locally, which will eventually cause system performance to degrade severely or even become unavailable. Summary of the invention

[0009] In view of the defects of the prior art, an embodiment of the present invention provides a method, system, intelligent terminal and storage medium for HTTP file restoration in a small file local storage scenario, which reduces the number of files stored in HTTP file restoration by aggregating and storing small files for HTTP file restoration.

[0010] The present invention is achieved through the following technical solutions:

[0011] In a first aspect, an embodiment of the present invention provides an HTTP file restoration method in a small file local storage scenario, comprising:

[0012] Get the data packet to be analyzed;

[0013] Perform protocol identification on the data packet to be analyzed to determine whether the transport layer protocol of the data packet is TCP and the application layer protocol is HTTP;

[0014] If not, return to obtain the data packet to be analyzed;

[0015] If yes, the TCP session in the data packet is reassembled;

[0016] Extract the HTTP file content, and check whether there is compression or encoding in the HTTP response according to the HTTP file content. If there is compression or encoding, decode or decompress it to obtain the original file.

[0017] In a second aspect, an HTTP file restoration system in a small file local storage scenario provided by an embodiment of this aspect includes: a traffic collection module, a protocol identification module, a TCP reassembly module and an HTTP file restoration module;

[0018] The traffic collection module is used to obtain data packets to be analyzed;

[0019] The protocol identification module is used to identify the protocol of the data packet to be analyzed;

[0020] The TCP reassembly module is used to reassemble the TCP session in the data packet;

[0021] The HTTP file restoration module is used to extract HTTP file content and check whether there is compression or encoding in the HTTP response according to the HTTP file content. If there is compression or encoding, decoding or decompression is performed to obtain the original file.

[0022] In a third aspect, an embodiment of the present invention provides an intelligent terminal, comprising a processor, an input device, an output device and a memory, wherein the processor is connected to the input device, the output device and the memory respectively, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method described in the above embodiment.

[0023] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the above embodiment.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] The embodiments of the present invention provide an HTTP file restoration method, system, intelligent terminal and storage medium in a small file local storage scenario, which reduces the number of files stored on the local disk during HTTP file restoration, alleviates the burden on the system disk, and does not affect the efficiency of IO operations; during the continuous HTTP file restoration process, the overall system performance will not decrease as the number of local disk files increases sharply. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 A flowchart of a method for restoring HTTP files in a small file local storage scenario provided by the first embodiment of the present invention;

[0028] Figure 2 A flowchart of locally storing an original file in an HTTP file restoration method in a small file local storage scenario provided by another embodiment of the present invention;

[0029] Figure 3 A flowchart of a small file download in an HTTP file restoration method in a small file local storage scenario provided by another embodiment of the present invention;

[0030] Figure 4 A structural block diagram of an HTTP file restoration system in a small file local storage scenario provided by another embodiment of the present invention;

[0031] Figure 5 A structural block diagram of an intelligent terminal provided in another embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0033] like Figure 1 As shown, a first embodiment of the present invention provides an HTTP file restoration method in a small file local storage scenario, comprising the following steps:

[0034] Obtaining data packets to be analyzed by reading the data packets to be analyzed on a disk or capturing mirrored traffic data packets on a target device or network node;

[0035] Perform protocol analysis on the captured data packets, and judge the protocol of the data packets based on the media and protocol characteristics to determine whether the transport layer protocol of the data packets is TCP and the application layer protocol is HTTP.

[0036] If not, return to obtain the data packet to be analyzed;

[0037] If yes, the TCP session in the data packet is reassembled;

[0038] Extract the HTTP file content, cache the HTTP payload portion in memory, and check whether there is compression (such as Gzip) or encoding (such as Base64) in the HTTP response based on the HTTP file content. If so, decode or decompress the response as needed to obtain the original file content.

[0039] Due to the complexity of the network environment, the original TCP session may have packet disorder, packet loss and retransmission. The TCP session data packets are reassembled through TCP SEQ numbers to restore the correct order of the data packets for subsequent file restoration.

[0040] like Figure 2 As shown, another embodiment of the present invention provides an HTTP file restoration method in a small file local storage scenario, which also includes local storage of the original file, specifically including:

[0041] Set small file size threshold;

[0042] Set the merge file size threshold;

[0043] Determine whether the original file size is less than or equal to the small file size threshold;

[0044] If not, the original file is a large file, and the large file is stored in the local disk, and the absolute path of the large file is recorded;

[0045] If yes, the original file is a small file, and a small file record is generated, wherein the small file record includes a four-tuple, an absolute path of the merged file, an offset of the small file in the merged file, and a size of the small file;

[0046] Merge small files into a merged file;

[0047] Determine whether the merged file size exceeds the merged file size threshold;

[0048] If so, the merged file will no longer be written, and a new merged file will be reopened later to write the small file into the new merged file;

[0049] If not, do nothing.

[0050] During HTTP file restoration, the restored file content is cached in memory. If the file size exceeds the small file size threshold during HTTP restoration, the restored file is not a small file and can be stored locally independently. When HTTP restoration ends, if the file content size in memory does not exceed the small file size threshold, the restored file is a small file and the file content is written into the merged file.

[0051] like Figure 3 As shown, in another embodiment of the present invention, after the step of locally storing the original file, the small file download is also included, which specifically includes:

[0052] Get the command to open the specified merged file;

[0053] Determine whether to download the specified small file or all the small files of the merged file according to the instruction;

[0054] If a specified small file is to be downloaded, the content of the small file is found and downloaded according to the offset of the small file in the merged file and the size of the small file in the small file record;

[0055] If all small files of the merged file are downloaded, determine whether the merged file has been read completely;

[0056] If the file is not read completely, it will move to the starting point of the next small file, find the content of the specified small file, and download the small file.

[0057] If reading or downloading is completed, the operation ends.

[0058] The embodiment of the present invention provides an HTTP file restoration method in a small file local storage scenario, which reduces the number of files stored in the local disk during HTTP file restoration, reduces the burden on the system disk, and does not affect the efficiency of IO operations; in the continuous HTTP file restoration process, the overall system performance will not decrease as the number of local disk files increases sharply. And the function of viewing and downloading small files is realized.

[0059] like Figure 4 As shown, another embodiment of the present invention provides an HTTP file restoration system in a small file local storage scenario, including: a traffic collection module, a protocol identification module, a TCP reassembly module and an HTTP file restoration module; the traffic collection module is used to obtain data packets to be analyzed; the protocol identification module is used to parse the protocol of the data packets to be analyzed; the TCP reassembly module is used to reassemble the TCP sessions in the data packets and restore the normal order between the data packets; the HTTP file restoration module is used to extract the HTTP file content, cache the HTTP payload part in the memory, and check whether there is compression or encoding in the HTTP response according to the HTTP file content. If compression or encoding exists, decoding or decompression is performed to obtain the original file.

[0060] The system also includes a local storage module, which includes: a threshold setting submodule, a comparison submodule and a file storage submodule. The threshold setting submodule is used to set a small file size threshold and a merged file size threshold; the comparison submodule is used to compare the original file size with the small file size threshold and the merged file size threshold respectively. When the original file is larger than the small file threshold, it is determined to be a large file, and the large file is stored in the local disk; when the original file size is less than or equal to the small file threshold, it is a small file, and a small file record is generated. The small file record includes a four-tuple, an absolute path of the merged file, an offset of the small file in the merged file and a small file size, and the small files are merged and written into the merged file. When the merged file size exceeds the merged file size threshold, the write operation is closed, a new file is reopened, and the small files are merged and written into the new file; when the merged file size does not exceed the merged file size threshold, the small file continues to be written into the current merged file.

[0061] The system also includes a small file download module, which includes a file association submodule, a download submodule and a merged file expansion submodule; the file association submodule is used to generate a file association record for storage by using the quadruple of the small file record, the absolute path of the merged file, the offset of the small file in the merged file, and the small file size information when the restored file is a small file; the download submodule is used to find the corresponding small file content for downloading according to the instruction to download the specified small file; the merged file expansion submodule is used to find the file association record stored in the system according to the absolute path of the merged file, and download all small files in the merged file.

[0062] The embodiment of the present invention provides an HTTP file restoration system in a small file local storage scenario, which reduces the number of files stored in the local disk during HTTP file restoration, reduces the burden on the system disk, and does not affect the efficiency of IO operations; in the continuous HTTP file restoration process, the overall system performance will not decrease as the number of local disk files increases sharply. And the function of viewing and downloading small files is realized.

[0063] like Figure 5 As shown, a structural block diagram of an intelligent terminal according to another embodiment of the present invention is shown, the terminal includes a processor, an input device, an output device and a memory, the processor is connected to the input device, the output device and the memory respectively, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the above embodiment.

[0064] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0065] Input devices may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and output devices may include a display (LCD, etc.), a speaker, etc.

[0066] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0067] In a specific implementation, the processor, input device, and output device described in the embodiments of the present invention may execute the implementation described in the method embodiment provided in the embodiments of the present invention, or may execute the implementation described in the system embodiment described in the embodiments of the present invention, which will not be described in detail here.

[0068] In another embodiment of the present invention, an embodiment of a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the method described in the above embodiment.

[0069] The computer-readable storage medium may be an internal storage unit of the terminal described in the foregoing embodiments, such as a hard disk or memory of the terminal. The computer-readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0070] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed terminal and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for restoring HTTP files in a small file local storage scenario, characterized in that: include: Get the data packet to be analyzed; Perform protocol identification on the data packets to be analyzed; Determine whether the transport layer protocol of the data packet is TCP and the application layer protocol is HTTP; If not, return to obtain the data packet to be analyzed; If yes, the TCP session in the data packet is reassembled; Extract the HTTP file content, and check whether there is compression or encoding in the HTTP response according to the HTTP file content. If there is compression or encoding, decode or decompress it to obtain the original file.

2. The HTTP file restoration method in the small file local storage scenario according to claim 1 is characterized in that: Local storage of original files is also included.

3. The HTTP file restoration method in the small file local storage scenario according to claim 2 is characterized in that: The specific method for locally storing the original file includes: Set small file size threshold; Set the merge file size threshold; Determine whether the original file size is less than or equal to the small file size threshold; If not, the original file is a large file, and the large file is stored in the local disk; If yes, the original file is a small file, and a small file record is generated, wherein the small file record includes a four-tuple, an absolute path of the merged file, an offset of the small file in the merged file, and a size of the small file; Merge small files into a merged file; Determine whether the merged file size exceeds the merged file size threshold; If so, the small file is no longer written into the current merged file, a new file is reopened, and the small file is merged and written into the new file; If not, no action is required.

4. The HTTP file restoration method in the small file local storage scenario according to claim 3 is characterized in that: The method further includes downloading small files after the step of locally storing the original files.

5. The HTTP file restoration method in the small file local storage scenario according to claim 4 is characterized in that: The specific method of downloading the small file includes: Get the command to open the specified merged file; Determine whether to download the specified small file according to the instruction; If so, find the content of the specified small file according to the content in the small file record and download the small file content; If not, determine whether the merged file has been read; If the file is not read completely, it will move to the starting point of the next small file, find the content of the specified small file, and download the small file. If reading or downloading is completed, the operation ends.

6. An HTTP file restoration system in a small file local storage scenario, characterized in that: include: Traffic collection module, protocol identification module, TCP reassembly module and HTTP file restoration module; The traffic collection module is used to obtain data packets to be analyzed; The protocol identification module is used to parse the protocol of the data packet to be analyzed; The TCP reassembly module is used to reassemble the TCP session in the data packet; The HTTP file restoration module is used to extract HTTP file content and check whether there is compression or encoding in the HTTP response according to the HTTP file content. If there is compression or encoding, decoding or decompression is performed to obtain the original file.

7. The HTTP file restoration system in the small file local storage scenario according to claim 6 is characterized in that: It also includes a local storage module, which includes: a threshold setting submodule, a comparison submodule and a file storage submodule, and the threshold setting submodule is used to set a small file size threshold and a merged file size threshold; The comparison submodule is used to compare the original file size with the small file size threshold and the merged file size threshold respectively. When the original file is larger than the small file threshold, it is determined to be a large file and the large file is stored in the local disk; when the original file size is less than or equal to the small file threshold, it is a small file and a small file record is generated. The small file record includes a four-tuple, the absolute path of the merged file, the offset of the small file in the merged file and the small file size. The small files are merged and written into the merged file. When the merged file size exceeds the merged file size threshold, the write operation is closed, a new file is reopened, and the small files are merged and written into the new file; when the merged file size does not exceed the merged file size threshold, the small file continues to be written into the current merged file.

8. The HTTP file restoration system in the small file local storage scenario according to claim 7, characterized in that: The system also includes a small file download module, which includes a file association submodule, a download submodule and a merged file expansion submodule; The file association submodule is used to generate a file association record for storage by storing the quadruple of the small file record, the absolute path of the merged file, the offset of the small file in the merged file, and the small file size information when the restored file is a small file; The download submodule is used to find the corresponding small file content and download it according to the instruction of downloading the specified small file; The merged file expansion submodule is used to find the file association record stored in the system according to the absolute path of the merged file, and download all small files in the merged file.

9. An intelligent terminal, comprising a processor, an input device, an output device and a memory, wherein the processor is connected to the input device, the output device and the memory respectively, and the memory is used to store a computer program, wherein the computer program comprises program instructions, and wherein: The processor is configured to call the program instructions to execute the method according to any one of claims 1-5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images