Method and device for realizing FTP file restoration

Abstract

The invention discloses a method and a device for realizing FTP (File Transfer Protocol) file restoration, on one hand, continuous data streams are restored under the conditions of disorder, retransmission, short-time packet loss and the like by analyzing an FTP control channel and combining indexed backtracking and TCP (Transmission Control Protocol) stream recombination, so that a restoration gap is reduced to the maximum extent, and the integrity and the success rate of FTP file restoration are improved; and on the other hand, by pre-filtering the non-preset protocol and cooperating with establishment of the file index information relationship of the original message corresponding to the quintuple information, the original message retention amount and disk write amplification are significantly reduced, so that the storage and I / O overhead is reduced, the system throughput is improved, and the historical retention window is prolonged.

Description

TECHNICAL FIELD

[0001] The present application relates to, but is not limited to, the technical field of computer technology, and in particular to a method and device for realizing FTP file restoration. BACKGROUND

[0002] File Transfer Protocol (FTP) is an application layer protocol used for file exchange between two computers in a Transmission Control Protocol / Internet Protocol (TCP / IP) network, and is one of the file transfer methods widely used in the early Internet. Through the FTP client, users can send requests to the FTP server to realize file download from the server, file upload to the server, and new directory creation or directory modification on the server.

[0003] With the rapid development of information technology, the application of networks in various industries is becoming more and more widespread, and network security problems are becoming increasingly prominent. In particular, in the field of network traffic analysis and security monitoring, the security of FTP transmission files has become one of the important concerns. Since FTP protocol often uses plaintext in the transmission process, it is extremely easy to be intercepted, tampered with or carry malicious files, so in network security protection, it is of great significance to accurately identify and restore FTP transmission files. SUMMARY

[0004] The present application provides a method and device for realizing FTP file restoration, which can improve the completeness and success rate of FTP file restoration, while reducing storage and I / O overhead.

[0005] The present application provides a method for realizing FTP file restoration, comprising: parsing the commands of a File Transfer Protocol (FTP) control channel to obtain five-tuple information of an FTP data channel used for transmitting files; saving original messages of non-pre-set protocols and file index information of the original messages corresponding to the five-tuple information; when it is necessary to restore files, querying the file index information of the original messages according to the five-tuple information, and obtaining the original messages of the FTP data channel according to the queried file index information; generating a target file from the data obtained after flow recombination according to the obtained original messages.

[0006] In an exemplary example, the five-tuple information of the FTP data channel used for transmitting files is obtained, comprising: obtaining mirror traffic; for the case that the obtained mirror traffic is Transmission Control Protocol (TCP) traffic, for the FTP control channel, parsing the commands of the FTP control channel to extract the five-tuple information of the corresponding FTP data channel; Add the obtained five-tuple information of the FTP data channel to an FTP file transfer list.

[0007] In an example, the method further includes saving the original packet of the non-pre-set protocol and saving file index information of the original packet corresponding to the five-tuple information.

[0008] In an example, the method further includes, for a case that the current original packet belongs to the packet of the pre-set protocol, discarding the current original packet.

[0009] In an example, the method further includes, for a case that the current original packet belongs to the packet of the pre-set protocol, discarding the current original packet.

[0010] In an example, the method further includes, for a case that the current original packet belongs to the packet of the pre-set protocol, discarding the current original packet.

[0011] Embodiments of the present application also provide a computer readable storage medium storing computer executable instructions for implementing the method for restoring the FTP file.

[0012] Embodiments of the present application also provide a computer device including a memory and a processor, wherein the memory stores instructions executable by the processor, and the instructions are used for implementing the method for restoring the FTP file.

[0013] Embodiments of the present application also provide an apparatus for restoring the FTP file, including an obtaining module, a saving module, and a processing module. The obtaining module is configured to parse a command of an FTP control channel and obtain five-tuple information of an FTP data channel used for transferring a file. ​​​​The saving module is used for saving original packets of non-pre-set protocols and saving file index information of the original packets corresponding to the five-tuple information; The processing module is used for querying file index information of original packets according to the five-tuple information when a file needs to be restored, obtaining original packets of the FTP data channel according to the queried file index information, and generating a target file from data obtained by performing stream recombination on the original packets.

[0014] In an exemplary instance, the saving module is used for: The mirror traffic is read packet by packet to form an original packet stream, and the mirror traffic is filtered according to the pre-set protocol; for a case that a current original packet is not a packet of the pre-set protocol, the current original packet is saved to a packet file, and index information of a file in which the original packet corresponding to the five-tuple is written into a database.

[0015] The method for implementing FTP file restoration provided by the embodiment of the application has the following advantages. On the one hand, the continuous data stream is recovered under the conditions of disordering, retransmission, and short-time packet loss by analyzing the FTP control channel and combining index backtracking and TCP stream recombination, so that the restoration gap is minimized, and the completeness and success rate of FTP file restoration are improved. On the other hand, the amount of original packets saved and the disk write amplification are significantly reduced by pre-filtering non-pre-set protocols and establishing the index information relationship of the file in which the original packet corresponding to the five-tuple is located, so that the storage and I / O overheads are reduced, the system throughput is improved, and the historical retention window is prolonged.

[0016] Additional features and advantages of the application are set forth in the description that follows, and in part will become apparent to those skilled in the art upon examination of the description or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the technical solutions of the application, and do not constitute a limitation on the technical solutions of the application.

[0018] Figure 1 A flowchart of the method for implementing FTP file restoration in the embodiment of the application; Figure 2 An embodiment flowchart of the method for saving original packets in the implementation of FTP file restoration in the embodiment of the application; Figure 3 An embodiment flowchart of the method for restoring FTP files in the embodiment of the application; Figure 4A flowchart of a device for implementing FTP file restoration in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. It should be explained that the embodiments in the present application and the features in the embodiments can be combined with each other as long as there is no conflict.

[0020] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings. The embodiments of the present application are shown in the drawings. However, the present application can be implemented in many different forms, and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0022] It can be understood that the terms "first", "second" used in the present application are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0023] It can be understood that "connection" in the following embodiments means "electrical connection", "communication connection", etc. if the connected circuits, modules, units, etc. have the transmission of electrical signals or data between each other.

[0024] As used herein, the singular forms "a", "an" and "the" can include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "comprising" or "including" or "having" or the like specifies the presence of stated features, integers, steps, operations, components, parts, or combinations thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in the specification includes any and all combinations of the related listed items.

[0025] The steps shown in the flowchart of the drawing can be performed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown here.

[0026] In practical applications, network administrators often collect network traffic data through the mirror port of the core switch, and analyze the traffic using protocol analysis technology. For FTP traffic, the transmission content needs to be restored to the original file for further malicious code detection, content review or forensic analysis. However, due to the complexity of the network environment, transmission data fragmentation, packet loss and other factors, the existing technology still has deficiencies in FTP file restoration rate and restoration integrity, so there is an urgent need for a technical solution that can improve the FTP file restoration effect.

[0027] Figure 1 The flowchart of the method for implementing FTP file restoration in the embodiments of the present application, as shown in Figure 1 , can include: Step 100: Analyze the command of the FTP control channel to obtain the quintuple information of the FTP data channel used for transmitting files.

[0028] In an exemplary example, step 100 can include: Obtain mirror traffic; For the case where the obtained mirror traffic is TCP traffic, for the FTP control channel, analyze the command of the FTP control channel to extract the quintuple information of the corresponding FTP data channel; Add the obtained quintuple information of the FTP data channel to the FTP file transmission list.

[0029] Through step 100, the command on the FTP control connection (TCP / 2 or custom port) is analyzed, the quintuple of the FTP data channel corresponding to each file transmission is identified, and the FTP file transmission list is written.

[0030] The FTP control channel is used to transmit commands and response information between the client and the server. The client sends login, download, upload and other operation instructions to the server through the FTP control channel, and the server returns the operation result through the FTP control channel. The FTP control channel will remain connected during the entire FTP session, uses TCP protocol, and the default port number is 21.

[0031] FTP data channel, for transmitting actual data. For example, the data involved in operations such as uploading files, downloading files, and listing directory contents are transmitted through the FTP data channel. The FTP data channel is different from the FTP control channel. The FTP data channel is established on demand, and is only established when data needs to be transmitted. After the data transmission is completed, the FTP data channel is closed. The FTP data channel is also based on the TCP protocol, and its port number is negotiated through the command of the FTP control channel.

[0032] The five-tuple is used to identify a network session, and includes: source IP, source port, destination IP, destination port, and transport layer protocol.

[0033] Step 101: Save the original packet of the non-preset protocol, and save the file index information where the original packet corresponding to the five-tuple information is located.

[0034] In an exemplary instance, step 101 can include: reading the mirror traffic packet by packet to form an original packet stream, and filtering according to a preset protocol (such as filtering common protocol packets unrelated to FTP). For the case that the current original packet is not a preset protocol packet, the current original packet is saved to a packet file, and the index information of the file where the original packet corresponding to the five-tuple is located is written to the database. Specifically, it can include: filtering the preset protocol packets; saving the original packets of the unfiltered protocol to a file, and saving the index information of the file where the original packet corresponding to the five-tuple is located to the database. In the embodiment of the application, the original packet refers to a single data packet record obtained from the mirror traffic one by one, without being parsed and rewritten.

[0035] Step 101 can also include: for the case that the current original packet is a preset protocol packet, discarding the current original packet.

[0036] Through step 101, based on the core switch mirror traffic, the original packets of non-preset protocols (i.e. unfiltered protocols, including target FTP data channels) are sequentially written to a file, and the correspondence between the five-tuple and the index of the file where the packet is located is written to the database. In this step, filtering refers to discarding non-FTP packets of preset protocols, and retaining FTP control / data channel related and other protocol packets not included in the preset protocols.

[0037] In an embodiment, the preset protocol can include, but is not limited to, non-FTP protocols such as: user datagram protocol (UDP) based encrypted transmission protocols (such as hypertext transfer protocol secure (HTTPS), secure shell protocol (SSH), etc.), hypertext transfer protocol (HTTP), simple mail transfer protocol (SMTP), server message block protocol (SMB), etc.

[0038] Step 102: When the file needs to be restored, the file index information of the original message is queried according to the quintuple information, and the original message of the FTP data channel is obtained according to the queried file index information.

[0039] In an exemplary instance, step 102 can include: when the file of a certain FTP transmission needs to be restored, the quintuple recorded in step 100 is taken as a key to find the position of the corresponding original message in the file index database (i.e. index library) where the original message corresponding to the quintuple of step 101 is stored, and the original message is pulled.

[0040] Step 103: The target file is generated according to the data obtained after the stream recombination of the obtained original message.

[0041] In an exemplary instance, step 103 can include: performing TCP stream recombination on the pulled original message, and writing the recombined payload into the target file, thereby realizing the restoration of the FTP file.

[0042] In an embodiment, steps 102-103 can include: Timing task triggering, traversing the FTP file transmission list; Querying the message index information of the original message according to the quintuple in the FTP file transmission list; After locating and reading all original messages according to the queried message index, performing TCP stream recombination, extracting the payload of the FTP data channel, and writing the recombined data into the target file to restore the TPF file.

[0043] In an embodiment, for the case where the message index information of the original message is not queried, the step of traversing the FTP file transmission list is returned.

[0044] The method for realizing FTP file restoration provided by the embodiment of the application, on the one hand, restores the continuous data stream under the conditions of disorder, retransmission, short-time packet loss, etc. by analyzing the FTP control channel and combining the indexing backtracking and TCP stream recombination, maximally reduces the restoration gap, and improves the completeness and success rate of FTP file restoration; on the other hand, by pre-filtering the non-pre-set protocol, cooperating with the establishment of the file index information relationship of the original message corresponding to the quintuple information, the original message saving amount and disk write amplification are significantly reduced, thereby reducing the storage and I / O overhead, improving the system throughput, and prolonging the historical retention window.

[0045] The application further provides a computer readable storage medium storing computer executable instructions for executing the method for realizing FTP file restoration according to any one of the above.

[0046] The application further provides a computer device comprising a memory and a processor, wherein the memory stores instructions executable by the processor, for performing the steps of the method for implementing FTP file restoration.

[0047] Figure 2 For the embodiment process of saving original messages in the implementation of FTP file restoration in the embodiments of the application, the specific implementation examples of steps 100-101 are shown, as shown in Figure 2 , comprising: Step 200: Obtain mirror traffic. This step can continuously collect network messages from the mirror port of the core switch to enter the packet-by-packet / stream-by-stream processing.

[0048] Step 201: Determine whether the obtained mirror traffic is TCP traffic. If yes, go to step 202; if not, end the process. Through this step, non-TCP messages irrelevant to FTP are early screened out, the irrelevant analysis and storage overheads are reduced, and the overall processing throughput is improved.

[0049] Step 202: Determine whether it is an FTP control channel. If yes, go to step 203; if not, go to step 205.

[0050] Step 203: Obtain the five-tuple of the FTP data channel. In this step, the five-tuple (including source IP, source port, destination IP, destination port, and protocol) and related metadata of the data channel to be used for file transmission can be extracted by analyzing FTP control commands (such as PASV / EPSV, PORT / EPRT, RETR / STOR, etc.).

[0051] Step 204: Add the FTP file transmission list, that is, write the five-tuple information obtained in step 203 into the FTP file transmission list, and end the process.

[0052] In the embodiments of the application, the file transmission instruction and the actual data channel are accurately bound through steps 203 and 204, which reduces the mismatch and stream mixing in complex scenarios such as concurrency, NAT, and port multiplexing, lays a foundation for accurate input for subsequent five-tuple backtracking and recombination, and improves the success rate and completeness of FTP file restoration.

[0053] Step 205: Determine whether the current message belongs to a preset protocol. If yes, discard the message of the preset protocol and end the process; if not, go to step 206.

[0054] The preset protocol can include but is not limited to non-FTP protocols such as encrypted transmission protocols of UDP (such as HTTPS, SSH, etc.), HTTP, SMTP, SMB, etc.

[0055] In the embodiments of the present application, the messages determined as the preset protocol are not saved and indexed, which significantly reduces the amount of original message saving and disk write amplification, reduces storage and I / O / CPU consumption, improves system throughput and prolongs the historical retention window, without affecting the effectiveness of the FTP restoration link.

[0056] Step 206: save the current original message to the message file, and write the index information of the file where the original message corresponding to the five-tuple is located into the database, which is used for subsequent fast backtracking and restoration according to the five-tuple. Thus, the message processing is ended this time, and the next round of processing can be entered.

[0057] Through the processing of step 206, the traceable link of original message→index→on-demand backtracking is formed, which supports subsequent fast positioning and extraction of target data channel messages, shortens the restoration delay; at the same time, the evidence chain (file name, timestamp, offset, etc.) is preserved, which improves the verifiability and auditability of the results, and is beneficial to the scalability and stable throughput in a large flow environment.

[0058] Figure 3 For the embodiment flowchart of restoring the FTP file in the embodiments of the present application, a specific implementation example of steps 102-103 is shown, as shown in Figure 3 , which includes: Step 300: the timing task is triggered, and the FTP file transfer list is traversed.

[0059] In one embodiment, the current file restoration process can be started according to a preset period (such as every 5 minutes / according to a scheduled task).

[0060] In one embodiment, the records to be processed (including five-tuple, time window, direction, TYPE / REST, etc. metadata) can be taken out one by one.

[0061] Step 301: according to the five-tuple in the FTP file transfer list, the message index information of the original message is queried, if found, step 302 is entered; if not found, step 300 is returned to continue traversing the FTP file transfer list.

[0062] In this step, the five-tuple (if necessary, including the time range) in the record in step 300 is used to search the corresponding original message position in the index library. For the case that the original message index information is not queried, step 300 is returned to continue traversing the next one (or waiting for the next timing task); for the case that the original message index is queried, step 302 is entered.

[0063] In the embodiments of the present application, the key-based retrieval based on the index realizes fast positioning of the target data channel, which significantly reduces the retrieval delay and read amplification; the back-off record when the hit is missed also avoids misprocessing, and enhances the robustness and recoverability of the process.

[0064] Step 302: Obtain original packets and complete restoration. In this step, after all original packets are located and read according to the index, TCP stream recombination is performed, the payload of the FTP data channel is extracted, and the recombined data is written into a target file, i.e., a restored TPF file. At this point, the current round of timing task processing is completed, and the next period or external trigger can be waited for.

[0065] Figure 4 A flowchart of an apparatus for implementing FTP file restoration in an embodiment of the present application is shown in FIG. 3, which can include an obtaining module, a saving module, and a processing module. The obtaining module is configured to parse commands of an FTP control channel and obtain five-tuple information of an FTP data channel used for transmitting files. The saving module is configured to save original packets of non-pre-set protocols and save file index information of the original packets corresponding to the five-tuple information. The processing module is configured to, when a file needs to be restored, query file index information of original packets according to the five-tuple information, obtain original packets of the FTP data channel according to the queried file index information, and generate a target file from data obtained after stream recombination according to the obtained original packets. Figure 4 The obtaining module is configured to parse commands of an FTP control channel and obtain five-tuple information of an FTP data channel used for transmitting files. The saving module is configured to save original packets of non-pre-set protocols and save file index information of the original packets corresponding to the five-tuple information. The processing module is configured to, when a file needs to be restored, query file index information of original packets according to the five-tuple information, obtain original packets of the FTP data channel according to the queried file index information, and generate a target file from data obtained after stream recombination according to the obtained original packets.

[0066] In an exemplary instance, the obtaining module can be configured to: obtain mirror traffic; for the case that the obtained mirror traffic is TCP traffic, for an FTP control channel, parse commands of the FTP control channel to extract five-tuple information of a corresponding FTP data channel; and add the obtained five-tuple information of the FTP data channel to an FTP file transmission list.

[0067] In an exemplary instance, the saving module can be configured to: read original packet streams packet by packet, filter the mirror traffic according to a pre-set protocol, for the case that a current original packet is a packet of a non-pre-set protocol, save the current original packet to a packet file, and write index information of a file where the original packet corresponding to the five-tuple is located into a database.

[0068] Further, the saving module can be further configured to, for the case that the current original packet is a packet of a pre-set protocol, discard the current original packet.

[0069] In an exemplary instance, the processing module can be configured to: ​The timing task triggers, traverses the FTP file transfer list; queries the message index information of the original message according to the five-tuple in the FTP file transfer list; after reading all the original messages according to the queried message index, performs TCP stream recombination, extracts the payload of the FTP data channel, and writes the recombined data into the target file to restore the TPF file.

[0070] Further, the processing module can also be configured to: for the case that the message index information of the original message is not queried, return to execute the step of traversing the FTP file transfer list.

[0071] The device for restoring the FTP file provided by the embodiment of the application can restore the continuous data stream under the conditions of disorder, retransmission, short-time packet loss, etc. by analyzing the FTP control channel and combining the indexing backtracking and TCP stream recombination, thereby minimizing the restoration gap and improving the completeness and success rate of the FTP file restoration. On the other hand, by pre-filtering the non-pre-set protocol and establishing the file index information relationship of the original message corresponding to the five-tuple information, the original message storage amount and disk write amplification are significantly reduced, thereby reducing the storage and I / O overhead, improving the system throughput, and prolonging the historical retention window.

[0072] Although the embodiments disclosed in the present application are as above, the content described is only the embodiments adopted for the purpose of facilitating the understanding of the present application, and is not intended to limit the present application. Any person skilled in the art of the present application can make any modification and change in the implementation form and details without departing from the spirit and scope of the present application. The patent protection scope of the present application shall be subject to the scope defined by the appended claims.

Claims

1. A method for restoring FTP files, characterized in that, include: The command parses the FTP control channel of the File Transfer Protocol (FTP) to obtain the 5-tuple information of the FTP data channel used for file transfer. Save the original message of the non-preset protocol, and save the file index information of the original message corresponding to the five-tuple information; When a file needs to be restored, the file index information of the original message is queried based on the 5-tuple information, and the original message of the FTP data channel is obtained based on the queried file index information; The target file is generated based on the data obtained after stream reassembly of the original messages.

2. The method according to claim 1, wherein, The process of obtaining the 5-tuple information of the FTP data channel used for file transfer includes: Obtain mirror traffic; If the obtained mirrored traffic is Transmission Control Protocol (TCP) traffic, for the FTP control channel, parse the commands of the FTP control channel to extract the corresponding 5-tuple information of the FTP data channel; Add the obtained 5-tuple information of the FTP data channel to the FTP file transfer list.

3. The method according to claim 2, wherein, The process of saving the original message that does not conform to the preset protocol, and saving the file index information of the original message corresponding to the five-tuple information, includes: The mirrored traffic is read packet by packet to form an original message stream, and filtered according to the preset protocol. If the current original message does not belong to the preset protocol, the current original message is saved to a message file, and the index information of the file where the original message corresponding to the five-tuple is located is written to the database.

4. The method according to claim 3, further comprising: If the current original message belongs to the preset protocol, the current original message is discarded.

5. The method according to claim 2, wherein, When a file needs to be restored, the file index information of the original message is queried based on the 5-tuple information. The data obtained after stream reassembly of the original message is used to generate the target file, including: A scheduled task is triggered to iterate through the FTP file transfer list; The message index information of the original message is retrieved based on the five-tuple in the FTP file transfer list; Locate and read all original packets according to the queried packet index, perform TCP stream reassembly, extract the payload of the FTP data channel, and write the reassembled data to the target file to restore the TPF file.

6. The method according to claim 5, further comprising: If the message index information of the original message is not found, return to the step of traversing the FTP file transfer list.

7. A computer-readable storage medium storing computer-executable instructions for performing the method for restoring FTP files as described in any one of claims 1-6.

8. A computer device comprising a memory and a processor, wherein, The memory stores the following instructions executable by a processor: steps for performing the method for restoring FTP files as described in any one of claims 1-6.

9. An apparatus for restoring FTP files, characterized in that, include: The module includes an acquisition module, a storage module, and a processing module; among them, The acquisition module is used to parse commands from the FTP control channel and obtain the 5-tuple information of the FTP data channel used for file transfer. The storage module is used to store the original messages that are not based on the preset protocol, and to store the file index information of the original message corresponding to the five-tuple information. The processing module is used to query the file index information of the original message based on the five-tuple information when a file needs to be restored, obtain the original message of the FTP data channel based on the queried file index information, and generate the target file from the data obtained by stream reconstruction of the obtained original message.

10. The apparatus according to claim 9, wherein, The storage module is used for: The mirrored traffic is read packet by packet to form the original message stream, and the mirrored traffic is filtered according to the preset protocol. If the current original message does not belong to the preset protocol, the current original message is saved to the message file, and the index information of the original message file corresponding to the five-tuple is written to the database.

Images