Method for extracting tcp session message content according to features based on python

Abstract

The invention discloses a method for extracting tcp session message content according to features based on python, and the method comprises the following steps: extracting each piece of field information of a tcp data message header according to a tcp protocol; according to the method, different sessions are identified through field combinations, code values corresponding to the fields are calculated, complete tcp session content is recombined, the sessions are further analyzed based on application features, and therefore the purposes of rapidly analyzing tcp messages and extracting message content are achieved. The method effectively solves the problems that manual participation is needed in the whole process of a traditional package capture analysis process, manual checking is needed for data correctness checking, the process is tedious and time-consuming and excessively depends on manual analysis, the human input in the software development process can be greatly reduced, the development cost of a project period is saved, and the development efficiency is improved. The error rate of manual data checking is reduced, the working efficiency of employees is improved, and human resources are released to expand other business functions.

Description

Technical Field

[0001] The invention relates to network communication technology, and in particular to a method for extracting TCP session message content by features based on Python. Background Art

[0002] With the rapid development of the Internet, applications that use the Internet for social interaction and entertainment are emerging in an endless stream. Communication networks have become one of the most important infrastructures supporting modern society.

[0003] The main architecture for communication between applications is the client / server architecture. A server (S) is a series of hardware or software that provides the required "services" for one or more clients (C). Its only purpose is to wait for client requests, respond to them, and then wait for more requests. On the other hand, the client contacts the server for a specific request, sends the necessary data, waits for the server's response, and finally completes the request, giving the reason for the failure if the request fails. Before the server responds to the client's request, it first creates a communication endpoint that enables the server to continuously listen for requests. The client creates its single communication endpoint for it to establish a connection to the server. A socket is a computer network data structure that is used to implement the communication endpoint between the server and the client. Before any type of communication can begin, the network application must create a socket. They can be compared to telephone jacks, without which no communication can take place.

[0004] Applications that use the above C / S architecture and are programmed using network socket technology often use some packet capture tools to capture and analyze the data packets generated during the interaction process during the software development and testing to the final release to verify the correctness of the interaction process. However, the traditional packet capture analysis process has a series of problems, such as the need for manual participation throughout the process; the need to deploy the packet capture environment in advance before the interaction process occurs and start packet capture; after the interaction process is completed, the data packets captured during the interaction need to be parsed and viewed using tools such as wireshark; in addition, the correctness of the data must also be manually checked, and the entire process is lengthy, time-consuming, and overly dependent on manual analysis. Therefore, in order to solve the above problems, how to control the interaction process of network programming to achieve a data packet analysis method that can accurately screen and efficiently analyze is the key. Summary of the invention

[0005] The present invention proposes a method for extracting TCP session message content by features based on Python. A data packet of TCP communication between client A and server B is captured from the network in advance, and the link building packets (syn or syn+ack) in two directions (A->B or B->A) in the data packet are specifically numbered respectively, and the directional IP and the number value are used as unique identifiers (hereinafter collectively referred to as key identifiers) to record a single complete TCP unidirectional session information in the data packet.

[0006] The request and response message formats of conventional applications in TCP communication follow general principles and consist of a message header with custom characteristics (including message code and message length) and a message body of corresponding length. For the messages of these applications, we can further extract the payload data actually needed from the complete session content associated with the above key identifier by using the characteristics of the application message.

[0007] The specific steps are as follows: Step 1: Use Python to call the packet protocol analysis interface of tshark, and count the message content of the TCP protocol message based on the message position in the data packet (hereinafter referred to as the message index). The statistical information includes allLen (message length), dataLen (payload length), dataContent (payload content), seq (sequence number), ack (acknowledgement number), SYN (syn packet flag), and ips (ipA->ipB).

[0008] Step 2: Traverse the messages at all positions, group the messages with the same key identifier together, sort them according to the size of the seq value, and record a group of message index values ​​associated with each key identifier (hereinafter collectively referred to as a message set).

[0009] Step 3: Analyze the message set associated with each key identifier, and make validity judgment on the dataContent of each message in the message set for subsequent message reassembly. The judgment process starts from the syn packet in the message set, and dataseek is used to represent the total data length of the current valid reassembled content. If the seq value of the current index message is equal to the dataseek value, it is determined to be a message to be completely reassembled; if the seq value is less than the dataseek value, and seq+dataLen is greater than dataseek, it is determined to be a message to be partially reassembled, and the dataContent statistics of the message are modified and updated; if the seq value is less than the dataseek value, and seq+dataLen is less than or equal to dataseek, it is determined to be a message that does not need to be reassembled, and the message index is removed from the message set associated with the key identifier; if the seq value is greater than the dataseek value, it is determined that some intermediate messages are missing, an error is prompted, and the message reassembly is terminated.

[0010] Step 4: The above step 3 selects the valid message set associated with each key identifier and updates the correct message dataContent statistics. By accumulating the dataContent of all messages in the message set, the information content of the complete TCP one-way session can be obtained. The session information may contain multiple messages in the session process. Since a request process or response process (single or multiple data packets) in the TCP protocol is based on the same ack value, the message set can be divided based on the ack value to obtain multiple messages contained in the session process.

[0011] Step 5: Each message in the conversation process generally has obvious application characteristics. Messages generally consist of a message header and a message body. By performing corresponding feature analysis and length analysis on the messages of specific applications, you can extract the specific content of the message header and message body. If there is a detailed document that explains the structure of the message content, you can further extract the information of each field contained in the message body, divide the fields based on some characteristics (message codes defined by different messages), and make statistics, which can be of great help for the subsequent analysis and comparison work.

[0012] Compared with the prior art, the technical solution of the present invention has the following significant beneficial effects: 1. The present invention is based on Python's TCP message code value calculation and a message analysis method for extracting content through key features, which solves a series of problems such as the need for manual participation in the traditional packet capture and analysis process, the need for manual verification of the correctness of the data, the lengthy and time-consuming process, and excessive reliance on manual analysis.

[0013] 2. The present invention can greatly reduce the manpower input in the software development process, save the development cost of the project cycle, reduce the error rate of manual data verification, improve the work efficiency of employees, and release human resources to expand other business functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in every figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of a method for extracting TCP session message content by features based on Python, provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose and technical solution of the present disclosure more clearly understood, the present disclosure is further elaborated in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0016] The flowchart of the method for extracting TCP session message content by features based on Python is as follows: Figure 1 As shown. A data packet of TCP communication between client A and server B is captured from the network in advance, and the link building packets (syn or syn+ack) in the two directions (A->B or B->A) in the data packet are specifically numbered, and the directional IP and number value are used as unique identifiers (hereinafter collectively referred to as key identifiers) to record a single complete TCP unidirectional session information in the data packet.

[0017] The request and response message formats of conventional applications in TCP communication follow general principles and consist of a message header with custom characteristics (including message code and message length) and a message body of corresponding length. For the messages of these applications, we can further extract the payload data actually needed from the complete session content associated with the above key identifier by using the characteristics of the application message.

[0018] The specific steps are as follows: Step 1: Use Python to call the packet protocol analysis interface of tshark, and count the message content of the TCP protocol message based on the message position in the data packet (hereinafter referred to as the message index). The statistical information includes allLen (message length), dataLen (payload length), dataContent (payload content), seq (sequence number), ack (acknowledgement number), SYN (syn packet flag), and ips (ipA->ipB).

[0019] Step 2: Traverse the messages at all positions, group the messages with the same key identifier together, sort them according to the size of the seq value, and record a group of message index values ​​associated with each key identifier (hereinafter collectively referred to as a message set).

[0020] Step 3: Analyze the message set associated with each key identifier, and make validity judgment on the dataContent of each message in the message set for subsequent message reassembly. The judgment process starts from the syn packet in the message set, and dataseek is used to represent the total data length of the current valid reassembled content. If the seq value of the current index message is equal to the dataseek value, it is determined to be a message to be completely reassembled; if the seq value is less than the dataseek value, and seq+dataLen is greater than dataseek, it is determined to be a message to be partially reassembled, and the dataContent statistics of the message are modified and updated; if the seq value is less than the dataseek value, and seq+dataLen is less than or equal to dataseek, it is determined to be a message that does not need to be reassembled, and the message index is removed from the message set associated with the key identifier; if the seq value is greater than the dataseek value, it is determined that some intermediate messages are missing, an error is prompted, and the message reassembly is terminated.

[0021] Step 4: The above step 3 selects the valid message set associated with each key identifier and updates the correct message dataContent statistics. By accumulating the dataContent of all messages in the message set, the information content of the complete TCP one-way session can be obtained. The session information may contain multiple messages in the session process. Since a request process or response process (single or multiple data packets) in the TCP protocol is based on the same ack value, the message set can be divided based on the ack value to obtain multiple messages contained in the session process.

[0022] Step 5: Each message in the conversation process generally has obvious application characteristics. Messages generally consist of a message header and a message body. By performing corresponding feature analysis and length analysis on the messages of specific applications, you can extract the specific content of the message header and message body. If there is a detailed document that explains the structure of the message content, you can further extract the information of each field contained in the message body, divide the fields based on some characteristics (message codes defined by different messages), and make statistics, which can be of great help for the subsequent analysis and comparison work.

[0023] Various aspects of the present invention are described in this disclosure with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the present disclosure are not necessarily defined to include all aspects of the present invention. It should be understood that the various concepts and embodiments introduced above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed by the present invention are not limited to any implementation. In addition, some aspects disclosed by the present invention can be used alone, or in any appropriate combination with other aspects disclosed by the present invention.

[0024] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A method for extracting TCP session message content by features based on Python. It is characterized in that The method comprises extracting information of each field in the header of a TCP data message according to the TCP protocol; identifying different sessions by field combination and calculating the corresponding code value of each field, reorganizing the complete TCP session content, and further parsing the session based on application characteristics, so as to achieve the purpose of quickly analyzing the TCP message and extracting the message content; the specific steps are as follows: Step 1: Use Python to call the packet protocol analysis interface of tshark, and count the message content of the TCP protocol message based on the message position in the data packet (hereinafter referred to as the message index). The statistical information includes allLen (message length), dataLen (payload length), dataContent (payload content), seq (sequence number), ack (acknowledgement number), SYN (syn packet flag), ips (ipA->ipB); Step 2: Traverse all messages at all positions, group messages with the same key identifier together, sort them according to the size of the seq value, and record a group of message index values ​​associated with each key identifier (hereinafter collectively referred to as a message set); Step 3: Analyze the message set associated with each key identifier, and make validity judgment on the dataContent of each message in the message set for subsequent message reassembly. The judgment process starts from the syn packet in the message set, and uses dataseek to represent the total data length of the current valid reassembled content. If the seq value of the current index message is equal to the dataseek value, it is judged to be a message to be completely reassembled; if the seq value is less than the dataseek value, and seq+dataLen is greater than dataseek, it is judged to be a message to be partially reassembled, and the dataContent statistics of the message are modified and updated; If the seq value is less than the dataseek value, and seq+dataLen is less than or equal to dataseek, it is determined to be a message that does not need to be reassembled, and the message index is removed from the message set associated with the key identifier; If the seq value is greater than the dataseek value, it is determined that some intermediate messages are missing, an error is reported, and message reassembly is terminated; Step 4: In step 3 above, filter out the valid message sets associated with each key identifier and update the correct message dataContent statistics. Accumulate the dataContent of all messages in the message set to obtain the complete information content of the TCP one-way session. The session information may contain multiple messages in the session process. Since a request process or response process (single or multiple data packets) in the TCP protocol is based on the same ack value, the message set is further divided based on the ack value to obtain multiple messages contained in the session process. Step 5: Each message in the conversation process generally has obvious application characteristics. The message is generally composed of a message header and a message body. By performing corresponding feature analysis and length analysis on the messages of specific applications, you can extract the specific content of the message header and message body. If there is a detailed document that explains the structure of the message content, you can further extract the information of each field contained in the message body, divide the fields based on some characteristics (message codes defined by different messages), and make statistics, which can be of great help for the subsequent analysis and comparison work.

Images