Audit method and system for active learning analysis rule of custom protocol

Abstract

The invention discloses an auditing method and system for an active learning analysis rule of a custom protocol, and relates to the field of custom protocols. The method specifically comprises the following steps: collecting a data packet of a custom protocol from network traffic; using a deep learning algorithm and a pattern matching technology to identify the data packet structure of the data packet and the first appearance positions of the plurality of key fields; obtaining a preset encryption rule of each key field for encryption; sequentially splicing the encrypted key fields according to the first appearance position, marking field numbers and field types to obtain an analysis key, and forming a preliminary analysis rule set of the analysis key; and decrypting the data packet of the custom protocol based on the parsing key and performing threat review. According to the method, the precision and the automation degree of self-defined protocol analysis are improved, an automatic analysis mechanism is provided, and the efficiency and the detection capability of protocol auditing are improved.

Description

Technical Field

[0001] The present invention relates to the field of user-defined protocols, and in particular to an audit method and system for actively learning and parsing rules of user-defined protocols. Background Art

[0002] With the rapid development of Internet technology, communication methods are becoming more and more complex, and various custom protocols are widely used, including some well-known application layer protocols, such as TFTP, FTP, SIP, etc., as well as some private protocols defined by enterprises themselves. Existing security protection systems usually need to parse custom protocols in the network in order to detect and defend against malicious traffic. In order to ensure that data is not tampered with or stolen during transmission, many communication protocols use encryption protection. Most of the existing custom protocol parsing methods are manual methods, which cannot keep up with the development of communication protocols, are time-consuming and labor-intensive, and are difficult to achieve real-time performance for massive data. They are prone to errors and have high maintenance costs. After a new communication protocol is designed, it is usually used for intranet communication first, and will gradually be promoted to the external network after it stabilizes. At this intranet stage, attackers often launch attacks on custom protocols, such as forging and tampering with the content of protocol fields, in order to evade detection. Therefore, at this stage, it is necessary to parse and audit the custom protocol to discover malicious behavior. Existing custom protocol parsing methods often adopt the following methods: 1. Completely parse the custom protocol, and then find abnormal behavior by analyzing the parsing results; 2. Manually define parsing rules according to the format of the custom protocol, and then find abnormal behavior according to the parsing rules; 3. Use machine learning algorithms to partially parse the custom protocol, and find abnormal behavior based on the parsing results and abnormal features; 4. Parsing methods based on association rule mining extract features by discovering the association relationship between data, thereby identifying malicious behavior. The above-mentioned solutions in the prior art have the following shortcomings: 1. Solution 1 is usually based on clustering and classification parsing methods: These methods perform anomaly detection by dividing traffic data into different clusters, or classify traffic data into normal and abnormal categories. This method has a large amount of calculation, resulting in low detection efficiency; and for complex and changeable data traffic, especially for situations involving polymorphic attacks, these methods have low robustness and accuracy. 2. Solution 2 lacks flexibility and can only detect traffic that meets the parsing rules; 3. Solution 3 filters out malicious traffic by identifying these features; however, due to the lack of dynamics, this method can be easily overcome by bypass or adaptive attacks. For example, attackers can simply rearrange the protocol fields or modify the length of the packet header to evade detection. In addition, the use of machine learning algorithms requires a large amount of labeled training data. 4. Although the method based on association rule mining in Solution 4 can effectively reveal data features, it needs to continuously update and improve the association rule library when facing new protocols or variant forms. These construction and maintenance work will bring additional expenses.

[0003] The prior art discloses a customizable protocol parsing method for network interfaces (application number: 201710402897.4; authorization announcement number: CN106515276B), which is used to solve the problem of parsing non-standard, unregulated, and unstable communication protocols. The key points of the method are to use regular expressions to pre-analyze data packets to determine the key elements in the data packets, and then construct a parsing template based on the relationship between the key elements and the data packet content, and finally match the parsing template with the data packet content to obtain the parsing result. The disadvantages of the above-mentioned prior art are: relying on data packet content matching for custom protocol parsing, and still relying on manual participation in the analysis and parsing rules of key elements, which requires a lot of manpower and time to study new protocols, and lacks sufficient popularity and ecological support. In view of this, the present invention provides an audit method and system for actively learning parsing rules for custom protocols that can automatically generate parsing rules based on deep learning. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the existing technology cannot dynamically meet the needs of different custom protocols and relies on manually formulated protocol parsing rules. The detection efficiency of custom protocols is low. The purpose is to provide an audit method and system for actively learning parsing rules for custom protocols, which improves the accuracy and automation of protocol parsing, and provides an automatic parsing mechanism, thereby improving the efficiency and detection capability of custom protocol audits and solving the above-mentioned technical problems.

[0005] The present invention is achieved through the following technical solutions:

[0006] An audit method for actively learning parsing rules of a custom protocol, comprising:

[0007] Collect packets of custom protocols from network traffic and remove noise data;

[0008] Using deep learning algorithms and pattern matching technology, the packet structure and the first occurrence of multiple key fields of the data packet are identified;

[0009] Obtaining the preset encryption rules for each of the above key fields, encrypting each of the above key fields; sequentially splicing the encrypted key fields according to the above first appearance positions, marking the field number and field type of each of the above key fields to obtain a parsing key, and forming a preliminary parsing rule set for the above parsing key;

[0010] Decrypt the data packet of the custom protocol based on the parsing key; and perform threat review on the data packet of the custom protocol.

[0011] The above-mentioned collection of data packets of custom protocols from network traffic includes: receiving data packets containing port information, and using service port configuration items to generate a service acceleration table, which is stored in the form of a nested binary tree; through the above-mentioned service acceleration table, identifying data packets of custom protocols that match the above-mentioned port information.

[0012] The above uses deep learning algorithms and pattern matching technology to identify the packet structure of the data packet and the first occurrence of multiple key fields, including:

[0013] Identify the data packet structure and the first occurrence position of multiple key fields of the data packet through a regular expression or a specific matching algorithm;

[0014] The protocol feature model constructed by the deep learning algorithm is used to obtain the above-mentioned data packet structure and multiple key fields of the data packet of the custom protocol.

[0015] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key includes:

[0016] According to the field number and field type corresponding to each key field at the first appearance position, the preliminary parsing rule set is searched to obtain the parsing key, and the data packet of the custom protocol is decrypted using the parsing key;

[0017] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key also includes:

[0018] A search index of the above-mentioned preliminary parsing rule set is established for the above-mentioned parsing key that has been successfully decrypted to form a custom protocol search index table; when multiple parsing keys are found, each of the above-mentioned parsing keys is used to decrypt in sequence, and the above-mentioned search index of the above-mentioned parsing key that has been successfully decrypted is added to the above-mentioned custom protocol search index table; when the above-mentioned preliminary parsing rule set is searched, the search is performed according to the above-mentioned search index of the above-mentioned custom protocol.

[0019] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key also includes:

[0020] When the parsing key found is unable to decrypt the data packet, obtain the preset encryption rules specified by each of the key fields, and encrypt each of the key fields; sequentially splice the encrypted key fields according to the first appearance position, mark the field number and field type of each of the key fields, and generate a temporary key; decrypt the data packet of the custom protocol based on the temporary key.

[0021] The above threat review of the packets of the above custom protocols includes:

[0022] When the above parsing key cannot be found, the data packet is intercepted and the IP address of the port where the data packet is sent is added to the blacklist.

[0023] An audit system for active learning parsing rules of custom protocols, including:

[0024] Active learning module and Suricata parsing module; the above active learning module includes traffic collection module, feature extraction module and preliminary parsing rule generation module:

[0025] The above-mentioned traffic collection module collects data packets of the custom protocol from the network traffic and removes noise data;

[0026] The feature extraction module uses deep learning algorithms and pattern matching techniques to identify the data packet structure and the first occurrence position of multiple key fields of the data packet;

[0027] The above-mentioned module for generating preliminary parsing rules obtains the preset encryption rules of each of the above-mentioned key fields, encrypts each of the above-mentioned key fields; sequentially splices the above-mentioned encrypted key fields according to the above-mentioned first appearance positions, marks the field number and field type of each of the above-mentioned key fields to obtain a parsing key, and forms a preliminary parsing rule set for the above-mentioned parsing key;

[0028] The Suricata parsing module decrypts the data packets of the custom protocol based on the parsing key, and performs threat review on the data packets of the custom protocol.

[0029] The above system also includes a manual rule selection module; the above manual rule selection and editing module includes:

[0030] A rule viewing module for users to view the rule set generated by active learning;

[0031] The rule editing and optimization module allows users to edit the above generated parsing rules;

[0032] Multi-version rule management allows users to save and switch different versions of the above parsing rules.

[0033] An electronic device includes a memory, a processor, and a computer program running on the processor. When the processor executes the computer program, the steps of the audit method for actively learning and parsing rules of any one of the custom protocols are implemented.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of an audit method for actively learning and parsing rules of any one of the above-mentioned custom protocols.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] The present invention proposes an audit method for active learning parsing rules of a custom protocol, collects data packets of the custom protocol from network traffic, and removes noise data; uses deep learning algorithms and pattern matching technology to identify the data packet structure and the first appearance position of multiple key fields of the data packet, thereby obtaining the structure and content characteristics of the data packet; uses the preset encryption rules of each key field to encrypt, and sequentially splices the encrypted key fields according to the first appearance position, and marks the field number and field type of each key field to obtain a parsing key, and forms a preliminary parsing rule set for the parsing key; based on the parsing rules, the flow parsing and threat review of the custom protocol are realized. The present invention uses the automatic extraction of the flow characteristics of the custom protocol to generate parsing rules for further flow parsing and threat detection, improves the accuracy and automation of protocol parsing, and provides an automatic parsing mechanism, which improves the efficiency and detection capability of the protocol audit system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0038] Figure 1 A flowchart of an audit method for actively learning parsing rules of a custom protocol according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0040] Example

[0041] like Figure 1 As shown, the present invention is further described below in conjunction with the accompanying drawings. The application embodiment provides an audit method for actively learning parsing rules of a custom protocol, including:

[0042] Collect packets of custom protocols from network traffic and remove noise data;

[0043] Using deep learning algorithms and pattern matching technology, the packet structure and the first occurrence of multiple key fields of the data packet are identified;

[0044] Obtaining the preset encryption rules for each of the above key fields, encrypting each of the above key fields; sequentially splicing the encrypted key fields according to the above first appearance positions, marking the field number and field type of each of the above key fields to obtain a parsing key, and forming a preliminary parsing rule set for the above parsing key;

[0045] Decrypt the data packet of the custom protocol based on the parsing key; and perform threat review on the data packet of the custom protocol.

[0046] The data packets of custom protocols usually include the following data components: Frame header: used to identify the beginning of the data packet, usually contains a special byte sequence to facilitate the location of the starting position of the data packet in the data stream. Address information: includes source address and destination address, used to identify the sender and receiver of the data. In a one-to-many communication system, it may only contain the destination address information; in a many-to-many communication system, it contains both the source address and the destination address. Data type: used to identify the type of data carried in the data packet, such as text, image, audio, etc. Data length: indicates the number of bytes of valid data to help the receiving end correctly parse the data packet. Data block: the actual data content, located after the data type and data length. Check code: used to verify the integrity and correctness of the data. Common check codes include parity check, cyclic redundancy check (CRC), etc. Frame tail: used to identify the end of the data packet. Similar to the frame header, the frame tail also contains a special byte sequence to facilitate the location of the end of the data packet in the data stream.

[0047] Identify the content of one or more of the above data components, obtain the preset specific key fields, and record the first appearance position of the key fields. Encrypt each key field, sort the encrypted key fields according to the first appearance position, mark the field number and corresponding field type of each key field, and generate a parsing key. Among them, the field type includes numeric type, string type, date and time type, etc. The parsing key can be expressed in S-box form. Optionally, commonly used key fields and corresponding field numbers are specified according to user habits, and can be flexibly adjusted according to different periods to facilitate regular updating of encryption keys.

[0048] Optionally, the preset encryption rules for each key field can be all password rules except the empty password, which are set separately for each key field. The preset encryption rules for each key field can be set randomly, and different key fields can select the same preset encryption rules. The preset encryption rules can be selected by a technician in this field as a conventional encryption algorithm suitable for the field type. Specific examples are as follows: 3DES (Triple DES): It is a symmetric algorithm based on DES. A block of data is encrypted three times with three different keys, which has higher strength. RC2 and RC4: Symmetric algorithms, which use variable-length keys to encrypt large amounts of data, which is faster than DES. IDEA (International Data Encryption Algorithm) international data encryption algorithm, uses 128-bit keys to provide very strong security. RSA: It is a public key algorithm that supports variable-length keys. The length of the file block to be encrypted is also variable, and it is an asymmetric algorithm. AES (Advanced Encryption Standard): Advanced Encryption Standard, a symmetric algorithm, is the next generation of encryption algorithm standard, with fast speed and high security level.

[0049] Among them, after the encrypted key fields are spliced ​​in sequence according to the first appearance position, the field number and field type of each key field are marked to obtain the parsing key, and a multidimensional vector can be used to represent the field number and field type of each key field, so as to traverse each key field in sequence.

[0050] The above-mentioned collection of data packets of custom protocols from network traffic includes: receiving data packets containing port information, and using service port configuration items to generate a service acceleration table, which is stored in the form of a nested binary tree; through the above-mentioned service acceleration table, identifying data packets of custom protocols that match the above-mentioned port information.

[0051] When removing noise data from a data packet, a noise filter design can be used: Self-tracking filter solution: An intelligent controllable filter structure is used, whose signal input receives the custom protocol data signal to be filtered, and receives the control signal through the control input to adjust the center frequency of the filter's passband output. The control unit is coupled with the control input of the filter to generate a control signal in real time to ensure that the center frequency of the filter is basically consistent with the frequency of the input signal and dynamically track the changes in the data. This design enables the filter to effectively remove noise components in the data, especially in the data stream of the custom protocol, and can accurately capture and eliminate disturbance signals, thereby improving the accuracy and stability of signal processing.

[0052] Through the service acceleration table, you can quickly locate the custom protocol type and speed up the parsing of the custom protocol. Among them, the general service port configuration items mainly include the following aspects: Determine the required port: Determine the port number that needs to be opened according to the services and applications provided by the server. Common ports include port 80 for HTTP service and port 21 for FTP service. Log in to the server: Use SSH or other remote login tools to log in to the server as an administrator. Open the firewall settings: Open the corresponding firewall settings interface according to the server's operating system and firewall software. Add port rules: Add new port rules in the firewall settings interface. The rules should include the port number, protocol type (TCP or UDP), and allowed or denied operations. Save and apply settings: Save the firewall settings and apply the new port rules. Make sure that the settings are still valid after the server is restarted. Test port connectivity: Use telnet or other network tools to test whether the ports on the server are open and available. Security: When opening ports, be sure to pay attention to security issues. Avoid opening unnecessary ports to reduce potential security risks. Performance optimization: Reasonably set the number of ports and parameters based on the actual needs and performance of the server. Too many open ports may affect the performance and stability of the server. Backup and restore: Before setting the port, it is recommended to back up the relevant configuration files and settings.

[0053] The above-mentioned use of deep learning algorithms and pattern matching technology to identify the data packet structure of a data packet and the first occurrence position of multiple key fields includes: identifying the data packet structure of a data packet and the first occurrence position of multiple key fields through regular expressions or specific matching algorithms; obtaining the above-mentioned data packet structure and multiple above-mentioned key fields of the data packet of the custom protocol through a protocol feature model constructed by a deep learning algorithm.

[0054] Pattern matching technology can identify key fields in data packets through regular expressions or specific matching algorithms. For example, regular expressions can be used to match specific protocol header information. Optionally, if the custom protocol is identified as an online banking transaction protocol, the key fields of the protocol can be the transaction type, primary account number, and secondary account number obtained by identifying the data block. The protocol feature model is a deep neural network model pre-trained based on massive data, which can accurately identify the categories of the components of the data packet and the key field information contained. The use of deep learning algorithms can better and faster identify artificially constructed feature rules, avoiding misjudgments and rule omissions caused by insufficient human experience. Optionally, the deep learning algorithm uses a stacked neural network, because the network structure will not be affected by the number of nodes in the middle layer, which can solve the problem of reduced fitting accuracy of other machine learning algorithms on high-dimensional data and improve generalization capabilities.

[0055] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key includes:

[0056] According to the field number and field type corresponding to each key field at the first appearance position, the preliminary parsing rule set is searched to obtain the parsing key, and the data packet of the custom protocol is decrypted using the parsing key;

[0057] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key also includes:

[0058] A search index of the above-mentioned preliminary parsing rule set is established for the above-mentioned parsing key that has been successfully decrypted to form a custom protocol search index table; when multiple parsing keys are found, each of the above-mentioned parsing keys is used to decrypt in sequence, and the above-mentioned search index of the above-mentioned parsing key that has been successfully decrypted is added to the above-mentioned custom protocol search index table; when the above-mentioned preliminary parsing rule set is searched, the search is performed according to the above-mentioned search index of the above-mentioned custom protocol.

[0059] When a new custom protocol appears, you only need to add a corresponding search index.

[0060] The above-mentioned decryption of the data packet of the above-mentioned custom protocol based on the above-mentioned parsing key also includes:

[0061] When the parsing key found is unable to decrypt the data packet, obtain the preset encryption rules specified by each of the key fields, and encrypt each of the key fields; sequentially splice the encrypted key fields according to the first appearance position, mark the field number and field type of each of the key fields, and generate a temporary key; decrypt the data packet of the custom protocol based on the temporary key.

[0062] The above threat review of the packets of the above custom protocols includes:

[0063] When the above parsing key cannot be found, the data packet is intercepted and the IP address of the port where the data packet is sent is added to the blacklist.

[0064] In summary, the present application embodiment provides an audit method and system for actively learning parsing rules of a custom protocol:

[0065] The present invention proposes an audit method for active learning parsing rules of a custom protocol, collects data packets of the custom protocol from network traffic, and removes noise data; uses deep learning algorithms and pattern matching technology to identify the data packet structure and the first appearance position of multiple key fields of the data packet, thereby obtaining the structure and content characteristics of the data packet; uses the preset encryption rules of each key field to encrypt, and sequentially splices the encrypted key fields according to the first appearance position, and marks the field number and field type of each key field to obtain a parsing key, and forms a preliminary parsing rule set for the parsing key; based on the parsing rules, the flow parsing and threat review of the custom protocol are realized. The present invention uses the automatic extraction of the flow characteristics of the custom protocol to generate parsing rules for further flow parsing and threat detection, improves the accuracy and automation of protocol parsing, and provides an automatic parsing mechanism, which improves the efficiency and detection capability of the protocol audit system.

[0066] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An audit method for active learning parsing rules of a custom protocol, characterized in that: include: Collect packets of custom protocols from network traffic and remove noise data; Using deep learning algorithms and pattern matching technology, the packet structure and the first occurrence of multiple key fields of the data packet are identified; Obtaining preset encryption rules for each of the key fields, and encrypting each of the key fields; Sequentially splicing the encrypted key fields according to the first appearance positions, marking the field number and field type of each key field to obtain a parsing key, and forming a preliminary parsing rule set for the parsing key; The data packet of the custom protocol is decrypted based on the parsing key; and a threat review is performed on the data packet of the custom protocol.

2. According to claim 1, the audit method for active learning parsing rules of a custom protocol is characterized in that: The method of collecting data packets of a custom protocol from network traffic includes: receiving a data message containing port information, and using a service port configuration item to generate a service acceleration table, which is stored in the form of a nested binary tree; and identifying data packets of a custom protocol that match the port information through the service acceleration table.

3. According to claim 1, the audit method for active learning parsing rules of a custom protocol is characterized in that: The deep learning algorithm and pattern matching technology are used to identify the data packet structure and the first occurrence position of multiple key fields of the data packet, including: Identify the data packet structure and the first occurrence position of multiple key fields of the data packet through a regular expression or a specific matching algorithm; The protocol feature model constructed by the deep learning algorithm is used to obtain the data packet structure and multiple key fields of the data packet of the custom protocol.

4. According to claim 1, the audit method for active learning parsing rules of a custom protocol is characterized in that: The decrypting the data packet of the custom protocol based on the parsing key comprises: According to the field number and the field type corresponding to each key field at the first appearance position, searching the preliminary parsing rule set to obtain the parsing key, and using the parsing key to decrypt the data packet of the custom protocol; The decrypting the data packet of the custom protocol based on the parsing key also includes: A search index of the preliminary parsing rule set is established for the successfully decrypted parsing key to form a custom protocol search index table; when multiple parsing keys are found, each parsing key is used to decrypt in turn, and the search index of the successfully decrypted parsing key is added to the custom protocol search index table; when searching the preliminary parsing rule set, the search is performed according to the search index of the custom protocol.

5. According to claim 4, the audit method for active learning parsing rules of a custom protocol is characterized in that: The decrypting the data packet of the custom protocol based on the parsing key also includes: When the parsing key found cannot decrypt the data packet, the preset encryption rule specified by each key field is obtained to encrypt each key field; the encrypted key fields are sequentially spliced ​​according to the first appearance position, the field number and field type of each key field are marked, and a temporary key is generated; and the data packet of the custom protocol is decrypted based on the temporary key.

6. According to claim 5, the audit method for active learning parsing rules of a custom protocol is characterized in that: The threat review of the data packet of the custom protocol includes: When the parsing key cannot be found, the data packet is intercepted and the IP address of the port where the data packet is sent is added to a blacklist.

7. An audit system for active learning parsing rules of a custom protocol, characterized in that: It includes an active learning module and a Suricata parsing module; the active learning module includes a traffic collection module, a feature extraction module and a module for generating preliminary parsing rules: The traffic collection module collects data packets of the custom protocol from the network traffic and removes noise data; The feature extraction module uses a deep learning algorithm and pattern matching technology to identify the data packet structure and the first occurrence position of multiple key fields of the data packet; The generating module of preliminary parsing rules obtains preset encryption rules of each key field and encrypts each key field; Sequentially splicing the encrypted key fields according to the first appearance positions, marking the field number and field type of each key field to obtain a parsing key, and forming a preliminary parsing rule set for the parsing key; The Suricata parsing module decrypts the data packet of the custom protocol based on the parsing key; and performs threat review on the data packet of the custom protocol.

8. The audit system for active learning and parsing rules of a custom protocol according to claim 7 is characterized in that: The system also includes a manual rule selection module; The manual rule selection and editing module includes: A rule viewing module for users to view the rule set generated by active learning; A rule editing and optimization module allows the user to edit the generated parsing rules; Multi-version rule management allows users to save and switch between different versions of the parsing rules.

9. An electronic device comprising a memory, a processor, and a computer program running on the processor, characterized in that: When the processor executes the computer program, the steps of the audit method for active learning parsing rules of a custom protocol as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of an audit method for actively learning parsing rules of a custom protocol as described in any one of claims 1 to 6 are implemented.

Images