HTTP message rule matching method, system, device and medium

Abstract

The present invention discloses an HTTP message rule matching method, system, device and medium. The method comprises: parsing an acquired HTTP message to obtain the parsed HTTP message; reading and parsing a configuration file for HTTP message rule matching to generate a custom rule set; extracting a head part to obtain extraction information; performing string matching on a body part according to the custom rule set based on a multi-mode matching algorithm improved by an AC automaton; and recording the number of occurrences and positions of the matched string in the body part information in each matching rule; determining a preliminary matching result according to the extracted information and the multi-mode matching result; performing a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, i.e., obtaining an HTTP message rule matching result. The present invention is lightweight, has no special requirements on the environment and equipment, has high rule matching efficiency and a low false alarm rate.

Description

Technical Field

[0001] The present invention belongs to the field of communication network technology, relates to communication network technology, and in particular to an HTTP message rule matching method, system, device and medium. Background Art

[0002] In computer networks, HTTP is a widely used application layer protocol that defines the message format for exchanging information and resource requests / responses between clients and servers. Generally, HTTP messages include a start line, header fields, and an optional body. To achieve web protection, threat detection, and user profiling, the content of HTTP messages needs to be filtered, detected, and modified, and information extraction and rule matching needs to be performed on HTTP messages in network devices or software.

[0003] Currently, regular expressions are commonly used as matching patterns when matching HTTP messages to describe the strings or character sequences that need to be found or replaced. Regular expressions have powerful and flexible expressive capabilities, but they also bring high computational complexity and memory consumption. In the field of network traffic monitoring, single-mode matching is usually used to match HTTP messages to achieve the identification and monitoring of special information. Although single-mode matching has a wide coverage range, it has low performance and high false positive and false negative rates.

[0004] To address the above issues, those skilled in the art have proposed technical solutions based on high-performance regular expression matching libraries. High-performance regular expression matching is a tool that can quickly match multiple regular expressions in large amounts of data. Examples include Intel's open-source matching library Hyperscan, based on hybrid automata technology, and Google's open-source matching library RE2, based on finite state automata (FSA). Intel's open-source matching library Hyperscan, based on hybrid automata technology, supports Perl Compatible Regular Expression (PCRE) syntax, simultaneous matching of regular expression groups, and stream operations, but requires specific hardware support, has a broad coverage range, and is not targeted. Google's open-source matching library RE2, based on finite state automata (FSA), supports RE2 regular expression syntax and guarantees linear time complexity and limited memory usage, but does not support some regular expression syntaxes in PCRE, Perl, or Python, and requires additional steps and dependencies, such as C++ compilers and Make tools. Furthermore, in the field of web protection, there are also some technical solutions that use multi-mode matching algorithms to segment HTTP packets, perform feature analysis, and forward them. Although these solutions are targeted, they suffer from shortcomings such as false positives and a special dependence on equipment.

[0005] In summary, the current technical solutions for rule matching on HTTP messages lack specificity for HTTP messages, have high false positive and false negative rates, and are particularly dependent on the environment and equipment. Summary of the Invention

[0006] In response to the technical problems existing in the prior art, the present invention provides an HTTP message rule matching method, system, device and medium to solve the technical problems that the current technical solution for rule matching of HTTP messages lacks specificity for HTTP messages, has a high false alarm rate and missed alarm rate, and has special dependence on the environment and equipment.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] The present invention provides an HTTP message rule matching method, comprising:

[0009] Parse the obtained HTTP message to obtain the parsed HTTP message;

[0010] Read and parse a configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules;

[0011] Extracting the Head portion of the parsed HTTP message to obtain extraction information;

[0012] Based on the multi-mode matching algorithm improved by the AC automaton, a string matching is performed on the body part of the parsed HTTP message according to the custom rule set to obtain a multi-mode matching result; and the number and position of the occurrence of the matched string in the body part of the parsed HTTP message in each matching rule are recorded to obtain the position information;

[0013] Determining a preliminary matching result based on the extracted information and the multi-modal matching result;

[0014] According to the location information, a location matching operation is performed on the preliminary matching result to obtain an accurate matching result, that is, the HTTP message rule matching result.

[0015] Furthermore, the parsed HTTP message includes an HTTP Request data packet and an HTTP Response data packet; and rule matching is performed on the HTTP Request data packet and the HTTP Response data packet respectively in a parallel processing manner.

[0016] Furthermore, the process of extracting the Head portion of the parsed HTTP message and obtaining the extracted information is as follows:

[0017] Using a programming language standard library and according to the requirements of the custom rule set on the Head part, the Head part in the parsed HTTP message is extracted to obtain the extracted information.

[0018] Furthermore, based on the multi-mode matching algorithm improved by the AC automaton, a string matching is performed on the body part of the parsed HTTP message according to the custom rule set to obtain a multi-mode matching result, as follows:

[0019] According to the key-value string of the body part involved in the custom rule set, a dictionary tree and a fail pointer are established;

[0020] Perform multi-mode matching on the body part of the parsed HTTP message according to the dictionary tree and the fail pointer, and record the matched strings into the results string array of the matching results, thereby obtaining the multi-mode matching results;

[0021] During the multi-mode matching process, it is checked whether the next matched string already exists in the results string array of the matching results; if not, the next matched string is added to the results string array of the matching results; if so, the next matched string is ignored.

[0022] Furthermore, the number of occurrences and positions of the matched string in the body of the parsed HTTP message in each matching rule are recorded. The process of obtaining the position information is as follows:

[0023] Constructing a custom structure; wherein the custom structure includes the character strings involved in the corresponding matching rules and the index slices recording the positions of the matched character strings;

[0024] Using a custom function to return the index of the string involved in the corresponding matching rule in the custom structure;

[0025] According to the index, obtain the index position of the matched string in the custom structure;

[0026] The acquisition position information can be obtained by adding the index position to the index slice and recording the number of times the index position is added.

[0027] Furthermore, the process of determining the preliminary matching result based on the extracted information and the multi-mode matching result is as follows:

[0028] According to the extracted information and the multi-mode matching result, a map function is used to perform a query to determine the matching rules that do not match the parsed HTTP message;

[0029] The unmatched matching rules are removed from the multi-mode matching results to obtain a preliminary matching result.

[0030] Furthermore, based on the location information, a location matching operation is performed on the preliminary matching result to obtain an accurate matching result, that is, a process of obtaining the HTTP message rule matching result, which is specifically as follows:

[0031] According to the position information, determine whether the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, and filter the preliminary matching result according to the determination result. The filtered preliminary matching result is the exact matching result;

[0032] The process of screening the preliminary matching results according to the judgment result is as follows:

[0033] If the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, the match is correct and the corresponding matching rule is retained; if not, the corresponding matching rule is removed.

[0034] The present invention also provides an HTTP message rule matching system, comprising:

[0035] The message parsing module is used to parse the acquired HTTP message and obtain the parsed HTTP message;

[0036] A reading and parsing module, configured to read and parse a configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules;

[0037] An information extraction module is used to extract the Head part of the parsed HTTP message to obtain extracted information;

[0038] a multi-mode matching module configured to perform string matching on the body portion of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by the AC automaton to obtain a multi-mode matching result; and to record the number and position of occurrences of the matched string in the body portion of the parsed HTTP message in each matching rule to obtain position information;

[0039] A preliminary matching module, configured to determine a preliminary matching result based on the extracted information and the multi-modal matching result;

[0040] The location matching module is used to perform a location matching operation on the preliminary matching result according to the location information to obtain an accurate matching result, that is, to obtain the HTTP message rule matching result.

[0041] The present invention also provides an HTTP message rule matching device, comprising:

[0042] memory for storing computer programs;

[0043] A processor is used to implement the steps of the HTTP message rule matching method when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the HTTP message rule matching method.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention provides an HTTP message rule matching method and system. By reading a configuration file for HTTP message rule matching, the method and system realize programming technical means for capturing, parsing, field extraction and rule matching of HTTP messages, which has the characteristics of precise coverage and strong professionalism. By extracting the Head part of the parsed HTTP message, the extracted information of the relevant fields for rule matching is screened out. The body part of the parsed HTTP message is matched with the multi-mode matching algorithm improved based on AC automaton to obtain the multi-mode matching result and the position information of the matched string, and the extraction information and the multi-mode matching result are used to perform preliminary matching judgment, thereby effectively improving the rule matching rate of the HTTP message. At the same time, by introducing the position information to perform the accurate judgment process of the position matching operation, the false alarm rate of the HTTP message rule matching is effectively reduced. Secondly, a modular, easy-to-implement lightweight programming algorithm is adopted, combined with existing open source technology, so that HTTP messages can be captured, parsed, information extracted and rule matched on a single server. The method and system have the characteristics of being lightweight, easy to deploy and having no special dependence on the environment and equipment.

[0047] Furthermore, in the multi-mode matching algorithm improved based on the AC automaton, in the process of performing string matching on the body part of the parsed HTTP message according to the custom rule set, a dictionary tree with a fail pointer is established using the set of all strings involved in the matching rules in the custom rule set, which is used for multi-mode matching of the HTTP message; at the same time, a position recording step is introduced into the multi-mode matching algorithm improved based on the AC automaton to record the index position of the matched string in the message, thereby obtaining position information; and then, after the preliminary matching result, the position information is used to further judge the form in which the fields involved in the matched rule appear in the HTTP message: for fields that appear in the form required by the rule set, the initial matching result is retained; otherwise, it is judged as a misjudgment and the matching result is corrected. Since the multi-mode matching algorithm has the characteristics of high speed and the unified reading of the strings involved in each matching rule in the rule set has the characteristics of low false positives; due to the addition of the judgment of the string appearance form after multi-mode matching, the requirements of the position relationship between fields in the rule set are utilized, which has the characteristics of low false positives, thereby effectively improving the algorithm performance and reducing the false positive rate and false negative rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the HTTP message rule matching method described in the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail in the following specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] As attached Figure 1 As shown, the present invention provides an HTTP message rule matching method, comprising the following steps:

[0051] Step 1: Parse the acquired HTTP message to obtain the parsed HTTP message. The acquired HTTP message is parsed using a programming method; specifically, the acquired HTTP message is parsed using a programming language standard library to obtain the parsed HTTP message.

[0052] Step 2: Divide the parsed HTTP message into packets to obtain HTTP Request packets and HTTP Response packets, and perform rule matching on the HTTP Request packets and the HTTP Response packets respectively in a parallel processing manner. Specifically, steps 3-7 below are performed on the HTTP Request packets and the HTTP Response packets respectively to achieve rule matching.

[0053] Step 3: Read and parse the configuration file for HTTP message rule matching to generate a custom rule set. The custom rule set includes a plurality of matching rules. Specifically, the configuration file for HTTP message rule matching is read and compiled using a programming method to generate a code-readable rule set format, thereby obtaining the custom rule set. When reading and parsing the configuration file, the matching rules in the configuration file are converted into a code-readable format so that, when matching rules, valid character strings can be extracted and matched according to the corresponding matching rules.

[0054] Step 4: Extract the Head portion of the parsed HTTP message to obtain extraction information.

[0055] The extraction process is as follows:

[0056] Using a programming language standard library and according to the requirements of the custom rule set on the Head part, the Head part in the parsed HTTP message is extracted to obtain the extracted information.

[0057] Step 5. Based on the multi-mode matching algorithm improved by the AC automaton, perform string matching on the body part of the parsed HTTP message according to the custom rule set to obtain a multi-mode matching result; and record the number and position of the appearance of the matched string in the body part information of the parsed HTTP message in each matching rule to obtain the position information.

[0058] Specifically, the process of performing string matching and obtaining multi-mode matching results is as follows:

[0059] Step 501: Create a dictionary tree and a fail pointer based on the key-value string of the body part involved in the custom rule set;

[0060] Step 502: Perform multi-mode matching on the body part of the parsed HTTP message according to the dictionary tree and the fail pointer, and record the matched string in the results string array of the matching result, so as to obtain the multi-mode matching result; wherein, during the multi-mode matching process, query whether the string matched next time already exists in the results string array of the matching result; if not, add the string matched next time to the results string array of the matching result; if so, ignore the string matched next time.

[0061] In step 5 of the present invention, in the process of performing string matching on the body part of the parsed HTTP message according to the custom rule set in the multi-mode matching algorithm improved based on the AC automaton, a code-implemented position recording operation step is introduced to record the number and position of the occurrence of the matched string in the body part information of the parsed HTTP message for subsequent position matching operations.

[0062] Specifically, the process of obtaining location information is as follows:

[0063] Step 511: Construct a custom structure; wherein the custom structure includes the character strings involved in the corresponding matching rules and the index slices recording the positions of the matched character strings;

[0064] Step 512: using a custom function to return the index of the character string involved in the corresponding matching rule in the custom structure;

[0065] Step 513: Obtain the index position of the matched string in the custom structure according to the index;

[0066] Step 514: Add the index position to the index slice and record the number of times the index position is added to obtain the acquisition position information.

[0067] Step 6: Determine a preliminary matching result based on the extracted information and the multi-mode matching result. Specifically, a map function is used to query the extracted information and the multi-mode matching result to determine matching rules that do not match the parsed HTTP message; the matching rules that do not match are removed from the multi-mode matching result to obtain a preliminary matching result.

[0068] Step 7: Perform a location matching operation on the preliminary matching result according to the location information to obtain an accurate matching result, that is, obtain the HTTP message rule matching result.

[0069] Specifically, based on the position information, it is determined whether the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, and the preliminary matching result is filtered according to the determination result. The filtered preliminary matching result is the exact matching result.

[0070] The process of screening the preliminary matching results according to the judgment result is as follows:

[0071] If the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, the match is correct and the corresponding matching rule is retained; if not, the corresponding matching rule is removed.

[0072] In step 7 of the present invention, a mathematical method is applied to establish an index relationship for the matching rule of the matched string, and it is judged whether the matched string appears in the text in the form of "Key=Value", thereby determining whether the rule is matched; based on the position information, for the rule that preliminarily matches the string involved, the position relationship of its Key-Value string in the text is reconfirmed to reduce false matches.

[0073] The HTTP message rule matching method of the present invention obtains a preliminary matching result based on the multi-mode matching result and the extracted information. Furthermore, a position matching operation is performed on the preliminary matching result based on the position information recorded during the multi-mode matching process to determine whether the matched string meets the key-value pair requirements in the rule set, thereby determining the correctly matched rule, thereby reducing the false positive rate and improving the HTTP rule matching efficiency. A programming method and an improved multi-mode matching algorithm based on AC automata are used to extract partial information involved in the rule set from the parsed HTTP message, thereby improving the HTTP message rule matching rate.

[0074] It should be noted that the process of extracting the Head part of the parsed HTTP message and establishing a dictionary tree and a fail pointer based on the Key-Value string of the body part involved in the custom rule set is related to the specific content of the custom rule set. The extraction information, dictionary tree and fail pointer contain all the fields involved in the custom rule set; when using location information to perform location matching operations, it is guided by the specific application scenarios of HTTP rule matching, including but not limited to: Case 1: When a rule is matched, a label is added and the end is ended; Case 2: Multiple rules are matched at the same time, multiple labels are added and the end is ended.

[0075] Performance testing:

[0076] In the present invention, in order to verify the target requirements of high rate and low false positive of the HTTP message rule matching method, the method performance test is carried out in the following manner. The specific process is as follows:

[0077] First, we configured the environment to visualize the results of HTTP message rule matching and the packet loss during the matching process. We deployed PF_RING on the server to capture HTTP message data packets. We deployed Prometheus to monitor packet loss during code execution. We deployed Elasticsearch to output the extracted HTTP message information and the labeled results after rule matching to ES. We also deployed Grafana and imported the Prometheus data source to visualize the packet loss during code execution.

[0078] Next, we conducted performance testing. In this test, we replayed HTTP packets of different sizes and containing different message contents at different speeds on the target network card. For false positive detection, we visualized the results of HTTP message rule matching and detected them by observing the information extracted from ES and the labels added by rule matching. For rate detection, we detected the packet loss displayed on the Grafana performance monitoring panel.

[0079] The HTTP message rule matching method described in the present invention performs multi-mode matching based on a multi-mode matching algorithm improved by an AC automaton, thereby improving the rule matching rate of HTTP messages. By adding location records and location matching operations, the false alarm rate of HTTP message rule matching is reduced, which can effectively solve the problems of low efficiency and high false alarms in HTTP message parsing and information extraction, and rule matching application scenarios.

[0080] The present invention also provides an HTTP message rule matching system, which includes a message parsing module, a reading and parsing module, an information extraction module, a multi-mode matching module, a preliminary matching module and a position matching module.

[0081] The message parsing module is used to parse the acquired HTTP message to obtain the parsed HTTP message; the reading and parsing module is used to read and parse the configuration file for HTTP message rule matching to generate a custom rule set; wherein, the custom rule set includes a plurality of matching rules; the information extraction module is used to extract the Head part of the parsed HTTP message to obtain extraction information; the multi-mode matching module is used to perform string matching on the body part of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by AC automaton to obtain a multi-mode matching result; and record the number and position of the occurrence of the matched string in the body part information of the parsed HTTP message in each matching rule to obtain position information; the preliminary matching module is used to determine the preliminary matching result based on the extraction information and the multi-mode matching result; the position matching module is used to perform a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, that is, the HTTP message rule matching result.

[0082] The present invention also provides an HTTP message rule matching device, comprising: a memory for storing a computer program; a processor for implementing the steps of an HTTP message rule matching method when executing the computer program. When the processor executes the computer program, the steps of the HTTP message rule matching method described above are implemented, for example: parsing an acquired HTTP message to obtain a parsed HTTP message; reading and parsing a configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules; extracting a header portion of the parsed HTTP message to obtain extraction information; performing string matching on the body portion of the parsed HTTP message according to the custom rule set based on a multi-mode matching algorithm improved by an AC automaton to obtain a multi-mode matching result; and recording the number of occurrences and positions of the matched string in each matching rule in the body portion of the parsed HTTP message to obtain position information; determining a preliminary matching result based on the extraction information and the multi-mode matching result; and performing a position matching operation on the preliminary matching result based on the position information to obtain an exact matching result, i.e., the HTTP message rule matching result.

[0083] Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned system are implemented, for example: a message parsing module, used to parse the obtained HTTP message to obtain the parsed HTTP message; a reading and parsing module, used to read and parse the configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules; an information extraction module, used to extract the Head part of the parsed HTTP message to obtain extraction information; a multi-mode matching module, used to perform string matching on the body part of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by AC automaton, to obtain a multi-mode matching result; and record the number and position of the occurrence of the matched string in the body part information of the parsed HTTP message in each matching rule to obtain position information; a preliminary matching module, used to determine a preliminary matching result based on the extraction information and the multi-mode matching result; and a position matching module, used to perform a position matching operation on the preliminary matching result based on the position information to obtain an exact matching result, that is, the HTTP message rule matching result.

[0084] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing preset functions, and the instruction segments are used to describe the execution process of the computer program in the HTTP message rule matching device.

[0085] For example, the computer program may be divided into a message parsing module, a reading and parsing module, an information extraction module, a multi-mode matching module, a preliminary matching module, and a position matching module. The specific functions of each module are as follows:

[0086] The message parsing module is used to parse the acquired HTTP message to obtain the parsed HTTP message; the reading and parsing module is used to read and parse the configuration file for HTTP message rule matching to generate a custom rule set; wherein, the custom rule set includes a plurality of matching rules; the information extraction module is used to extract the Head part of the parsed HTTP message to obtain extraction information; the multi-mode matching module is used to perform string matching on the body part of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by AC automaton to obtain a multi-mode matching result; and record the number and position of the occurrence of the matched string in the body part information of the parsed HTTP message in each matching rule to obtain position information; the preliminary matching module is used to determine the preliminary matching result based on the extraction information and the multi-mode matching result; the position matching module is used to perform a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, that is, the HTTP message rule matching result.

[0087] The HTTP message rule matching device can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The HTTP message rule matching device can include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the above are merely examples of HTTP message rule matching devices and do not constitute a limitation on the HTTP message rule matching device. The device can include more components than those described above, or a combination of certain components, or different components. For example, the HTTP message rule matching device can also include input and output devices, network access devices, buses, and the like.

[0088] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the HTTP message rule matching device, and connects various parts of the entire HTTP message rule matching device using various interfaces and lines.

[0089] The memory may be used to store the computer program and / or module, and the processor implements various functions of the HTTP message rule matching device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0090] The memory may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0091] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the HTTP message rule matching method. If the module / unit integrated into the HTTP message rule matching system is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0092] Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned HTTP message rule matching method by using a computer program to instruct relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-mentioned HTTP message rule matching method. The computer program includes computer program code, which can be in source code form, object code form, executable file, or a preset intermediate form.

[0093] The computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0094] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0095] Example 1

[0096] In this embodiment 1, the HTTP rule set whose programming language is golang and whose format is packetbeat is used as an example to match the HTTP message rules of a WeChat public platform, specifically includes the following steps:

[0097] Step 1: Use the programming language standard library to parse the acquired HTTP message to obtain the parsed HTTP message.

[0098] Step 2: Divide the parsed HTTP message into packets to obtain HTTP Request packets and HTTP Response packets. Since there are structural differences between the HTTP Request packets and the HTTP Response packets, there are differences in the Key fields when information is extracted. Therefore, steps 3-7 below are performed on the HTTP Request packets and the HTTP Response packets, respectively, to achieve rule matching.

[0099] Step 3: Read and parse the configuration file for HTTP message rule matching to generate a custom rule set. In this embodiment, the HTTP matching rules in packetbeat format and golang programming language are used as an example to automatically read and parse the configuration file in yaml format to meet different rule matching requirements. The specific process is as follows:

[0100] Customize the configuration file in YAML format, and create a custom structure according to the form of the configuration file in YAML format;

[0101] The matching rule information of the configuration file in the custom YAML format is read, and the Unmarshal function method of the YAML standard package of the Golang programming language is used to fill the matching rule information read in binary form into the custom structure to obtain the custom rule set, so as to perform valid string extraction and string matching when the rules are matched.

[0102] Step 4: Extract the Head portion of the parsed HTTP message to obtain extraction information. Specifically, use the http.Request.Header.Get() function in the net / http standard library of the golang programming language to extract the Head portion of the HTTPRequest packet to obtain extraction information corresponding to the HTTP Request packet; use the http.Response.Header.Get() function in the net / http standard library of the golang programming language to extract the Head portion of the HTTPResponse packet to obtain extraction information corresponding to the HTTP Response packet; and output the extraction information corresponding to the HTTP Request packet and the extraction information corresponding to the HTTP Response packet to ES for subsequent rule matching.

[0103] Specifically, the process of extracting the Head part of the HTTP Request data packet using the http.Request.Header.Get() function under the net / http standard library of the golang programming language is as follows: using the Get method in Request.Header, according to the Key value in the standard format of the HTTP Request data packet, obtains its corresponding Value value.

[0104] In this embodiment 1, the Get method in Request.Header is used to obtain the corresponding Value value according to the Key value in the standard format of the HTTP Request data packet. The passed Key value includes the "Accept" field, the "Accept-Encoding" field, the "Accept-Language" field, the "Authorization" field, the "If-Match / If-None-Match" field, the "If-Modified-Since / If-Unmodified-Since" field, the "If-Range" field and the "Referer" field.

[0105] Among them, the "Accept" field is used to inform the server user agent of the media type that can be processed and the corresponding priority of the media type; the "Accept-Encoding" field is used to inform the server client of the content encoding that can be supported and the priority of the corresponding content encoding; the "Accept-Language" field is used to inform the server client of the natural language set that can be processed and the priority of the corresponding language set; the "Authorization" field is used to inform the server user end authentication information; the "If-Match / If-None-Match" field is used to tell the server whether to process the message based on whether the following ETag matches or not. The "If-Modified-Since / If-Unmodified-Since" field is followed by a date. If-Modified-Since indicates that if the resource is updated after this date, the server will process the request. If-Unmodified-Since has the opposite logic. The "If-Range" field is followed by an Etag and must be used in conjunction with the Range field. If the Etag matches, the requested content is returned within the range specified by the Range field; otherwise, the entire content is returned. The "Referer" field is followed by a URI, which is the URI that initiated the request.

[0106] Specifically, the process of extracting the Head part of the HTTP Response data packet using the http.Response.Header.Get() function under the net / http standard library of the golang programming language is as follows: using the Get method in Request.Header, according to the Key value in the standard format of the HTTP Response data packet, obtains its corresponding Value value.

[0107] In this embodiment 1, the Get method in Request.Header is used to obtain the corresponding Value value according to the Key value in the standard format of the HTTP Response data packet. The passed-in Key value includes the "Accept-Ranges" field, the "Age" field, the "Etag" field, the "Location" field, the "Server" field, the "Vary" field, the "WWW-Authenticate" field, the "Last-Modified" field and the "Set-Cookie" field.

[0108] Among them, the "Accept-Ranges" field is used to inform the client whether the server supports range requests; among them, the Range field is used in the request for partial content; when the value of Accept-Ranges is bytes, it means that the server supports range requests; when it is none, it means that the server does not support the client's range requests; the "Age" field is used to inform the client how long ago the source server created the response; the "Etag" field is a unique string corresponding to the server resource currently requested by the server; the Etags of different resources are different, and the Etag will also be updated when the resource is updated; the "Location" field is used in conjunction with redirection; the "Server" field is used to indicate the server model used by the server; The "Vary" field is used to control the cache. The source server uses this field to convey commands about how to use the local cache to the proxy server. The "WWW-Authenticate" field is used for HTTP access authentication. The status code 401 Unauthorized must contain this field. This field is used to specify the client's authentication scheme, such as Basic / Digest. The "Last-Modified" field is followed by a time, marking the time when the requested resource was last modified on the server. The "Allow" field is used by the server to notify the client of all request methods supported by the server, such as GET and POST. The "Set-Cookie" field is used by the server to inform the client of various information in advance when it is about to start managing the client's status.

[0109] In this embodiment 1, the Get method in Request.Header is used to obtain the corresponding Value value according to the Key value in the standard format of the HTTP Request data packet, and the Get method in Request.Header is used to obtain the corresponding Value value according to the Key value in the standard format of the HTTP Response data packet. The input Key value also includes common fields; the common fields include the "Cache-Control" field, the "Connection" field, the "Transfer-Encoding" field, the "Via" field, the "Content-Encoding" field, the "Content-Language" field, and the "Content-Length" field.

[0110] Among them, the "Cache-Control" field is used to operate the working mechanism of the cache; the "Connection" field is used to control the header fields that are not forwarded to the proxy server and to manage persistent connections; the "Transfer-Encoding" field is used to indicate the encoding method used for the message during transmission, which is only valid for block encoding during HTTP / 1.1 message transmission; the "Via" field is used to track the transmission path of request and response messages. When the message passes through a proxy or gateway, the server information will be added to the Via field before forwarding it; the "Content-Encoding" field is used to indicate the encoding method of the message entity; the "Content-Language" field is used to indicate the natural language used by the message entity; the "Content-Length" field is used to specify the byte length of the message entity.

[0111] Step 5: Based on the multi-mode matching algorithm improved by the AC automaton, perform string matching on the body part of the parsed HTTP message according to the custom rule set to obtain a multi-mode matching result; and record the number and position of the occurrence of the matched string in the body part of the parsed HTTP message in each matching rule to obtain position information; wherein, the steps include:

[0112] Step 501: Create a dictionary tree and a fail pointer based on the key-value string of the body part involved in the custom rule set;

[0113] Step 502: Perform multi-mode matching on the body part of the parsed HTTP message according to the dictionary tree and the fail pointer, and record the matched string in the results string array of the matching result, so as to obtain the multi-mode matching result; wherein, during the multi-mode matching process, query whether the string matched next time already exists in the results string array of the matching result; if not, add the string matched next time to the results string array of the matching result; if so, ignore the string matched next time.

[0114] Step 503: Construct a custom structure patState; wherein the custom structure patState includes the character strings involved in the corresponding matching rule and the index slice pos[] recording the position of the matched character strings;

[0115] Step 512: Use the custom function InSlice to return the index index of the string involved in the corresponding matching rule in the custom structure;

[0116] Step 513: According to the index index, obtain the index position of the matched string in the custom structure patState;

[0117] Step 514: Add the index position to the index slice pos[] and record the number of times the index position is added to obtain the acquisition position information.

[0118] Step 6: Determine a preliminary matching result based on the extracted information and the multi-mode matching result. Specifically, a map function is used to query the extracted information and the multi-mode matching result to determine matching rules that do not match the parsed HTTP message; the matching rules that do not match are removed from the multi-mode matching result to obtain a preliminary matching result.

[0119] Step 7: Perform a location matching operation on the preliminary matching result according to the location information to obtain an accurate matching result, that is, obtain the HTTP message rule matching result.

[0120] In this embodiment 1, taking the Golang programming language as an example, a position matching operation is implemented, and the preliminary matching results are further accurately processed; that is, for the rules that match the corresponding strings, a mathematical method is applied to establish an index relationship, and it is judged whether the matched related strings conform to the "Key-Value" format, thereby determining whether the rules are matched; specifically, the following steps are included:

[0121] Based on the position information, it is determined whether the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result. The preliminary matching result is filtered based on the determination result, and the filtered preliminary matching result is the exact matching result.

[0122] The process of screening the preliminary matching results according to the judgment result is as follows:

[0123] If the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, the match is correct and the corresponding matching rule is retained; if not, the corresponding matching rule is removed.

[0124] For example, for "key1=abcd", if both key1 and abcd exist in results, the subsequent processing is:

[0125] (1) Use InSlice() to query the index x and index y of key1 and abcd in patterns;

[0126] (2) In patState, get the position record poskey where key1 appears in content and the position record posvalue where abcd appears in content from x and y;

[0127] (3) Check whether there is a satisfying condition between poskey and posvalue;

[0128] (4) If the element of posvalue[j]-poskey[i]=len("key1")+1 exists, it means that the rule is indeed matched. If it does not exist, it means it is a false positive and is ignored.

[0129] The description of the relevant parts of the HTTP message rule matching system, device and computer-readable storage medium provided in this embodiment 1 can be found in the detailed description of the corresponding parts of the HTTP message rule matching method described in this embodiment 1, and will not be repeated here.

[0130] Example 2

[0131] This embodiment 2 also provides a method for extracting information from an HTTP message, including the following steps:

[0132] Parse the obtained HTTP message to obtain the parsed HTTP message;

[0133] Obtaining an HTTP message rule matching result according to the HTTP message rule matching method described in Example 1;

[0134] According to the HTTP message rule matching result, an information extraction operation is performed on the parsed HTTP message to obtain an information extraction result of the HTTP message.

[0135] The HTTP message rule matching method described in the present invention adopts programming technology means for capturing, parsing, field extraction and rule matching of HTTP messages, and has the characteristics of precise coverage and strong professionalism, thereby realizing refined and efficient processing of HTTP message rule matching. The rule matching of HTTP messages has changed from a small functional component module of the overall rule matching technology to the object that the technology itself specifically solves.

[0136] In the present invention, an improved multi-mode matching algorithm based on AC automata is adopted. A dictionary tree with a fail pointer is established using the set of all strings involved in the matching rules in the rule set to perform multi-mode matching on HTTP messages. At the same time, a position recording operation is introduced to record the index position of the matched string in the message, that is, to obtain position information. After obtaining the preliminary matching result, the position information is used to further judge the form in which the fields involved in the matched rules appear in the HTTP message: for fields that appear in the form required by the rule set, the initial matching result is retained; otherwise, it is judged as a misjudgment and the matching result is corrected. Therefore, high-speed rule matching is achieved. The unified reading of the strings involved in each rule in the rule set has the characteristic of low missed detection. The addition of the judgment of the string appearance form after multi-mode matching utilizes the positional relationship requirements between fields in the rule set, which has the characteristic of low false positive. This technology is used to solve the above problems and can bring about improvements in performance, reduction of false positive rate and missed detection rate.

[0137] In the present invention, a modular, easy-to-implement lightweight programming algorithm is adopted, combined with existing open source technologies, to capture, parse, extract information and match rules on HTTP messages on a single server. It is lightweight, easy to deploy, and has no special dependence on the environment and equipment. It has the advantages of simplifying preparation work and reducing construction costs.

[0138] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention.

Claims

1. A HTTP message rule matching method, characterized in that: include: Parse the obtained HTTP message to obtain the parsed HTTP message; Read and parse a configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules; Extracting the Head portion of the parsed HTTP message to obtain extraction information; Based on the multi-mode matching algorithm improved by the AC automaton, a string matching is performed on the body part of the parsed HTTP message according to the custom rule set to obtain a multi-mode matching result; and the number and position of the occurrence of the matched string in the body part of the parsed HTTP message in each matching rule are recorded to obtain the position information; Determining a preliminary matching result based on the extracted information and the multi-modal matching result; Performing a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, that is, obtaining the HTTP message rule matching result; The process of performing string matching on the body part of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by AC automaton to obtain the multi-mode matching result is as follows: According to the key-value string of the body part involved in the custom rule set, a dictionary tree and a fail pointer are established; Perform multi-mode matching on the body part of the parsed HTTP message according to the dictionary tree and the fail pointer, and record the matched strings into the results string array of the matching results, thereby obtaining the multi-mode matching results; In the multi-mode matching process, it is checked whether the next matched string already exists in the results string array of the matching results; if not, the next matched string is added to the results string array of the matching results; if it exists, the next matched string is ignored; The process of performing a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, that is, obtaining the HTTP message rule matching result, is as follows: According to the position information, determine whether the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, and filter the preliminary matching result according to the determination result. The filtered preliminary matching result is the exact matching result; The process of screening the preliminary matching results according to the judgment result is as follows: If the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, the match is correct and the corresponding matching rule is retained; if not, the corresponding matching rule is removed.

2. The HTTP message rule matching method according to claim 1, wherein: The parsed HTTP message includes an HTTP Request data packet and an HTTP Response data packet; rule matching is performed on the HTTP Request data packet and the HTTP Response data packet respectively in a parallel processing manner.

3. The HTTP message rule matching method according to claim 1, wherein: The process of extracting the Head part of the parsed HTTP message and obtaining the extracted information is as follows: Using a programming language standard library and according to the requirements of the custom rule set on the Head part, the Head part in the parsed HTTP message is extracted to obtain the extracted information.

4. The HTTP message rule matching method according to claim 1, wherein: Record the number of occurrences and positions of the matched string in the body of the parsed HTTP message in each matching rule. The process of obtaining the position information is as follows: Constructing a custom structure; wherein the custom structure includes the character strings involved in the corresponding matching rules and the index slices recording the positions of the matched character strings; Using a custom function to return the index of the string involved in the corresponding matching rule in the custom structure; According to the index, obtain the index position of the matched string in the custom structure; The acquisition position information can be obtained by adding the index position to the index slice and recording the number of times the index position is added.

5. The HTTP message rule matching method according to claim 1, wherein: The process of determining the preliminary matching result based on the extracted information and the multi-mode matching result is as follows: According to the extracted information and the multi-mode matching result, a map function is used to perform a query to determine the matching rules that do not match the parsed HTTP message; The unmatched matching rules are removed from the multi-mode matching results to obtain a preliminary matching result.

6. An HTTP message rule matching system, characterized in that: include: The message parsing module is used to parse the acquired HTTP message and obtain the parsed HTTP message; A reading and parsing module, configured to read and parse a configuration file for HTTP message rule matching to generate a custom rule set; wherein the custom rule set includes a plurality of matching rules; An information extraction module is used to extract the Head part of the parsed HTTP message to obtain extracted information; a multi-mode matching module configured to perform string matching on the body portion of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by the AC automaton to obtain a multi-mode matching result; and to record the number and position of occurrences of the matched string in the body portion of the parsed HTTP message in each matching rule to obtain position information; A preliminary matching module, configured to determine a preliminary matching result based on the extracted information and the multi-modal matching result; A location matching module is used to perform a location matching operation on the preliminary matching result according to the location information to obtain an accurate matching result, that is, to obtain the HTTP message rule matching result; The process of performing string matching on the body part of the parsed HTTP message according to the custom rule set based on the multi-mode matching algorithm improved by AC automaton to obtain the multi-mode matching result is as follows: According to the key-value string of the body part involved in the custom rule set, a dictionary tree and a fail pointer are established; Perform multi-mode matching on the body part of the parsed HTTP message according to the dictionary tree and the fail pointer, and record the matched strings into the results string array of the matching results, thereby obtaining the multi-mode matching results; In the multi-mode matching process, it is checked whether the next matched string already exists in the results string array of the matching results; if not, the next matched string is added to the results string array of the matching results; if it exists, the next matched string is ignored; The process of performing a position matching operation on the preliminary matching result according to the position information to obtain an accurate matching result, that is, obtaining the HTTP message rule matching result, is as follows: According to the position information, determine whether the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, and filter the preliminary matching result according to the determination result. The filtered preliminary matching result is the exact matching result; The process of screening the preliminary matching results according to the judgment result is as follows: If the matched character string meets the key-value pair requirements of the matching rule in the preliminary matching result, the match is correct and the corresponding matching rule is retained; if not, the corresponding matching rule is removed.

7. An HTTP message rule matching device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of an HTTP message rule matching method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the HTTP message rule matching method as described in any one of claims 1 to 5 are implemented.

Images