HTTPS flow monitoring method and device based on rule tree

Abstract

The invention relates to an HTTPS flow monitoring method and device based on a rule tree, and the method comprises the steps: constructing a matching rule base based on machine learning; constructing a balanced binary rule tree by the matching rule base; performing service conversion on the HTTPS service according to the matching rule base through the balanced binary rule tree; and carrying out HTTP flow monitoring on the HTTPS service after the service conversion. According to the method and the device, the HTTPS service is converted into the HTTP service after the corresponding rule matching is performed on the proxy based on the rule tree, so that the real-time HTTPS service traffic degradation is realized, the real-time monitoring of the HTTPS traffic is realized by monitoring the degraded HTTP traffic in real time, and the monitoring accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of network traffic management, and in particular relates to a HTTPS traffic monitoring method and device based on a rule tree. Background Art

[0002] Since the traditional HTTP protocol transmits data in plain text, the data can be easily stolen or tampered with during the transmission process. In order to avoid the above problems, application service providers usually use the HTTPS protocol for data transmission. For example, when accessing a website, even if the HTTP protocol is used to access the website, it will be forced to jump to the HTTPS protocol for access.

[0003] However, this also brings challenges to network traffic monitoring, because traditional traffic monitoring methods (such as DPI) cannot directly decrypt and analyze HTTPS traffic. Therefore, how to effectively monitor HTTPS traffic has become an urgent problem to be solved.

[0004] At present, client HTTPS traffic monitoring usually uses machine learning to analyze bypass traffic. For example, the C4.5 decision tree model is trained with information such as the packet length, packet interval, and TCP sequence number of the traffic packet, and the trained model is used to classify the bypass traffic. This classification method has certain false positives and false negatives, and the accuracy is not high. Some other client HTTPS traffic processing methods use fake certificates or illegally obtain official certificate private keys to decrypt the traffic and then perform detection or analysis. For example, forged server certificates are used to encrypt and decrypt traffic to complete detection. This method requires the installation of a self-made root certificate on the client in advance, so that the proxy server can establish HTTPS connections with the client and server respectively and provide normal services. In addition to the fake certificate, SSL / TSL encryption and decryption are required twice on the HTTPS proxy server, which greatly increases the computing burden of the proxy server and has low performance. Summary of the invention

[0005] In view of the above deficiencies in the prior art, the object of the invention is to provide a HTTPS traffic monitoring method and device based on a rule tree.

[0006] A first aspect of the present invention provides a rule-tree-based HTTPS traffic monitoring method, comprising:

[0007] S1: Build a matching rule library based on machine learning;

[0008] S2: constructing a balanced binary rule tree from the matching rule base;

[0009] S3: performing service conversion on the HTTPS service according to the matching rule base through the balanced binary rule tree;

[0010] S4: Monitor the HTTP traffic of the HTTPS service after service conversion.

[0011] According to a rule tree-based HTTPS traffic monitoring method provided by the present invention, step S1 further includes:

[0012] S11: configure a transparent proxy for the client, and select two groups of websites for proxying respectively;

[0013] S12: respectively collecting and obtaining HTTP communication information between the client and the proxy, wherein the HTTP communication information includes first HTTP communication information and second HTTP communication information;

[0014] S13: manually marking errors on the first HTTP communication information, and filling the modification rules corresponding to the first HTTP communication information and the error marking into a feature library to obtain a first data set; performing machine learning error marking on the first HTTP communication information, and filling the modification rules corresponding to the second HTTP communication information and the error marking into a feature library to obtain a second data set;

[0015] S14: Construct a matching rule library using the first data set and the second data set.

[0016] According to a rule-tree-based HTTPS traffic monitoring method provided by the present invention, the HTTP communication information in step S12 specifically includes: message information, status code, HTTP request header, HTTP response header, and control exception information.

[0017] According to a rule tree-based HTTPS traffic monitoring method provided by the present invention, the matching rule base in step S1 specifically includes: an attribute type table, a characteristic attribute and type relationship table, a characteristic table, and a matching rule table.

[0018] According to a rule tree-based HTTPS traffic monitoring method provided by the present invention, the balanced binary rule tree in step S2 specifically includes: a request message balanced binary rule tree and a response message balanced rule binary tree.

[0019] According to a rule tree-based HTTPS traffic monitoring method provided by the present invention, in step S3, when the proxy receives client data, the request message is balanced with the binary rule tree to match the rule base, and the service conversion is performed according to the matched rules; when the proxy receives server data, the request message is balanced with the binary rule tree to match the rule base, and the service conversion is performed according to the matched rules.

[0020] According to a rule tree-based HTTPS traffic monitoring method provided by the present invention, step S3 further includes:

[0021] S31: The proxy receives an input HTTPS service request from the client, and redirects the input HTTPS service request to an HTTP service request;

[0022] S32: the proxy performs general rule matching and specified rule matching on the HTTP service request according to the matching rule library to obtain an output HTTPS service request;

[0023] S33: The server receives the output HTTPS service request and returns a response message;

[0024] S34: The proxy performs general rule matching and specified rule matching on the response message according to the matching rule library to obtain a modified message;

[0025] S35: The proxy returns the modified message to the client based on the HTTP protocol.

[0026] The second aspect of the present invention further provides an HTTPS traffic monitoring device based on a rule tree, comprising:

[0027] Rule base construction module: used to build a matching rule base based on machine learning;

[0028] A rule tree construction module: used to construct a balanced binary rule tree from the matching rule base;

[0029] Conversion module: used to perform service conversion on the HTTPS service according to the matching rule library through the balanced binary rule tree;

[0030] Monitoring module: used to monitor HTTP traffic of HTTPS service after service conversion.

[0031] The third aspect of the present invention further provides an HTTPS traffic monitoring device based on a rule tree, comprising:

[0032] A memory and at least one processor, wherein instructions are stored in the memory;

[0033] At least one of the processors calls the instructions in the memory so that the rule-tree-based HTTPS traffic monitoring device executes the rule-tree-based HTTPS traffic monitoring method as described in any one of the above items.

[0034] A fourth aspect of the present invention further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the rule-tree-based HTTPS traffic monitoring method as described in any one of the above items is implemented.

[0035] The beneficial effects of the present invention are as follows:

[0036] The present invention provides a rule-tree-based HTTPS traffic monitoring method, device, equipment and storage medium, which adds a forward HTTPS proxy between a client browser and an HTTPS server, and the forward HTTPS proxy establishes a normal HTTPS connection with a real HTTPS server. After matching corresponding rules based on the rule tree on the proxy, the HTTPS service is converted into an HTTP service, so that the client browser and the HTTPS proxy server establish an HTTP connection, and real-time HTTPS service traffic degradation is achieved, so that real-time monitoring of HTTPS traffic can be achieved by real-time monitoring of the downgraded HTTP traffic.

[0037] The present invention realizes that there is no need for a proxy server to establish HTTPS connections with a client and a server respectively, thereby improving the efficiency of communication. In addition, when constructing a rule base, a semi-supervised machine learning method is adopted to automatically mark abnormal messages and fill in rules, and the matching rules of abnormal messages are automatically adapted, which is beneficial to improving the efficiency of adaptation. A balanced binary rule tree is adopted during matching to improve the speed at which the proxy server processes messages. Finally, the message sent to the client after matching based on the rule tree is an HTTP message, and the work of HTTPS traffic monitoring is completed in combination with an HTTP traffic monitoring device. During monitoring, decryption and analysis of a false certificate are not required, which not only ensures the security and integrity of user data between the client and the proxy server, but also improves the monitoring performance, reduces missed reports and false reports, and improves the accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0039] Figure 1 A schematic diagram of a flow chart of a rule-tree-based HTTPS traffic monitoring method provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the structure of a rule-tree-based HTTPS traffic monitoring device provided in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the service flow principle based on forward proxy provided in an embodiment of the present invention.

[0042] Reference numerals:

[0043] 100. Rule base construction module; 200. Rule tree construction module; 300. Conversion module; 400. Monitoring module. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0045] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0048] In order to better understand the present invention, the terms appearing in the embodiments of the present invention are first explained below.

[0049] HTTP (Hypertext Transfer Protocol): is a simple request-response protocol. It usually runs on top of TCP and specifies what kind of messages the client may send to the server and what kind of response it will get. The main function of the HTTP protocol is to stipulate the information transmission specifications between the WWW server and the browser. It is a protocol that both of them abide by. It allows communication and data transmission between the client and the server. It is the most widely used network protocol on the Internet.

[0050] TLS (Transport Layer Security): It is a transport layer security protocol, and its predecessor, Secure Sockets Layer (SSL), is a security protocol that aims to provide security and data integrity protection for Internet communications.

[0051] HTTPS (Hypertext Transfer Protocol Secure): It is an HTTP channel with security as its goal. It ensures the security of the transmission process through transmission encryption and identity authentication based on HTTP. HTTPS is based on HTTP with SSL / TLS added. The security foundation of HTTPS is SSL / TLS, so the details of encryption require SSL / TLS. HTTPS has a default port different from HTTP and an encryption / authentication layer (between HTTP and TCP). This system provides authentication and encryption communication methods. It is widely used for security-sensitive communications on the World Wide Web.

[0052] In order to better understand the present invention, the research background of the present invention is explained below.

[0053] Since the traditional HTTP protocol transmits data in plain text, the data can be easily stolen or tampered with during the transmission process. In order to avoid the above problems, application service providers usually use the HTTPS protocol for data transmission and adopt encryption algorithms.

[0054] The specific process is as follows: the user initiates an HTTP service request. The user enters http: / / www.xxx1.com in the browser. The Upgrade-Insecure-Requests in the request header is 1, which requires upgrading to a secure request. The xxx1 server will return a response with a status code of 30x and redirect to the location of https: / / www.xxx1.com. The user is forced to use the HTTPS protocol to access the service. The browser then initiates a server request to https: / / www.xxx1.com. The xxx1 server returns a response message based on the https protocol.

[0055] Traffic monitoring for HTTPS is divided into server monitoring and client monitoring; the server provides HTTPS services usually by using a combination of HTTPS reverse proxy services and ordinary HTTP application services. The reverse proxy server has the CA certificate of the service and can encrypt and decrypt the traffic. The traffic between the reverse proxy server and the application server is HTTP traffic, which can be monitored on this link. At this time, all traffic accessing the server is monitored. The method of server-side HTTPS traffic monitoring is relatively mature, and IDS detection uses the above method.

[0056] Client HTTPS traffic monitoring usually uses machine learning to analyze bypass traffic, such as using information such as the packet length, packet interval, TCP sequence number, etc. of the traffic packet to train the C4.5 decision tree model, and using the trained model to classify the bypass traffic. This method has certain false positives and false negatives, and the accuracy rate is not high; some other client HTTPS traffic processing methods are to use fake certificates or illegally obtain official certificate private keys to decrypt the traffic and then perform detection or analysis, such as using forged server certificates to encrypt and decrypt traffic to complete detection. This method requires the installation of a self-made root certificate on the client in advance, so that the proxy server can establish HTTPS connections with the client and server respectively to provide normal services. In addition to the fake certificate, it is also necessary to perform SSL / TSL encryption and decryption twice on the HTTPS proxy server, which greatly increases the computing burden of the proxy server and has low performance.

[0057] In order to solve the problem of how to perform real-time HTTPS traffic monitoring and ensure normal and efficient communication of the client when the proxy server can only establish an HTTPS connection with the server but cannot establish a normal connection with the client when there is no real server certificate, the present invention proposes an HTTPS traffic monitoring method and device based on rule tree matching.

[0058] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, the first aspect of the present invention provides a HTTPS traffic monitoring method based on a rule tree, comprising:

[0060] S1: Build a matching rule library based on machine learning.

[0061] The matching rule base in step S1 specifically includes: an attribute type table, a characteristic attribute and type relationship table, a characteristic table, and a matching rule table.

[0062] Furthermore, the rule base is used to store the list of message modification contents required to convert HTTPS to HTTP service protocol on the forward proxy. The rule base of the present invention is designed with four tables, namely, attribute type table, feature attribute and type relationship table, feature table, and matching rule table. The content of the feature table is the feature attribute information of the machine learning feature library, and the feature attributes can be increased or decreased according to different analysis scenarios.

[0063] Wherein, step S1 further comprises:

[0064] S11: Configure a transparent proxy for the client and select two groups of websites for proxying respectively.

[0065] S12: respectively collecting and obtaining HTTP communication information between the client and the proxy, wherein the HTTP communication information includes first HTTP communication information and second HTTP communication information.

[0066] The HTTP communication information in step S12 specifically includes: message information, status code, HTTP request header, HTTP response header, and control exception information.

[0067] S13: Manually mark the errors of the first HTTP communication information, and fill the modification rules corresponding to the first HTTP communication information and the error markings into the feature library to obtain a first data set; perform machine learning error marking on the first HTTP communication information, and fill the modification rules corresponding to the second HTTP communication information and the error markings into the feature library to obtain a second data set.

[0068] S14: Construct a matching rule library using the first data set and the second data set.

[0069] In steps S11 to S14, a transparent proxy is first configured for the client, and the rules in the proxy are empty at this time; several websites are selected, and the message information, status code, HTTP request header, response header, and console exception information of the HTTP communication between the client and the proxy are manually obtained and filled into the corresponding attribute fields of the feature table; then other websites are accessed by an automated tool, and the message information, status code, HTTP request header, response header, and console exception information between the abnormal client and the proxy are also extracted and filled into a new feature table isomorphic to the feature table; a semi-supervised machine learning method is used to annotate the error type of the data accessing other websites and fill in the modification rules, and the data is merged into the feature library after verification; a matching rule library is constructed according to the feature library.

[0070] Among them, when performing semi-supervised learning, it is necessary to construct a data set. The specific steps are to merge the first data set and the second data set, assign different weights to each attribute according to the attribute type, normalize the features of the data set, and then use the K-Means algorithm to perform K classification on the merged data set. The specific clustering steps are as follows.

[0071] First, determine the K value. The parameter K of the K-Means algorithm represents the number of generated clusters, so the algorithm will involve the selection of the K value. According to actual data verification, the sum of the distance within the cluster is inversely proportional to the number of clusters. When K is less than the number of real clusters, the sum of the distance of the cluster will decrease sharply as the K value increases. When K is equal to or higher than the number of real clusters, the trend of the sum within the cluster will decrease very slowly. The sum of the distance within the cluster after K-Means clustering is calculated for K = 1, 2, 3, 4, ..., n. When the sum of the distance within the cluster reaches a relatively stable value, the determined K value is close to the actual value.

[0072] After selecting the K value, randomly select K sample points from the sample as the initial cluster center, calculate the distance between each sample point and the K cluster center points, obtain the cluster center closest to the sample point, and assign the sample point to the cluster, using the center point of all sample points in each cluster to represent the cluster center;

[0073] Repeat the above steps until the cluster center remains unchanged or is less than a certain threshold, at which point K-Means clustering ends.

[0074] Traverse each class, calculate the number of C records with annotations in the cluster, and the types of each label, calculate the label type with a higher proportion, and then use the rules of this label type to process abnormal messages in this category. After modification, verify whether it is normal. If it is normal, move the attribute features, labels and rules to the feature table. Repeat the above steps if it is abnormal. If it is still abnormal after three iterations, mark it as unprocessable and manually process it later.

[0075] If there is no labeled data in the cluster, it is necessary to manually process several abnormal data in the cluster and label them. Subsequent iterations can be processed automatically. So far, through the above steps, only a small part of the data is required for manual standardization, and the construction of the feature table and rules is completed through the iterative execution of the above algorithm, which greatly improves the efficiency of manual standardization work.

[0076] S2: constructing a balanced binary rule tree from the matching rule base.

[0077] As the number of adapted websites or services increases, the number of feature attributes in the feature table will continue to increase. In order to perform rule matching more quickly or efficiently, the feature table is converted into the following type of matching rule table, and the rules contained in the Rule field in the feature table are extracted into the general rules, and other non-general parts are retained. A feature vector corresponds to a rule in the matching rule table, and is extracted as the rule type according to the attrType field in the Rule field.

[0078] The structure of the matching rule table is as follows. The core fields include the following:

[0079] Rule UUID: describes the unique ID of the rule;

[0080] Domain name: describes the scope of application of the rule, usually expressed in the form of a domain name. A domain name of * indicates a general rule that all requests need to match;

[0081] Subdomain: describes the subdomain under the main domain name to which this rule applies;

[0082] Path: describes the specific path of the request or response to which the rule applies;

[0083] Rule type: The rule types mainly include General, Request Header, Request Content, Response Header, Response Content and Custom. General corresponds to the general information of the request, Request Header corresponds to the request header information, Request Content corresponds to the request message content, Response Header corresponds to the response header information, Response Content corresponds to the response message content, and the rule corresponding to Custom is a customized code, which usually solves more complex rule matching;

[0084] Rule content: The rule content describes the specific rules to be executed.

[0085] The balanced binary rule tree in step S2 specifically includes: a request message balanced binary rule tree and a response message balanced binary rule tree.

[0086] When there are enough adapted websites, the rule table will increase dramatically, and the real-time query of the matching rule table will take a long time, affecting the query performance. In order to improve the query performance, the matching rule table is converted into a balanced binary rule tree, and the time complexity is increased from n to log n.

[0087] According to the content of the matching rule table, two balanced binary rule trees are constructed, corresponding to the balanced binary rule tree of the request message and the balanced binary rule tree of the response message. The request message rule tree contains rules of rule types General, RequestHeader, and Request Content, and the response message rule tree contains rules of rule types Response Header, Response Content, and Custom. First, the rules are divided into request rules, i.e., Request, and response rules, according to the rule type field.

[0088] The following takes the Request type as an example to illustrate the idea of ​​building a balanced binary rule tree.

[0089] First, determine the Key-Value content of the balanced binary tree node. The Key value is a string consisting of "domain name & subdomain name & Path", and the Value value is a List type, which is composed of a specific rule list.

[0090] Traverse the rule table of the Request type. If the domain name is *, add it to the passed rule list. Otherwise, enter the process of constructing a balanced binary tree.

[0091] When building a balanced binary tree, if the Key value of the node to be inserted is the same as the Key value of a node in the balanced binary tree, it means that there are multiple specific rules corresponding to the Key. In this case, there is no need to insert the node into the balanced binary tree. Instead, you only need to insert the corresponding rule into the specific rule list of the Value value. Otherwise, proceed to the next step.

[0092] When the key value of the node to be inserted is different from the key value of the balanced binary tree node, the node is inserted into the binary tree and the binary tree remains a balanced binary tree.

[0093] When the balanced binary tree is constructed, the rule list will be inserted into each node of the balanced binary tree to ensure that the general rules match.

[0094] According to the above steps, a balanced binary rule tree of the Request type can be constructed. Similarly, a balanced binary rule tree of the Response type can also be constructed.

[0095] S3: Performing service conversion on the HTTPS service according to the matching rule base through the balanced binary rule tree.

[0096] Among them, in step S3, when the proxy receives client data, it matches the rule base through the request message balancing binary rule tree, and performs service conversion according to the matched rules; when the proxy receives server data, it matches the rule base through the request message balancing binary rule tree, and performs service conversion according to the matched rules.

[0097] Wherein, step S3 further comprises:

[0098] S31: The proxy receives an input HTTPS service request from the client, and redirects the input HTTPS service request to an HTTP service request;

[0099] S32: the proxy performs general rule matching and specified rule matching on the HTTP service request according to the matching rule library to obtain an output HTTPS service request;

[0100] S33: The server receives the output HTTPS service request and returns a response message;

[0101] S34: The proxy performs general rule matching and specified rule matching on the response message according to the matching rule library to obtain a modified message;

[0102] S35: The proxy returns the modified message to the client based on the HTTP protocol.

[0103] In step S3, high-speed HTTPS service protocol conversion is implemented through the matching rule base in the aforementioned step S1 and the balanced binary rule tree in step S2. The specific matching process is as follows.

[0104] When the proxy receives a message, it first determines the type of the message, whether it is Request or Response; assigns the rule type that needs to be matched to the message ruleType; traverses the balanced binary rule tree corresponding to ruleType; determines the number of matching nodes after traversing the balanced binary rule tree, if it is equal to 1, executes subsequent steps, otherwise ends; executes the general rules and specific rule lists of the matching nodes, and the ruleType rules of a message are now executed.

[0105] Step S3 of the present invention aims to add a rule-tree-based forward proxy between the client browser and the HTTPS server, perform rule matching on the HTTPS message on the proxy, and convert the matched message into an HTTP message. Taking access to xxx1 as an example, the implementation of HTTPS service protocol conversion is described in detail in combination with the description of the above steps S31 to S34.

[0106] like Figure 3 As shown in the figure, the user initiates an HTTPS service request and enters https: / / www.xxx1.com in the browser; after receiving the HTTPS request, the proxy redirects the request to http: / / www.xxx1.com, forcing the proxy to establish an HTTP connection with the client, where Browser represents the user browser and Server represents the server.

[0107] After the client browser is redirected, the HTTP service request first reaches the proxy. The balanced binary rule tree of the request message is queried on the proxy. After matching the binary tree node, the general rule in the node is matched first, and then the specific rule of the www.xxx1.com service is matched. After matching, http: / / www.xxx1.com is converted to https: / / www.xxx1.com, and the proxy sends the HTTPS-based request to the xxx1 server.

[0108] The specific matching rules are stored in the rule list of the binary tree node. The rule examples are as follows.

[0109] Delete the cache policy parameters If-Modified-Since and Cache-Control; delete the security policy parameter Upgrade-Insecure-Requests for upgrading to https; replace http in the request URL with https and the port with port 443.

[0110] After receiving the https: / / www.xxx1.com request, the xxx1 server returns a response message. The proxy queries the balanced binary rule tree of the response message. After matching the node, it traverses the node's general rules and specific rules to process the message. The rule examples are as follows.

[0111] Delete security-related parameters such as Cross-Origin-Opener-Policy, Cross-Origin-Embedder-Policy, Cross-Origin-Resource-Policy, Access-Control-Allow-Origin, Cross-Origin-Opener-Policy-Report-Only, Strict-Transport-Security, X-Frame-Options, Public-Key-Pins, etc. in the response header to prevent the browser from forcibly converting HTTP to HTTPS. When a specific service sets a response policy, the content of the response header also needs to be modified accordingly. For example: Access-Control-Allow-Origin cannot be *. In this case, Access-Control-Allow-Origin needs to be set to the value of the request header Origin or other values; the HTTPS protocol in the URL, Refer, and Location of the response message header is changed to HTTP; the link referenced in the response message content is changed to HTTP protocol, and the message type may be Document or Script. After modifying the message content, HTTPS request services will no longer be introduced; modify the properties of specific cookies, remove the SameSite attribute, and remove cookies with specific tags.

[0112] Finally, the proxy returns a modified response message based on the HTTP protocol.

[0113] S4: Monitor the HTTP traffic of the HTTPS service after service conversion.

[0114] After steps S1 to S3, a forward proxy is added between the user browser and the xxx1 server, and rule matching is performed on the proxy based on the rule base for the accessed service. After the rule matching, the HTTPS service can be downgraded to the HTTP service. At this time, the HTTPS traffic is converted into HTTP traffic, and HTTPS traffic monitoring can be achieved using traditional traffic monitoring equipment or software.

[0115] like Figure 2 As shown, the present invention also provides an HTTPS traffic monitoring device based on a rule tree, comprising:

[0116] A rule base construction module 100 is used to construct a matching rule base based on machine learning;

[0117] A rule tree construction module 200 is used to construct a balanced binary rule tree from the matching rule base;

[0118] Conversion module 300: used to perform service conversion on the HTTPS service according to the matching rule base through the balanced binary rule tree;

[0119] Monitoring module 400: used to monitor HTTP traffic of the HTTPS service after service conversion.

[0120] The present invention also provides a rule-tree-based HTTPS traffic monitoring device, comprising:

[0121] A memory and at least one processor, wherein instructions are stored in the memory;

[0122] At least one of the processors calls the instructions in the memory so that the rule-tree-based HTTPS traffic monitoring device executes the rule-tree-based HTTPS traffic monitoring method as described in any one of the above items.

[0123] The present invention also provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, the rule-tree-based HTTPS traffic monitoring method as described in any one of the above items is implemented.

[0124] Furthermore, a rule-tree-based HTTPS traffic monitoring device provided by the present invention may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU), for example, one or more processors and memories, one or more storage media for storing applications or data, such as one or more massive storage devices, wherein the memories and storage media may be short-term storage or persistent storage, and the program stored in the storage medium may include one or more modules, each module may include a series of instruction operations in the rule-tree-based HTTPS traffic monitoring device, and further, the processor may be configured to communicate with the storage medium to execute a series of instruction operations in the storage medium on the rule-tree-based HTTPS traffic monitoring device.

[0125] It may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the rule-tree-based HTTPS traffic monitoring device structure provided by the present invention does not constitute a limitation on the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0126] The present invention provides a rule-tree-based HTTPS traffic monitoring method, device, equipment and storage medium. By building a matching rule base and using a balanced binary rule tree to perform service conversion, the key information in the HTTPS traffic can be matched and identified more accurately and quickly, thereby improving the accuracy and efficiency of traffic monitoring. Compared with the traditional monitoring method, complex HTTPS communication data can be processed more effectively. In the process of building the matching rule base, the present invention adopts machine learning technology to automatically mark errors in HTTP communication information and generate corresponding modification rules, thereby greatly enhancing the automation and intelligence level of the system, reducing the need for manual intervention and improving work efficiency. By configuring the proxy, the present invention can achieve seamless conversion and monitoring of HTTPS traffic without additional configuration or modification of the client, thereby ensuring the normal user experience and achieving effective monitoring of HTTPS traffic. The matching rule base and balanced binary rule tree design of the present invention have good scalability and flexibility, and can be dynamically adjusted and expanded according to actual needs, which helps to adapt to the ever-changing network security environment and business needs.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A rule-tree-based HTTPS traffic monitoring method, characterized in that: include: S1: Build a matching rule library based on machine learning; S2: constructing a balanced binary rule tree from the matching rule base; S3: performing service conversion on the HTTPS service according to the matching rule base through the balanced binary rule tree; S4: Perform HTTP traffic monitoring on the HTTPS service after service conversion.

2. According to the rule tree-based HTTPS traffic monitoring method of claim 1, it is characterized in that: Step S1 further comprises: S11: configure a transparent proxy for the client, and select two groups of websites for proxying respectively; S12: respectively collecting and obtaining HTTP communication information between the client and the proxy, wherein the HTTP communication information includes first HTTP communication information and second HTTP communication information; S13: manually marking errors on the first HTTP communication information, and filling the modification rules corresponding to the first HTTP communication information and the error marking into a feature library to obtain a first data set; performing machine learning error marking on the first HTTP communication information, and filling the modification rules corresponding to the second HTTP communication information and the error marking into a feature library to obtain a second data set; S14: Construct a matching rule library using the first data set and the second data set.

3. According to claim 2, a rule-tree-based HTTPS traffic monitoring method is characterized in that: The HTTP communication information in step S12 specifically includes: message information, status code, HTTP request header, HTTP response header, and control exception information.

4. According to the rule tree-based HTTPS traffic monitoring method of claim 1, it is characterized in that: The matching rule base in step S1 specifically includes: an attribute type table, a characteristic attribute and type relationship table, a characteristic table, and a matching rule table.

5. The HTTPS traffic monitoring method based on a rule tree according to claim 1 is characterized in that: The balanced binary rule tree in step S2 specifically includes: a request message balanced binary rule tree and a response message balanced rule binary tree.

6. The HTTPS traffic monitoring method based on rule tree according to claim 5 is characterized in that: In step S3, when the proxy receives client data, it matches the rule base through the request message balancing binary rule tree, and performs service conversion according to the matched rules; when the proxy receives server data, it matches the rule base through the request message balancing binary rule tree, and performs service conversion according to the matched rules.

7. The HTTPS traffic monitoring method based on a rule tree according to claim 1 is characterized in that: Step S3 further comprises: S31: The proxy receives an input HTTPS service request from the client, and redirects the input HTTPS service request to an HTTP service request; S32: the proxy performs general rule matching and specified rule matching on the HTTP service request according to the matching rule library to obtain an output HTTPS service request; S33: The server receives the output HTTPS service request and returns a response message; S34: The proxy performs general rule matching and specified rule matching on the response message according to the matching rule library to obtain a modified message; S35: The proxy returns the modified message to the client based on the HTTP protocol.

8. A rule-tree-based HTTPS traffic monitoring device, used to execute the method according to any one of claims 1 to 7, characterized in that: include: Rule base construction module: used to build a matching rule base based on machine learning; A rule tree construction module: used to construct a balanced binary rule tree from the matching rule base; Conversion module: used to perform service conversion on the HTTPS service according to the matching rule library through the balanced binary rule tree; Monitoring module: used to monitor HTTP traffic of HTTPS service after service conversion.

9. A rule-tree-based HTTPS traffic monitoring device, characterized in that: include: A memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory to enable the rule-tree-based HTTPS traffic monitoring device to execute the rule-tree-based HTTPS traffic monitoring method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed by the processor, the rule-tree-based HTTPS traffic monitoring method according to any one of claims 1 to 7 is implemented.

Images