A conditional probability-based recursive decoding pruning optimization method

Abstract

The present application relates to the technical field of recursive decoding, in particular to a recursive decoding pruning optimization method based on conditional probability, comprising: collecting website data sets of various websites; analyzing the website data sets to determine the encoding algorithm of the website data sets; determining the encoding order of the website data sets according to the encoding algorithm, and determining the decoding order of the website data sets based on the encoding order; counting the decoding order to determine the conditional probability of the decoding algorithm in the website data sets; defining a load, the decoding layers of the load are N layers, and the decoding algorithm of each layer is M; determining the conditional probability of the M decoding algorithms corresponding to the load according to the conditional probability of the website data sets; comparing the conditional probability of the M decoding algorithms with a probability threshold, and decoding the load according to the comparison result. The present application can adapt to the encoding style of different websites, and improve the speed and accuracy of the recursive decoding algorithm.

Description

Technical Field

[0001] This invention relates to the field of recursive decoding technology, and in particular to a recursive decoding pruning optimization method based on conditional probability. Background Technology

[0002] In the fields of network security and cloud security, network attack detection is primarily achieved by configuring corresponding security detection rule sets to match data transmitted over the network; if a match is found, it is considered an attack. These security detection rules are mainly written in rule languages ​​to describe the attack characteristics in the attack payload. Common network attack detection products using this method include WAF (Web Application Firewall) and IDS (Intrusion Detection System).

[0003] As the game between attackers and defenders unfolds, hackers have discovered that they can generate variant payloads by using multi-layered nested encoding to hide attack features from the original payload, ultimately bypassing attack detection rules. Multi-layered nested encoding means that each layer can use multiple types of encoding algorithms, and multiple layers of encoding algorithms can be nested. Assuming there are m supported encoding methods and n encoding layers, the number of variant attack payloads is:

[0004]

[0005] As the formula above shows, the number of variants increases exponentially with the number of layers. If the defender adds a large number of corresponding rules, the number of rules will reach tens of thousands, far exceeding the number of rules normally provided by the device. This not only makes it impossible to completely enumerate all variants, but also leads to a decrease in attack detection efficiency as the number of rules increases.

[0006] Recursive decoding to extract the original payload is a common method for solving the above problems. The core of recursive decoding lies in which algorithm to use for decoding next, which paths that do not need decoding to be removed, and when to stop decoding, which is commonly referred to as "pruning". CN114745206A proposes a nested coding attack payload detection method, system, device, and storage medium. It first scans the decoder feature set, and then calls the corresponding decoder to automatically and intelligently decode the multi-layered coded attack payload. It determines whether to continue decoding by calling an intelligent detection method that matches the detection rules with the payload. CN110598846A proposes a hierarchical recurrent neural network decoder and decoding method. The decoder divides the hierarchical recurrent neural network into layers, sets an internal decoding unit at the top layer, and sets a conjugate decoding unit at each layer except the top layer. This decoder can well complete the task of outputting "semi-structured data".

[0007] However, the above methods still have the following shortcomings: 1) After decoding each layer, it is necessary to determine whether to continue decoding by matching the result with the detection rules. This repeated "guess-verify" method of restoring the attack payload leads to a decrease in decoding efficiency. In fact, the two encoding algorithms that appear before and after the encoding layers often have a certain conditional probability relationship. This relationship can be used to "cut off" erroneous decoding paths, thereby improving decoding efficiency; 2) Each website has its own encoding style, and the existing decoding methods cannot adapt to the needs of different website encoding styles. For the same data, the same number of decoding probes are required for different scenarios, which cannot give full play to the advantages of different websites' encodings. Summary of the Invention

[0008] The purpose of this invention is to provide a recursive decoding pruning optimization method based on conditional probability. This method uses the conditional probability relationship before and after the number of encoding layers to determine the next decoding algorithm and to "prune" erroneous decoding paths. It also adapts to the encoding styles of different websites, thereby improving the speed and accuracy of the recursive decoding algorithm.

[0009] This invention provides a recursive decoding pruning optimization method based on conditional probability, comprising:

[0010] Collect website datasets from various websites;

[0011] The website dataset is analyzed to determine the encoding algorithm of the website dataset;

[0012] The encoding order of the website dataset is determined according to the encoding algorithm, and the decoding order of the website dataset is determined based on the encoding order;

[0013] The decoding order is statistically analyzed to determine the conditional probability of the decoding algorithm in the website dataset;

[0014] Define a load, wherein the load has N decoding layers and each layer has M decoding algorithms;

[0015] The conditional probabilities of the M decoding algorithms corresponding to the payload are determined based on the conditional probabilities of the website dataset.

[0016] The conditional probabilities of the M decoding algorithms are compared with the probability thresholds, and the load is decoded based on the comparison results.

[0017] In some embodiments of this application, website datasets from various websites are collected, including:

[0018] Write a web crawler to collect data sources from various websites, and set the data sources as the website datasets.

[0019] In some embodiments of this application, the website dataset is analyzed to determine the encoding algorithm of the website dataset, including:

[0020] The website dataset was analyzed using expert experience to determine its encoding characteristics.

[0021] The encoding algorithm for each layer of the website dataset is determined based on the encoding characteristics.

[0022] In some embodiments of this application, determining the encoding order of the website dataset according to the encoding algorithm, and determining the decoding order of the website dataset based on the encoding order, includes:

[0023] The encoding algorithm is represented by English letters, and the encoding order of the website dataset is determined according to the encoding algorithm corresponding to each layer of the website dataset;

[0024] Determine the reverse sequence of the encoding order, and set the reverse sequence as the decoding order of the website dataset.

[0025] In some embodiments of this application, the method for determining the conditional probability of the decoding algorithm in the website dataset is as follows:

[0026] ;

[0027] in, This represents the probability that the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is set to X.

[0028] In some embodiments of this application, the The method for determining it is as follows:

[0029]

[0030] in, This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is A. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is B. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is C. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is D, ...

[0031] In some embodiments of this application, determining the conditional probabilities of the M decoding algorithms corresponding to the payload based on the conditional probabilities of the website dataset includes:

[0032] The decoding algorithm of the payload is compared with the decoding algorithm of the website dataset;

[0033] If the decoding algorithm of the payload is the same as the centralized decoding algorithm of the website dataset, then the conditional probability of the centralized decoding algorithm of the website dataset is set as the conditional probability of the decoding algorithm of the payload.

[0034] In some embodiments of this application, the conditional probabilities of the M decoding algorithms are compared with probability thresholds, and the load is decoded based on the comparison results, including:

[0035] The conditional probabilities of the M decoding algorithms are compared sequentially with the probability thresholds;

[0036] If the conditional probability of the current decoding algorithm is greater than the probability threshold, then the current decoding algorithm is used for decoding. At this time, the number of decoding layers is incremented by one, and the load is replaced with the decoded load.

[0037] If the conditional probability of the current decoding algorithm is less than the probability threshold, the current decoding algorithm is skipped, and the conditional probability of the next decoding algorithm is compared with the probability threshold until decoding stops.

[0038] In some embodiments of this application, the method further includes: after decoding stops, matching the decoded payload with the detection rules; if the decoded payload matches the detection rules, then putting the decoded payload into the website dataset and recalculating the conditional probability of the website dataset.

[0039] In some embodiments of this application, if the decoded payload does not match the detection rules, the decoded payload is not retained.

[0040] The advantages and effects of this application compared to the prior art are as follows:

[0041] 1) Instead of performing decoding detection based on the matching results of detection rules after each decoding, the conditional probability is judged to determine whether the next decoding is needed. This can "cut off" the wrong decoding path more quickly and improve the efficiency of decoding detection.

[0042] 2) For data from different websites, this invention statistically analyzes the decoding conditional probability of each website, making full use of the encoding style of each website to improve the adaptability of decoding detection;

[0043] 3) Once decoding is complete, if it matches the detection rules, it will be put into the original data source to iteratively update the data and improve the accuracy of conditional probability statistics.

[0044] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a recursive decoding pruning optimization method based on conditional probability according to the present invention. Detailed Implementation

[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the ordinary meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof, without excluding other elements or objects. The terms "first," "second," and similar words used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "upper," "lower," "left," "right," "front," "rear," "vertical," "horizontal," "side," and "bottom," indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are merely relational terms determined for the convenience of describing the structural relationships of the various components or elements of the present invention, and do not specifically refer to any component or element in the invention, nor should they be construed as limiting the invention. Terms such as "fixed," "connected," and "linked," etc., should be interpreted broadly, indicating that it can be a fixed connection, an integral connection, or a detachable connection; it can be a direct connection or an indirect connection through an intermediate medium. For researchers or technicians in the field, the specific meaning of the above terms in this invention can be determined according to the specific circumstances, and they should not be construed as limitations on this invention.

[0049] Example

[0050] This invention provides a recursive decoding pruning optimization method based on conditional probability, such as... Figure 1 As shown, it includes:

[0051] S1 collects website datasets from various websites.

[0052] S2, Analyze the website dataset to determine the encoding algorithm of the website dataset.

[0053] S3, determine the encoding order of the website dataset according to the encoding algorithm, and determine the decoding order of the website dataset based on the encoding order.

[0054] S4, Statistically analyze the decoding order to determine the conditional probability of the decoding algorithm in the website dataset.

[0055] S5, define a payload, the number of decoding layers of the payload is N, and the number of decoding algorithms for each layer is M.

[0056] S6. Determine the conditional probabilities of the M decoding algorithms corresponding to the load based on the conditional probabilities of the website dataset.

[0057] S7. The conditional probabilities of the M decoding algorithms are compared with the probability thresholds, and the load is decoded based on the comparison results.

[0058] In some embodiments of this application, website datasets from various websites are collected, including:

[0059] Write a web crawler to collect data sources from various websites, and set the data sources as the website datasets.

[0060] In this embodiment, in order to achieve website adaptability and make full use of the coding style of each website, the present invention collects the data sources of each website by writing web crawler software.

[0061] In some embodiments of this application, the website dataset is analyzed to determine the encoding algorithm of the website dataset, including:

[0062] The website dataset was analyzed using expert experience to determine its encoding characteristics.

[0063] The encoding algorithm for each layer of the website dataset is determined based on the encoding characteristics.

[0064] In some embodiments of this application, determining the encoding order of the website dataset according to the encoding algorithm, and determining the decoding order of the website dataset based on the encoding order, includes:

[0065] The encoding algorithms are represented by English letters, and the encoding order of the website dataset is determined according to the encoding algorithms corresponding to each layer of the website dataset.

[0066] Determine the reverse sequence of the encoding order, and set the reverse sequence as the decoding order of the website dataset.

[0067] In this embodiment, common encoding algorithms include URL encoding, Base64 encoding, Unicode encoding, JSON encoding, and Escape / Unescape encoding. This invention defines the use of 26 English letters to represent different encoding types, such as A for URL encoding, B for Base64 encoding, C for Unicode encoding, and D for JSON encoding. Therefore, for each piece of collected data, a letter sequence can be used to represent the encoding algorithm used at each layer, for example, ABD indicates that the first layer uses URL encoding, the second layer uses Base64 encoding, and the third layer uses JSON encoding.

[0068] Each encoding has its own encoding characteristics. For example, URL encoding uses %xx, Base64 encoding uses +, / , and =, Unicode encoding uses %, %u, and +, and JSON encoding uses [, {, and ", etc. Based on these encoding characteristics, this invention can first obtain the encoding algorithm for each layer of each data through expert experience, then obtain the reverse sequence of the encoding, i.e., the decoding sequence, and then calculate the conditional probability of each decoding situation. For example, if the encoding order of a data is AB, the conditional probability is calculated based on the decoding direction, i.e., the conditional probability of the decoding order being BA.

[0069] In some embodiments of this application, the method for determining the conditional probability of the decoding algorithm in the website dataset is as follows:

[0070] ;

[0071] in, This represents the probability that the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is set to X.

[0072] In this embodiment, the current decoding algorithm X can be one or more.

[0073] In some embodiments of this application, the The method for determining it is as follows:

[0074]

[0075] in, This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is A. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is B. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is C. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is D, ...

[0076] In this embodiment, the statistics for other conditional probabilities are similar to those described above, for example, That is, we need to count the number of times the next layer decoding algorithm is C when the current first layer decoding algorithm and the current second layer decoding algorithm are A and B respectively, and count the number of times the current decoding algorithm is all decoding cases when the current first layer decoding algorithm and the current second layer decoding algorithm are A and B respectively.

[0077] In some embodiments of this application, determining the conditional probabilities of the M decoding algorithms corresponding to the payload based on the conditional probabilities of the website dataset includes:

[0078] The decoding algorithm of the payload is compared with the centralized decoding algorithm of the website dataset.

[0079] If the decoding algorithm of the payload is the same as the centralized decoding algorithm of the website dataset, then the conditional probability of the centralized decoding algorithm of the website dataset is set as the conditional probability of the decoding algorithm of the payload.

[0080] In some embodiments of this application, the conditional probabilities of the M decoding algorithms are compared with probability thresholds, and the load is decoded based on the comparison results, including:

[0081] The conditional probabilities of the M decoding algorithms are compared with the probability thresholds in turn.

[0082] If the conditional probability of the current decoding algorithm is greater than the probability threshold, then the current decoding algorithm is used for decoding. At this time, the number of decoding layers is incremented by one, and the load is replaced with the decoded load.

[0083] If the conditional probability of the current decoding algorithm is less than the probability threshold, the current decoding algorithm is skipped, and the conditional probability of the next decoding algorithm is compared with the probability threshold until decoding stops.

[0084] In this embodiment, the probability threshold is 10%.

[0085] Furthermore, assuming that after a payload is decoded using algorithm A, the conditional probability that it can be decoded using algorithm B next time is... Here, P(A) and P(B|A) are the conditional probabilities calculated previously. If P(AB) is greater than the probability threshold, then it means that the B algorithm can be tried for decoding next time; otherwise, it is skipped.

[0086] Similarly, due to After decoding with AB, if it is necessary to determine whether the C algorithm can be used for decoding next time, it is only necessary to calculate the probability of P(ABC), where P(AB) is usable. This is deduced through reasoning. By calculating this probability, erroneous decoding paths can be "cut off," thereby improving decoding and detection efficiency.

[0087] In some embodiments of this application, the method further includes: after decoding stops, matching the decoded payload with the detection rules; if the decoded payload matches the detection rules, then putting the decoded payload into the website dataset and recalculating the conditional probability of the website dataset.

[0088] In some embodiments of this application, if the decoded payload does not match the detection rules, the decoded payload is not retained.

[0089] In this embodiment, when decoding stops, if the decoded payload matches the detection rule, the decoded payload is added to the website dataset as new statistical data, and the conditional probability of each decoding case is iteratively updated according to the decoding situation, thereby achieving the purpose of optimizing the conditional probability.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

[0091] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

Claims

1. A recursive decoding pruning optimization method based on conditional probability, characterized in that, include: Collect website datasets from various websites; The website dataset is analyzed to determine the encoding algorithm of the website dataset; The encoding order of the website dataset is determined according to the encoding algorithm, and the decoding order of the website dataset is determined based on the encoding order; The decoding order is statistically analyzed to determine the conditional probability of the decoding algorithm in the website dataset; Define a load, wherein the load has N decoding layers and each layer has M decoding algorithms; The conditional probabilities of the M decoding algorithms corresponding to the payload are determined based on the conditional probabilities of the website dataset. The conditional probabilities of the M decoding algorithms are compared with the probability thresholds, and the load is decoded based on the comparison results.

2. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, Collect website datasets from various websites, including: Write a web crawler to collect data sources from various websites, and set the data sources as the website datasets.

3. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, Analyzing the website dataset to determine its encoding algorithm includes: The website dataset was analyzed using expert experience to determine its encoding characteristics. The encoding algorithm for each layer of the website dataset is determined based on the encoding characteristics.

4. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, Determining the encoding order of the website dataset according to the encoding algorithm, and determining the decoding order of the website dataset based on the encoding order, includes: The encoding algorithm is represented by English letters, and the encoding order of the website dataset is determined according to the encoding algorithm corresponding to each layer of the website dataset; Determine the reverse sequence of the encoding order, and set the reverse sequence as the decoding order of the website dataset.

5. The recursive decoding pruning optimization method based on conditional probability according to claim 4, characterized in that, The method for determining the conditional probability of the decoding algorithm in the website dataset is as follows: ； in, This represents the probability that the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is Y. This indicates the number of times the current decoding algorithm is set to X.

6. The recursive decoding pruning optimization method based on conditional probability according to claim 5, characterized in that, The The method for determining it is as follows: in, This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is A. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is B. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is C. This indicates the number of times the current decoding algorithm is X and the next layer decoding algorithm is D, ...

7. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, The conditional probabilities of the M decoding algorithms corresponding to the payload are determined based on the conditional probabilities of the website dataset, including: The decoding algorithm of the payload is compared with the decoding algorithm of the website dataset; If the decoding algorithm of the payload is the same as the centralized decoding algorithm of the website dataset, then the conditional probability of the centralized decoding algorithm of the website dataset is set as the conditional probability of the decoding algorithm of the payload.

8. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, The conditional probabilities of the M decoding algorithms are compared with probability thresholds, and the load is decoded based on the comparison results, including: The conditional probabilities of the M decoding algorithms are compared sequentially with the probability thresholds; If the conditional probability of the current decoding algorithm is greater than the probability threshold, then the current decoding algorithm is used for decoding. At this time, the number of decoding layers is incremented by one, and the load is replaced with the decoded load. If the conditional probability of the current decoding algorithm is less than the probability threshold, the current decoding algorithm is skipped, and the conditional probability of the next decoding algorithm is compared with the probability threshold until decoding stops.

9. The recursive decoding pruning optimization method based on conditional probability according to claim 1, characterized in that, The method further includes: when decoding stops, matching the decoded payload with the detection rules; if the decoded payload matches the detection rules, then putting the decoded payload into the website dataset and recalculating the conditional probability of the website dataset.

10. The recursive decoding pruning optimization method based on conditional probability according to claim 9, characterized in that, If the decoded payload does not match the detection rules, the decoded payload will not be retained.

Images