Methods, apparatus, electronic devices and storage media for determining data sources
By constructing decision trees and selecting information gain, a rule-based tree model is built, which solves the problem of high false alarm rate in data source identification in existing technologies and achieves high accuracy in data source identification.
Patent Information
- Application Number
- CN202310842168.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-10
AI Technical Summary
Existing methods for determining data sources have a high false alarm rate, making it difficult to accurately identify the source of data from illegitimate requests.
A decision tree construction method is adopted. By acquiring the rule set of network traffic data, a decision tree model is constructed. Information gain is used to select nodes, generate a rule tree model, analyze the characteristic rules of network traffic data, and score to determine the data source.
It improves the accuracy of data source identification, reduces the false alarm rate, and achieves efficient and accurate source identification of network traffic data.
Smart Images

Figure CN117014330B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for determining data sources. Background Technology
[0002] With the rapid development of technologies such as 5G and cloud computing, and the widespread interconnection of the Internet of Things, more and more basic information exposure points and surfaces are being revealed. Attackers often use illegal surveying and mapping to launch attacks on these exposed surfaces, leading to the leakage of critical basic information and causing huge losses to individuals and enterprises. Quickly identifying and locating illegal requests can block attacks at the source and protect critical basic information from being leaked. How to quickly determine the data source of illegal requests is crucial for protecting network information security.
[0003] In related technologies, clustering is typically used to group similar traffic data to determine the data source. However, this method has a high false positive rate.
[0004] Therefore, accurately determining the data source and reducing the false alarm rate of data source determination is an urgent problem to be solved. Summary of the Invention
[0005] To address the problems existing in the prior art, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for determining the data source.
[0006] This invention provides a method for determining the source of data, comprising:
[0007] Obtain a first set of rules corresponding to network traffic data, wherein the first set of rules is a first set of words extracted from the network traffic data;
[0008] If the first rule set fails to match the preset whitelist, the first rule set is input into the decision tree to obtain the first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source.
[0009] If the network traffic data does not belong to the same source, the score value of the network traffic data belonging to each data source is determined based on the first result.
[0010] Optionally, the decision tree is constructed in the following manner:
[0011] Step A: Preprocess the network traffic data to generate preprocessed target network traffic data, wherein the target network traffic data includes the first rule set;
[0012] Step B: Calculate the first information gain for each word in the first word set, and select the first word with the largest first information gain as the root node of the decision tree;
[0013] Step C: Calculate the second information gain corresponding to each word in the second word set, select the second word with the largest second information gain as the split node of the root node, and the second word set is the word set in the first rule set after removing the first word;
[0014] Step D: Calculate the third information gain corresponding to each word in the third word set, select the third word with the largest third information gain as the sub-split node of the split node, repeat step D until all words in the first word set are selected, or the number of layers of the decision tree reaches a preset threshold, and generate the leaf node of the decision tree; the third word set is the word set in the second word set after removing the second word.
[0015] Optionally, the step of preprocessing the network traffic data to generate preprocessed target network traffic data includes:
[0016] The non-readable bytes in the network traffic data are deleted to obtain the valid data payload corresponding to the network traffic data;
[0017] The valid data payload is decoded, and dynamic data is removed to generate the decoded valid data payload;
[0018] The decoded valid data payload is segmented to generate the target network traffic data.
[0019] Optionally, the first result includes at least one of the following: the number of valid data payloads corresponding to the network traffic data; the total number of data sources containing the same rule; the weight value of each rule; the probability of each data source in each valid data payload; and the rule representing any path from the root node to the leaf node in the decision tree.
[0020] The step of determining the score value of each data source based on the first result includes:
[0021] The score is determined based on the number of valid data payloads corresponding to the network traffic data, the total number of data sources containing the same rule, the weight value of each rule, and the probability of each data source in each valid data payload.
[0022] Optionally, the weight value is obtained in the following way:
[0023] Obtain the number of times each rule appears in each of the stated data sources;
[0024] Based on the frequency of occurrence, the weight value of each rule is determined using the entropy method.
[0025] Optionally, the method further includes:
[0026] The first rule set is matched with the second rule set in the whitelist to obtain a matching result; the second rule set is the rule set associated with each of the data sources.
[0027] If the first rule set and the second rule set are successfully matched, the target data source corresponding to the network traffic data is determined based on the matching result.
[0028] The present invention also provides a device for determining the source of data, comprising:
[0029] The first acquisition module is used to acquire a first rule set corresponding to network traffic data, wherein the first rule set is a first word set extracted from the network traffic data;
[0030] The input module is used to input the first rule set into a decision tree when the first rule set fails to match the preset whitelist, and obtain the first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source.
[0031] The first determining module is used to determine, based on the first result, the score value of the network traffic data belonging to each data source when the network traffic data does not belong to the same source.
[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for determining the data source as described above.
[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the data source as described in any of the above.
[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for determining the data source as described in any of the above.
[0035] The present invention provides a method, apparatus, electronic device, and storage medium for determining data sources. By acquiring a first set of rules corresponding to network traffic data, and when the first set of rules fails to match a preset whitelist, the first set of rules is input into a decision tree to obtain a first result indicating whether the network traffic data belongs to the same data source. The decision tree is constructed based on the first set of rules. By constructing the decision tree, the disordered rules in the network traffic data are integrated according to certain rules, avoiding rule conflicts between different data sources. When network traffic data does not belong to the same source, the network traffic data is analyzed based on the first result output by the decision tree. The score value represents the probability that the network traffic data belongs to each data source, improving the accuracy of determining the data source and avoiding a large false alarm rate. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0037] Figure 1 This is one of the flowcharts illustrating the method for determining the data source provided by the present invention;
[0038] Figure 2 This is a schematic diagram of the process for constructing a decision tree provided by the present invention;
[0039] Figure 3 This is the second flowchart illustrating the method for determining the data source provided by the present invention;
[0040] Figure 4 This is the third flowchart illustrating the method for determining the data source provided by this invention;
[0041] Figure 5 This is a schematic diagram of the decision tree structure provided by the present invention;
[0042] Figure 6 This is a schematic diagram of the structure of the data source determination device provided by the present invention;
[0043] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0045] To facilitate a clearer understanding of the various embodiments of this application, some relevant knowledge will be introduced as follows.
[0046] To quickly locate and identify the source of current abnormal behavior, including the platform or organization to which the behavior belongs, it is necessary to analyze the network traffic data of the current behavior and locate the source of the behavior based on the key feature information in the network traffic data.
[0047] In related technologies, clustering methods are commonly used to group similar traffic, as traffic from the same platform or organization often exhibits similarities. Additionally, a custom rule base can be used to determine the source of current behavior by matching rules within that base; alternatively, classification algorithms can be used to learn and build a classification model based on rule base features to categorize sources. Among these, matching techniques primarily include PCRE-based regular expression matching, Hyperscan multi-modal regular expression matching, ACBM multi-modal string matching, and string matching.
[0048] However, the above method has the following drawbacks:
[0049] (1) The clustering method divides data with similar traffic characteristics into the same platform. However, the clustering parameters are difficult to determine, and different platforms may have the same traffic characteristics. Therefore, the false alarm rate of this method is high.
[0050] (2) The method of building a custom rule base has the disadvantages of difficulty in rule construction and conflict between rules of different platforms.
[0051] In addition, rule matching methods also have their limitations. Hyperscan does not fully support all regular expressions. ACBM has certain advantages for multi-string matching, but it also has limitations for discrete feature rule matching. PCRE and string matching do not have advantages for multi-feature rule matching.
[0052] In summary, in order to accurately determine the data source and reduce the false alarm rate of data source determination, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for determining the data source.
[0053] The method for determining the data source provided in this invention is based on a decision tree, which displays the decision path of feature rules. It summarizes the seemingly disordered and chaotic feature rules and constructs a rule tree model. Each path from the root node (the rule that contributes the most to the final score result) to the leaf node (the final score result) represents a rule decision path. When the effective load of a certain source IP matches the feature rules of multiple different platforms, the platform to which the source IP belongs can be scored according to the rule tree model.
[0054] The following is combined Figures 1 to 4 The method for determining the data source provided by this invention will be described in detail. Figure 1 This is one of the flowcharts illustrating the method for determining the data source provided by this invention. See [link / reference]. Figure 1 As shown, the method includes steps 101-103, wherein:
[0055] Step 101: Obtain the first rule set corresponding to the network traffic data. The first rule set is the first word set extracted from the network traffic data.
[0056] First, it should be noted that the subject of this invention can be any electronic device capable of determining the data source, such as any kind of smartphone, smartwatch, desktop computer, laptop, etc.
[0057] In this embodiment of the invention, network traffic data can be real-time network traffic data or historical network traffic data, such as current real-time mapping data, historical mapping data, etc. The first rule set corresponding to the network traffic data of the current source IP can be represented as Relu_ip.
[0058] Where Rule_ip = {rule_ip1, rule_ip2, ..., rule_ipM}, M represents that the network traffic data of the current source IP contains M valid data payloads, and rule_ipM represents the first set of rules matched by the Mth valid payload.
[0059] It should be noted that the first rule set is the first set of words extracted from the network traffic data. For example, the network traffic data includes the following:
[0060] http: / / www.xxx.com / word1 / 2020.1.1 / 192.168.0.1.
[0061] Therefore, the first word set can include: http, www, xxx, com, word.
[0062] Step 102: If the first rule set fails to match the preset whitelist, input the first rule set into the decision tree to obtain the first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source.
[0063] In this embodiment of the invention, if the first rule set fails to match the preset whitelist, it indicates that the network traffic data does not belong to the data source in the whitelist.
[0064] In this scenario, the first set of rules corresponding to the network traffic data is input into the decision tree. The decision tree is traversed until all rules in the decision tree are matched, or until a leaf node of the decision tree is reached. The traversal ends, and the first result output by the decision tree is obtained. Here, any rule in the decision tree refers to any path from the root node to a leaf node of the decision tree.
[0065] The first result is used to characterize whether network traffic data belongs to the same platform or organization. If the network traffic data belongs to the same platform or organization, the source of the target data can be determined directly based on the source IP of the network traffic data, that is, which platform or organization the network traffic data specifically comes from.
[0066] Step 103: If the network traffic data does not belong to the same source, based on the first result, determine the score value of the network traffic data belonging to each data source.
[0067] In this embodiment of the invention, each path from the root node to the leaf node represents a rule decision path. When the first result indicates that the network traffic data matches the rules of multiple different platforms, that is, the network traffic data does not belong to the same source, the data source can be determined by scoring each data source corresponding to the source IP network traffic according to the decision tree.
[0068] It should be noted that when network traffic data belongs to the same source, the score of the target data source determined based on the source IP of the network traffic data is 1.
[0069] The method for determining the data source provided by this invention obtains a first set of rules corresponding to network traffic data. If the first set of rules fails to match a preset whitelist, it is input into a decision tree to obtain a first result indicating whether the network traffic data belongs to the same data source. The decision tree is constructed based on the first set of rules. By constructing the decision tree, the disordered rules in the network traffic data are integrated according to certain rules, avoiding rule conflicts between different data sources. Even when network traffic data does not belong to the same source, the network traffic data is analyzed based on the first result output by the decision tree. The score value represents the probability that the network traffic data belongs to each data source, improving the accuracy of determining the data source and avoiding a large false alarm rate.
[0070] Optionally, before inputting the first set of rules into the decision tree, it is also necessary to match the network traffic data with a preset whitelist, specifically including the following steps:
[0071] Step 1) Match the first rule set with the second rule set in the whitelist to obtain the matching result; the second rule set is the rule set associated with each of the data sources.
[0072] Step 2) If the first rule set and the second rule set are successfully matched, the target data source corresponding to the network traffic data is determined based on the matching result.
[0073] In this embodiment of the invention, a whitelist needs to be established in advance. The whitelist refers to the unique labeling rules (i.e., the second rule set) of each data source platform or organization. The second rule set includes a set of words that have a unique association with the data source platform or organization.
[0074] If any payload data of a source IP network traffic data matches the tag rules in the whitelist, that is, if the network traffic data successfully matches the preset whitelist, then the source of the IP network traffic data is unique.
[0075] It should be noted that the source IP of network traffic data can be used to determine which platform or organization the data originates from, and the corresponding score for that platform or organization is 1.
[0076] Optionally, the decision tree is constructed in the following manner, specifically including steps A to D:
[0077] Step A: Preprocess the network traffic data to generate preprocessed target network traffic data, wherein the target network traffic data includes the first rule set.
[0078] In this embodiment of the invention, preprocessing network traffic data means extracting the effective data payload from the network traffic data, and then extracting the first rule set from the effective data payload.
[0079] Optionally, the network traffic data can be preprocessed to generate preprocessed target network traffic data, which can be achieved through the following steps [1]-[3]:
[0080] Step [1]: Delete the unreadable bytes in the network traffic data to obtain the valid data payload corresponding to the network traffic data;
[0081] Step [2]: Decode the valid data payload and remove dynamic data to generate the decoded valid data payload;
[0082] Step [3]: Perform word segmentation on the decoded valid data payload to generate the target network traffic data.
[0083] Specifically, for example, network traffic data includes the following:
[0084] http: / / www.xxx.com / word1 / 2020.1.1 / 192.168.0.1.
[0085] First, non-readable bytes need to be removed from the network traffic data to obtain the valid data payload, represented as follows: http, www, xxx, com, word1, 202011, 19216801.
[0086] The valid data payload is then decoded, for example, by Base64 decoding, to generate text composed of words. Dynamic data is then removed to generate the decoded valid data payload. Dynamic data includes, for example, dates, variable numbers, IP addresses, port numbers, and dynamic strings.
[0087] The decoded valid data payload is then segmented to generate target network traffic data: http, www, xxx, com, word.
[0088] In the above implementation, by preprocessing the network traffic data, the effective data payload in the network traffic can be extracted, and then the regular features of the network traffic data can be extracted from the effective data payload, providing data support for the subsequent determination of the data source.
[0089] Step B: Calculate the first information gain for each word in the first word set, and select the first word with the largest first information gain as the root node of the decision tree.
[0090] In other words, it is necessary to calculate the first information gain for each word in the first rule set, and select the word with the largest information gain as the first word associated with the root node of the decision tree.
[0091] Step C: Calculate the second information gain corresponding to each word in the second word set, select the second word with the largest second information gain as the split node of the root node, and the second word set is the word set in the first rule set after removing the first word.
[0092] Step D: Calculate the third information gain corresponding to each word in the third word set, select the third word with the largest third information gain as the sub-split node of the split node, repeat step D until all words in the first word set are selected, or the number of layers of the decision tree reaches a preset threshold, and generate the leaf node of the decision tree; the third word set is the word set in the second word set after removing the second word.
[0093] In this embodiment of the invention, if the root nodes belong to the same category, a decision tree is returned; otherwise, the information gain of the remaining words is calculated according to the method for finding the root node in the first rule set, after removing the first word associated with the root node. The word with the largest information gain is selected as the split node. The above steps are repeated until all words in the first rule set are selected or the number of tree levels reaches the preset maximum number of levels. Finally, a rule tree composed of the first rule set is obtained (each node in the rule tree represents a word, and the path in the rule tree represents a rule).
[0094] The following section details the specific process of constructing a decision tree. First, for the rule set All_rules, calculate the information gain and select the word node with the largest information gain as the root node. If the root nodes belong to the same category, return the rule tree; otherwise, calculate the split nodes based on the information gain, i.e., the child nodes of the root node. Repeat step 4 until no words can be selected or the maximum level of the rule tree is reached, ultimately obtaining a rule tree.
[0095] The specific process is as follows:
[0096] Step 1: Define the first rule set matched by the current network traffic data (e.g., mapping data) as: rule, rule = {rule1, rule2, rule3, ..., rulelen}, calculate the information gain, and select the word with the largest information gain in the first rule set as the root node.
[0097] Step 2: Determine the category (i.e., platform or organization) of the root node. If the root node belongs to the same category, return the category; otherwise, calculate the split node based on information gain, i.e., the child node of the root node. Repeat Step 2 to build a decision tree. Continue until there are no words to choose from, or the maximum number of levels in the rule tree is reached, and finally a decision tree is obtained.
[0098] In the process of constructing a rule-based tree model, the information gain of the root node and split nodes is calculated as follows:
[0099] First, the information entropy is calculated based on formula (1), which is expressed as follows:
[0100]
[0101] Among them, |C k | represents the number of samples in the k-th class, |D| represents the number of data sets D, and n represents the number of classes, i.e., the number of platform classes.
[0102] Then, the conditional entropy of each rule is calculated based on formula (2), which is expressed as follows:
[0103]
[0104] Where n is the number of categories, K is the number of categories to which the current rule belongs, and |D i | represents the number of samples in the i-th class of the dataset.
[0105] Finally, the information gain Info(D) is calculated based on formula (3), which is expressed as follows:
[0106] Info(D) = Ent(D) - Ent(D|rule) i (3)
[0107] In the above implementation, by constructing a decision tree, the source platforms and organizations of cyberspace mapping and attack data can be analyzed. Based on the path analysis of the decision tree, the source of mapping data can be analyzed, and the seemingly disordered and chaotic feature rules can be summarized. Each path from the root node to the leaf node in the decision tree represents a rule decision path. When the effective data payload of a certain source IP network data matches the feature rules of multiple different platforms, the source IP network data can be identified by scoring the platform to which it belongs based on the decision tree.
[0108] Figure 2 This is a schematic diagram of the process for constructing a decision tree provided by the present invention. See [link / reference]. Figure 2 As shown, it specifically includes steps 201-207, wherein:
[0109] Step 201: Obtain network traffic data and set the maximum number of layers k in the decision tree.
[0110] Step 202: Delete the unreadable bytes in the network traffic data to obtain the valid data payload corresponding to the network traffic data.
[0111] Step 203: Decode the valid data payload and remove dynamic data to generate the decoded valid data payload.
[0112] Step 204: Perform word segmentation on the decoded valid data payload to generate target network traffic data, wherein the target network traffic includes the first rule set.
[0113] Step 205: Calculate the information gain of each word in the first rule set, and take the word with the largest information gain as the root node or split node.
[0114] Step 206: Determine whether the network traffic data belongs to the same category. If yes, proceed to step 208; otherwise, return to step 205.
[0115] Step 207: Determine if the decision tree has reached the maximum number of levels. If yes, proceed to step 208; otherwise, return to step 205.
[0116] It should be noted that the execution order of steps 206 and 207 is not important.
[0117] Step 208: Decision tree construction complete.
[0118] Optionally, the first result includes at least one of the following:
[0119] a) The number of valid data payloads corresponding to the network traffic data;
[0120] b) The total number of data sources containing the same rules;
[0121] c) The weight value of each rule;
[0122] d) The probability of each data source in each valid data payload;
[0123] Here, a rule represents any path from the root node to a leaf node in the decision tree.
[0124] When the network traffic data does not belong to the same source, based on the first result, the score value of the network traffic data belonging to each data source is determined, which is specifically achieved through the following steps:
[0125] The score is determined based on the number of valid data payloads corresponding to the network traffic data, the total number of data sources containing the same rule, the weight value of each rule, and the probability of each data source in each valid data payload.
[0126] In this embodiment of the invention, based on the number of valid data payloads corresponding to network traffic data, the total number of data sources containing the same rules, the weight value of each rule, and the probability of each data source in each valid data payload, the score value of network traffic data belonging to each data source can be determined using formula (4); wherein, formula (4) is expressed as follows:
[0127]
[0128] Where, score represents the rating value; Platforms[i] represents the i-th data source; M represents the number of valid data payloads; n represents the total number of identical rules contained in any data source; w j p represents the weight of the j-th rule; i This represents the probability of the i-th data source.
[0129] Optionally, since different rules have different influencing factors on different data sources, in order to avoid potential conflicts between rules from different platforms, it is necessary to assign a corresponding weight to each rule in the decision tree; the weight values are obtained in the following way:
[0130] Step 1) Obtain the number of times each rule appears in each of the aforementioned data sources;
[0131] Step 2) Based on the number of occurrences, determine the weight value of each rule using the entropy method.
[0132] Specifically, since each rule has different factors affecting the platform's rating, the weighting method can adopt the entropy method, using each rule (except for the rules in the whitelist) as an indicator, and the number of times each indicator appears on each platform as the indicator value, and the weight value of different indicators is calculated by the entropy method.
[0133] Figure 3 This is the second flowchart illustrating the method for determining the data source provided by this invention. See also... Figure 3 As shown, the method includes steps 301-305, wherein:
[0134] Step 301: Obtain the first rule set corresponding to the network traffic data, wherein the first rule set is the first word set extracted from the network traffic data.
[0135] Step 302: Match the first rule set with the second rule set in the preset whitelist, wherein the second rule set is the rule set associated with each data source.
[0136] If the match is successful, proceed to step 305; if the match fails, proceed to step 303.
[0137] Step 303: Input the first set of rules into the decision tree to obtain the first result output by the decision tree. The decision tree is constructed based on the first set of rules, and the first result is used to characterize whether network traffic data belongs to the same data source.
[0138] If the network traffic data belongs to the same source, proceed to step 305; if the network traffic data does not belong to the same source, proceed to step 304.
[0139] Step 304: Based on the first result, determine the score value of each data source for the network traffic data.
[0140] Step 305: Based on the source IP of the network traffic data, determine the target data source corresponding to the network traffic data.
[0141] Figure 4 This is the third flowchart illustrating the method for determining the data source provided by this invention. See also... Figure 4 As shown, the method includes steps 401-405, wherein:
[0142] Step 401: Obtain network traffic data.
[0143] Step 402: Determine whether the network traffic data meets the whitelist; if yes, proceed to step 405; if no, proceed to step 403.
[0144] Step 403: Traverse the decision tree.
[0145] Step 404: Determine whether the network traffic data belongs to the same platform; if yes, proceed to step 405; if no, proceed to step 406.
[0146] Step 405: Based on the source IP of the network traffic data, determine the target data source corresponding to the network traffic data, where the target source has a score of 1.
[0147] Step 406: Calculate the score of network traffic data belonging to each data source.
[0148] Specifically, the score can be calculated using the formula (1) above, based on the number of valid data payloads corresponding to network traffic data, the total number of identical rules for any data source, the weight values of the first and second rules, and the probability of each data source.
[0149] The method for determining the data source provided by the present invention will be described below with reference to specific embodiments:
[0150] Assume the matching rules are distributed as shown in Table 1 below:
[0151] Table 1
[0152] R1 R2 R3 R4 R5 label 1 0 0 0 0 1 0 0 1 0 0 2 0 0 1 1 0 2 1 0 0 0 0 1 0 0 0 1 1 1 0 1 0 0 1 1 1 0 0 0 0 2
[0153] R1, R2, R3, R4, and R5 represent the rule sets of the current dataset; the label indicates the platform to which the rule belongs; 1 indicates that the rule exists, and 0 indicates that the rule does not exist.
[0154] The rule tree creation process is as follows:
[0155] 1. Calculate information gain to determine the root node.
[0156] H(D)=-5 / 8log25 / 8-2 / 8log2 / 8=0.954434002924965;
[0157] H(D|1)=3 / 8*(-2 / 3*math.log2(2 / 3)-1 / 3*math.log2(1 / 3))+5 / 8*(-2 / 5*math.log2(2 / 5)-3 / 5*math.log2(3 / 5))=0.9512050593046015;
[0158] H(D|2)=7 / 8*(4 / 7*math.log2(4 / 7)-3 / 7*math.log2(3 / 7))+1 / 8=0.1797196969723659;
[0159] H(D|3)=6 / 8*(5 / 6*math.log2(5 / 6)-1 / 6*math.log2(1 / 6))+2 / 8=0.4087238089440234;
[0160] H(D|4)=0.23
[0161] H(D|5)=0.55
[0162] Info(D) = H(D) - H(D|i)
[0163] Choose the largest value of Info(D) as the root node, i.e., R2 as the root node.
[0164] 2. Calculate the split nodes
[0165] With R2 as the root node, there are two cases: 0 and 1. When R2 = 1, label = 1; when R2 = 0, a split is required. Calculate the information gain of the split node:
[0166] H(R2)=-4 / 7math.log2(4 / 7)-3 / 7*math.log2(3 / 7)=0.9852281360342516; H(R2|1)=3 / 7*(-2 / 3*math.log2 (2 / 3)-1 / 3*math.log2(1 / 3))+4 / 7*(-2 / 4*math.log2(2 / 4)-2 / 4*math.log2(2 / 4))=0.9649839288804954;
[0167] H(R2|3)=0.41
[0168] H(R²|⁴)=0.31
[0169] H(R2|5)=0.62
[0170] Info(D) = H(R2) - H(R2|i), therefore node R4 is chosen as the split node.
[0171] 3. Repeat step 2 to calculate the split nodes until there are no more attributes to choose from, finally obtaining the decision tree as shown below. Figure 5 As shown, Figure 5 This is a schematic diagram of the decision tree structure provided by the present invention.
[0172] The apparatus for determining the data source provided by the present invention will be described below. The apparatus for determining the data source described below can be referred to in correspondence with the method for determining the data source described above. Figure 6 This is a schematic diagram of the structure of the data source determination device provided by the present invention, as shown below. Figure 6 As shown, the data source determination device 600 includes: a first acquisition module 601, an input module 602, and a first determination module 603, wherein:
[0173] The first acquisition module 601 is used to acquire a first rule set corresponding to network traffic data, wherein the first rule set is a first word set extracted from the network traffic data.
[0174] The input module 602 is used to input the first rule set into a decision tree when the first rule set fails to match the preset whitelist, and obtain a first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source.
[0175] The first determining module 603 is used to determine, based on the first result, the score value of the network traffic data belonging to each data source when the network traffic data does not belong to the same source.
[0176] The data source determination device provided by this invention acquires a first set of rules corresponding to network traffic data. If the first set of rules fails to match a preset whitelist, it inputs the first set of rules into a decision tree to obtain a first result indicating whether the network traffic data belongs to the same data source. The decision tree is constructed based on the first set of rules. By constructing the decision tree, the disordered rules in the network traffic data are integrated according to certain rules, avoiding rule conflicts between different data sources. When the network traffic data does not belong to the same source, the device analyzes the network traffic data based on the first result output by the decision tree, using the score value to represent the probability that the network traffic data belongs to each data source, thus improving the accuracy of determining the data source and avoiding a large false alarm rate.
[0177] Optionally, the device further includes:
[0178] The preprocessing module is used to preprocess the network traffic data to generate preprocessed target network traffic data, wherein the target network traffic data includes the first rule set.
[0179] The first selection module is used to calculate the first information gain corresponding to each word in the first word set, and select the first word with the largest first information gain as the root node of the decision tree;
[0180] The second selection module is used to calculate the second information gain corresponding to each word in the second word set, and select the second word with the largest second information gain as the split node of the root node. The second word set is the word set in the first rule set after removing the first word.
[0181] The third selection module is used to calculate the third information gain corresponding to each word in the third word set, select the third word with the largest third information gain as the sub-split node of the split node, and repeat the steps of the first selection module until all words in the first word set are selected, or the number of layers of the decision tree reaches a preset threshold, and generate the leaf node of the decision tree; the third word set is the word set in the second word set after removing the second word.
[0182] Optionally, the preprocessing module is further configured to:
[0183] The non-readable bytes in the network traffic data are deleted to obtain the valid data payload corresponding to the network traffic data;
[0184] The valid data payload is decoded, and dynamic data is removed to generate the decoded valid data payload;
[0185] The decoded valid data payload is segmented to generate the target network traffic data.
[0186] Optionally, the first result includes at least one of the following: the number of valid data payloads corresponding to the network traffic data; the total number of data sources containing the same rule; the weight value of each rule; the probability of each data source in each valid data payload; and the rule representing any path from the root node to the leaf node in the decision tree.
[0187] The first determining module 603 is further configured to:
[0188] The score is determined based on the number of valid data payloads corresponding to the network traffic data, the total number of data sources containing the same rule, the weight value of each rule, and the probability of each data source in each valid data payload.
[0189] Optionally, the device further includes:
[0190] The acquisition module is used to obtain the number of times each rule appears in each of the data sources;
[0191] The second determining module is used to determine the weight value of each rule based on the number of occurrences using the entropy method.
[0192] Optionally, the device further includes:
[0193] The matching module is used to match the first rule set with the second rule set in the whitelist to obtain a matching result; the second rule set is the rule set associated with each of the data sources.
[0194] The third determining module is used to determine the target data source corresponding to the network traffic data based on the matching result when the first rule set and the second rule set are successfully matched.
[0195] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a method for determining the data source. This method includes: acquiring a first rule set corresponding to network traffic data, the first rule set being a first set of words extracted from the network traffic data; if the first rule set fails to match a preset whitelist, inputting the first rule set into a decision tree to obtain a first result output by the decision tree, the decision tree being constructed based on the first rule set, the first result being used to characterize whether the network traffic data belongs to the same data source; if the network traffic data does not belong to the same source, determining a score value for each data source based on the first result.
[0196] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0197] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data source determination method provided by the above methods. The method includes: obtaining a first rule set corresponding to network traffic data, wherein the first rule set is a first word set extracted from the network traffic data; if the first rule set fails to match a preset whitelist, inputting the first rule set into a decision tree to obtain a first result output by the decision tree, wherein the decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source; if the network traffic data does not belong to the same source, determining the score value of the network traffic data belonging to each data source based on the first result.
[0198] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for determining the data source provided by the above-described methods. This method includes: acquiring a first set of rules corresponding to network traffic data, the first set of rules being a first set of words extracted from the network traffic data; if the first set of rules fails to match a preset whitelist, inputting the first set of rules into a decision tree to obtain a first result output by the decision tree, the decision tree being constructed based on the first set of rules, the first result being used to characterize whether the network traffic data belongs to the same data source; if the network traffic data does not belong to the same source, determining a score value for each data source based on the first result.
[0199] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for determining the source of data, characterized in that, include: Obtain a first set of rules corresponding to network traffic data, wherein the first set of rules is a first set of words extracted from the network traffic data; If the first rule set fails to match the preset whitelist, the first rule set is input into the decision tree to obtain the first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source. If the network traffic data does not belong to the same source, based on the first result, determine the score value of the network traffic data belonging to each data source. The decision tree is constructed in the following way: Step A: Preprocess the network traffic data to generate preprocessed target network traffic data, wherein the target network traffic data includes the first rule set; Step B: Calculate the first information gain for each word in the first word set, and select the first word with the largest first information gain as the root node of the decision tree; Step C: Calculate the second information gain corresponding to each word in the second word set, select the second word with the largest second information gain as the split node of the root node, and the second word set is the word set in the first rule set after removing the first word; Step D: Calculate the third information gain corresponding to each word in the third word set, select the third word with the largest third information gain as the sub-split node of the split node, repeat step D until all words in the first word set are selected, or the number of layers of the decision tree reaches a preset threshold, and generate the leaf node of the decision tree; the third word set is the word set in the second word set after removing the second word.
2. The method for determining the data source according to claim 1, characterized in that, The step of preprocessing the network traffic data to generate preprocessed target network traffic data includes: The non-readable bytes in the network traffic data are deleted to obtain the valid data payload corresponding to the network traffic data; The valid data payload is decoded, and dynamic data is removed to generate the decoded valid data payload; The decoded valid data payload is segmented to generate the target network traffic data.
3. The method for determining the data source according to claim 1 or 2, characterized in that, The first result includes at least one of the following: the number of valid data payloads corresponding to the network traffic data; the total number of data sources containing the same rule; the weight value of each rule; the probability of each data source in each valid data payload; and the rule representing any path from the root node to the leaf node in the decision tree. The step of determining the score value of each data source based on the first result includes: The score is determined based on the number of valid data payloads corresponding to the network traffic data, the total number of data sources containing the same rule, the weight value of each rule, and the probability of each data source in each valid data payload.
4. The method for determining the data source according to claim 3, characterized in that, The weight values are obtained in the following way: Obtain the number of times each rule appears in each of the stated data sources; Based on the frequency of occurrence, the weight value of each rule is determined using the entropy method.
5. The method for determining the data source according to claim 1 or 2, characterized in that, The method further includes: The first rule set is matched with the second rule set in the whitelist to obtain a matching result; the second rule set is the rule set associated with each of the data sources. If the first rule set and the second rule set are successfully matched, the target data source corresponding to the network traffic data is determined based on the matching result.
6. A device for determining the source of data, characterized in that, include: The first acquisition module is used to acquire a first rule set corresponding to network traffic data, wherein the first rule set is a first word set extracted from the network traffic data; The input module is used to input the first rule set into a decision tree when the first rule set fails to match the preset whitelist, and obtain the first result output by the decision tree. The decision tree is constructed based on the first rule set, and the first result is used to characterize whether the network traffic data belongs to the same data source. The first determining module is used to determine, based on the first result, the score value of the network traffic data belonging to each data source when the network traffic data does not belong to the same source. The decision tree is constructed in the following way: Step A: Preprocess the network traffic data to generate preprocessed target network traffic data, wherein the target network traffic data includes the first rule set; Step B: Calculate the first information gain for each word in the first word set, and select the first word with the largest first information gain as the root node of the decision tree; Step C: Calculate the second information gain corresponding to each word in the second word set, select the second word with the largest second information gain as the split node of the root node, and the second word set is the word set in the first rule set after removing the first word; Step D: Calculate the third information gain corresponding to each word in the third word set, select the third word with the largest third information gain as the sub-split node of the split node, repeat step D until all words in the first word set are selected, or the number of layers of the decision tree reaches a preset threshold, and generate the leaf node of the decision tree; the third word set is the word set in the second word set after removing the second word.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for determining the data source as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for determining the data source as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for determining the data source as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Interception feedback processing method based on big data analysis interception
CN115168848A
