Deletable weighted learning Bloom filter and malicious URL detection method

Abstract

This application provides a removable weighted learning Bloom filter and a method for detecting malicious URLs. The removable weighted learning Bloom filter includes a PH table, a weighted learning model, and a partitioned backup Bloom filter. The PH table stores some benign URLs that have been deleted from the database based on data popularity. If a URL to be detected is in the PH table, it indicates that the URL has been deleted from the database, alerting the user that it is a malicious URL and no further detection is needed. The weighted learning model assigns weights to URLs based on data popularity for training. During detection, a score is obtained through the learning model. If the score is greater than a set threshold, it indicates that the URL is benign and can be accessed normally; otherwise, it is initially judged as malicious. The partitioned backup Bloom filter uses multiple sub-Bloom filters to store URLs judged as malicious by the learning model for further detection. If a URL is malicious, the user is alerted; otherwise, it can be accessed normally. This application effectively reduces the FPR of URL detection and improves the detection performance after data deletion.

Description

Technical Field

[0001] This application relates to the field of computer communication technology, specifically to a removable weighted learning Bloom filter and a method for detecting malicious URLs. Background Technology

[0002] A Bloom filter is a technique that uses bit arrays and hash functions to map data and detect the existence of data in a set. As a data structure with high space efficiency and query time efficiency, it is widely used in malicious URL detection, name lookup, IP address lookup, and database query optimization. In big data environments, traditional Bloom filters require large bit arrays for hash mapping, consuming significant memory. To address this issue, Kraska et al. proposed a learning-based Bloom filter at the 2018 SIGMOD conference. This filter combines machine learning with traditional Bloom filters, effectively reducing memory usage while maintaining a low false positive rate and improving the overall performance of the filter. This has attracted widespread attention and research from scholars both domestically and internationally.

[0003] In malicious URL detection, learned Bloom filters train a learning model, such as a random forest classifier, using features extracted from URLs. A traditional Bloom filter is then used as a backup to prevent false positives. By classifying URLs using machine learning models, learned Bloom filters can more accurately identify malicious URLs, effectively reducing memory overhead and providing a new approach to malicious URL detection.

[0004] The following are some existing Bloom filters: (1) Counting Bloom Filter Counting Bloom Filter A variant of the traditional Bloom filter, it achieves data deletion by replacing each bit in the bit array with a small counter, thus solving the problem that the traditional Bloom filter (SBF) cannot delete data. When k bits are mapped using k hash functions, the counter for the corresponding bit array is incremented by 1. If an element is deleted, the counter for the corresponding k bits is decremented by 1.

[0005] Figure 1 This illustrates the working principle of CBF, with three data points shown in the figure. , , To insert them into the bit array, the hash function needs to be mapped to the corresponding positions in the bit array and accumulated, such as... Figure 1 The non-zero positions in the bit array shown. When data is deleted... hour, The counter for the mapped hash position is decremented by 1, such as Figure 1The new bit array is shown below. When querying data separately... , , During the query, due to and The mapping to all positions in the bit array is non-zero, therefore it is judged as positive, but in practice... This data does not exist in the database and is considered a false positive. Meanwhile, The mapped position contains 0, therefore it is judged as negative. This demonstrates that CBF possesses the general characteristics of a Bloom filter: it has no false negatives, but false positives exist. Furthermore, CBF effectively supports data deletion.

[0006] Because it uses counters, CBF requires more memory than SBF. When reserving space for the CBF bit array, an appropriate number of counters needs to be chosen to ensure a low false positive rate (FPR) while minimizing memory usage. Furthermore, choosing the optimal number of hash functions, k, is equally important. The optimal number of hash functions is: Equation (1) Where m represents the number of counters, and n represents the number of data items stored. The number of times the i-th counter increments is: Equation (2) in, This means selecting j times from nk hash maps. This indicates the probability of selecting the i-th counter for the j-th hash map. Let represent the probability that the other nk-j hash maps do not select the i-th counter. Therefore, the probability that the i-th counter increases by more than j times is: Equation (3) Combining equations (1) and (3), we can obtain: Equation (4) If each counter is allocated 4 bits, it will overflow when the counter reaches 16 bits. At this time, according to equation (4), we can obtain... This probability is already small enough that a smaller FPR can be obtained by allocating according to the above method.

[0007] (2) Learning-type Bloom filter Kraska et al. proposed the Learned Bloom Filter (LBF) at the 2018 SIGMOD conference. This method combines machine learning with traditional Bloom filters, aiming to improve the accuracy of set membership queries and reduce the initial probability per scatter (FPR). In LBF, when the classifier's predicted score is above a certain popularity threshold, an element is directly identified as a member of the set without further Bloom filter testing. This approach not only improves efficiency but also allows LBF to maintain performance when dealing with dynamic datasets, especially in data streaming applications such as duplicate detection, malicious URL inspection, and web caching. Figure 2 As shown, LBF mainly consists of two parts: the first part uses a machine learning model to simulate a traditional Bloom filter as the main filter; the second part combines the first part with a traditional Bloom filter (called a backup Bloom filter) to prevent false negatives. When training the learning model, a set of datasets from a database is typically given. As positive data, given a dataset that does not exist in the database. As negative data and The input is fed into the learning model f(x) for training, following these rules: Equation (5) The learning model in LBF can employ gradient boosting (GB), recurrent neural networks (RNNs), or convolutional neural networks (CNNs), etc. Then, the loss function is minimized. The optimal training model is obtained to determine whether the queried data is in the set. The result is a probability assessment value, which is then passed through a set popularity threshold. and The output results are compared to determine the size relationship. If the queried data x is in the database, then it is assumed that the data x is in the database; if Then, x is further input into the backup Bloom filter for judgment, and the final judgment result is obtained. Most of the data has been filtered by the learning model, so the backup Bloom filter stores less data, thereby reducing memory consumption. The FPR of LBF can be expressed as: Equation (6) in, The representation learning model FPR, This represents the FPR (Flat Rate) of the backup Bloom filter. LBF (Backup Bloom Filter) opens up a new direction for the development of BFs (Browser Filters) and has significant advantages over traditional BFs. First, in big data environments, traditional BFs require more memory to reduce the FPR, while LBF uses machine learning models to evaluate the data, ensuring that the FPR does not change with the growth of data volume. Second, LBF is more flexible and can dynamically adjust the model parameters according to different data distribution patterns. Third, compared to traditional BFs requiring a fixed bit array, LBF only needs to store the parameters of the learned model, reducing the memory space required.

[0008] (3) Optimized learning Bloom filter In recent years, many learning-based Bloom filters have been developed based on LBF optimization. Mitzenmacher proposed a sandwiched learned Bloom filter (SandwichedLBF) based on LBF, which reduces the false positive rate by adding an initial Bloom filter at the start of the model. Rae et al. proposed a neural Bloom filter that applies meta-learning to learn an approximate membership set of a dataset, achieving a higher data compression rate than traditional Bloom filters. Furthermore, some learning-based methods focus on partitioning Bloom filter regions while adaptively setting hash functions to improve model performance. Dai et al. proposed an adaptive learned Bloom filter (Ada-BF) that reduces the false positive rate by using the complete spectrum of the score regions. Vaidya et al. proposed a partitioned learned Bloom filter (PLBF) that clusters the score space into different regions through dynamic programming and assigns a different Bloom filter to each region, thereby reducing memory usage while maintaining a low false positive rate. Bhattacharya et al. proposed a Projection Hash Bloom filter (PHBF), which effectively reduces the false positive rate and query time consumption by replacing the standard hash function and working with SBF.

[0009] None of the methods described above consider the impact of data popularity on model performance. On one hand, the design of these methods fails to account for the influence of data popularity on the learning effectiveness of the learning model. On the other hand, these models do not adequately consider the issue that deleting hot data will significantly increase the false positive rate and query time, thus affecting the overall model performance after data deletion.

[0010] (4) Application of Bloom filters in malicious URL detection In malicious URL detection, by storing known malicious URL characteristics in a Bloom filter, the system can quickly determine whether a given URL is likely malicious, thus providing warnings and blocking before users access it. Furthermore, the dynamic nature of Bloom filters allows the system to continuously update the set of malicious URLs over time to address evolving cyber threats. Although there is a certain false positive rate, this probabilistic nature of Bloom filters makes them an effective tool for initial screening of malicious URLs, especially in resource-constrained scenarios or those requiring rapid response.

[0011] In recent years, scholars both domestically and internationally have proposed various optimized Bloom filters for malicious URL detection. Feng et al. proposed a novel multi-level counting Bloom filter for network-based URL filtering to improve caching efficiency and reduce bandwidth consumption. Patgiri et al. combined deep learning with Bloom filters to propose deepBF, designing a two-dimensional Bloom filter to store malicious and benign URLs separately for malicious URL detection. Dai et al. proposed an adaptive learning Bloom filter that reduces the false positive rate of malicious URL detection by using the full spectrum of the fractional region. Shuai et al. proposed a fast matching method based on Bloom filters for matching trusted whitelists, improving matching efficiency and reducing computational resource consumption. Gebretsadik et al. proposed a novel enhanced Bloom filter to improve memory efficiency and query performance for IoT intrusion detection. Dinakar et al. proposed a hybrid model combining traditional machine learning and deep learning methods, and evaluated the combined performance of shallow (traditional machine learning) and deep (deep learning) learning methods in malicious URL detection, explaining the advantages and limitations of different methods.

[0012] The methods described above use learning models to design Bloom filters, effectively improving their performance in malicious URL detection. However, none of these methods consider the important characteristic of URL popularity and are therefore effective in improving the accuracy and efficiency of malicious URL detection.

[0013] Existing learning-based Bloom filters have the following problems: The initial learning model did not support data updates, and updating after data insertion and deletion was very costly. There is too little optimization in improving the accuracy of the learning model. The learning model is an important component of a learning Bloom filter, but most optimization methods focus on the design of the overall model or backup Bloom filters, ignoring the impact of the learning model on the overall performance. The impact of data features on model performance is not considered. Furthermore, the learning Bloom filter does not consider the impact of data popularity, i.e., the frequency of data access, on the performance of the learning model, nor the problem of model performance degradation after data deletion.

[0014] During URL access, the data popularity of different URLs varies greatly. Existing learning-based Bloom filters, when targeting malicious URL detection tasks, have failed to fully utilize this characteristic to improve the overall performance of the filters. Summary of the Invention

[0015] This application provides a removable weighted learning Bloom filter, which can solve the technical problem that existing learning Bloom filters fail to fully utilize the data heat difference characteristics of URLs to improve the overall performance of the filter when performing malicious URL detection tasks.

[0016] Firstly, this application provides a removable weighted learning Bloom filter, comprising: The PH table is used to store some benign URLs that have been deleted from the database based on data popularity. URLs that are deleted are identified as malicious URLs and added to the PH table. The weighted learning model is used to assign weights to URLs to be detected based on their popularity and perform preliminary detection to obtain a detection score. The preliminary detection result is obtained based on the detection score. If the detection score is greater than a set threshold, it is considered a benign URL and can be accessed normally; otherwise, it is input into the partition backup Bloom filter for further detection. The partition backup Bloom filter includes multiple sub-count Bloom filters. These sub-count Bloom filters are used to store URLs that were not initially identified as malicious for subsequent detection and to obtain the detection results.

[0017] In conjunction with the first aspect, in one implementation, the loss function L of the weighted learning model is minimized as follows:

[0018] In the formula, This represents the input field, which is the URL to be tested. Let y be the weight of the input item X, f(x) represent the judgment result output by the weighted learning model, and y be the actual judgment result.

[0019] Secondly, this application provides a method for detecting malicious URLs applied to the removable weighted learning Bloom filter described above, characterized by comprising the following steps: Get the URL to be tested; The PH table is used to filter the URLs to be detected. Based on the PH table, URLs that have been deleted from the database are initially screened out and identified as malicious URLs to alert users. A weighted learning model is used to perform preliminary detection on the filtered URLs, obtain detection scores, and obtain the malicious URLs and / or URLs that were not initially determined to be malicious based on the detection scores; Based on the detection score of the learning model, the groups of URLs judged as malicious by the learning model in the partition backup Bloom filter are obtained, and detection is performed in the corresponding groups. If a URL is detected as malicious, the user is notified; if a URL is detected as benign, normal access is allowed.

[0020] In conjunction with the second aspect, in one implementation, the step of using a pH table to filter the URLs to be detected and obtaining the filtered URLs and the filtered malicious URLs specifically includes the following steps: The URLs to be tested are filtered using a pH meter; If the URL to be detected exists in the PH table, then the URL to be detected is determined to be a malicious URL; If the URL to be detected does not exist in the PH table, then the URL to be detected will be used as the filtered URL for the next step of detection.

[0021] In conjunction with the second aspect, in one implementation, the step of using a weighted learning model to perform preliminary detection on the filtered URLs, obtaining a detection score, and obtaining the URLs initially determined to be malicious and / or not initially determined to be malicious based on the detection score, specifically includes the following steps: A weighted learning model is obtained by training based on minimizing the loss function; The weighted learning model obtained from training is used to detect and obtain the detection score of each filtered URL; Based on the detection scores and data delineation boundaries, the filtered URLs are initially detected to obtain preliminary detection results, including malicious URLs and / or URLs that are not directly identified as malicious.

[0022] In conjunction with the second aspect, in one implementation, the step of grouping URLs that were not initially determined to be malicious based on detection scores and storing them in multiple sub-count Bloom filters of the partition backup Bloom filter for further detection to obtain detection results specifically includes the following steps: URLs that were not initially determined to be benign were grouped according to their detection scores. URLs that were not initially determined to be benign are stored in the corresponding sub-Bloom filters; For each group of URLs, different numbers of independent hash functions are used for testing to obtain detection results including malicious URLs and / or non-malicious URLs.

[0023] In conjunction with the second aspect, in one implementation, storing benign URLs that have been deleted from the database and benign URLs that have been mistakenly identified as malicious in the PH table based on data popularity specifically includes the following steps: Sort benign URLs that have been removed from the database based on their popularity; Set the minimum popularity threshold for URLs stored in the PH table; Access frequency higher than The deleted URLs are stored in the PH table.

[0024] In conjunction with the second aspect, in one implementation, the data popularity is calculated as follows:

[0025] In the formula, The input item X represents the historical number of visits, and N represents the total number of visits to all URLs.

[0026] In conjunction with the second aspect, in one implementation, after grouping and storing the detection scores of URLs that were not initially determined to be malicious into multiple sub-count Bloom filters of the partitioned backup Bloom filter, and performing malicious detection again to obtain the results of the second detection, the method further includes the following steps: Based on data popularity, benign URLs that have been deleted from the database are stored in PH.

[0027] In conjunction with the second aspect, in one implementation, storing the benign URLs that have been deleted from the database according to data popularity in PH specifically includes the following steps: Malicious URLs are sorted based on their popularity. Set a minimum heat threshold; Benign URLs that are accessed more frequently than the minimum popularity threshold and are subsequently deleted from the database are classified as malicious URLs.

[0028] Thirdly, this application provides a computer-readable storage medium storing a malicious URL detection program, wherein when the malicious URL detection program is executed by a processor, it implements the steps of the malicious URL detection method as described above.

[0029] The beneficial effects of the technical solutions provided in this application include at least the following: This application addresses the problem of large-scale malicious URL detection by proposing a removable weighted learning Bloom filter, which effectively reduces the FPR of URL detection and data deletion, while also saving data detection time after data deletion. A weighted learning model based on data popularity was designed to improve the judgment accuracy of the learning model. A pH table based on data popularity was designed to improve the detection performance after data deletion. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the working principle of CBF in the existing technology; Figure 2 This is a diagram of the LBF framework in the existing technology; Figure 3 A DWLBF framework diagram provided for embodiments of this application; Figure 4 A flowchart of the malicious URL detection method provided in this application embodiment; Figure 5 A comparison chart of the FPR of DWLBF and five Bloom filters at different data deletion ratios on three malicious URL datasets, provided for embodiments of this application; Figure 6 A comparison chart of detection times of DWLBF and five Bloom filters on three malicious URL datasets at different data deletion ratios, provided for embodiments of this application; Figure 7 A comparison chart of FPR (Frequency Rate) with and without using the PH (Profile Table) when detecting three malicious URL datasets provided in this application embodiment; Figure 8 A comparison chart of detection time using and not using the PH table when detecting three malicious URL datasets provided in the embodiments of this application; Figure 9 FPR comparison chart of three malicious URL datasets with different heat thresholds for detection in the embodiments of this application; Figure 10 A comparison chart showing the detection time when setting different popularity thresholds for detection on three malicious URL datasets provided in this application embodiment. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0032] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0033] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0034] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0035] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0036] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0037] DWLBF: Deletable Weighted Learned Bloom Filter; PH: Perfect Hash; FPR: False Positive Rate. Ada-BF: Adaptive Learned Bloom Filter; URL: Uniform Resource Locator.

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] To support deletion operations and ensure a low overall FPR for the Bloom filter, this application uses a partition backup filter as a backup filter in DWLBF, which was originally designed in Ada-BF.

[0040] Firstly, such as Figure 3 As shown, this application provides a removable weighted learning Bloom filter, which mainly consists of the following three parts: The PH table is used to store some benign URLs that have been deleted from the database based on data popularity, reducing the FPR of the model after data deletion and the time consumption for URL detection. Specifically, it stores benign URLs that have been deleted from the database with a data popularity greater than a popularity threshold, and identifies them as malicious URLs in the PH table after deletion. The PH table is a special hash table that ensures that for a given set of keys, the hash function maps each key to a unique bucket or slot, and no two keys share the same hash value. In other words, the PH table achieves zero collisions during insertion. Ideally, the query time complexity of the PH table is O(1). A weighted learning model is used to assign weights to URLs to be detected based on data popularity and perform preliminary detection to obtain a detection score. Based on the detection score, a preliminary detection result is obtained, and malicious URLs detected in the preliminary detection are deleted. The partitioned backup Bloom filter includes multiple sub-count Bloom filters. These sub-count Bloom filters store URLs that were not initially identified as benign for re-detection. The filter retrieves the detection results and alerts the user if a URL is detected as malicious. If a URL is detected as benign, it can be accessed normally. The filter also has a deletion function, which effectively reduces the FPR of the model detection.

[0041] The removable weighted learning Bloom filter provided in this application performs malicious detection on the URLs to be detected by assigning weights to the data based on the data popularity characteristics, effectively improving the detection accuracy of the learning model.

[0042] The total memory size M occupied by the DWLBF in this application is represented as: Equation (7) in, , and These represent the memory usage of the PH table, the learning model, and the partition backup filter, respectively. Therefore, only the top 25% of the most popular data is stored. This data is used to control total memory usage. Furthermore, the stored deleted data represents high-frequency data, maximizing resource utilization while improving query performance.

[0043] In one embodiment, the learning model uses gradient descent (GB) to optimize the loss function L. Equation (8) definition The weight of input item X, i.e., the URL to be detected, is calculated using the following formula: Equation (9) in, For the first The historical access count of each URL, where N represents the total access count of all URLs.

[0044] This application considers the impact of data popularity characteristics, i.e., data access frequency, on the performance of the learning model. Combining equations (8) and (9), the minimum loss function L of the weighted learning model in this application is: Equation (10) In the formula, y represents the input item, f(x) represents the URL to be detected, f(x) represents the judgment result output by the weighted learning model, and y is the actual judgment result.

[0045] Secondly, based on the same inventive concept, this application provides a method for URL-related operations applied to the removable weighted learning Bloom filter described above, including URL detection, deletion from the database, and insertion of URLs. The following details the different specific operations: A method for detecting malicious URLs applied to a removable weighted learning Bloom filter as described above, such as... Figure 4 As shown, it includes the following steps: Step S1: Obtain the URL to be detected, i.e., the original URL; Step S2: Use the PH table to filter the URLs to be detected and obtain the filtered URLs. If the URL exists in the PH table, it is considered a malicious URL and can be accessed normally without further processing. Otherwise, proceed to the next step. Step S3: Use a weighted learning model to perform preliminary detection on the filtered URLs, obtain detection scores, and obtain the malicious URLs and / or URLs that were not initially determined to be malicious based on the detection scores; Step S4: Based on the detection score of the learning model, obtain the groups of URLs identified as malicious by the learning model in the partition backup Bloom filter, and perform detection in the corresponding groups. If a malicious URL is detected, notify the user; if a benign URL is detected, normal access is allowed.

[0046] This application uses PH table filtering, weighted learning model for preliminary detection, and Bloom filters for each sub-count of the backup filter for further detection to gradually detect malicious URLs from the database and ensures filtering performance even if the URL is deleted from the database.

[0047] In one embodiment, step S1: obtaining the URL to be detected, i.e., the original URL, specifically includes the following steps: The original URL is obtained by querying a perfect hash function.

[0048] In one embodiment, step S2: using a PH table to filter the URLs to be detected, and obtaining the filtered URLs and the filtered malicious URLs, specifically includes the following steps: Step S21: Map the URLs to be detected to the PH table using a perfect hash function for filtering; Step S22A: If the URL to be detected exists in the PH table, it indicates that the URL to be detected has been deleted from the database and is a malicious URL. No further detection is needed to determine whether it is a malicious URL. Step S22B: If the URL to be detected does not exist in the PH table, then the URL to be detected is used as the URL filtered by the PH table for the next step of detection to determine whether it is a malicious URL.

[0049] In one embodiment, step S3: using a weighted learning model to perform preliminary detection on the filtered URLs, obtaining a detection score, and obtaining the malicious URLs initially determined and / or URLs not initially determined to be malicious based on the detection score, specifically includes the following steps: Step S31: Train and obtain a weighted learning model based on minimizing the loss function; Step S32: Obtain the detection score for each filtered URL based on the weighted learning model obtained during training. ; Step S33: Perform preliminary detection on the filtered URLs based on the detection scores and data delineation boundaries to obtain preliminary detection results including malicious URLs and / or URLs not directly identified as malicious; more specifically, let the data delineation boundaries be... , The Bloom filter is used to define the boundary between negative and positive data for partition backup; if If the URL is positive, then x is directly identified as a malicious URL; otherwise, x is inserted into the specified sub-counting Bloom filter (CBF) to further determine whether it is a malicious URL.

[0050] In one embodiment, step S4: grouping URLs that were not initially determined to be malicious based on the detection score and storing them in multiple sub-count Bloom filters of the partition backup Bloom filter for re-detection to obtain the detection results, specifically includes the following steps: Step S41: Group the URLs that were not initially identified as malicious and obtain the grouped URLs; specifically, compare the detection scores of the URLs that were not initially identified as malicious with their corresponding scores. Component popularity threshold URLs that were not initially identified as malicious were divided into d groups; Step S42: Store the URLs that were not initially identified as malicious in each group in the corresponding sub-bloom filter; Step S43: Test each group of URLs using a different number of independent hash functions to obtain detection results including malicious URLs and / or non-malicious URLs; specifically, for the j-th group of URLs, use... Each hash function was tested independently; the number of hash functions can be adjusted based on the characteristics of the URLs within each group. The specific values ​​are used to control the expected FPR for each group; adjusting the parameters specifically includes adjusting the hash function for each group of URL tests. Number of items, score heat threshold used for grouping Hyperparameters are used to make the expected FPR of different groups more balanced.

[0051] In one embodiment, step S4, which involves grouping URLs initially identified as malicious based on their detection scores and storing them in multiple sub-count Bloom filters of a partitioned backup Bloom filter, performing malicious detection again, and obtaining the results of the second detection, further includes the following steps: Step S5: Store benign URLs that have been deleted from the database based on data popularity, and identify them as malicious URLs after deletion.

[0052] In one embodiment, to reduce the FPR and detection time consumption after data deletion, step S5: storing the benign URLs deleted from the database in PH according to data popularity, specifically includes the following steps: Step S51: Sort malicious URLs according to data popularity; Step S52: Set the minimum heat threshold ; Step S53: Store items based on their access frequency, which is higher than the minimum hotness threshold. Benign URLs that are deleted are identified as malicious URLs and added to the PH table.

[0053] In one embodiment, the calculation of data heat in step S51 is shown in the following formula:

[0054] In the formula, The input item X represents the historical number of visits, and N represents the total number of visits to all URLs.

[0055] The following are pseudocode examples of algorithms applied to the URL detection, deletion, and insertion operations in the removable weighted learning Bloom filter described above: The URL detection algorithm is shown below:

[0056] The URL detection process described in Algorithm 1 mainly consists of the following steps: First, using the PH table... For query data Perform the test (lines 2 and 3 of the algorithm). If... The hash value in In the middle, it indicates For URLs that have been deleted from the database, i.e., URLs that are not in the whitelist, The state is set to False (lines 4 and 5). Otherwise, Input into the weighted learning model (Lines 6 and 7); then, using get Score (Line 8). If Representation learning model decision In the database, then The state is set to True. Otherwise, according to... Will Input to the corresponding sub-backup filter (Lines 9-12) Third, use To test (Line 13), and according to Result settings The state (line 14). Fourth, return. The result (line 16).

[0057] The algorithm for deleting a URL is as follows:

[0058] Algorithm 2 describes the process of URL deletion, which mainly consists of two steps: First, the URL is deleted... Input into the weighted learning model In the middle, we get Score (Lines 2 and 3). If Then we get Popularity ranking value (Lines 4-6). If This means The popularity Above that, rehash And Stored in In, and will The state is set to True (lines 7-10). Otherwise, according to... Will Input to the corresponding sub-backup filter (Line 12); Second step, from Delete and will The state is set to True (lines 13-14).

[0059] This approach does not consider exceptions such as URLs not being stored in the database or having been previously deleted. Furthermore, for batch URL deletion, the sorting operation in line 5 of Algorithm 2 is placed at the very beginning of the algorithm to save time overhead during deletion.

[0060] The algorithm for URL insertion is as follows:

[0061] Algorithm 3 describes the URL insertion process, which mainly consists of the following steps: First, using the PH table... For the inserted URL Perform the test (lines 2-3). If The hash value in In the middle, then prove This is hot data that has already been deleted; at this point, from Remove from (lines 4-6); otherwise, Input into the weighted learning model (Lines 7-8); then, using get Score (Line 9). If This indicates that the learning model determines... Since it's a benign URL, it can be directly... Set to True (lines 10-11). Otherwise, according to... Will Input to the corresponding sub-backup filter Middle (lines 12-13); Third, will insert and will Setting it to True indicates successful insertion (lines 14-15).

[0062] To verify the above effects, all experiments were conducted using 68,909 URLs obtained from Kaggle, including benign URLs as positive data and malicious URLs (including malware URLs, phishing URLs, and spam URLs) as negative data. To simulate different search frequencies for different URLs, each URL was assigned a different number of searches. Details of all URLs are described in Table 1. During the training phase, all positive samples and 85% of the negative samples were used to train the model, and 10% of the negative samples were used for validation. During the testing phase, all positive data and 5% of the negative samples were used for testing.

[0063] Table 1. Detailed information about the URLs used in the experiment.

[0064] This application compares the proposed DWLBF method with five Bloom filter methods, and the comparison method is described below: Counting Bloom Filter (CBF): CBF is a traditional Bloom filter structure that optimizes upon SBF by expanding each bit of the bit array into a counter so that the data structure can support dynamic set deletion.

[0065] Learning Bloom Filters (LBF): LBF is a new type of Bloom filter that combines a machine learning model with the traditional BF. It uses predicted probability scores to reduce the number of keys that need to be mapped into a bit array, thereby reducing FPR and memory usage.

[0066] Sandwiched Bloom Filter (SandwichedLBF): SandwichedLBF is a model that is further optimized based on LBF. It optimizes the performance of the filter by adding an initial filter before learning the model.

[0067] Adaptive Bloom Filter (Ada-BF): Ada-BF is an improved LBF that optimizes performance by adaptively adjusting the backup filter parameters for different score regions, utilizing the full score range. Compared to the LBF method, it has lower FPR and memory usage.

[0068] Partition Learning Bloom Filter (PLBF): PLBF is also an improved LBF. It uses dynamic programming to cluster the score space into different regions and assigns a different BF to each region, thereby reducing memory usage while maintaining a low FPR.

[0069] The comparative experiments were conducted on an Ubuntu 20.04 LTS system with an Intel(R) Xeon(R) CPU E3-1245 v3@3.40GHz processor and 32GB of memory. Furthermore, Python 3 was used as the programming language for all methods.

[0070] The performance of the filter was compared in terms of detection FPR and detection time before data deletion, as well as the filter's detection performance after partial data deletion, pH meter usage performance, and heat threshold setting. The specific comparisons are as follows: I. Comparison of filter detection performance without data deletion: 1.1 FPR Comparison: This section compares the FPR of all methods during the query process, considering only the case where the URL was not deleted. The bit array size of the backup filter is set to... Between, the interval is Furthermore, the SandwichedLBF method in comparison has an initial filter and a backup filter, with the size of the bit array used in the initial filter set to one-tenth of the total bit array size, and the remainder used for the backup filter.

[0071] The FPR comparison results of the detection process on three malicious URL datasets are shown in Tables 2 to 4. These tables show that, with a fixed bit array size, the DWLBF method has the lowest FPR, while the traditional Bloom filter (CBF) has a very high FPR. For the three URLs, when the bit array size used by the Bloom filter is set to... At this time, the FPR of CBF can reach 81.0739%. When the bit array size is small, the FPR of other comparison methods is higher, while that of DWLBF is the lowest. Furthermore, the FPR of all methods decreases with increasing bit array size and then tends to stabilize, while the FPR of DWLBF stabilizes at a small value when the bit array size is large. The comparison results clearly show that the DWLBF method has the smallest FPR when using the same bit array size. Especially in Tables 2 and 3, when the bit array size is greater than... and At that time, the FPR of the DWLBF method is 0.

[0072] Table 2. Comparison of FPR results on Malware without data deletion (FPR: %)

[0073] Table 3. Comparison of FPR results on Phishing without data deletion (FPR: %)

[0074] Table 4. Comparison of FPR results on Spam without data deletion (FPR: %)

[0075] Several factors contribute to these results. First, the traditional Bloom filter (CBF) uses a bit array to determine whether a URL is positive, so the size of the given bit array significantly impacts the FPR. Second, for the comparative learning Bloom filter, SandwichedLBF adds an initial filter, which can reduce the FPR to some extent. However, the initial filter must occupy some bits, resulting in a higher FPR when memory space is limited. Ada-BF and PLBF use partitioning to optimize the partitioned backup Bloom filter, but they ignore the impact of the learning model's learning performance on the FPR. Third, the proposed DWLBF method uses data heat for weighting during model training, which effectively improves the accuracy of the learning model, thus achieving the lowest FPR compared to other Bloom filters.

[0076] 1.2 Comparison of Detection Time This application further compares the query time consumption when no data is deleted, with the bit array size set to... The comparison results are shown in Table 5. On the one hand, since CBF only needs to confirm that the hash value at the corresponding position in the bit array is not 0, while the learned Bloom filter needs to verify through a combination of the learned model and the backup Bloom filter, it consumes more detection time. Therefore, CBF has the lowest time consumption. On the other hand, the time consumption of all learned Bloom filters is very similar, and the time consumption of the proposed DWLBF method is not more than that of other learned Bloom filters. Therefore, considering the results of FPR, the DWLBF method is superior to other methods in improving query performance.

[0077] Table 5. Comparison of detection times without data deletion (time: seconds)

[0078] II. Comparison of Filter Detection Performance After Deleting Partial Data 2.1 FPR Comparison This section compares the filter's FPR after removing different percentages of data from three malicious URL datasets. The percentage of deleted data from the corresponding URL datasets in the database ranges from 5% to 30%, with intervals of 5%, and the bit array size is set to [value missing]. Set a data popularity threshold. Set to 25%, all heat values ​​greater than Deleted URL data is stored in the PH table. Experimental results are shown in Figure 5(a). Figure 7 As shown in (c), it is evident that the DWLBF method exhibits the best performance among all learning methods, while the FPR of other learning methods gradually increases with the proportion of deleted URLs, the FPR of DWLBF remains low.

[0079] Figure 5 (a) is a comparison of the FPR of DWLBF and five Bloom filters on the Malware malicious URL dataset at different data deletion ratios; Figure 5 (b) is a comparison of the FPR of DWLBF and five Bloom filters on the Phishing malicious URL dataset under different data deletion ratios. Figure 5 (c) is a comparison of the FPR of DWLBF and five Bloom filters on the Spam malicious URL dataset at different data deletion ratios.

[0080] There are several reasons for the above results: First, DWLBF uses a PH table to store a portion of the deleted hot data. When querying this part of the data stored in the PH table, it will be filtered out directly to avoid being input into the subsequent model and being judged as positive data. Secondly, the PH table stores hot data, which is more likely to be queried than other data, thus effectively controlling the rise of FPR; Furthermore, although CBF is a traditional Bloom filter, it removes the marker at the corresponding position in the bit array of the hash map after data deletion, thus keeping the false positive rate stable. However, when the amount of data is too large and the size of the bit array used is small, its FPR is still relatively high.

[0081] In summary, the DWLBF method proposed in this application can effectively solve the problem of increased FPR after data deletion.

[0082] 2.2 Comparison of Detection Times This application further compares the detection time under data deletion conditions, with all settings identical to those in Section D.2.3.2(1). The comparison results are as follows: Figure 6 As shown, since the CBF method is a traditional Bloom filter method, its detection time is still lower than that of the learning-based methods after data deletion. For all learning-based methods, the query times are very similar before data deletion. However, after data deletion, the DWLBF method shows better performance because some of the deleted data is stored in the PH table. This data does not need to be detected by the learning model and the backup Bloom filter, thus greatly reducing the query time.

[0083] Figure 6 (a) is a comparison of the detection time of DWLBF and five Bloom filters on the Malware malicious URL dataset under different data deletion ratios; Figure 6 (b) is a comparison of the detection time of DWLBF and five Bloom filters on the Phishing malicious URL dataset under different data deletion ratios; Figure 6 The middle (c) graph shows the comparison of detection time between DWLBF and five Bloom filters on the Spam malicious URL dataset under different data deletion ratios.

[0084] III. Performance Testing of pH Meters 3.1 FPR Test This application tests the impact of the PH table on the FPR (Frequency Rate) after data deletion. Experiments were conducted on three malicious URL datasets, comparing the FPR results of URL detection using and without the PH table after data deletion. The percentage of data deletion ranged from 10% to 30%, with intervals of 10%. The bit array size was set to a fixed value for all URL datasets. The comparison results are as follows: Figure 7As shown, it is clear that the FPR was effectively reduced after using the PH table, therefore the PH table has a good effect on reducing the FPR after data deletion.

[0085] Figure 7 (a) is a comparison chart of FPR in the Phishing URL dataset after data deletion, with and without the PH table. Figure 7 (b) is a comparison chart of FPR after data deletion in the Malware URL dataset, with and without the PH table. Figure 7 (c) is a comparison chart of FPR after deleting data from the Spam URL dataset, with and without the PH table.

[0086] 3.2 Detection Time Test This application tests the effect of pH meter on detection time after data deletion. All experimental settings are the same as in Section D.2.3.3(1). Experimental results are as follows: Figure 8 As shown in the results, using the pH table can effectively reduce the detection time consumed by the model after data deletion. Combined with the FPR comparison results, using the pH table can effectively improve the detection performance of the model after data deletion.

[0087] Figure 8 (a) is a comparison of detection time in the Phishing URL dataset after data deletion, with and without the PH table. Figure 8 (b) is a comparison chart of detection time with and without the PH table after data deletion in the Malware URL dataset; Figure 8 (c) is a comparison of detection time with and without the PH table after data deletion in the Spam URL dataset.

[0088] IV. Heat Threshold Settings 4.1 FPR Test This application tests with different heat thresholds. Impact on FPR results. Experiments were conducted on three malicious URL datasets, with different heat thresholds. The percentage of deleted data is between 5% and 25%, with intervals of 10%; the percentage of deleted data is between 10% and 30%, with intervals of 10%; and the bit array size on all URL datasets is set to [value missing]. The experimental results are as follows: Figure 9 As shown, regardless of whether the percentage of data deleted is small or large, the FPR increases accordingly. The value increases and then decreases. This is because... As the value increases, the amount of heat stored in the pH meter is greater than that stored in the pH meter. The larger the dataset, the more data will be deleted during testing and detected as negative, resulting in a lower FPR.

[0089] Figure 9 (a) shows a comparison of FPR (Functional Per Count) when different thresholds are set for detection on the Phishing URL dataset; Figure 9 (b) shows a comparison of FPR (Failure Rate) when different thresholds are set for detecting the Malware URL dataset; Figure 9 (c) shows a comparison of FPR (Functional Per Count) when different thresholds are set for detecting Spam URLs. 4.2 Detection Time Test This application tests with different heat thresholds. The experimental setup for the impact on detection time is the same as in section 4.1. The experimental results are as follows: Figure 10 As shown, regardless of whether the percentage of data deleted is small or large, the time consumption increases accordingly. The value increases and then decreases. This is because... As the value increases, the amount of heat stored in the pH meter is greater than that stored in the pH meter. The larger the dataset, the more deleted data will be detected as negative during testing. Deleted data stored in the PH table does not need to be detected again through the learning model, thus shortening the detection time. Considering the results of 4.1 FPR, the heat threshold... The settings have a certain impact on the model performance after data deletion. Furthermore, the more deleted data stored in the PH table, the larger the memory space usage; therefore, only a subset of this setting is used in the experiment. .

[0090] Figure 10 (a) is a comparison of detection times when different thresholds are set for the Phishing URL dataset; Figure 10 (b) is a comparison chart of detection time when different thresholds are set for detecting Malware URL datasets; Figure 10 (c) is a comparison of detection time when setting different thresholds for the Spam URL dataset.

[0091] Thirdly, embodiments of this application provide a removable weighted learning Bloom filter device for malicious URL detection. The removable weighted learning Bloom filter device for malicious URL detection can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0092] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces for interconnecting internal components of the deleteable weighted learning Bloom filter device used for malicious URL detection, as well as interfaces for interconnecting the deleteable weighted learning Bloom filter device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0093] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0094] The processor can be a general-purpose processor, which can call a program for a removable weighted learning Bloom filter for malicious URL detection stored in memory and execute the removable weighted learning Bloom filter provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the program for the removable weighted learning Bloom filter for malicious URL detection is called can be referred to the various embodiments of the removable weighted learning Bloom filter in this application, and will not be repeated here.

[0095] Fourthly, embodiments of this application also provide a readable storage medium.

[0096] This application has a readable storage medium storing a program for detecting malicious URLs that can be deleted, a weighted learning Bloom filter program for detecting malicious URLs, which, when executed by a processor, implements the steps of the weighted learning Bloom filter program as described above.

[0097] The method implemented when the program for detecting malicious URLs using a removable weighted learning Bloom filter is executed can be referred to in the various embodiments of the removable weighted learning Bloom filter of this application, and will not be repeated here.

[0098] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0100] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A removable weighted learning Bloom filter, characterized in that, include: A perfect hash table is used to store some benign URLs that have been deleted from the database based on data popularity. URLs that have been deleted are identified as malicious URLs. The perfect hash table is used to filter URLs to be detected. If a URL exists in the perfect hash table, it is considered a malicious URL. The weighted learning model is used to assign weights to URLs to be detected based on their popularity and to perform preliminary detection to obtain a detection score. If the detection score is greater than a set threshold, the URL is determined to be a benign URL; if the detection score is less than the set threshold, the URL is preliminarily determined to be a malicious URL. The partition backup Bloom filter includes multiple sub-count Bloom filters. These sub-count Bloom filters are used to store URLs that are initially identified as malicious for re-detection. The results of the re-detection are obtained. If the URL is identified as malicious again, a notification command is sent to the user. Otherwise, normal access is granted to the URL. The process of storing benign URLs that have been deleted from the database based on data popularity, and then identifying them as malicious URLs after deletion, specifically includes the following steps: Sort URLs based on data popularity; Set a minimum heat threshold; URLs that have been deleted from the database and whose access frequency is higher than the minimum popularity threshold are sorted and stored as benign URLs. In this case, the URL is identified as a malicious URL. The loss function L of the weighted learning model is minimized as follows: In the formula, x represents the input item, and represents the URL to be detected. Let y be the weight of the input item x, f(x) represent the judgment result output by the weighted learning model, and y be the actual judgment result. The data popularity The calculation is shown in the following formula: In the formula, represents the historical access count of input item x, and N represents the total access count of all URLs.

2. A method for detecting malicious URLs applied in a removable weighted learning Bloom filter as described in claim 1, characterized in that, Includes the following steps: Get the URL to be tested; A perfect hash table is used to filter the URLs to be detected. If the URL exists in the perfect hash table, it is considered a malicious URL and no further action is required; otherwise, proceed to the next step. A weighted learning model is used to perform preliminary detection on the filtered URLs and obtain a detection score. If the detection score is greater than a set threshold, the URL is considered a benign URL and is granted normal access without further action. Otherwise, proceed to the next step. Based on the detection scores, the URLs identified as malicious by the weighted learning model are grouped in the partition backup Bloom filter, and then re-detected in the corresponding groups. If a URL is detected as malicious, a notification is sent to the user; if a URL is detected as benign, normal access is granted to that URL.

3. The malicious URL detection method as described in claim 2, characterized in that, Obtaining the URL to be detected specifically includes the following steps: The URL to be checked is obtained by querying a hash function.

4. The malicious URL detection method as described in claim 2, characterized in that, The process of using a weighted learning model to perform preliminary detection on the filtered URLs, obtaining detection scores, and then identifying malicious URLs and / or URLs not initially identified as malicious based on these scores includes the following steps: A weighted learning model is obtained by training based on minimizing the loss function; The weighted learning model obtained from training is used to detect and obtain the detection score of each filtered URL; Based on the detection scores and data delineation boundaries, the filtered URLs are initially detected to obtain preliminary detection results, including malicious URLs and / or URLs that are not directly identified as malicious.

5. The malicious URL detection method as described in claim 2, characterized in that, The process of grouping URLs that were not initially identified as malicious based on their detection scores and storing them in multiple sub-count Bloom filters of the partition backup Bloom filter for further detection to obtain the detection results includes the following steps: URLs that were not initially identified as malicious are stored in the corresponding sub-count Bloom filter; For each group of URLs, different numbers of independent hash functions are used for testing to obtain detection results including malicious URLs and / or non-malicious URLs.

6. The malicious URL detection method as described in claim 2, characterized in that, The process of grouping and storing URLs that were not initially identified as malicious into multiple sub-count Bloom filters of a partitioned backup Bloom filter based on their detection scores, and then performing malicious detection again to obtain the results of the second detection, further includes the following steps: Based on the data popularity, benign URLs that have been deleted from the database are identified as malicious URLs after being deleted.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a malicious URL detection program, wherein when the malicious URL detection program is executed by a processor, it implements the steps of the malicious URL detection method as described in any one of claims 2 to 6.

Images