Web URL malicious real-time detection system based on continuous learning and fusion large model

By adopting continuous learning and fusion of big models in the Web URL malicious detection system, combining RAG+ big model assisted judgment and real-time log processing, the problem of incomplete data sets is solved, the detection accuracy and adaptability are improved, and the unknown is achieved Effective identification and real-time detection of threats.

CN119004453APending Publication Date: 2024-11-22DONGGUAN UNIV OF TECH +2
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411010770.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the malicious detection of Web URLs, the detection accuracy is low and it is difficult to effectively identify unknown threats. The lightweight model cannot invest a large amount of data for training, which limits the training ability of the model.

Method used

Using a system based on continuous learning and fusion of big models, combined with RAG+ big models to assist judgment, malicious real-time detection of Web URLs and Continue to learn.

Benefits of technology

It significantly improves detection accuracy, continuous learning ability, data set coverage, real-time and response speed, enhances the scalability and flexibility of the system, and can effectively identify unknown threats and adapt to new threat patterns.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004453A_ABST
    Figure CN119004453A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, and relates to application of deep learning in natural language processing, in particular to a Web URL malicious real-time detection system based on continuous learning and a fused large model. Comprising a deep learning model, an RAG URL data knowledge base, a large model with a good public effect, a real-time log extraction module, a detection and judgment module, auxiliary judgment logic, an analysis providing module and a continuous learning module. According to the Web URL malicious real-time detection method, the precision, the continuous learning ability, the data set coverage range, the real-time performance and the response speed of the Web URL malicious real-time detection system and the expandability and the flexibility of the system are remarkably improved through a continuous learning and large model fusion method. Due to the beneficial effects, the method has higher practical value and competitiveness in practical application.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to an application of deep learning in natural language processing, and more particularly to a Web URL malicious real-time detection system based on continuous learning and fusion of large models. Background Art

[0002] In the field of natural language processing, deep learning models provide new solutions for malicious Web URL detection. They can learn unknown threats and significantly improve detection accuracy. The RAG+ large model makes large language models more practical and reliable in practical applications by alleviating hallucination problems and enhancing the traceability of content generation. This approach can focus on specific areas. By continuously learning new data through incremental updates, catastrophic forgetting problems are reduced, allowing the model to continuously learn new malicious threats and improve detection accuracy.

[0003] However, in the actual environment, due to the incompleteness of the collection and classification of Web log data sets by the user units, and the inability to invest a large amount of data for training like large language models, there are limitations in training malicious detection models. Therefore, a new method is needed to solve the problem of incomplete training data sets for lightweight models, improve the accuracy of the overall system, and achieve real-time malicious detection of Web URLs. Summary of the invention

[0004] To solve the above problems, the present invention proposes a Web URL malicious real-time detection system based on continuous learning and fusion big model. The main purpose of the system is to increase the judgment of unknown threats and improve the accuracy of the overall system by combining RAG+big model to assist judgment. At the same time, the continuous learning method is used to achieve malicious real-time detection of Web URLs.

[0005] The technical solution adopted by the present invention is: a Web URL malicious real-time detection system based on continuous learning and fusion of large models, the Web URL malicious real-time detection system includes: a deep learning model for training the collected data set;

[0006] A RAG URL data knowledge base containing examples of malicious URL-related web attack techniques and the benign portion of the collected training dataset;

[0007] A large model with good public performance as an auxiliary judgment module;

[0008] A real-time log extraction module, used to extract and normalize URL fields in real-time logs;

[0009] A detection and judgment module, used to detect and judge whether the URL in the window is malicious or benign when the window queue reaches a certain value;

[0010] An auxiliary judgment logic is used to further assist judgment through the RAG URL data knowledge base + large model when the URL is judged to be benign;

[0011] An analysis providing module that records the relevant analysis of the RAG URL data knowledge base + large model for professional managers to view and analyze;

[0012] A continuous learning module is used to achieve continuous learning of the model by fusing intermediate vectors.

[0013] The deep learning model is obtained by training an initial data set, and the trained model weights and the vocabulary dictionary and character dictionary made by the initial data set are recorded.

[0014] The RAG URL data knowledge base is obtained by integrating examples of malicious URL-related Web attack techniques and the benign part of the collected training data set.

[0015] The auxiliary judgment logic includes:

[0016] If the RAG URL data knowledge base + large model determines that it is malicious, the URL information will be labeled malicious and recorded; and the analysis results will be provided to the analysis providing module.

[0017] The analysis providing module comprises:

[0018] The analysis results are stored for review and analysis by professional managers.

[0019] The continuous learning module includes:

[0020] A cluster center vector acquisition module is used to obtain the cluster center vector of the initial data set and select a certain number of points closest to the cluster center;

[0021] An incremental dictionary implementation module, which is used to implement the incremental dictionary by adding new key values ​​to meet the vector conversion of new URLs;

[0022] A model weight fusion update module is used to adopt replacement-based incremental update to update the model weights by taking new vector values ​​and corresponding labels and a certain amount of cluster center vector values ​​and corresponding labels.

[0023] The cluster center vector acquisition module finds the cluster center of the initial data set by clustering, and records a certain amount of vector points closest to the cluster center.

[0024] The purpose of the incremental dictionary implementation module is to allow more unseen words to be remembered by the model, and also to make vectorization more accurate.

[0025] The model weight fusion update module also includes:

[0026] After each update, the cluster center of the current data set is found again through clustering, and a certain number of vector points closest to the cluster center are recorded as the replacement part of the next incremental learning based on replacement.

[0027] A Web URL malicious real-time detection system based on continuous learning and fusion of large models also includes a report recording module for directly reporting and recording when a URL is judged to be malicious, and the recorded content can be manually checked again.

[0028] A Web URL malicious real-time detection system based on continuous learning and fusion of large models also includes a window mechanism module, which is used to continuously read Web log information through the window mechanism, and extract and regularize URL field information through regular matching as input of the deep learning model.

[0029] The technical effect achieved by the present invention is: the present invention significantly improves the accuracy, continuous learning ability, data set coverage, real-time performance and response speed of the Web URL malicious real-time detection system, as well as the scalability and flexibility of the system through the method of continuous learning and integration of large models. These beneficial effects make the present invention have higher practical value and competitiveness in practical applications. Compared with the existing technology, it has the following significant beneficial effects:

[0030] (I) Improved detection accuracy: By combining the RAG URL data knowledge base + large model auxiliary judgment, the present invention increases the ability to judge unknown threats and effectively improves the detection accuracy of the overall system. This is particularly important for defending against new and unknown Web attacks.

[0031] (ii) Enhanced continuous learning capability of the system: The present invention adopts a method of continuous learning by fusing intermediate vectors, which enables the model to continuously learn new data while reducing the problem of catastrophic forgetting. This means that the system can continuously adapt to new threat patterns and maintain a high detection efficiency.

[0032] (III) Expanded data set coverage: By integrating examples of malicious URL-related Web attack techniques and the benign portion of the collected training data set as a URL data knowledge base, the present invention reduces the false positives of the model and expands the data coverage of the Web URL malicious real-time detection system through more malicious URL data examples, thereby compensating for the problem that the lightweight model training data set is not comprehensive enough.

[0033] (IV) Improved real-time performance and response speed of the system: The present invention reads Web log information in real time through a window mechanism and can promptly judge whether a URL is malicious or benign. This helps to promptly discover and respond to Web attacks and reduce potential security risks.

[0034] (V) Enhanced scalability and flexibility of the system: The design of the present invention allows different large models to be selected as auxiliary judgment modules, and can be replaced or updated according to actual needs. This enables the system to adapt to different application scenarios and detection requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an architecture diagram combining RAG URL data knowledge base + large model assistance proposed by the present invention;

[0036] Figure 2 A system diagram for implementing continuous learning by fusing intermediate vectors proposed by the present invention. DETAILED DESCRIPTION

[0037] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings:

[0038] like Figure 1-2 As shown, the present invention proposes a real-time malicious Web URL detection system based on continuous learning and fusion of large models, aiming to solve the problem of incomplete lightweight model training data set, and through RAG+ large model auxiliary judgment, increase the judgment of unknown threats and improve the accuracy of the overall system. The following is a specific implementation of the present invention:

[0039] 1. Initial preparation

[0040] Collect the log data set that can be collected as the initial data set. Most of the existing Web system log collection methods use attack tools for testing and then log. Although this method has a strong defense capability against previous threats, it cannot effectively defend against unknown vulnerabilities such as 0day due to the limitations of attack tools.

[0041] A deep learning model with good results in natural language processing is used as a URL lightweight model to train the initial data set, and the model weights after training and the vocabulary dictionary and character dictionary made by the initial data set are recorded. The cluster center of the initial data set is found by clustering, and a certain number of vector points closest to the cluster center are recorded as the replacement part of incremental learning based on replacement.

[0042] 2. System Architecture and Workflow

[0043] RAG URL data knowledge base + big model is designed as an auxiliary module of URL lightweight model. The content of URL data knowledge base adopts the integration of examples of malicious URL-related Web attack techniques and the benign part of the collected training data set. Through various Web attack techniques and the URL field positions that may be modified by related examples, the big model can better decide the maliciousness of the URL, and at the same time provide the benign part of the training data set, in order to reduce the false positives of the existing labeled data. The example of RAG + big model adopts the mode of FASTGPT + ONEAPI, and the selection of big model can select the big model with better test results.

[0044] Obtain Web log information from CSNS in real time, continuously read Web log information through the window mechanism, and extract and regularize URL field information through regular matching.

[0045] Input the regularized URL field information into the URL lightweight model and output the results through the model.

[0046] Determine whether the URL information is malicious or benign. If it is malicious, no auxiliary judgment is made, and the report is recorded directly, and the recorded content can be manually checked again. If it is benign, the URL information is further judged by RAG URL data knowledge base + large model.

[0047] In the auxiliary judgment stage, if the RAG URL data knowledge base + large model judges it to be malicious, the URL information will be labeled malicious and recorded, and the recorded content can be manually checked again. At the same time, the corresponding malicious analysis reasons will be stored for review by professional managers.

[0048] 3. Continuous Learning and Model Updating

[0049] For the newly recorded malicious URL information, we use the method of adding key values ​​to implement the incremental dictionary. This is to allow more unseen words to be remembered by the model and to make vectorization more accurate.

[0050] Vectorize the new malicious URL information. This part is used as the new incremental part and merged with the previously prepared replacement part to update the URL lightweight model weight. Record the updated model weight data, and find the cluster center of the current data set again through clustering. Record a certain number of vector points closest to the cluster center as the replacement part of the next incremental learning based on replacement.

[0051] Through the above specific implementation methods, the present invention can realize real-time malicious detection of Web URLs, while continuously improving the detection accuracy and the ability to cope with unknown threats.

[0052] Embodiment 1

[0053] This embodiment proposes a Web URL malicious real-time detection system based on continuous learning and fusion of large models. The following is a specific embodiment of the system:

[0054] 1. Initial preparation

[0055] Collect log data sets: Use existing Web system log collection methods, combine with attack tools for testing, and record logs as the initial data set. Although this method has strong defense capabilities against known threats, it has limitations and requires further processing. Train URL lightweight model: Use a deep learning model that works well in natural language processing to train the initial data set, and record the model weights after training and the vocabulary dictionary and character dictionary made from the initial data set.

[0056] Cluster analysis: Find the cluster center of the initial data set through clustering, and record a certain number of vector points closest to the cluster center as the replacement part of incremental learning based on replacement.

[0057] 2. System Architecture and Workflow

[0058] Design auxiliary modules: Build a RAG URL data knowledge base and integrate examples of malicious URL-related Web attack techniques and the benign parts of the collected training data set. Select a large model with good test results, such as the FASTGPT+ONEAPI architecture and llama3:8b as an auxiliary large model.

[0059] Real-time log processing: Obtain web log information from CSNS in real time, and continuously read and process log information through the window mechanism. Use regular matching to extract and regularize URL field information as input for the URL lightweight model.

[0060] Model detection and judgment: The URL lightweight model outputs the results to determine whether the URL information is malicious or benign. If it is malicious, it will be directly reported and recorded, and can be manually checked again. If it is benign, further auxiliary judgment will be carried out.

[0061] Auxiliary judgment: benign URL information is input into the RAG URL data knowledge base + large model auxiliary module for judgment. If the large model judges it as malicious, the URL information will be marked with a malicious label and recorded; and the relevant analysis results judged as malicious will be stored for review by professional management personnel.

[0062] 3. Continuous Learning and Model Updating

[0063] Incremental dictionary construction: For the newly recorded malicious URL information, an incremental dictionary is implemented by adding key values ​​to expand the vocabulary of the model and improve the vectorization accuracy.

[0064] Vectorization and model update: Vectorize the new malicious URL information and merge it with the previously prepared replacement part to update the URL lightweight model weight. Record the updated model weight data and perform cluster analysis again to prepare for the next incremental learning based on replacement.

[0065] Through the above specific implementation methods, this embodiment can realize real-time malicious detection of Web URLs, while continuously improving detection accuracy and the ability to cope with unknown threats.

[0066] Embodiment 2

[0067] This embodiment proposes a Web URL malicious real-time detection system based on continuous learning and fusion of large models. The following is another embodiment of the system:

[0068] 1. Data preparation and model training

[0069] Data collection: Collect web URL datasets from various sources, including public and classified URL datasets, and log datasets obtained through attack tool testing.

[0070] Data preprocessing: Preprocess the collected data, including cleaning, deduplication, and format unification, to form an initial data set that can be used for model training.

[0071] Model training: Select a deep learning model that performs well in natural language processing as the URL lightweight model and train it using the initial dataset. The original weights and vocabulary dictionary of the model are recorded during the training process.

[0072] Cluster analysis: Perform cluster analysis on the initial data set, find the cluster center, and record a certain number of vector points closest to the cluster center for subsequent incremental learning.

[0073] 2. System Construction and Real-time Detection

[0074] Auxiliary module construction: Build a RAG URL data knowledge base, integrate examples of malicious URL-related Web attack techniques and the benign parts of the collected training data set, and select a large model with good test results as an auxiliary judgment module.

[0075] Real-time log acquisition and processing: Obtain log information from the target Web system (such as CSNS) in real time, read and process the logs through the window mechanism, extract the URL field and format it.

[0076] Model detection and preliminary judgment: The processed URL field is input into the URL lightweight model to make a preliminary judgment on whether it is malicious or benign. If it is judged to be malicious, it will be directly reported and recorded; if it is judged to be benign, further auxiliary judgment will be made.

[0077] Auxiliary judgment and result processing: The URL information initially judged as benign is input into the RAG URL data knowledge base + large model auxiliary module for further judgment. According to the judgment result of the large model, the URL information is finally judged as malicious or benign, and the corresponding records and processing are performed, and the analysis results are stored at the same time.

[0078] 3. Continuous Learning and Model Update Strategy

[0079] Incremental dictionary construction and update: For newly emerging malicious URL information, incremental dictionaries are constructed by adding key values ​​to expand the model's vocabulary and improve vectorization accuracy. At the same time, incremental dictionaries are regularly updated to adapt to new malicious URL patterns.

[0080] Vectorization and model weight update: New malicious URL information is vectorized and merged with the put-back part obtained during previous training to update the URL lightweight model weight. Through continuous learning and updating, the model's detection accuracy and response capabilities for unknown threats are improved. At the same time, the updated model weight data is regularly recorded for subsequent use.

Claims

1. A real-time malicious Web URL detection system based on continuous learning and fusion of large models, Features: The Web URL malicious real-time detection system includes: A deep learning model, trained on the collected dataset; A RAG URL data knowledge base containing examples of malicious URL-related web attack techniques and the benign portion of the collected training dataset; A large model with good public performance as an auxiliary judgment module; A real-time log extraction module, used to extract and normalize URL fields in real-time logs; A detection and judgment module, used to detect and judge whether the URL in the window is malicious or benign when the window queue reaches a certain value; An auxiliary judgment logic is used to further assist judgment through the RAG URL data knowledge base + large model when the URL is judged to be benign; An analysis providing module that records the relevant analysis of the RAG URL data knowledge base + large model for professional managers to view and analyze; A continuous learning module is used to achieve continuous learning of the model by fusing intermediate vectors.

2. According to claim 1, a Web URL malicious real-time detection system based on continuous learning and fusion of large models, Features: The deep learning model is obtained by training an initial data set, and the trained model weights and the vocabulary dictionary and character dictionary made by the initial data set are recorded.

3. According to claim 1, a Web URL malicious real-time detection system based on continuous learning and fusion of large models, Features: The RAG URL data knowledge base is obtained by integrating examples of malicious URL-related Web attack techniques and the benign part of the collected training data set.

4. According to claim 1, a Web URL malicious real-time detection system based on continuous learning and fusion of large models, Features: The auxiliary judgment logic includes: If the RAG URL data knowledge base + large model determines that it is malicious, the URL information will be labeled malicious and recorded; and the analysis results will be provided to the analysis providing module.

5. According to claim 1, a Web URL malicious real-time detection system based on continuous learning and fusion of large models, Features: The analysis providing module stores the analysis results for review and analysis by professional managers.

6. According to claim 1, a Web URL malicious real-time detection system based on continuous learning and fusion of large models, Features: The continuous learning module includes: A cluster center vector acquisition module is used to obtain the cluster center vector of the initial data set and select a certain number of points closest to the cluster center; An incremental dictionary implementation module, which is used to implement the incremental dictionary by adding new key values ​​to meet the vector conversion of new URLs; A model weight fusion update module is used to adopt replacement-based incremental update to update the model weights by taking new vector values ​​and corresponding labels and a certain amount of cluster center vector values ​​and corresponding labels.

7. A Web URL malicious real-time detection system based on continuous learning and fusion of large models according to claim 6, Features: The cluster center vector acquisition module finds the cluster center of the initial data set by clustering, and records a certain amount of vector points closest to the cluster center.

8. A Web URL malicious real-time detection system based on continuous learning and fusion of large models according to claim 6, Features: The purpose of the incremental dictionary implementation module is to allow more unseen words to be remembered by the model, and also to make vectorization more accurate.

9. A Web URL malicious real-time detection system based on continuous learning and fusion of large models according to claim 6, Features: The model weight fusion update module also includes: after each update, re-finding the cluster center of the current data set by clustering, and recording a certain amount of vector points closest to the cluster center as the replacement part of the next replacement-based incremental learning.

10. A Web URL malicious real-time detection system based on continuous learning and fusion of large models according to claim 1, Features: It also includes a report recording module, which is used to directly record the report when the URL is judged to be malicious, and the recorded content can be manually checked again.

11. A Web URL malicious real-time detection system based on continuous learning and fusion of large models according to any one of claims 1 to 10, Features: It also includes a window mechanism module, which is used to continuously read Web log information through the window mechanism, and extract and regularize URL field information through regular matching as the input of the deep learning model.

Citation Information

Cited By

  • Method for detecting malicious domain name, computer equipment, program product and medium

    CN119766558A