Network security big model knowledge base construction method and device, and storage medium

Abstract

The invention relates to the field of network security, and discloses a network security big model knowledge base construction method and device and a storage medium, and the method comprises the following steps: obtaining an input network security problem through a user interaction module; reconstructing the network security question according to a synonym mapping table in the network security field, querying a pre-cached database according to the reconstruction question, if an answer exists, directly returning, otherwise, comparing the similarity between the reconstruction question and a pre-stored question in the database, and if the similarity exceeds a preset value, returning to the database; if yes, calling a machine learning module to obtain an answer output by the machine learning module, and otherwise, converting the reconstruction question into a question vector; querying a text with the highest association degree in a large model knowledge base according to the question vector, and when the association degree exceeds a preset value, combining the text and a context thereof, and inputting the combined text and context into the large model for processing to obtain a large model output answer; and storing the answers and the corresponding questions in a database. According to the invention, the accuracy and efficiency of question answering in the network security field are improved.

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to a method for constructing a network security large model knowledge base. Background Art

[0002] With the development of the information age, AI technology has made great progress. In the field of network security, a large amount of log data will be monitored in real time. The analyzed data will generate corresponding security events and be stored in the project system for statistical analysis.

[0003] In order to facilitate network engineers, security experts, and operation and maintenance personnel to efficiently and quickly analyze and locate problems in the current environment based on the time range, log aggregation will be performed. This can reduce the amount of data and speed up the processing and analysis rate. However, in addition to this, how to reasonably handle security incidents, such as blocking the corresponding IP through a firewall, patching the system with corresponding vulnerabilities, or some more complex processing methods, is very dependent on the experience of the experts who handle the task. Even if the system retains relevant records, it depends on the relevant personnel to accumulate experience and learn by themselves, and cannot be effectively utilized. At this time, introducing AI to process and improve work efficiency is certainly good, but for some professional field problems, there may be random answers. In this case, how to connect the local knowledge base and the large model to build an exclusive AI knowledge base to answer questions is obviously a better solution.

[0004] At present, the security incident handling methods and technologies in the field of network security are mostly handled manually, with AI used for assistance. The following is some background status of current technologies:

[0005] 1. Handling a large number of security incidents: In some cases, even after log aggregation, there are still a large number of security incidents, which puts tremendous pressure on operation and maintenance personnel. When the processing task volume is large, errors may occur;

[0006] 2. Personal ability deviation: Different people are responsible for different areas when handling security incidents. If the person handling the incident is weak in this area, it may take more time to handle the problem, or even adopt the wrong solution;

[0007] 3. General large models cannot meet the actual business needs of enterprises: The actual business needs of enterprises also involve issues such as knowledge limitations and information security. Enterprises obviously cannot upload private domain data to third-party platforms for training. For example, a more extreme example: company users ask general large models for the password to connect to the local database. Usually, the answer is impossible to be correct, and the enterprise is unlikely to provide such sensitive information for training. In this case, how to connect the local knowledge base and the large model to build a dedicated AI knowledge base has become a key issue;

[0008] 4. Performance bottleneck: Sometimes a large number of questions in the environment are actually very similar, but a large model is required to respond one by one. This solution leads to low efficiency of AI answers after the enterprise's knowledge base is enhanced;

[0009] 5. Accuracy issues: When introducing the company's own knowledge base, there will be problems with accuracy. Although it is feasible to simply import security field knowledge to build a knowledge base, the accuracy is obviously lacking. Summary of the invention

[0010] The purpose of the present invention is to propose a method for constructing a network security large model knowledge base to solve the technical problem that the existing technology cannot effectively connect the local knowledge base with the large model and reasonably and quickly handle network security related technical problems.

[0011] A method for constructing a network security large model knowledge base includes the following steps:

[0012] S1. Network security issues of obtaining input through user interaction modules;

[0013] S2. Reconstruct the network security problem according to a synonym mapping table in the network security field to obtain a reconstructed problem;

[0014] S3. Query the pre-cached redis database according to the reconstructed question. If there is an answer, return the answer directly. Otherwise, compare the similarity between the reconstructed question and the pre-stored question in the redis database. If the similarity exceeds the preset value, call the machine learning module to obtain the answer output by the machine learning module. Otherwise, convert the reconstructed question into a question vector.

[0015] S4, searching the large model knowledge base for the text with the highest relevance according to the question vector, and when the relevance exceeds a preset value, combining the text and its context, and inputting the text into the large model for processing, to obtain the large model output answer;

[0016] S5. Store the answers and corresponding questions output by the machine learning module and the answers and corresponding questions output by the large model in the redis database.

[0017] A storage medium stores instructions and data for implementing a method for constructing a network security large model knowledge base.

[0018] A network security large model knowledge base construction device includes: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a network security large model knowledge base construction method.

[0019] The beneficial effects provided by the present invention are: in the field of network security, for hot issues, the reply speed of the present invention can be in milliseconds, and for similar questions that conform to the knowledge base or the handling suggestions generated by clicking on the security incident handling, answers can be given within seconds, and the accuracy of the answers will not be affected, avoiding the situation where the large model answers inaccurately. Only completely irrelevant questions will be answered by the large model, which is close to the rate of normal large model AI answers. Overall, the present invention improves the accuracy and efficiency of answering questions in the field of network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic flow chart of the method of the present invention;

[0021] Figure 2 It is a working schematic diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0023] Before formally describing the present invention, the scheme of the present invention is first generally described for easy understanding.

[0024] Please refer to Figure 1 , Figure 1 It is a schematic flow diagram of the method of the present invention;

[0025] The present invention provides a method for constructing a network security large model knowledge base, comprising the following steps:

[0026] S1. Network security issues of obtaining input through user interaction modules;

[0027] It should be noted that the network security questions input in step S1 include questions input by users and security questions generated by relevant types of security events.

[0028] As an embodiment, assuming that the input network security question is a security question generated by a related type of security event, and the security event is a basic information collection event, the present invention will further convert the security event into a security question, that is, how to handle the basic information collection event.

[0029] S2. Reconstruct the network security problem according to a synonym mapping table in the network security field to obtain a reconstructed problem;

[0030] As an embodiment, with respect to the security question (how to handle basic information collection events) input in step S1, the present invention converts it into how to handle basic information collection events through a synonym mapping table in the network security field.

[0031] It should be noted that the synonym mapping table in the network security field is pre-constructed and defined, and the mapping table contains various synonyms related to the network security field. The present invention will convert different texts into their corresponding synonym questions by searching the mapping table to make the input questions more standardized.

[0032] S3. Query the pre-cached redis database according to the reconstructed question. If there is an answer, return the answer directly. Otherwise, compare the similarity between the reconstructed question and the pre-stored question in the redis database. If the similarity exceeds the preset value, call the machine learning module to obtain the answer output by the machine learning module. Otherwise, convert the reconstructed question into a question vector.

[0033] It should be noted that the redis database stores general hot issues, that is, some common hot issues in the field of network security. Since hot issues are frequently queried, their answers are relatively rich, and the answers obtained by querying the redis database are also relatively fast.

[0034] As an embodiment, in the present invention, after reconstructing the security question, a synonymous question is used to query the redis database. If a corresponding answer exists in the redis database, the answer to the question is directly returned.

[0035] It should be noted that the tf-idf algorithm is used to compare the similarity between the reconstructed question and the pre-stored questions in the redis database in step S3.

[0036] TF-IDF (term frequency-inverse document frequency): is a commonly used weighting technique for information retrieval and text mining.

[0037] If the corresponding answer does not exist in the redis database, the present invention uses another method, that is, calling the machine learning module to obtain the answer output by the machine learning module.

[0038] The machine learning module in the present invention has two forms: supervised and unsupervised.

[0039] Supervised learning: Provide a training set with correct results. Based on the training set, the algorithm will generalize "how to correctly respond to all possible inputs", that is, there is a clear output for the test set. It is also called "learning from examples".

[0040] Unsupervised learning: The training set does not provide the correct results, but allows the algorithm to try to identify similarities between different data, so that data with common characteristics can be classified together. Implementing supervised learning in a statistical way is also called "density estimation".

[0041] It should be noted that the machine learning module in the present invention is the key to optimizing the efficiency of large model answers. The module classifies security-related questions stored in the system and then conducts training, so that when questions are raised, they are classified into different types.

[0042] For example, when it is type 1, the present invention does not need to call the big model to answer, and directly extracts the answer from the knowledge base. When it is type 2, the big model is called again to answer.

[0043] As an example, for example, how to handle basic information collection events, which are classified as Type 1 by the machine learning module, the present invention uses a big model to maintain the answers to such security questions on a daily basis according to the knowledge base, rather than calling the big model for real-time answers.

[0044] It should be noted that the large model (also called the foundation model) refers to a machine learning model with a large number of parameters and complex structure, which can process massive data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, etc.

[0045] In addition, compared with the previous common machine learning model training method, which uses nlp word segmentation and then calls the word2vec method for training, the machine learning module in the present invention directly calls the embedded engine of the large model (nomic-embed-txt in the present invention) to perform text vectorization. Compared with previous solutions, it can train a more accurate model and avoid problem classification errors that lead to wrong answers.

[0046] Embedded Engine: When embedding an entire paragraph or document, the embedding process takes into account the overall context and the relationships between sentences and phrases in the text. This can produce a more comprehensive vector representation that captures the broader meaning and theme of the text. On the other hand, larger input text sizes may introduce noise or dilute the importance of individual sentences or phrases, making it more difficult to find exact matches when querying the index.

[0047] S4, searching the large model knowledge base for the text with the highest relevance according to the question vector, and when the relevance exceeds a preset value, combining the text and its context, and inputting the text into the large model for processing, to obtain the large model output answer;

[0048] As an embodiment, when the reconstructed problem cannot be well classified by the machine learning module, the present invention will convert the problem into a corresponding vector to vector database (i.e., the big model knowledge base) to query the text with the highest correlation. When the text matching degree is higher than a certain value, the text will be submitted to the big model for processing as a context-combined problem. When the matching degree of the document with the highest text matching degree is still lower than a certain value (for example: the weather today will have a low matching degree with any data in the security knowledge base), it represents a problem that is completely unrelated to the enterprise's private knowledge base. The context is directly set to empty and the problem is submitted to the big model to obtain an answer.

[0049] S5. Store the answers and corresponding questions output by the machine learning module and the answers and corresponding questions output by the large model in the redis database.

[0050] It should be noted that the answers to the questions will be stored in redis so that they can be answered quickly the next time the same question is asked.

[0051] The optimization point of steps S2 to S4 is to improve the questions by replacing words that are strongly related to the knowledge base, increase the matching degree of using questions to query the knowledge base and the prediction of the machine learning model, use redis to cache hot questions to improve the efficiency of answering, and use the traditional text comparison algorithm (tf-idf) to quickly filter out irrelevant data, avoid unnecessary performance consumption caused by calling the machine model and inaccurate answers caused by forced classification, and improve the accuracy and performance of the answers.

[0052] It should also be noted that the large model knowledge base of the present invention will form chunks according to the handling records of the corresponding event types in the database when it is constructed. For example, the handling of all event record texts of the basic information collection type is a chunk, and the pre-defined security domain synonym mapping table will be queried to reconstruct the chunk. For example, the handling of basic information collection events requires the deletion of local malicious program attacks -> the handling of basic information collection events deletes the local malicious information collection program. After that, the chunk will be vectorized through the embedded model provided by ollma and stored in the vector database lancedb.

[0053] CHUNK means: After document parsing is completed, some structured or semi-structured data can be obtained. The main task at present is to break this data into smaller chunks (a term used to describe data or information divided into small pieces or fragments) and then extract features, and then convert the features extracted from the data into a form that can capture its semantics.

[0054] The above process is different from the existing one in that:

[0055] Most commonly used data chunking methods are rule-based, using techniques such as fixed chunk size (chunking data or text into chunks of fixed size) or overlap of adjacent chunks (allowing adjacent data chunks to have overlapping content to ensure that information is not lost).

[0056] However, in practical applications, since the predefined rules (such as chunk size or size of overlapping parts) are too rigid, the rule-based data chunking method can easily lead to problems such as incomplete retrieved contexts or data chunks that are too large and contain noise (unnecessary, interfering information or data that may interfere with or mislead the analysis or processing).

[0057] The knowledge base construction program divides the text into chunks by security event category, so that each chunk is strongly related to the question, removing unnecessary noise and forming a more accurate knowledge base in the field of network security. At the same time, after updating the knowledge base, the system will also issue general type questions to the large model (for example: How to handle basic information collection events in type 1?), and enter the answers into the database after receiving the answers after the knowledge base is updated, so that machine learning can obtain more accurate answers to various types of questions.

[0058] See also Figure 2 , Figure 2 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a network security large model knowledge base construction device 401, a processor 402 and a storage medium 403.

[0059] A network security large model knowledge base construction device 401: The network security large model knowledge base construction device 401 implements the network security large model knowledge base construction method.

[0060] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the method for constructing a network security large model knowledge base.

[0061] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the method for constructing a network security large model knowledge base.

[0062] The beneficial effects of the present invention are: in the field of network security, for hot issues, the reply speed of the present invention can be in milliseconds, and for similar questions that conform to the knowledge base or the handling suggestions generated by clicking on the security incident handling, answers can be given within seconds, and the accuracy of the answers will not be affected, avoiding the situation where the large model answers inaccurately. Only completely irrelevant questions will be answered by the large model, which is close to the rate of normal large model AI answers. Overall, the present invention improves the accuracy and efficiency of answering questions in the field of network security.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a large network security model knowledge base, Features: The following steps are involved: S1. Network security issues of obtaining input through user interaction modules; S2. Reconstruct the network security problem according to a synonym mapping table in the network security field to obtain a reconstructed problem; S3. Query the pre-cached redis database according to the reconstructed question. If there is an answer, return the answer directly. Otherwise, compare the similarity between the reconstructed question and the pre-stored question in the redis database. If the similarity exceeds the preset value, call the machine learning module to obtain the answer output by the machine learning module. Otherwise, convert the reconstructed question into a question vector. S4, searching the large model knowledge base for the text with the highest relevance according to the question vector, and when the relevance exceeds a preset value, combining the text and its context, and inputting the text into the large model for processing, to obtain the large model output answer; S5. Store the answers and corresponding questions output by the machine learning module and the answers and corresponding questions output by the large model in the redis database.

2. A method for constructing a network security large model knowledge base as claimed in claim 1, Features: The network security questions input in step S1 include questions input by users and security questions generated by security events of related types.

3. A method for constructing a network security large model knowledge base as claimed in claim 1, Features: In step S3, the machine learning module stores a pre-trained machine learning model; the machine learning model is used to classify issues related to the field of network security and obtain answers to corresponding questions based on the classification.

4. A method for constructing a network security large model knowledge base as claimed in claim 3, Features: The training process of machine learning models uses an embedded engine for large models.

5. A method for constructing a network security large model knowledge base as claimed in claim 1, Features: In step S3, the similarity between the reconstructed question and the pre-stored questions in the redis database is compared using the tf-idf algorithm.

6. A method for constructing a network security large model knowledge base as claimed in claim 1, Features: When constructing the large model knowledge base in step S4, the text is divided into blocks according to the security event categories.

7. A storage medium, Features: The storage medium stores instructions and data for implementing a method for constructing a network security large model knowledge base as described in any one of claims 1 to 6.

8. A network security large model knowledge base construction device, Features: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a method for constructing a network security large model knowledge base as described in any one of claims 1 to 6.

Images