A honeypot log analysis method, system and medium based on a large language model

Abstract

The application discloses a method and system for analyzing honeypot logs based on a large language model and a medium, the method comprising: obtaining log data of a target system based on honeypot technology; performing cleaning processing on the log data, and constructing a vector index library based on the cleaned log data; in response to input text of a target object, obtaining a target log entry from the vector index library; based on the target log entry, arranging a prompt text through a preset template; inputting the prompt text into a pre-trained language model to obtain an answer content. The application uses honeypot technology to accurately collect relevant log data, and uses the accurate text analysis and evaluation capability of the large language model technology to solve the defects of the existing log analysis tools in terms of accuracy, real-time processing efficiency, data processing capability and flexibility. The application can accurately analyze logs and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a honeypot log analysis method, system and medium based on a large language model. Background Technology

[0002] Existing log analysis methods for threat trapping systems have three main drawbacks:

[0003] (1) High false alarm rate and high false alarm rate

[0004] Threat trapping logs are diverse in type and massive in volume. Many log analysis tools rely on static rules or known attack signatures, which may be insufficient to accurately distinguish between normal behavior and malicious activity, especially failing to identify new or complex attack patterns. Large language modeling techniques, however, can understand and parse complex textual data and behavioral patterns in network logs. Through continuous learning and adaptive capabilities, they can extract deeper meaning from logs, going beyond simply matching predefined rules or signatures. Therefore, high false negative and high false positive rates often lead to systems incorrectly labeling normal activity, requiring security teams to spend significant time and resources investigating these harmless events.

[0005] (2) Low real-time query efficiency

[0006] With the continuous growth of data volume, especially in large network environments, log data accumulates rapidly, leading to insufficient real-time analysis capabilities during processing and analysis. Existing log analysis tools are more suitable for historical data analysis than real-time data processing, lacking efficient indexing and search mechanisms. Therefore, the problem of low real-time query efficiency will affect the flexibility and efficiency of threat trapping log analysis.

[0007] (3) Low data processing capability

[0008] In large networks and complex IT environments, massive amounts of log data require powerful data processing capabilities, which traditional log analysis tools struggle to handle effectively. By utilizing large language model technology, with its capabilities in language understanding, generation, text evaluation, text conversion, and adaptation, the data quality and processing capabilities of threat trapping logs can be significantly improved. Therefore, low data processing capabilities can introduce time delays during log analysis and increase cybersecurity risks.

[0009] Application content

[0010] The main objective of this application is to propose a honeypot log analysis method, system, and medium based on a large language model, which can accurately perform log analysis.

[0011] To achieve the above objectives, one aspect of this application proposes a honeypot log analysis method based on a large language model, the method comprising:

[0012] Obtaining log data from the target system using honeypot technology;

[0013] The log data is cleaned and processed, and a vector index library is built based on the cleaned log data.

[0014] In response to the input text of the target object, the target log entry is obtained by matching from the vector index library;

[0015] Based on the target log entries, the prompt text is generated by organizing them using a preset template;

[0016] The prompt text is input into a pre-trained language model to generate a response and obtain the answer content.

[0017] In some embodiments, obtaining log data of the target system based on honeypot technology includes:

[0018] By utilizing the host virtualization and network topology virtualization functions of the honeypot, operating system characteristics and service types are specified for virtual nodes. Based on the specified operating system characteristics and service types, all relevant events and interactions are monitored and recorded, and log data is compiled. The log data includes service status logs and packet logs.

[0019] In some embodiments, cleaning the log data includes:

[0020] Use regular expressions to remove special symbols and irrelevant information from log data.

[0021] In some embodiments, a vector index library is constructed based on the cleaned log data, including:

[0022] The cleaned log data is segmented into words, and the log documents are compiled based on the results of the word segmentation; each log document contains several words.

[0023] Convert the log document into vector form;

[0024] Log documents in vector form are stored in a pre-defined database, and an index is built for the vector data in the database to obtain a vector index library.

[0025] In some embodiments, the method further includes:

[0026] The input text is cleaned, and the cleaned input text is then vectorized.

[0027] In some embodiments, in response to input text of the target object, matching to obtain the target log entry from a vector index includes:

[0028] Based on the query function of the vector index, the similarity between each log entry in the vector index and the input text is obtained by traversing the preset similarity calculation method.

[0029] The log entry with the highest similarity to the input text in the vector index is used as the target log entry.

[0030] In some embodiments, based on the target log entry, the prompt text is prepared using a preset template and includes:

[0031] Extract key information from target log entries and then integrate the key information into a preset Prompt template;

[0032] The Prompt template is optimized using a large-scale language model programming framework to obtain the prompt text; the optimization includes sentence structure adjustment and information coherence processing.

[0033] To achieve the above objectives, another aspect of this application proposes a honeypot log analysis system based on a large language model, the system comprising:

[0034] The first module is used to obtain log data from the target system based on honeypot technology;

[0035] The second module is used to clean and process log data, and to build a vector index library based on the cleaned log data.

[0036] The third module is used to obtain target log entries by matching the vector index library in response to the input text of the target object.

[0037] The fourth module is used to generate prompt text based on target log entries using a preset template;

[0038] The fifth module is used to input the prompt text into the pre-trained language model to respond and obtain the response content.

[0039] In some embodiments, the system further includes:

[0040] The sixth module is used to clean the input text and then vectorize the cleaned input text.

[0041] To achieve the above objectives, another aspect of the embodiments of this application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0042] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0043] The embodiments of this application include at least the following beneficial effects: This application provides a honeypot log analysis method, system, and medium based on a large language model. This scheme acquires log data from a target system using honeypot technology; cleans the log data and constructs a vector index library based on the cleaned log data; in response to input text from the target object, it matches and obtains target log entries from the vector index library; based on the target log entries, it organizes them using a preset template to obtain prompt text; and it inputs the prompt text into a pre-trained language model to obtain the response content. The embodiments of this application first utilize honeypot technology to accurately collect relevant log data, and then leverage the powerful language understanding and generation capabilities, flexible text conversion and generation capabilities, and accurate text analysis and evaluation capabilities of large language model technology to overcome the shortcomings of existing log analysis tools in terms of accuracy, real-time processing efficiency, data processing capabilities, and flexibility. The embodiments of this application can accurately perform log analysis. Attached Figure Description

[0044] Figure 1 This is a flowchart of the honeypot log analysis method based on a large language model provided in the embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the architecture of the threat trapping log data collection module provided in the embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the architecture principle of the threat trapping log analysis module based on large language model technology provided in the embodiments of this application;

[0047] Figure 4 This is a schematic diagram of the structure of the honeypot log analysis system based on a large language model provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0050] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0051] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0053] The honeypot log analysis method based on a large language model provided in this application relates to the field of data processing technology. This honeypot log analysis method based on a large language model can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the honeypot log analysis method based on a large language model, but is not limited to the above forms.

[0054] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0055] Figure 1 This is an optional flowchart of the honeypot log analysis method based on a large language model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S100 to S500.

[0056] S100. Obtain log data of the target system based on honeypot technology;

[0057] It should be noted that in some embodiments, step S100 may include: using the host virtualization and network topology virtualization functions of the honeypot to specify operating system characteristics and service types for virtual nodes, monitoring and recording all relevant events and interaction processes based on the specified operating system characteristics and service types, and compiling log data; the log data includes service status logs and packet logs.

[0058] For example, in some specific embodiments, threat trapping system log data is acquired: the threat trapping system is configured to record all relevant events and interactions. The acquired log information contains sufficient details, such as timestamps, source and target IP addresses, port numbers, protocol types, etc.; in some specific embodiments, the logs are also sliced ​​at fixed line intervals, and the sliced ​​log data is exported in a format suitable for processing.

[0059] In some preferred embodiments, log data can be obtained through a threat trapping log data collection module built based on honeypot technology, specifically:

[0060] The threat trapping log data collection module uses a typical honeypot as an example. Leveraging the honeypot's host virtualization and network topology virtualization capabilities, it statically or dynamically assigns operating system characteristics and service types to virtual nodes, monitoring and recording all relevant events and interactions. The recorded content includes service status logs and packet logs. Finally, the generated logs are sent to the threat trapping log analysis module based on large language model technology. For example, Figure 2 The diagram shown is an architectural schematic of the threat trapping log data collection module S1, which specifically includes:

[0061] S11: Deploy virtual hosts or virtual topologies on the honeypot host's network interface card (NIC) and set the NIC to obfuscated mode to monitor all traffic within the local area network.

[0062] S12: Honeypot simulated router, which generates arbitrary network topologies according to research needs.

[0063] S13: Unknown network requests for virtual hosts are handled by the central packet distributor.

[0064] S14: The central packet distributor compares the configuration database and distributes the data to the corresponding protocol processing unit according to the protocol type;

[0065] S15, S16, S17: The protocol processing unit executes the default response action according to the configuration file.

[0066] S18: The protocol processing unit responds to the port service specified in the configuration file.

[0067] S19: Log the response and interaction actions of threat trapping.

[0068] S200. Clean the log data and build a vector index library based on the cleaned log data.

[0069] It should be noted that in some embodiments, cleaning the log data may include using regular expressions to remove special symbols and irrelevant information from the log data.

[0070] For example, in some specific embodiments, regular expressions are used to remove special symbols and irrelevant information from log data, such as redundant punctuation and formatting symbols, to ensure data quality and consistency and to guarantee more accurate vector matching.

[0071] In some embodiments, constructing a vector index library based on the cleaned log data includes: performing word segmentation on the cleaned log data; organizing the log documents based on the word segmentation results; the log documents containing several words; converting the log documents into vector form; storing the vector-form log documents in a preset database; and constructing an index for the vector data in the database to obtain a vector index library.

[0072] For example, in some specific embodiments, the tokenizer built into the ChatGLM3-6B-32K model can be used to perform word segmentation on the cleaned log data, converting these tokens into numerical values ​​that the model can understand. LangChain is then applied to convert the processed documents into vector form, and a vector index library is defined. In this embodiment, Milvus can be chosen as the vector index library to store the vectorized data and build an index for the stored vector data, thereby optimizing subsequent retrieval and matching performance. LangChain is a programming framework that helps use large language models (LLMs) in applications.

[0073] S300, In response to the input text of the target object, obtain the target log entry by matching from the vector index library;

[0074] In some embodiments, the method may further include: cleaning the input text and vectorizing the cleaned input text.

[0075] It should be noted that in some embodiments, step S300 may include: based on the query function of the vector index library, combining a preset similarity calculation method to traverse and obtain the similarity between each log entry in the vector index library and the input text; and taking the log entry in the vector index library with the highest similarity to the input text as the target log entry.

[0076] For example, in some specific embodiments, taking Milvus as the vector index library, the vector representation of the user question is constructed into a suitable query format according to the requirements of the Milvus vector database to adapt to the vector database used. Using the query function of the vector database, the vector representation of the user question is compared with the log data vectors stored in the database. Based on the similarity calculation method provided by Milvus, the log entry that best matches the user question is identified.

[0077] S400: Based on the target log entries, the prompt text is generated by organizing them using a preset template;

[0078] It should be noted that in some embodiments, step S400 may include: extracting key information from the target log entry, and then integrating the key information into a preset Prompt template; optimizing the Prompt template through a large language model programming framework to obtain the prompt text; the optimization process includes sentence structure adjustment and information coherence processing.

[0079] For example, in some specific embodiments, a Prompt is constructed and user questions are processed: key information is extracted from the log entries that best match the user questions retrieved from the database, and this information is integrated into a selected Prompt template. Langchain is used to optimize the Prompt that combines user questions and log information, including adjusting the statement structure and ensuring information coherence.

[0080] S500: Input the prompt text into the pre-trained language model to generate a response and obtain the response content.

[0081] For example, in some specific embodiments, accurate answers are obtained through a Prompt template: The ChatGLM3-6B-32K model, trained on a large-scale network threat trapping related text dataset, is loaded to process and respond to relevant question queries input by the user. The model will generate answers based on the previously constructed Prompt content (i.e., prompt text) to help users quickly retrieve and understand key information in threat trapping logs.

[0082] To explain in detail the principles of the technical solution of this application, the overall process of this application will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principles of this application and should not be regarded as a limitation of this application.

[0083] First, it's important to note that with the development of internet technology and the acceleration of digital transformation, new forms of cybersecurity threats are constantly emerging. However, traditional passive defense technologies, such as firewalls and intrusion detection, are unable to effectively cope with the rapid evolution of security threats. Threat decoys, as a proactive defense technology, effectively help reverse the information asymmetry between defenders and attackers. Threat decoy systems deploy sensing nodes in the network to perceive the surrounding network environment in real time, and simultaneously store and visualize the sensing node logs in real time to achieve threat awareness in the network environment. However, the analysis of threat decoy system logs usually requires professional cybersecurity analysts, which to some extent limits the application of threat decoy systems.

[0084] Honeypots are a typical threat trapping system. Honeyd is a typical virtual honeypot framework capable of creating thousands of virtual honeypots simultaneously. Attackers can interact with each individual host over the network, while defenders can configure various functions through configuration files. Furthermore, Honeyd can simulate various routing topologies and configure network characteristics such as latency, packet loss, and bandwidth. Honeyd supports subsystem virtualization, allowing the execution of real UNIX applications, such as web servers and FTP servers, within the virtual namespace of the honeypot. Honeyd generates comprehensive logs, including packet-level and service-level logs, which record detailed interactions between attackers and simulated hosts. These logs reflect attack behaviors and strategies in diverse environments. Through these logs, cybersecurity teams can gain deep insights into attacker behavior and optimize their security defense strategies.

[0085] The main methods for analyzing logs from existing threat trapping systems include:

[0086] 1. Threat trapping log clustering: This method uses unsupervised machine learning algorithms to automatically group attack events in threat trapping logs, and classifies different attack types or attack scenarios based on similarity or distance metrics, thereby improving analysis efficiency and discovering potential attack patterns or abnormal behaviors.

[0087] 2. Threat Deception Based on Clustering and Association Rules: A Honeypot Log Analysis Method Based on Large Language Models. This method combines data mining and statistical analysis techniques to correlate attack events in threat deception logs to discover attack patterns. It reveals attackers' attack strategies, attack paths, and attack chains, and helps identify attack targets, vulnerabilities, and methods, providing a basis for defense and tracking.

[0088] 3. Attack profiling method: This method describes and classifies attack events in threat trapping logs, and constructs the characteristics and behavioral patterns of attackers from multiple dimensions (attack source, attack target, attack frequency, attack depth, attack complexity, attack stealth, etc.).

[0089] 4. Log Analysis Tools: Currently, threat trapping log analysis primarily relies on threat trapping technologies and product platforms, employing specific log analysis tools or utilizing the built-in log analysis functions of the threat trapping product. Among domestic and international projects, the Scan of Month project by the Honeynet project team stands out for its expertise in threat trapping log analysis. It analyzes system worms, nmap scans, IIS Unicode attacks, etc., using the Snort tool and Perl scripts for log analysis.

[0090] Given the shortcomings of the prior art in the background section, the main reasons for these shortcomings include the following aspects:

[0091] (1) Traditional log analysis tools rely on static rules or known attack signatures. These rules are usually based on past attack behaviors, which makes them less effective against new or complex attacks. In particular, they are difficult to accurately distinguish between normal behavior and malicious activities in complex network environments, resulting in high false negative and high false positive rates.

[0092] (2) As the network environment becomes more complex, threat trapping log data is increasing rapidly, leading to a significant increase in data variance and complexity, resulting in a large amount of data. Existing log analysis tools are mainly suitable for historical data analysis and are insufficient to meet the needs of real-time data processing. These tools are inefficient in the data processing process due to their lack of ability to process long text data and perform efficient real-time analysis.

[0093] (3) The lack of effective indexing and search mechanisms makes it difficult to retrieve information quickly and accurately from massive log data, which increases query time and reduces the feasibility of real-time analysis.

[0094] Furthermore, in recent years, the development of artificial intelligence and machine learning has brought revolutionary changes to the field of Natural Language Processing (NLP). In particular, the emergence of Large Language Models (LLMs) has enabled computers to understand and generate human language, providing a powerful tool for text analysis. With the rapid development of generative AI technology, the AI ​​capabilities demonstrated by large language model technology have attracted increasing attention. In the future, AI technology will continue to develop rapidly and gradually become more accessible to the general public.

[0095] ChatGLM3-6B-32K is a long-context version of the third-generation large language model series jointly launched by Zhipu AI and Tsinghua University's KEG Lab, supporting 32K context inputs. While retaining many excellent features of its predecessors, such as smooth dialogue and low deployment threshold, it offers significantly enhanced performance. Building upon ChatGLM3-6B-Base, ChatGLM3-6B-32K utilizes more diverse training data, more training steps, and a more reasonable training strategy. Tests on different datasets show that ChatGLM3-6B-32K performs superiorly. ChatGLM3-6B-32K adopts a newly designed Prompt format, natively supporting complex scenarios such as function calls, code execution, and agent tasks, in addition to normal multi-turn dialogues. For long text data files like logs, ChatGLM3-6B-32K offers significant advantages in both processing power and functional support.

[0096] Therefore, applying large language model technology to threat trapping log analysis, by leveraging the model's advanced language understanding capabilities, can more effectively parse and analyze threat trapping logs, automate the processing and analysis of threat trapping logs, understand the key information within them, and answer user questions in natural language, thereby greatly improving the efficiency and accuracy of threat trapping log analysis.

[0097] The purpose of this application is to address the shortcomings of traditional threat decoy log analysis techniques, such as high false positive rates, high missed detection rates, low real-time query efficiency, and low data processing capabilities. As threat decoy log analysis technology evolves, it requires continuous development and innovation. Therefore, this application applies large language modeling technology to threat decoy log analysis. Leveraging the powerful language understanding and generation capabilities, flexible text conversion and generation capabilities, and accurate text analysis and evaluation capabilities of large language modeling technology, this application addresses the deficiencies of existing log analysis tools in terms of accuracy, real-time processing efficiency, data processing capabilities, and flexibility. It reconstructs the threat decoy log analysis system, thereby providing a more efficient and accurate log analysis solution for the cybersecurity field.

[0098] Therefore, this application, through in-depth research on threat trapping log analysis technology, applies large language modeling technology to threat trapping log analysis, aiming to solve the problems of high false negative rate, high false positive rate, low real-time query efficiency, and insufficient data processing capabilities in traditional honeypot log analysis methods based on large language models. The overall process is as follows:

[0099] 1. Obtain Threat Enticement System Log Data: Configure the threat enticement system to record all relevant events and interactions. The obtained log information should contain sufficient details, such as timestamps, source and target IP addresses, port numbers, and protocol types. The logs should be sliced ​​at fixed line intervals, and the sliced ​​log data should be exported to a format suitable for processing.

[0100] 2. Knowledge Dataset Preprocessing: Regular expressions are used to remove special symbols and irrelevant information from the log data, such as redundant punctuation and formatting marks, ensuring data quality and consistency and guaranteeing more accurate vector matching. The built-in tokenizer of the ChatGLM3-6B-32K model is used to segment the cleaned log data, converting these tokens into numerical values ​​that the model can understand. LangChain is applied to convert the processed documents into vector form, and then a vector index library is defined. This specification uses Milvus as the vector index library to store the vectorized data and build an index for the stored vector data to optimize subsequent retrieval and matching performance. LangChain is a programming framework that helps use large language models (LLMs) in applications.

[0101] 3. User Question Matching Knowledge Base: The user's text input is preprocessed and vectorized using the same method as in step 2. Based on the requirements of the Milvus vector database, the vector representation of the user question is constructed into a suitable query format to fit the used vector database. Using the query function of the vector database, the vector representation of the user question is compared with the log data vectors stored in the database. Based on the similarity calculation method provided by Milvus, the log entry that best matches the user question is identified.

[0102] 4. Construct a Prompt and process user issues: Extract key information from the log entries that best match the user issue retrieved from the database and integrate this information into the selected Prompt template. Optimize the Prompt, which combines user issues and log information, using Langchain, including adjusting statement structure and ensuring information consistency.

[0103] 5. Obtain accurate answers through the Prompt template: Load the ChatGLM3-6B-32K model trained on a large-scale network threat trapping related text dataset to process and respond to relevant question queries input by the user. The model will generate answers based on the Prompt content constructed in step 4, helping users quickly retrieve and understand key information in threat trapping logs.

[0104] In some alternative implementations, embodiments of this application can achieve the above process through a threat trapping log data collection module based on honeypot technology and a threat trapping log analysis module based on large language model technology, specifically:

[0105] Threat trapping log data collection module:

[0106] The threat trapping log data collection module uses a typical honeypot as an example. Leveraging the honeypot's host virtualization and network topology virtualization capabilities, it statically or dynamically assigns operating system characteristics and service types to virtual nodes, monitoring and recording all relevant events and interactions. The recorded content includes service status logs and packet logs. Finally, the generated logs are sent to the threat trapping log analysis module based on large language model technology. For example, Figure 2 The diagram shown is an architectural schematic of the threat trapping log data collection module S1, which specifically includes:

[0107] S11: Deploy virtual hosts or virtual topologies on the honeypot host's network interface card (NIC) and set the NIC to obfuscated mode to monitor all traffic within the local area network.

[0108] S12: Honeypot simulated router, which generates arbitrary network topologies according to research needs.

[0109] S13: Unknown network requests for virtual hosts are handled by the central packet distributor.

[0110] S14: The central packet distributor compares the configuration database and distributes the data to the corresponding protocol processing unit according to the protocol type;

[0111] S15, S16, S17: The protocol processing unit executes the default response action according to the configuration file.

[0112] S18: The protocol processing unit responds to the port service specified in the configuration file.

[0113] S19: Log the response and interaction actions of threat trapping.

[0114] Threat trapping log analysis module based on large language model technology:

[0115] The threat trapping log analysis module based on large language modeling technology mainly includes log data slicing and processing, knowledge dataset preprocessing, user question matching with a knowledge base, prompt construction and question processing, and obtaining answers through prompt templates. This module applies large language modeling technology to threat trapping log analysis, aiming to improve the accuracy, real-time processing efficiency, data processing capabilities, flexibility, and adaptability of log analysis tools. For example... Figure 3 The diagram shown is an architectural principle diagram of the threat trapping log analysis module S2 based on large language model technology, which specifically includes:

[0116] S21: Use a honeypot to obtain log data, write a Python script to slice the log file at fixed line intervals, and export the sliced ​​log data into a format suitable for processing.

[0117] S22: Log data preprocessing is performed using regular expressions and the model's built-in tokenizer. The processed documents are then converted into vector form using langchain. The milvus vector index library is used to store the vectorized data and build an index for it.

[0118] S23: Preprocess and vectorize the user text. Using the query function of the vector database, compare the vector representation of the user's question with the log data vectors stored in the database, and identify the log entry that best matches the user's question based on the similarity calculation method provided by Milvus.

[0119] S24: Extract key information from the log entries that best match the user's question retrieved from the database, and integrate this information into the selected Prompt template. Langchain is used to further optimize the Prompt, which combines the user's question and log information, ensuring information consistency.

[0120] S25: Load the ChatGLM3-6B-32K pre-trained model and process specific types of queries. The model will generate relevant answers to threat trapping logs based on the Prompt content.

[0121] This application utilizes a threat decoy system to monitor malicious activities and log them to a database or file. It leverages the powerful contextual understanding and pattern recognition capabilities of large language modeling technology to more accurately distinguish between normal behavior and malicious activity, thereby reducing false positives and false negatives. Furthermore, it utilizes the efficient processing capabilities of large language modeling technology to optimize the threat decoy log data. Threat decoy logs from different sources, formats, and contents are uniformly vectorized and standardized to reduce data differences and complexity. Simultaneously, it utilizes the text analysis and evaluation capabilities of the ChatGLM3-6B-32K long text model to more effectively handle the context of long text data. By storing the vectorized dataset in a vector database, it provides rapid response to real-time query needs, accelerating the log data analysis process.

[0122] Existing threat trapping log analysis technologies suffer from high false negative and high false positive rates, and lack the ability to quickly process large-scale data and perform efficient real-time queries. This makes them difficult to adapt to increasingly complex network environments and ever-growing log data volumes. The embodiments of this application can bring at least the following benefits:

[0123] 1. This application, through the deep text analysis and understanding capabilities of a large language model, achieves greater accuracy in identifying novel or complex attack patterns compared to traditional methods based on static rules and attack signatures. This helps to more clearly distinguish between normal behavior and malicious activity, thereby reducing false positives and false negatives.

[0124] 2. This application utilizes the high-speed processing capabilities of large language models and vectorization technology to quickly analyze log data, significantly improve the query efficiency and response speed for large amounts of real-time data, and enhance the effectiveness of real-time monitoring and threat detection.

[0125] 3. This application effectively processes and analyzes large log datasets by optimizing data storage and indexing methods, thus overcoming the limitations of traditional tools in data processing.

[0126] 4. This application combines the dynamic learning, adaptability, and natural language processing capabilities of a large language model to continuously update the recognition patterns and algorithms. This enables it to adapt to new security threats and changing network environments, while providing intuitive user interaction.

[0127] In summary, existing threat trapping log analysis techniques have limited capabilities. Log analysis tools typically rely on static rules or known attack signatures for analysis. These rules are insufficient to accurately distinguish between normal behavior and malicious activity, and they cannot identify new or complex attack patterns, leading to high false negative and false positive rates. Traditional log analysis tools are primarily suited for historical data analysis, not real-time data processing. They lack efficient indexing and search mechanisms and often encounter performance bottlenecks when handling large log datasets.

[0128] This application provides a more comprehensive and accurate threat trapping log analysis solution. It applies large language modeling technology to threat trapping log analysis, leveraging the advanced language understanding capabilities of large language models for deep text analysis and comprehension. This effectively reduces false positives and false negatives common in static rule-based methods, while more accurately distinguishing between normal behavior and malicious activity. By combining large language models and vectorization technology, it improves query efficiency and response speed for large amounts of real-time generated log data, and provides users with a more intuitive and flexible interaction method. Through vectorized data storage and indexing methods, it can effectively process and analyze large log datasets, overcoming the limitations of traditional tools in big data processing capabilities. Utilizing the dynamic learning and adaptation capabilities of large language models, its recognition patterns and algorithms can be continuously updated to cope with rapidly evolving network threats, enhancing the system's flexibility and adaptability.

[0129] Please see Figure 4 This application also provides a honeypot log analysis system 500 based on a large language model, which can implement the above-mentioned honeypot log analysis method based on a large language model. The system includes:

[0130] The first module 510 is used to obtain log data of the target system based on honeypot technology;

[0131] The second module 520 is used to clean and process log data and build a vector index library based on the cleaned log data.

[0132] The third module 530 is used to obtain target log entries from the vector index library in response to the input text of the target object;

[0133] The fourth module, 540, is used to generate prompt text based on target log entries using a preset template.

[0134] The fifth module, 550, is used to input the prompt text into a pre-trained language model to generate a response and obtain the answer content.

[0135] In some embodiments, the system may further include:

[0136] The sixth module is used to clean the input text and then vectorize the cleaned input text.

[0137] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned honeypot log analysis method based on a large language model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0139] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Please see Figure 5 , Figure 5 The hardware structure of an electronic device 600 according to another embodiment is illustrated. The electronic device includes:

[0141] The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0142] The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the honeypot log analysis method based on a large language model according to the embodiments of this application.

[0143] The input / output interface 603 is used to implement information input and output;

[0144] The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0145] Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604);

[0146] The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0147] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described honeypot log analysis method based on a large language model.

[0148] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0149] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0150] This application provides a honeypot log analysis method, system, and medium based on a large language model. It acquires log data from a target system using honeypot technology; cleans the log data and constructs a vector index library based on the cleaned log data; in response to input text from the target object, it matches and obtains target log entries from the vector index library; based on the target log entries, it organizes them using a preset template to obtain prompt text; and it inputs the prompt text into a pre-trained language model to generate a response. This application first utilizes honeypot technology to accurately collect relevant log data, and then leverages the powerful language understanding and generation capabilities, flexible text conversion and generation capabilities, and accurate text analysis and evaluation capabilities of large language model technology to overcome the shortcomings of existing log analysis tools in terms of accuracy, real-time processing efficiency, data processing capabilities, and flexibility. This application can accurately perform log analysis.

[0151] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0152] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0155] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0156] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between the devices or units may be through some interfaces, and the indirect coupling or communication connection may be electrical, mechanical, or other forms.

[0158] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A honeypot log analysis method based on a large language model, characterized in that, The method includes: Obtaining log data from the target system using honeypot technology; The log data is cleaned, and a vector index library is built based on the cleaned log data. In response to the input text of the target object, the target log entry is obtained by matching from the vector index library; Based on the target log entries, prompt text is generated by organizing them using a preset template; The prompt text is input into a pre-trained language model to generate a response, and the response content is obtained. The acquisition of log data from the target system based on honeypot technology includes: The host virtualization and network topology virtualization functions of the honeypot are used to specify operating system characteristics and service types for virtual nodes. Based on the specified operating system characteristics and service types, all relevant events and interactions are monitored and recorded, and log data is compiled. The log data includes service status logs and packet logs. The step of obtaining target log entries from the vector index library in response to the input text of the target object includes: Based on the query function of the vector index library, the similarity between each log entry in the vector index library and the input text is obtained by traversing the library using a preset similarity calculation method. The log entry with the highest similarity to the input text in the vector index is taken as the target log entry; The step of obtaining the prompt text based on the target log entry and using a preset template includes: Extract key information from the target log entries, and then integrate the key information into a preset Prompt template; The Prompt template is optimized using a large-scale language model programming framework to obtain the prompt text; the optimization process includes sentence structure adjustment and information coherence processing.

2. The method according to claim 1, characterized in that, The cleaning process for the log data includes: Use regular expressions to remove special symbols and irrelevant information from the log data.

3. The method according to claim 1, characterized in that, The construction of a vector index library based on the cleaned log data includes: The cleaned log data is segmented into words, and the log document is obtained based on the results of the segmentation; the log document includes several words. Convert the log document into vector form; Log documents in vector form are stored in a pre-defined database, and an index is built for the vector data in the database to obtain the vector index library.

4. The method according to claim 1, characterized in that, The method further includes: The input text is cleaned, and the cleaned input text is then vectorized.

5. A honeypot log analysis system based on a large language model, characterized in that, The system applied to the honeypot log analysis method based on a large language model as described in claim 1 includes: The first module is used to obtain log data from the target system based on honeypot technology; The second module is used to clean the log data and build a vector index library based on the cleaned log data. The third module is used to obtain target log entries from the vector index library in response to the input text of the target object. The fourth module is used to generate prompt text based on the target log entries using a preset template; The fifth module is used to input the prompt text into a pre-trained language model to generate a response and obtain the response content.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images