Internal network threat intelligent analysis and prediction method and system based on large language model

Abstract

The invention provides an internal network threat intelligent analysis and prediction method and system based on a large language model, and relates to the technical field of network technology and security. According to the method, the long-range dependency processing capacity of the LLMs is utilized, global and long-range dependency analysis of the user behavior sequence is achieved, and the limitation of a traditional fixed session unit is broken through. Rich semantic features and context information are extracted from original log data through the deep semantic analysis ability of LLMs, and the feature expression ability is greatly improved. In combination with the reasoning capability of LLMs and a traditional anomaly detection algorithm, context-based deep semantic analysis is realized, and the recognition capability of complex threats is improved. In the aspects of real-time monitoring and response, the LLMs are introduced to assist in complex event processing and dynamic rule generation, and the intelligent level and efficiency are improved. Aiming at challenges of data management and analysis, an LLMs is utilized to enhance a data management strategy, a storage structure is optimized, and an intelligent data life cycle management and query interface is provided.

Description

Technical Field

[0001] The present invention relates to the fields of network technology and security technology, and in particular to a method and system for intelligent analysis and prediction of internal network threats based on a large language model. Background Art

[0002] As an important research topic in the field of cybersecurity, internal threats have become one of the most dangerous and common security threats faced by various institutions and companies in recent years. Internal threats refer to internal personnel (including current or former employees, contractors or business partners) using their access rights to intentionally or unintentionally conduct destructive activities, thereby adversely affecting the development of an organization. According to the findings of the 2023 Insider Threat Report, 74% of organizations have found an increase in internal attacks, and more than 50% of organizations have suffered internal threats in the past year, which reveals the increasing risks posed by high-risk insiders in remote and hybrid work environments.

[0003] Compared to external threats, insider threats pose a greater challenge because internal actors possess sophisticated system knowledge and can carefully carry out attacks after obtaining privileged access. Therefore, research on insider threat detection becomes crucial. However, identifying insider threats faces several major obstacles: First, the inherent potential atypical behaviors of organizational members further hinder the accurate diagnosis of their behavior. Second, the limited incidence of malicious incidents prevalent in various organizations leads to insufficient observable evidence for inspection, which raises a serious data disproportion problem.

[0004] Early methods for detecting insider threats mainly revolved around rule-based and traditional machine learning approaches, but both methods face certain limitations. Rule-based methods perform poorly when dealing with unknown threats that go beyond predefined rules. Although machine learning techniques have superior performance over rule-based methods, due to the complexity of high dimensionality, sparsity, and heterogeneity, traditional machine learning techniques have difficulty fully utilizing comprehensive user behavior data, and therefore often fail to capture subtle anomalies that reflect insider threats. In recent years, deep learning-based solutions, such as time-series deep learning models such as LSTM and GRU, have made progress in learning from user action sequences of varying lengths. However, these models still have flaws when applied to unbalanced, sparse, and irregular real-world data, especially in capturing long-term dependencies and complex contextual information, which is critical for accurately identifying insider threats.

[0005] In recent years, with the rapid development of artificial intelligence technology, especially the breakthrough progress made by large language models (LLMs) in natural language processing and understanding, new possibilities have been provided for solving many challenges in the field of internal threat detection. LLMs' powerful semantic understanding ability, long-range dependency processing ability, and knowledge reasoning ability have opened up new research directions for improving the accuracy, efficiency, and adaptability of internal threat detection. However, how to effectively combine the advantages of LLMs with traditional security analysis technology to build a comprehensive, intelligent, and efficient internal threat detection system is still a problem to be solved. It is in this context that the present invention proposes an innovative internal threat detection system based on LLMs, which aims to give full play to the advantages of LLMs in semantic understanding, context analysis, knowledge reasoning, etc. through the collaborative work of multiple modules, and combine advanced machine learning algorithms with traditional security analysis methods to comprehensively improve the ability of internal threat detection and provide organizations with more accurate, efficient, and explainable security protection solutions. Summary of the invention

[0006] Purpose of the invention: To propose an internal network threat intelligent analysis and prediction method and system based on a large language model, which is committed to overcoming the limitations of traditional methods in behavior sequence analysis, feature engineering, anomaly detection, real-time monitoring and response, data management and analysis, trend prediction, interpretability and decision support, so as to solve the above-mentioned problems existing in the prior art.

[0007] The present invention proposes an internal network threat intelligent analysis and prediction method based on a large language model, comprising the following steps: Collect and pre-process raw log data in real time through multiple channels, and convert the raw log data into structured feature description text data; Using large language models (LLMs) to perform semantic understanding on the feature description text data to generate a behavior sequence; using a neural network model to model the behavior sequence to generate a predicted behavior sequence for a predetermined period in the future; The complex event processing engine and LLMs are used to dynamically generate rules to analyze the predicted behavior sequence, generate recognition results, and trigger corresponding corresponding measures based on the recognition results.

[0008] In a further embodiment, the raw log data is collected by means of a RESTful API interface, deployment of a lightweight log collection agent, and a JDBC connection.

[0009] In a further embodiment, the collected raw log data is cleaned and standardized, and duplicate and erroneous data are removed using a hash algorithm, regular expressions, and fuzzy matching.

[0010] In a further embodiment, the use of a neural network model to model the behavior sequence to generate a predicted behavior sequence for a predetermined period in the future specifically includes: The transformer architecture is used to model the behavior sequence, and the sequence generation ability of large language models (LLMs) is used to predict possible future behavior sequences. LLMs are introduced as meta-learners to obtain the final threat assessment results.

[0011] In a further embodiment, the predicted behavior sequence is continuously received by monitoring the message queue, and the predicted behavior sequence is analyzed using a complex event processing engine and LLMs dynamically generated rules to form a recognition result; Based on the recognition results, the intelligent multi-level alarm system is immediately activated. LLMs assist in decision-making during this process and dynamically adjust the threat level, dividing the threat level into low, medium, and high levels according to the predetermined thresholds; For high-level threats, an intelligent multi-level alert system immediately sends personalized alerts through multiple channels, triggering automated response measures.

[0012] In a further embodiment, the intelligent multi-level alarm system also performs alarm aggregation and correlation analysis, using the deep semantic analysis capabilities of LLMs to identify potential large-scale attack patterns.

[0013] In addition, the present invention also discloses an internal network threat intelligent analysis and prediction system, which includes a data collection and preprocessing module, a behavior analysis module, a real-time monitoring and alarm module, a user interface and reporting module, and a threat data storage module.

[0014] The data collection and preprocessing module collects and preprocesses raw log data in real time through multiple channels, and converts the raw log data into structured feature description text data; The behavior analysis module uses large language models (LLMs) to perform semantic understanding on the feature description text data to generate a behavior sequence; a neural network model is used to model the behavior sequence to generate a predicted behavior sequence for a predetermined period in the future; The real-time monitoring and alarm module uses the complex event processing engine and LLMs to dynamically generate rules to analyze the predicted behavior sequence, generate recognition results, and trigger corresponding corresponding measures based on the recognition results.

[0015] The user interface and reporting module provides an interactive platform for security managers to display threat situations in the form of interactive charts and heat maps.

[0016] The threat data storage module is used to store and transfer the original log data, feature description text data, predicted behavior sequence, and recognition results.

[0017] In addition, the present invention also proposes a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on an electronic device, the electronic device executes the above-mentioned internal network threat intelligent analysis and prediction method based on a large language model.

[0018] Beneficial effects: The present invention utilizes the long-range dependency processing capability of LLMs to realize global and long-range dependency analysis of user behavior sequences, breaking through the limitations of traditional fixed session units. Through the deep semantic analysis capability of LLMs, rich semantic features and contextual information are extracted from the original log data, greatly improving the expressiveness of features. Combining the reasoning capability of LLMs and traditional anomaly detection algorithms, context-based deep semantic analysis is realized to improve the ability to identify complex threats. In terms of real-time monitoring and response, the present invention improves the level of intelligence and efficiency by introducing LLMs to assist in complex event processing and dynamic rule generation. In response to the challenges of data management and analysis, the present invention utilizes LLMs to enhance data management strategies, optimize storage structures, and provide intelligent data lifecycle management and query interfaces. Through the powerful reasoning and generation capabilities of LLMs, more advanced multimodal learning is achieved, which improves the foresight and adaptability of the system. Finally, the present invention makes full use of the natural language generation capability of LLMs to provide detailed and understandable threat analysis reports and decision-making recommendations, significantly improving the interpretability and decision support capabilities of the system. By solving these technical problems, the present invention aims to build a comprehensive, intelligent and efficient internal threat detection system, significantly improve the ability of organizations to deal with internal threats, and provide innovative solutions for the field of network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an architecture diagram of an internal network threat intelligent analysis and prediction system based on a large language model in an embodiment of the present invention.

[0020] Figure 2 It is a flow chart of an internal network threat intelligent analysis and prediction method based on a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0022] The applicant has found that the existing unsupervised internal threat detection method based on adversarial autoencoder splits the behavior sequence of the original log data of user behavior into a set of subsequences based on sessions, and extracts numerical features, which are divided into event frequency features and time features; the time features and event frequency features are respectively input into different adversarial autoencoders for modeling training to obtain the trained normal behavior model; the user log to be detected is extracted and input into the trained normal behavior model, the sum of the reconstruction errors of the numerical features is calculated and used as the threat score of the user behavior in the session to determine whether there is a threat in the session. Although this method has shown certain effects in specific scenarios, its limitations are increasingly prominent in the face of an increasingly complex internal threat environment. The primary problem lies in its behavior sequence processing strategy: simply splitting user behavior into a set of subsequences of fixed session units, this method inevitably leads to the loss of cross-session context information, which is particularly disadvantageous when dealing with long-term and progressive internal threat behaviors. Secondly, the feature engineering of this method is limited to event frequency and time features, and cannot fully capture the semantic connotation and logical association of user behavior. More importantly, using independent adversarial autoencoders to process time features and event frequency features separately makes it difficult to simulate the complex interaction between these two types of features, which is a significant flaw in multi-dimensional threat identification. Finally, relying solely on reconstruction errors as a threat judgment criterion is insufficient in its identification ability when faced with highly concealed and diverse threat behaviors.

[0023] In contrast, the internal threat detection system based on large language models (LLMs) proposed in this embodiment has achieved breakthrough innovations in multiple dimensions and effectively overcome the above limitations. First, we make full use of the powerful semantic understanding and long-range dependency processing capabilities of LLM to realize global and long-range dependency analysis of user behavior sequences, completely breaking through the limitations of traditional fixed session units. In terms of feature engineering, our system uses the deep semantic analysis capabilities of LLM to extract rich semantic features and contextual information from the original log data, far exceeding the limitations of traditional methods that only focus on surface statistical features. LLMs can understand and parse complex log semantics and automatically generate high-dimensional feature representations, including text representations and semantic embeddings of behavior patterns, enabling the system to identify more subtle and complex abnormal behavior patterns. In the threat detection stage, our system combines the reasoning capabilities of LLMs with traditional anomaly detection algorithms to achieve context-based deep semantic analysis. LLMs can understand complex behavioral contexts and identify hidden threats that are difficult to detect by simply relying on statistical features. For example, the system can understand the different meanings of a specific operation in different contexts, so as to more accurately determine whether it poses a threat. This multi-level detection mechanism is able to identify complex and hidden threats that are difficult to capture with traditional methods. In addition, our system innovatively utilizes the generation capability of LLMs to achieve dynamic threat pattern discovery. LLMs can generate potential new threat scenarios based on known threat patterns and the current security situation, greatly enhancing the system's foresight and adaptability. This feature is particularly important in dealing with rapidly evolving threat environments. Finally, our system not only provides highly accurate threat detection results, but also generates clear and easy-to-understand explanation reports. LLMs can convert complex technical analysis into natural language narratives, allowing security analysts to better understand and verify detection results, greatly enhancing the credibility and practicality of the system.

[0024] The innovative insider threat detection system based on large language models (LLMs) proposed in this paper is shown in Figure 1 , which consists of multiple modules working together, aims to comprehensively improve the accuracy, efficiency and adaptability of internal threat detection. The core architecture of the system includes data collection and preprocessing module, behavior analysis module, real-time monitoring and alarm module, user interface and reporting module, threat data storage module, historical data analysis and trend prediction module. The following are the responsibilities and implementation principles of each module: The data collection and preprocessing module is the foundation of the internal threat detection system. Its workflow can be further refined into five main steps: data collection, data cleaning, data parsing, data standardization, and feature engineering. In the data collection phase, the module uses a variety of technical means to obtain raw log data in real time. Specifically, it interacts with various application systems through the RESTful API interface, deploys lightweight log collection agents (such as Filebeat or Fluentd) to capture real-time logs on key servers, and directly extracts structured data from the database through the JDBC standard database connection protocol. The purpose of this step is to ensure comprehensive and timely acquisition of all data sources that may be related to internal threats, laying the foundation for subsequent analysis. The data cleaning phase aims to improve data quality and ensure the accuracy of subsequent analysis. A variety of technologies are used here, including the use of hash algorithms to quickly identify and delete duplicate records, the application of regular expressions and fuzzy matching algorithms to detect and correct format errors, and the use of statistical methods and domain knowledge to fill missing values. In addition, large language models (LLMs) are introduced to assist in identifying semantically abnormal or inconsistent data, improving the intelligence and accuracy of cleaning. The data parsing step focuses on extracting valuable information from semi-structured or unstructured log data. Here, regular expressions are used for pattern matching, natural language processing techniques such as named entity recognition (NER) are used to identify key information entities, and syntactic analysis techniques are applied to understand complex log sentence structures. At the same time, the introduction of pre-trained large language models (LLMs) for context-aware information extraction can better understand and parse complex log semantics. Data standardization and normalization processing aims to unify the format and measurement units of data from different sources to facilitate subsequent integrated analysis. This step includes converting timestamps to a unified UTC format, standardizing IP addresses, unifying user identifiers, and normalizing numerical data of different scales (such as network traffic) to the same range. Here, the powerful semantic understanding ability of large language models (LLMs) can be used to automatically identify data representations in different formats and generate conversion rules to improve the flexibility and adaptability of the standardization process. Feature engineering is a key step in improving data expression capabilities. Here, a variety of techniques are used to derive new features from raw data, including calculating statistics (such as behavior frequency and duration), extracting time series features (such as Fourier transform coefficients), and constructing graph structure features (such as social network centrality). In addition, the text generation capability of large language models (LLMs) can be used to transform unstructured text logs into structured feature descriptions.

[0025] The behavior analysis module is the core component of the internal threat detection system. It is responsible for in-depth analysis and anomaly detection of pre-processed employee behavior data. The workflow of this module can be divided into five main steps: semantic understanding, anomaly detection, time series analysis, rule matching, and comprehensive evaluation. Each step adopts advanced technical methods and makes full use of the capabilities of large language models (LLMs). The semantic understanding step focuses on in-depth analysis of text data. This step uses pre-trained large language models (such as GPT-3 or its variants) to perform semantic understanding and context analysis on text data such as employees' email content, chat records, document operations, etc. The module uses few-shot learning technology to guide large language models (LLMs) to understand specific threat contexts by providing a small number of examples related to internal threats. At the same time, the zero-shot learning ability of the model is used to dynamically generate semantic features related to potential threats. The purpose of this step is to capture potential threat intentions, sensitive information leakage, or abnormal communication patterns, and provide semantic insights for subsequent analysis. The anomaly detection step focuses on identifying anomalies in numerical and categorical data. Here, a variety of techniques are used in combination: the Isolation Forest algorithm is used to quickly identify global anomalies; the Local Outlier Factor method is used to detect local density anomalies; and the autoencoder neural network is applied to capture abnormal patterns in high-dimensional feature space. In addition, large language models (LLMs) are introduced to assist in the generation of dynamic anomaly detection rules, and complex anomaly patterns are defined by natural language descriptions to improve the flexibility and interpretability of anomaly detection. This step aims to discover anomalies in employee behavior, such as abnormal file access frequency, system login at unusual times, etc. The time series analysis step focuses on capturing the time series patterns of employee behavior. The transformer architecture is used here to model behavior sequences and identify long-term and short-term changes in behavior patterns. At the same time, the attention mechanism is introduced to focus on key time points that may be related to threats. In addition, the sequence generation ability of large language models (LLMs) is used to predict possible future behavior sequences and compare them with the actual observed sequences to discover potential abnormal behavior trajectories. The purpose of this step is to identify abnormal changes in employee behavior patterns, such as sudden changes in work habits or gradually escalating suspicious activities. The rule matching step implements detection methods based on expert knowledge. A series of threat indicators and behavior patterns are predefined here, such as frequent downloads of sensitive files, multiple failed attempts to escalate privileges, etc. By leveraging the natural language understanding capabilities of large language models (LLMs), complex rule descriptions can be converted into executable logic, improving the efficiency of rule formulation and updating. At the same time, the model's text generation capabilities are used to automatically generate new rule recommendations based on historical threat events. The purpose of this step is to capture known threat patterns and provide an explainable baseline for analysis. Finally, the comprehensive evaluation step integrates all the aforementioned analysis results to derive the final threat score and classification.Ensemble learning techniques, such as random forests or gradient boosting trees, are used here to synthesize the outputs of multiple models. At the same time, large language models (LLMs) are introduced as meta-learners to integrate and interpret the outputs of each model at the semantic level. The model can also generate a threat report in natural language, detailing the reasons and evidence chain for the threat assessment. The purpose of this step is to provide a comprehensive, reliable, and easy-to-understand threat assessment result.

[0026] The real-time monitoring and alert module is a key component of the internal threat detection system, responsible for continuously assessing threat conditions and responding to potential risks in a timely manner. The workflow of this module can be divided into six main steps: real-time data reception, complex event processing, threat level assessment, multi-channel alert sending, automatic response execution, and alert aggregation and correlation analysis. Each step uses advanced technical methods and combines the capabilities of large language models (LLMs) to optimize performance. The real-time data reception step continuously receives results from the behavior analysis module through a high-performance message queue system (such as Apache Kafka or RabbitMQ). This step adopts a stream processing architecture to ensure that a large amount of real-time data can be processed in a low-latency and high-throughput manner. At the same time, the natural language processing capabilities of large language models (LLMs) are used to parse and standardize the received unstructured or semi-structured analysis results in real time, providing a unified data format for subsequent processing. The complex event processing step uses a high-performance complex event processing (CEP) engine, such as Esper or Flink CEP, to identify complex event patterns in high-speed data streams. A series of event patterns are defined here, such as suspicious operations in a specific sequence, abnormal behaviors of multiple related systems, etc. Large language models (LLMs) are introduced to dynamically generate and optimize event pattern rules, describe complex event patterns through natural language, and improve the flexibility and adaptability of rule definition. The CEP engine can detect these patterns in real time and promptly discover potential complex threat scenarios. The threat level assessment step implements an intelligent multi-level alert system. Based on the preset thresholds and rules, the threats are initially divided into three levels: low, medium, and high. In this process, large language models (LLMs) are introduced to assist decision-making and dynamically adjust the threat level through in-depth understanding of the threat context. The model can consider multi-dimensional information such as historical data, current environment, and global threat intelligence to provide a more accurate threat level assessment. In addition, the model can also generate a detailed threat assessment report to explain the basis for the level division. The multi-channel alert sending step sends alerts to relevant personnel through different channels according to the threat level. For high-level threats, the system will immediately notify the security team through multiple methods (such as email, SMS, dedicated application push, voice call, etc.). Large language models (LLMs) are used here to generate personalized alert content for different recipients and threat types to ensure that the alert information is clear, accurate, and easy to understand. The model can also optimize the alert sending strategy based on historical response data, such as selecting the most effective notification channel and time. The automatic response execution step integrates a series of predefined security measures, such as temporary permission restrictions on suspicious accounts, network isolation, and mandatory multi-factor authentication. This step uses a rule-based automated response system combined with a machine learning model to dynamically adjust the response strategy. Large language models (LLMs) are introduced to interpret and execute complex response strategies, which can understand security policies described in natural language and convert them into specific execution steps.At the same time, the model can generate customized response suggestions based on real-time situations, helping security teams make quick decisions. The alert aggregation and correlation analysis step is designed to reduce false positives and identify potential large-scale attack patterns. This step uses advanced clustering algorithms and graph analysis techniques to intelligently aggregate and correlate alerts. The introduction of large language models (LLMs) for deep semantic analysis can discover seemingly unrelated but semantically related alerts, identify complex attack chains and potential APT (advanced persistent threat) activities. The model can also generate correlation analysis reports that detail the discovered patterns and potential attack strategies, providing security teams with a macro threat intelligence perspective.

[0027] The user interface and reporting module is a key interactive component of the internal threat detection system, providing security managers with a comprehensive, intuitive and highly interactive operating platform. The workflow of this module can be divided into six main steps: interface construction, data visualization, real-time update, report generation, event management, and intelligent decision-making assistance. Each step adopts advanced technical methods and cleverly combines the capabilities of large language models (LLMs) to enhance functionality and user experience. The interface construction step uses modern front-end frameworks (such as React or Vue.js) to create a responsive and modular user interface. The purpose of this step is to provide an intuitive, easy-to-use operating platform that adapts to different devices. Through component-based design, a highly customizable dashboard layout is achieved. Combined with the natural language processing capabilities of large language models (LLMs), the module also supports natural language-based interface customization instructions, allowing users to dynamically adjust the interface layout and functional components through simple text descriptions, greatly improving the flexibility and personalization of the interface. The data visualization step integrates the advanced data visualization library ECharts to generate rich interactive charts and heat maps to intuitively display the threat situation. The purpose of this step is to transform complex threat data into a visual representation that is easy to understand and analyze. Large language models (LLMs) are introduced to intelligently interpret chart content. The model can generate natural language chart descriptions and insights to help security personnel quickly understand the meaning behind the data. In addition, the model can automatically recommend the most suitable visualization type and configuration based on data characteristics and analysis purposes to improve the effectiveness and efficiency of data display. The real-time update step uses WebSocket technology to ensure the real-time and consistency of front-end data. The purpose of this step is to provide security personnel with the latest threat intelligence and system status. Combined with the capabilities of large language models (LLMs), the module can intelligently filter and prioritize the most important real-time update information, and generate concise natural language summaries to help users quickly grasp key information. The model can also analyze real-time data streams, predict possible trend changes, and provide users with forward-looking insights. The report generation step provides a flexible report generation system that supports custom report templates and can generate detailed reports in PDF or Excel format on demand. The purpose of this step is to meet reporting requirements at different levels and scenarios. Large language models (LLMs) are introduced to enhance the report generation function. The model can automatically generate a report outline based on the user's natural language description, intelligently select relevant data and charts, and generate professional report content. In addition, the model can automatically adjust the language style and technical depth of the report according to the report audience and purpose to ensure the pertinence and readability of the report. The incident management step integrates comprehensive incident tracking and processing functions, allowing security personnel to record investigation progress, add comments, and update incident status. The purpose of this step is to provide a centralized platform to manage and coordinate the response process of internal threat incidents.Combined with the capabilities of large language models (LLMs), the module can automatically generate event summaries and timelines, extract key information points, and provide processing suggestions based on historical cases. The model can also analyze event descriptions and annotations, automatically tag keywords and entities, and facilitate subsequent retrieval and association analysis. The intelligent decision-making step is an innovative feature of this module, which makes full use of the reasoning ability of large language models (LLMs) to assist security decisions. The purpose of this step is to provide intelligent decision-making support for security personnel. The model can generate detailed risk assessment reports and response recommendations based on the current threat situation, historical event data, and global threat intelligence. Through the natural language dialogue interface, security personnel can interact deeply with the model, explore complex security scenarios, and obtain personalized analytical insights. The model can also simulate different response strategies, predict possible outcomes, and help security teams develop optimal response plans.

[0028] The threat data storage module is the core data management component of the internal threat detection system, responsible for storing and managing a large amount of threat-related data securely and efficiently. The workflow of this module can be divided into seven main steps: data reception and preprocessing, distributed storage, data encryption, time series data management, data partitioning and index optimization, data lifecycle management, and API interface provision. Each step adopts advanced technical methods and cleverly combines the capabilities of large language models to enhance functionality and performance. The data reception and preprocessing step receives real-time data streams from other modules through a high-performance message queue system (such as Apache Kafka). The purpose of this step is to ensure reliable reception and preliminary processing of data. Here, a large language model is introduced for real-time data classification and labeling. The model can understand complex data contexts, automatically add semantic tags to incoming data, identify potential sensitive information, and perform preliminary anomaly detection. This intelligent preprocessing not only improves the efficiency of subsequent storage and query, but also provides richer contextual information for data analysis. The distributed storage step adopts advanced distributed storage architectures (such as Apache Hadoop HDFS or Ceph) to ensure high availability and scalability of the system. The purpose of this step is to build a robust storage infrastructure that can cope with large-scale data storage needs. Combined with the data understanding capabilities of the large language model, the system can intelligently determine the storage strategy for the data, such as dynamically adjusting the replication factor and storage location based on the importance, access frequency, and relevance of the data, and optimizing the efficiency of storage resource usage. The data encryption step uses advanced encryption technologies, such as transparent data encryption (TDE), to protect the security of static data. The purpose of this step is to ensure that the data remains confidential even when the storage medium is physically accessed. The large language model is introduced to enhance the intelligence of the encryption strategy. The model can analyze the data content and context, automatically identify sensitive information that requires additional protection, and recommend appropriate encryption algorithms and key management strategies for different types of data. The time series data management step uses a dedicated time series database (such as InfluxDB or TimescaleDB) to efficiently store and query log data and time series threat indicators. The purpose of this step is to optimize the management and analysis of time dimension data. Combined with the capabilities of the large language model, the system can intelligently design the time series data schema, automatically extract seasonal patterns and long-term trends in the time series, and provide support for anomaly detection and predictive analysis. The data partitioning and index optimization step implements complex partitioning strategies and dynamic indexing mechanisms to support fast historical data retrieval. The purpose of this step is to balance storage efficiency and query performance. By using a large language model to analyze query patterns and data access characteristics, the system can automatically design and optimize partition schemas and index structures, predict hot data, and preheat data in advance, greatly improving query response speed.The data lifecycle management step integrates automated data archiving and deletion functions to ensure that the system meets compliance requirements for data retention. The purpose of this step is to balance data value and storage costs. The introduction of a large language model to enhance this process can analyze the historical importance, legal requirements, and potential value of data, and formulate personalized lifecycle strategies for each type of data. The model can also predict future data access patterns and guide the hot and cold tiered storage of data. The API interface provision step implements a set of flexible and secure APIs to support other modules for efficient data reading and writing operations. The purpose of this step is to ensure data accessibility and system integrability. Combined with the capabilities of the large language model, the system can provide a natural language query interface that allows users to construct complex queries using near-natural language. The model can understand the query intent and automatically convert it into optimized database operations, greatly reducing the difficulty of using the API.

[0029] The historical data analysis and trend prediction module is an advanced analysis component of the internal threat detection system, responsible for deep mining and forward-looking analysis of long-term accumulated threat data. The workflow of this module can be divided into five main steps: data preparation and feature engineering, time series analysis, deep learning modeling, large language model-assisted analysis, and comprehensive report generation. Each step uses advanced analytical techniques and innovatively combines the capabilities of large language models to enhance analytical depth and insight. The data preparation and feature engineering step extracts and integrates historical data from the threat data storage module and performs advanced feature engineering. The purpose of this step is to build a high-quality analytical data set. Traditional methods use automatic feature extraction and feature selection algorithms, such as principal component analysis (PCA) and recursive feature elimination (RFE), to extract key indicators from raw data. Combined with the capabilities of large language models, the system can perform more intelligent feature engineering. The model automatically generates complex feature descriptions, such as text representations of employee behavior patterns or semantic embeddings of threat events, by understanding the semantics and context of the data. This approach not only captures explicit features, but also discovers latent, unstructured features, greatly enhancing the depth and breadth of subsequent analysis. The time series analysis step uses advanced time series models such as ARIMA, SARIMA, or Facebook's Prophet to identify long-term trends and seasonal patterns. The purpose of this step is to understand how threat indicators change over time. The introduction of a large language model to enhance time series analysis can provide additional contextual information for time series by analyzing text data related to the time series (such as security announcements, incident reports). This combination of quantitative and qualitative analysis can more accurately explain the fluctuations of time series and provide more insightful trend forecasts. The deep learning modeling step uses complex neural network architectures such as long short-term memory networks (LSTM), gated recurrent units (GRU), or temporal convolutional networks (TCN) to capture nonlinear and long-term dependent patterns. The purpose of this step is to build predictive models that can handle complex time series patterns. Combined with the capabilities of a large language model, the system can achieve more advanced multimodal learning. For example, combining time series data with related text descriptions (such as threat reports) to train a hybrid model that can handle both numerical and text inputs, resulting in more comprehensive and insightful forecasts. The large language model-assisted analysis step fully utilizes the powerful reasoning capabilities of GPT-like models to perform advanced semantic analysis and knowledge inference. The purpose of this step is to provide deep insights based on context. The model can analyze large amounts of unstructured data, such as security logs, incident reports, and external threat intelligence, to identify potential associations and patterns. By understanding historical events, the model is able to infer possible attack paths and motivations, predict possible threat scenarios in the future, and provide strategic defense recommendations. Finally, the comprehensive report generation step integrates all analysis results to generate a comprehensive and insightful trend analysis report.This step fully utilizes the natural language generation capabilities of the large language model to transform complex analysis results into structured, easy-to-read reports. The model can automatically identify key findings, generate an executive summary, and provide detailed data support. The report not only includes an analysis of historical trends, but also includes predictions and response recommendations for future threat situations.

[0030] Figure 2 The flowchart of the method for intelligent analysis and prediction of internal threats based on a large language model of the present invention is shown in the figure. As shown in the figure, the present invention provides a method for intelligent analysis and prediction of internal threats based on a large language model, including the following steps: Step 1. The system first collects employee behavior log data in real time through multiple channels, including using RESTful API interfaces, deploying lightweight log collection agents, and JDBC connections. The collected raw data is then cleaned and standardized, using hash algorithms, regular expressions, and fuzzy matching techniques to remove duplicate and erroneous data. In this process, LLMs are used to identify semantic anomalies and inconsistent data, significantly improving data quality. Next, the system applies natural language processing technology and the contextual understanding capabilities of LLMs to extract key information from semi-structured or unstructured logs. Finally, feature engineering is performed to convert raw logs into structured feature descriptions using statistical methods and the text generation capabilities of LLMs. This step lays a solid data foundation for subsequent behavioral analysis.

[0031] Step 2: Based on the high-quality data collected and preprocessed in step 1, the behavior analysis module begins to deeply analyze employee behavior. First, the pre-trained LLMs are used to perform semantic understanding and contextual analysis on text data to capture potential threat intentions and abnormal communication patterns. At the same time, the system uses a variety of machine learning algorithms, such as isolation forests and local anomaly factors, combined with dynamic rules generated by LLMs to identify anomalies in numerical and categorical data. Next, the transformer architecture is used to model the behavior sequence, and the sequence generation ability of LLMs is used to predict possible future behavior sequences. Finally, the outputs of multiple models are synthesized through ensemble learning technology, and LLMs are introduced as meta-learners to obtain the final threat assessment results. The output of this step flows directly into the real-time monitoring and alarm module to trigger the subsequent response mechanism.

[0032] Step 3: The real-time monitoring and alarm module continuously receives the results from the behavior analysis module in step 2 through a high-performance message queue system. Using the complex event processing (CEP) engine and the rules dynamically generated by LLMs, the system is able to identify complex threat patterns in high-speed data streams. Based on the identification results, the intelligent multi-level alarm system is immediately activated, and LLMs assist in decision-making and dynamically adjust the threat level in this process. For high-level threats, the system immediately sends personalized alerts through multiple channels and triggers automatic response measures. In addition, the system also performs alarm aggregation and correlation analysis, using the deep semantic analysis capabilities of LLMs to identify potential large-scale attack patterns. The output of this step not only directly affects the response actions of the security team, but also provides real-time updated information for the user interface.

[0033] Step 4: The user interface module provides an intuitive and interactive operating platform for security analysts. It displays the alarm and threat status from the real-time monitoring module in step 3 in real time, and allows analysts to drill down to historical data and trend forecast results. The interface is built with a modern front-end framework and supports natural language interaction driven by LLMs, making complex data queries simple and intuitive. The system also integrates a powerful report generation function, which uses LLMs to automatically generate structured and easy-to-read threat analysis reports. In addition, the intelligent decision-making support function enables analysts to have in-depth conversations with LLMs, explore complex security scenarios and obtain personalized analytical insights. This step not only shows the analysis results of the system, but also provides strong support for the security team's decision-making.

[0034] Step 5. The threat data storage module serves as the central data warehouse of the system and is responsible for managing all relevant data securely and efficiently. It continuously receives and stores data from data collection in step 1, behavior analysis in step 2, and real-time monitoring modules in step 3. The use of a distributed storage architecture ensures high availability and scalability of the system, while the introduction of LLMs makes data classification, sensitive information identification, and storage strategy formulation smarter. The system also implements complex data partitioning and index optimization strategies, greatly improving query performance. In addition, the integrated data lifecycle management function ensures that data storage meets compliance requirements. This module not only provides data support for real-time analysis, but also provides a rich data source for historical data analysis and trend prediction modules.

[0035] Step 6. As the advanced analysis component of the system, this module uses the large amount of historical data accumulated in the threat data storage module in step 5 for deep mining and forward-looking analysis. First, intelligent feature engineering is performed with the assistance of LLMs to extract key indicators from massive data. Then, advanced time series models and deep learning techniques are used, combined with the text analysis capabilities of LLMs, to conduct a comprehensive analysis of historical trends. The powerful reasoning capabilities of LLMs are used to analyze large amounts of unstructured data and identify potential attack patterns and motivations. Finally, the system generates comprehensive and insightful trend analysis reports to guide the organization's long-term security strategy. The analysis results of this step are not only used to update the detection rules of the real-time monitoring module, but also provide strategic insights to the security team through the user interface.

[0036] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. Although the present invention has been represented and described with reference to specific preferred embodiments, it shall not be interpreted as a limitation of the present invention itself. Various changes can be made to it in form and detail without departing from the spirit and scope of the present invention defined in the attached claims.

Claims

1. A method for intelligent analysis and prediction of internal network threats based on a large language model, characterized in that: The steps include: Collect and pre-process raw log data in real time through multiple channels, and convert the raw log data into structured feature description text data; Using large language models (LLMs) to perform semantic understanding on the feature description text data to generate a behavior sequence; using a neural network model to model the behavior sequence to generate a predicted behavior sequence for a predetermined period in the future; The complex event processing engine and LLMs are used to dynamically generate rules to analyze the predicted behavior sequence, generate recognition results, and trigger corresponding corresponding measures based on the recognition results.

2. The internal network threat intelligent analysis and prediction method based on a large language model according to claim 1, characterized in that: The raw log data is collected by means of a RESTful API interface, deployment of a lightweight log collection agent, and JDBC connection.

3. According to claim 1, a method for intelligent analysis and prediction of internal network threats based on a large language model is characterized in that: The collected raw log data is cleaned and standardized, and duplicate and erroneous data are removed using hash algorithms, regular expressions, and fuzzy matching.

4. According to claim 1, a method for intelligent analysis and prediction of internal network threats based on a large language model is characterized in that: The use of a neural network model to model the behavior sequence and generate a predicted behavior sequence for a predetermined period in the future specifically includes: The transformer architecture is used to model the behavior sequence, and the sequence generation ability of large language models (LLMs) is used to predict possible future behavior sequences. LLMs are introduced as meta-learners to obtain the final threat assessment results.

5. According to claim 1, a method for intelligent analysis and prediction of internal network threats based on a large language model is characterized in that: By monitoring the message queue, the predicted behavior sequence is continuously received, and the predicted behavior sequence is analyzed by using the complex event processing engine and LLMs to dynamically generate rules to form a recognition result; Based on the recognition results, the intelligent multi-level alarm system is immediately activated. LLMs assist in decision-making during this process and dynamically adjust the threat level, dividing the threat level into low, medium, and high levels according to the predetermined thresholds; For high-level threats, an intelligent multi-level alert system immediately sends personalized alerts through multiple channels, triggering automated response measures.

6. The method for intelligent analysis and prediction of internal network threats based on a large language model according to claim 5, characterized in that: The intelligent multi-level alarm system also performs alarm aggregation and correlation analysis, using the deep semantic analysis capabilities of LLMs to identify potential large-scale attack patterns.

7. An internal network threat intelligent analysis and prediction system, characterized in that: include: Data collection and preprocessing module; the data collection and preprocessing module collects and preprocesses raw log data in real time through multiple channels, and converts the raw log data into structured feature description text data; Behavior analysis module; the behavior analysis module uses large language models (LLMs) to perform semantic understanding on the feature description text data to generate a behavior sequence; a neural network model is used to model the behavior sequence to generate a predicted behavior sequence for a predetermined period in the future; Real-time monitoring and alarm module; The real-time monitoring and alarm module uses a complex event processing engine and LLMs to dynamically generate rules to analyze the predicted behavior sequence, generate recognition results, and trigger corresponding corresponding measures based on the recognition results.

8. The internal network threat intelligent analysis and prediction system according to claim 7, characterized in that: Also includes user interface and reporting modules; The user interface and reporting module provides an interactive platform for security managers to display threat situations in the form of interactive charts and heat maps.

9. The internal network threat intelligent analysis and prediction system according to claim 7, characterized in that: Also included is a threat data storage module; The threat data storage module is used to store and transfer the original log data, feature description text data, predicted behavior sequence, and recognition results.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on the electronic device, the electronic device executes the internal network threat intelligent analysis and prediction method based on a large language model as described in any one of claims 1 to 6.

Images