A network threat forensics method, program product, electronic device and storage medium

Abstract

The embodiment of the application provides a network threat forensics method, a program product, an electronic device and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining original data; preprocessing the original data to obtain an initial digital forensics report; detecting an abnormal event on the initial digital forensics report, and generating a text forensics file based on the abnormal event; performing event blocking and vector embedding on the text forensics file to obtain a vector file; receiving query information input by a user, and generating a forensics report based on the query information and the vector file. The method can automatically detect abnormal events and perform event blocking and vector embedding, can extract key information of the events, improves the efficiency and accuracy of threat forensics, and solves the problem of low efficiency of threat forensics in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a network threat forensics method, a program product, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of information technology, digital devices are widely used in people's life and work, and network security problems are increasingly prominent. Network attack means are constantly renewed, such as malicious software, phishing, DDoS attacks, etc. These attacks not only pose a serious threat to personal privacy and enterprise data security, but also may cause significant economic losses. In this context, the importance of threat forensics and incident response in maintaining digital security is increasingly significant. They are committed to finding evidence and reconstructing the event sequence, so as to effectively respond to network threats. However, the process of threat forensics and incident response faces many challenges, for example, in network incidents, the amount of data grows explosively, and the types of data are diverse, which brings great difficulties to data management and processing, and also makes the resource consumption of threat forensics and incident response serious, time-consuming and laborious, and requires high professional knowledge. Existing tools can usually only analyze artifacts one by one, cannot consider all related data at the same time, and are difficult to realize automatic analysis, which seriously affects the efficiency and accuracy of threat forensics. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a network threat forensics method, a program product, an electronic device and a storage medium, which can automatically detect abnormal events and perform event blocking and vector embedding, can extract key information of the event, improve the efficiency and accuracy of threat forensics, and solve the problem of low efficiency of threat forensics of the existing method.

[0004] The embodiments of the present application provide a network threat forensics method, which comprises:

[0005] obtaining original data;

[0006] preprocessing the original data to obtain an initial digital forensics report;

[0007] detecting abnormal events on the initial digital forensics report, and generating a text forensics file based on the abnormal events;

[0008] performing event blocking and vector embedding on the text forensics file to obtain a vector file;

[0009] receiving query information input by a user, and generating a forensics report based on the query information and the vector file.

[0010] In the implementation process, the more critical event information is extracted through the abnormal event detection and event embedding, the information loss or incomplete processing is avoided, the efficiency and accuracy of threat forensics are improved, and the problem of low efficiency of threat forensics in the prior art is solved.

[0011] Further, the original data includes text data and non-text data, and the preprocessing of the original data to obtain an initial digital forensics report includes:

[0012] Converting the non-text data into a form of text plus image metadata;

[0013] Generating a text file from the text data and event attributes;

[0014] Fusing the processed original data to obtain an initial digital forensics report.

[0015] In the implementation process, the non-text data is converted into text output, the event information can be more comprehensively obtained, the purpose of efficiently integrating multi-modal data is achieved, and the data processing efficiency and accuracy of the output result are improved.

[0016] Further, the initial digital forensics report is subjected to abnormal event detection, and a text forensics file is generated based on the abnormal event, including:

[0017] Comparing each event in the initial digital forensics report with a predefined rule;

[0018] If any event meets the predefined rule, the event is an abnormal event.

[0019] In the implementation process, the predefined rule can filter out abnormal events from the initial digital forensics report, avoiding unnecessary processing of a large number of normal events, and improving the analysis efficiency.

[0020] Further, the text forensics file is subjected to event blocking and vector embedding to obtain a vector file, including:

[0021] Blocking the abnormal event in the text forensics file to generate an event block, the event block including all attributes of the event.

[0022] In the implementation process, the event blocking retains the integrity and independence of each event, facilitating subsequent processing and analysis.

[0023] Further, the text forensics file is subjected to event blocking and vector embedding to obtain a vector file, including:

[0024] Performing vector embedding operation on the event block to obtain an event vector;

[0025] The event vector generates a vector file.

[0026] In the above implementation process, through the vector embedding operation, the semantic information in the time block can be converted into a vector, and each vector will serve as a mathematical representation of each event, facilitating processing and analysis.

[0027] Further, the query information input by the user is received, and a forensic report is generated based on the query information and the vector file, including:

[0028] Generate a query RI based on the query information;

[0029] Encode and embed the query RI to obtain a query vector VRI;

[0030] Based on the correlation between the query vector VRI and the event vectors in the vector file, the most relevant event is obtained;

[0031] Generate a forensic report based on the most relevant event and the corresponding original text data.

[0032] In the above implementation process, according to the query information input by the user, the most relevant event most relevant to the query information is obtained, achieving the purpose of providing comprehensive and accurate threat information for the user.

[0033] Further, based on the correlation between the query vector VRI and the event vectors in the vector file, the most relevant event is obtained, including:

[0034] Calculate the cosine similarity of the query vector VRI and all event vectors;

[0035] Get the top K events with the highest cosine similarity score as the most relevant events, where K is an adjustable value.

[0036] In the above implementation process, the most relevant event can be obtained by calculating the cosine similarity.

[0037] The application embodiment also provides a network threat forensics device, which comprises:

[0038] A data acquisition module for acquiring original data;

[0039] A preprocessing module for preprocessing the original data to obtain an initial digital forensic report;

[0040] An anomaly detection module for detecting abnormal events on the initial digital forensic report and generating a text forensic file based on the abnormal events;

[0041] A block and embedding module for event blocking and vector embedding of the text forensic file to obtain a vector file;

[0042] The report generation module is configured to receive user input query information and generate a forensic report based on the query information and the vector file.

[0043] In the implementation process, the more critical event information is extracted through the abnormal event detection and event embedding, the information loss or incomplete processing is avoided, the efficiency and accuracy of threat forensics are improved, and the problem of low efficiency of threat forensics in the prior art is solved.

[0044] The embodiment of the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the network threat forensics method.

[0045] The embodiment of the application further provides an electronic device, which comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to execute the network threat forensics method.

[0046] The embodiment of the application further provides a readable storage medium, which stores computer program instructions, and the computer program instructions are read and run by a processor to execute the network threat forensics method. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments of the application. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0048] Figure 1 A flowchart of a network threat forensics method provided by the embodiment of the application;

[0049] Figure 2 A preprocessing flowchart of raw data provided by the embodiment of the application;

[0050] Figure 3 An abnormal event detection flowchart provided by the embodiment of the application;

[0051] Figure 4 An event embedding flowchart provided by the embodiment of the application;

[0052] Figure 5 A generation flowchart of a forensic report provided by the embodiment of the application;

[0053] Figure 6The acquisition flowchart of the most relevant event provided for the embodiment of the present application is as follows:

[0054] Figure 7 The structural block diagram of a network threat forensics device provided for the embodiment of the present application is as follows.

[0055] Icon:

[0056] 100-data acquisition module; 200-preprocessing module; 300-anomaly detection module; 400-block and embedding module; 500-report generation module. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0058] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for differentiation, and cannot be understood as indicating or implying relative importance.

[0059] Please refer to Figure 1 , Figure 1 The flowchart of a network threat forensics method provided for the embodiment of the present application is as follows. The method can generate a corresponding forensics report based on the query of a user, including the following steps:

[0060] Step S100: acquiring original data;

[0061] Step S200: preprocessing the original data to obtain an initial digital forensics report;

[0062] Step S300: detecting abnormal events on the initial digital forensics report, and generating a text forensics file based on the abnormal events;

[0063] Step S400: performing event blocking and vector embedding on the text forensics file to obtain a vector file;

[0064] Step S500: receiving the query information input by the user, and generating a forensics report based on the query information and the vector file.

[0065] The method extracts more key event information through abnormal event detection and event embedding, avoids the situation of information loss or incomplete processing, improves the efficiency and accuracy of threat forensics, and solves the problem of low efficiency of threat forensics in the prior art.

[0066] For example, the present application can perform data processing and event correlation analysis based on multiple models in a large model Agent.

[0067] The raw data in step S100 includes text data and non-text data, including but not limited to text records in network device logs and system logs, email content, text information in documents, system files, and application data. Storing the raw data, such as in CSV file format, facilitates data organization and analysis.

[0068] like Figure 2 The diagram shows the preprocessing flowchart for the raw data. Step S200 specifically includes the following steps:

[0069] Step S201: Convert the non-text data into a format that includes text and image metadata;

[0070] For example, multimodal models can be used to process non-textual data. For multimodal information such as images contained in non-textual data, such as images captured by network surveillance cameras or binary images related to malware, multimodal large model selection (vLLM) models like the Lalava model can be used to output the analysis corresponding to the image. The analysis results are then converted into a form that combines text with image metadata (metadata such as image type, timestamp, etc.). For instance, for images containing malware characteristics, the Lalava model can identify and describe key features, such as specific code segments or file structures, and convert them into text.

[0071] Step S202: Generate a text file from the text data and event attributes;

[0072] For text-based data, such as text records in system logs, email content, and text information in documents, as well as event-related attributes, such as the event's unique identifier (ID), detailed description, specific content, severity level (e.g., urgent, high, medium, low), occurrence time, and date, the data is stored in a text file.

[0073] Step S203: Perform modal fusion on the processed raw data to obtain an initial digital forensics report.

[0074] Specifically, the processed non-text data and text data are merged to obtain an initial digital forensics report containing rich event information.

[0075] Preprocessing of raw data enables efficient integration of multimodal data: non-text data such as images are converted into text output, and fusion with other text data is achieved, enabling more comprehensive acquisition of event information. This effectively solves the problem that traditional tools have difficulty in handling multiple types of data and significantly improves data processing efficiency.

[0076] The abnormal event detection in step S300, i.e. screening relevant data from a large amount of data in the initial digital forensic report, thereby reducing the amount of data to be processed subsequently. As shown in Figure 3 FIG. 1 is a flowchart of an abnormal event detection process, specifically:

[0077] Step S301: comparing each event in the initial digital forensic report with a predefined rule;

[0078] Step S302: if any event meets the predefined rule, the event is an abnormal event.

[0079] Each event in the initial digital forensic report is detected, and the attributes of the event, such as event type, occurrence time, source IP address, operation behavior, etc. are compared with the corresponding conditions in the predefined rule. If the attributes of a certain event meet the abnormal conditions defined in the rule, the event is determined to be an abnormal event. For example, the event log: "ordinary privilege user Zhang San deleted the firewall configuration file", which matches the rule "a user account performs an operation that does not conform to its privilege range", so the event is identified as an abnormal event; for example, a process accesses a sensitive file of the system or a certain IP performs an abnormal frequent connection attempt in a short time, which will be determined as an abnormal event.

[0080] Once an abnormal event is detected, it will be selected and recorded in a new final text forensic file D. This file D will be used to store all events identified as abnormal, which will be the focus of further analysis and processing, providing key clues for in-depth investigation of network events, while avoiding unnecessary processing of a large number of normal events, improving analysis efficiency.

[0081] Event embedding, i.e. event chunking and vector embedding, in order to enable large language models (LLM) to obtain more accurate context information and generate forensic reports in combination with context information, the event needs to be chunked, and event chunking operation plays a crucial role in accurately and efficiently retrieving information from external knowledge base.

[0082] As shown in Figure 4 FIG. 2 is a flowchart of event embedding, specifically including the following steps:

[0083] Step S401: chunking the abnormal events in the text forensic file to generate event blocks, the event blocks including all attributes of the events.

[0084] Step S402: performing vector embedding operation on the event blocks to obtain event vectors;

[0085] Step S403: generating a vector file from the event vectors.

[0086] The event chunking operation divides the text forensics file D (which contains filtered abnormal events) into individual events (each log entry is considered an event), making each event a separate chunk. Each event chunk contains all relevant attributes of that event. This event-based chunking method ensures the integrity and independence of the information within each chunk, facilitating subsequent processing and analysis.

[0087] Vector embedding models such as mxbai-embed-large are used to embed the segmented text data, extracting a vector containing semantic information for each event. This transforms the event information, originally in text form, into a numerical form that the model can understand and compute, facilitating mathematical operations and semantic analysis. Finally, after segmentation and embedding, a dense vector file ED is output for the text evidence file D. Each vector serves as a mathematical representation of each event, used for subsequent comparison and analysis with data in the knowledge base, providing a foundation for accurately retrieving relevant information.

[0088] By employing event segmentation and embedding operations, the input / output token length limitation of the LLM model is effectively resolved. When processing long documents or complex cybersecurity reports, it can accurately handle all relevant information, avoiding information loss or incomplete processing, thus improving the performance of the LLM model in threat forensics and analysis tasks.

[0089] In summary, by using anomaly detection and event embedding, more critical event information can be extracted as the context of the LLM model, enabling the large model to output more accurate causal relationships and logical connections between events.

[0090] In step S500, the user can interact with the large model agent system to generate an event forensics report related to the user's query. Specifically, such as... Figure 5 The following is a flowchart of the evidence report generation process:

[0091] Step S510: Generate a query RI based on the query information;

[0092] Step S520: Encode and embed the query RI to obtain the query vector VRI;

[0093] Step S530: Based on the correlation between the query vector VRI and the event vectors in the vector file, obtain the most relevant event;

[0094] Step S540: Generate an evidence report based on the most relevant events and the corresponding original text data.

[0095] Among them, such as Figure 6 The diagram shown is a flowchart for obtaining the most relevant events. Specifically:

[0096] Step S531: Calculate the cosine similarity of the query vector VRI and all event vectors.

[0097] Step S532: Get the top K events with the highest cosine similarity scores as the most relevant events, where K is an adjustable value.

[0098] The user-input query information can be processed by a large language model (LLM) to generate a more contextually relevant query RI (Resource Identifier, a unique identifier for a certain resource), and an embedding model is used to encode and embed the query RI to generate a sentence vector VRI for the query, which is used for vector retrieval. This process measures the relevance of each event vector in the vector file ED and the query vector VRI by calculating their cosine similarity. The formula for calculating the cosine similarity is:

[0099]

[0100] where, is the query vector, is the event vector in the vector file ED.

[0101] Search for events with high similarity to VRI in the entire vector file ED, and select the most relevant events based on the calculated similarity scores. Since there may be multiple information with high relevance, select the top K events with the highest similarity scores (the value of K can be adjusted), and retrieve the original text information corresponding to the K events.

[0102] The most relevant events and corresponding original text information obtained are sent to the large language model (LLM) for further processing to identify and extract key information and mine potential associations between events. For example, by analyzing the time sequence, operation logic, and involved entities of events, causal relationships and logical connections between events are established. Finally, based on the analysis of events, the Agent system uses the LLM to generate an enhanced forensic report, including threat evidence, as well as detailed analysis of events, in-depth interpretation of event relevance, and mitigation measures for events. For example, for network security events, the LLM analyzes the type of attack, possible attack sources, and the scope of affected systems, and provides corresponding response recommendations based on these analysis results. The forensic report will display the timeline of events, detailed analysis of key events, relationships between events, and response recommendations in a clear and organized manner, providing comprehensive and accurate information to analysts to help them better understand the overall picture of the event and make reasonable decisions.

[0103] ​As one of the application scenarios, such as in the security operation scenario, first, various types of data related to the alarm are collected, including firewall logs, intrusion detection system (IDS) alarm information, server system logs, and part of the network traffic data packets related to the event, and other multi-modal data.

[0104] After the data collection is completed, the image-formatted metadata is analyzed by the Llava-Llama3 model, the effective information in the image is extracted and converted into text; at the same time, the log and alarm information in text format are cleaned and structured. All related data are fused by modalities to generate an initial report, which contains a preliminary description of the event and key information of each data.

[0105] Then, the abnormal event detection module filters the initial report according to the preset rules, such as abnormal port scanning mode, frequent authentication failure, etc., and records the events meeting the abnormal characteristics to file D.

[0106] Subsequently, event blocking is performed, and the events are reasonably divided according to the event attributes and length to ensure that each event block is suitable for subsequent processing. The blocked events are embedded into vectors by the mxbai-embed-large model. The technician initiates a query to understand the specific path and impact range of the intrusion behavior, retrieves relevant information from file D according to the optimized query, finds the most relevant content through cosine similarity calculation, generates a preliminary mining and analysis of the event; finally, according to the preliminary analysis of the event, a forensic report containing the time sequence of each stage of intrusion is generated to help security analysts quickly respond and handle the network security event.

[0107] This method can accurately associate events and reconstruct the time axis: it can mine the potential association between events, clearly sort out the logical relationship of events from different systems and different time periods, overcome the difficulty of traditional methods in associating scattered events, retrieve more information related to abnormal events from event records through event blocking, effectively supplement the context required for event analysis, greatly improve the accuracy of event association analysis, and thus realize more accurate time axis reconstruction.

[0108] The embodiments of the present application also provide a network threat forensics device applied to the network threat forensics method described above, as shown in FIG. 1, which is a structural block diagram of a network threat forensics device. The device includes but is not limited to: Figure 7

[0109] The data acquisition module 100 is used to acquire raw data;

[0110] The preprocessing module 200 is used to preprocess the raw data to obtain an initial digital forensic report;

[0111] ​Anomaly detection module 300, configured to perform anomaly event detection on the initial digital forensic report, and generate a text forensic file based on the anomaly event;

[0112] Chunking and embedding module 400, configured to perform event chunking and vector embedding on the text forensic file, and obtain a vector file;

[0113] Report generation module 500, configured to receive user input query information, and generate a forensic report based on the query information and the vector file.

[0114] For the specific implementation process of each module, it has been specifically described in the above method embodiments, and can also be applied to each module, and will not be described here.

[0115] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the network threat forensic method.

[0116] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to make the electronic device execute the network threat forensic method.

[0117] The embodiment of the present application further provides a readable storage medium, which stores computer program instructions, and the computer program instructions are read and run by a processor to execute the network threat forensic method.

[0118] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0119] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0120] If the functions are realized in the form of software functional modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0121] The above is only an embodiment of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0122] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0123] It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components. Additionally, the terms "comprise," "comprises," and "comprising," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless otherwise indicated herein, the terms "first," "second," "third," etc., are used herein merely as labels, and are not intended to impose ordinal import.

Claims

1. A network threat forensics method, characterized by, The method comprises: acquiring original data; preprocessing the original data to obtain an initial digital forensic report; detecting abnormal events from the initial digital forensic report and generating a text forensic file based on the abnormal events; performing event blocking and vector embedding on the text forensic file to obtain a vector file; receiving user input query information and generating a forensic report based on the query information and the vector file, specifically: generating a query RI based on the query information; performing encoding embedding on the query RI to obtain a query vector VRI; obtaining the most relevant event based on the correlation between the query vector VRI and the event vectors in the vector file; and generating a forensic report based on the most relevant event and the corresponding original text data.

2. The network threat forensics method of claim 1, wherein, The original data includes text data and non-text data, and the preprocessing of the original data to obtain an initial digital forensic report comprises: converting the non-text data into a text and image metadata form; generating a text file from the text data and event attributes; performing modal fusion on the processed original data to obtain an initial digital forensic report.

3. The network threat forensics method of claim 1, wherein, The abnormal event detection from the initial digital forensic report and the generation of a text forensic file based on the abnormal events comprise: comparing each event in the initial digital forensic report with a predefined rule; if any event meets the predefined rule, the event is an abnormal event.

4. The network threat forensics method of claim 1, wherein, The event blocking and vector embedding of the text forensic file to obtain a vector file comprise: blocking the abnormal events in the text forensic file to generate event blocks, the event blocks including all attributes of the events.

5. The network threat forensics method of claim 4, wherein, The event blocking and vector embedding of the text forensic file to obtain a vector file comprise: performing vector embedding operations on the event blocks to obtain event vectors; generating a vector file from the event vectors.

6. The network threat forensics method of claim 1, wherein, The obtaining of the most relevant event based on the correlation between the query vector VRI and the event vectors in the vector file comprises: calculating the cosine similarity of the query vector VRI and all event vectors; obtaining the top K events with the highest cosine similarity scores as the most relevant events, wherein K is an adjustable value.

7. A computer program product, characterised in that, The computer program product comprises a computer program that is executed by a processor to implement the network threat forensic method of any one of claims 1-6.

8. An electronic device, comprising: The electronic device comprises a memory for storing a computer program and a processor for running the computer program to enable the electronic device to perform the network threat forensic method of any one of claims 1-6.

9. A readable storage medium, characterized by, The readable storage medium stores computer program instructions that are read and run by a processor to perform the network threat forensic method of any one of claims 1-6.

Images