Original message intelligent storage method and system based on alarm event
By deploying bypass devices at the network egress point and selectively storing data based on alarm events, the high cost and invalid data management issues caused by full traffic storage are resolved. This achieves efficient reduction of storage pressure and rapid source tracing capabilities, improving the efficiency of locating network faults and attack sources.
Patent Information
- Application Number
- CN202511424103.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-30
- Publication Date
- 2025-11-18
AI Technical Summary
In large network and data center environments, full traffic collection and storage leads to high storage costs and management pressure. Existing technologies increase data management costs through full storage, while sampling storage lacks complete packet tracing capabilities, making it difficult to quickly locate network faults or attack sources.
Selective storage based on alarm events is achieved by deploying bypass acquisition devices at the network egress point to mirror the entire network traffic, analyze static rules, dynamic thresholds and transmission modes, detect alarm events, capture key event messages and generate an integrated 'event-message' document, which is stored independently and configured for long-term preservation as needed, reducing the storage of invalid data.
It reduces storage space usage by more than 80%, ensures the traceability of alarm events, quickly locates the source of faults or attacks, supports flexible storage strategies, and improves the efficiency and response time of tracing.
Smart Images
Figure CN120979903A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of network flow analysis and storage, and particularly relates to an original message intelligent storage method and system based on alarm events. BACKGROUND
[0002] In large network and data center environments, data collection and storage need to be carried out regularly; however, with the increasing size of network flow, full flow collection and storage often bring extremely high storage costs and management pressure.
[0003] In the prior art, when the full storage + regular coverage mode is used for flow data collection and storage, the data obtained is relatively complete, but it will lead to excessive storage pressure and increase data management costs; according to actual use statistics, more than 90% of network message data has no actual value in later analysis, and in this case, if the sampling storage mode is used for flow data collection and storage, the complete message tracing capability will be lacked, and when network failure, business performance decline or attack occurs, the original message related to the event is lacked, and it is difficult to quickly locate and trace the problem. SUMMARY
[0004] In view of the above defects or deficiencies in the prior art, the present application aims to provide an original message intelligent storage method and system based on alarm events, which selectively stores original messages based on alarm events, parallel to the full storage mechanism, improves the data filtering effect in time, retains the integrity of full data, improves the alarm event tracing and query efficiency, greatly reduces the long-term storage of invalid data while ensuring the traceability of alarm events, and reduces the storage pressure and cost.
[0005] In order to achieve the above purpose, the embodiments of the present application adopt the following technical solutions:
[0006] In a first aspect, the embodiments of the present application provide an original message intelligent storage method based on alarm events, which comprises the following steps:
[0007] Step S1: deploying a bypass collection device at a network exit to mirror collect full network flow;
[0008] Step S2: analyzing the current full network flow to extract static rules, dynamic thresholds, and transmission mode related information;
[0009] Step S3: detecting alarm events based on the analyzed related information, and the detection content includes static rule detection, dynamic threshold detection and / or learning model detection; when the detection result finds an alarm event, go to step S4; if no alarm event is found, go to step S8;
[0010] Step S4, intercepting the original message in a predetermined time window from the first predetermined time before the current alarm event to the second predetermined time after the current alarm event as the key event message;
[0011] Step S5, generating a unique ID for the current alarm event and binding it with the key event message to generate an "event-message" integrated document;
[0012] Step S6, independently storing the "event-message" integrated document and configuring a long-term storage mode as needed;
[0013] Step S7, the user manages and applies the stored "event-message" integrated document through the alarm interface;
[0014] Step S8, normally storing the current full network traffic.
[0015] In a preferred embodiment, the static rule related information in step 2 includes port information, IP; dynamic threshold related information includes traffic size, traffic rate, connection number, packet loss rate; transmission mode related information includes communication behavior mode, clustering model.
[0016] In a preferred embodiment, when detecting the alarm event in step S3, preset detection values are set from static rules, dynamic thresholds, and transmission modes, and when the parsed related information contains the preset detection values, it is determined that an alarm event is found.
[0017] In a preferred embodiment, the preset detection values from the perspective of static rules include malicious IP, port rules; the preset detection values from the perspective of dynamic thresholds include traffic surge, traffic rate threshold, connection number threshold, packet loss rate threshold; the preset detection values from the perspective of transmission mode include abnormal transmission mode, abnormal clustering mode or suspicious communication behavior detected and identified based on machine learning model.
[0018] In a preferred embodiment, the predetermined time window in step S4 uses fixed time or custom time.
[0019] In a preferred embodiment, the predetermined time window in step S4 is ±60 seconds or ±120 seconds.
[0020] In a preferred embodiment, when storing independently in step S6, the message file is stored in PCAP format; when storing independently, record event ID, time range, storage path, and associated message quantity information.
[0021] In a preferred embodiment, the strategy includes a default storage period, user manual marking, compliance archiving when security compliance requirements are met, and on-demand deletion when the "event-message" integrated document is managed in step S7.
[0022] In a preferred embodiment, the application of the "event-message" integrated document in step S7 includes:
[0023] When a certain event is clicked, the system displays the communication object, session data, and original message details related to the event, while supporting original message packet download;
[0024] The alarm interface displays basic alarm information, including time, source IP, destination IP, protocol type, and trigger rule.
[0025] Users can expand to view original message data, including session statistics, quintuple information, and packet details.
[0026] In a second aspect, the embodiments of the present application also provide an original message intelligent storage system based on alarm events, which includes a traffic collection module, an information analysis module, an alarm event determination module, a message interception module, a binding module, a storage module, an application module, and a normal storage module, wherein
[0027] The traffic collection module is used to deploy bypass collection devices at network exit to mirror and collect full network traffic.
[0028] The information analysis module is used to analyze the current full network traffic and extract static rules, dynamic thresholds, and transmission mode related information.
[0029] The alarm event determination module is used to detect alarm events from the current full network traffic based on the analyzed related information, and the detection content includes static rule detection, dynamic threshold detection, and / or learning model detection. When the detection result finds an alarm event, the message interception module is started. Otherwise, the normal storage module is started.
[0030] The message interception module is used to intercept original messages in a predetermined time window from the first predetermined time before the current alarm event to the second predetermined time after the current alarm event as key event messages.
[0031] The binding module is used to generate a unique ID for the current alarm event and bind it with the key event messages to generate an "event-message" integrated document.
[0032] The storage module is used to independently store the "event-message" integrated document and configure a long-term storage mode on demand.
[0033] The application module is used for the user to manage and apply the stored integrated document of the event-message through the alarm interface.
[0034] The normal storage module is used for normal storage of the current full network traffic.
[0035] The technical scheme provided by the embodiment of the application has the following beneficial effects:
[0036] The original message intelligent storage method and system based on the alarm event provided by the embodiment of the application avoid long-term storage of full traffic, reduce storage space occupation by more than 80%, guarantee traceability of the alarm event, quickly locate a fault or an attack source, support flexible storage strategies, take into account operation and maintenance and compliance requirements, improve traceability efficiency, and shorten alarm event response time.
[0037] Of course, implementing any product or method of the present application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0039] Figure 1 It is the original message intelligent storage method flowchart based on the alarm event according to the embodiment of the present application. DETAILED DESCRIPTION
[0040] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. It should be noted that the embodiments and features in the embodiments can be combined with each other without conflict.
[0041] It should be noted that: similar labels and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In the description of the present application, the terms "first", "second", "third", "fourth" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.
[0042] Based on the storage problem of large flow data in the prior art, the embodiment of the present application provides a kind of original message intelligent storage method and system based on alarm event, original text is filtered and stored based on alarm event, through alarm triggering+event learning+message windowing storage, while reducing storage pressure, guarantee abnormal event traceability and analysis.
[0043] As Figure 1 Indicated, the original message intelligent storage method based on alarm event described in the embodiment of the present application includes the following steps:
[0044] Step S1, bypass collection equipment is deployed at network outlet, and full network flow is mirror collected.
[0045] In this step, the mirror collection can use switch or network monitoring device with mirror function.
[0046] Step S2, the current full network flow is parsed, and static rules, dynamic threshold, transmission mode related information are extracted.
[0047] In this step, the static rule related information includes port information, IP, etc.;The dynamic threshold includes flow size, flow rate, connection number, packet loss rate, etc.;The transmission mode includes communication behavior mode, clustering model, etc.
[0048] Step S3, according to the related information parsed, alarm event detection is carried out on the current full network flow, and the detection content includes static rule detection, dynamic threshold detection and / or learning model detection. When the detection result finds alarm event, enter step S4. If no alarm event is found, go to step S8.
[0049] In this step, when alarm event detection is carried out, preset detection value is set from static rule, dynamic threshold, transmission mode, etc. When the related information parsed contains preset detection value, it is determined that alarm event is found. Among them, the preset detection value from the perspective of static rule includes: malicious IP, port rule, etc.;The preset detection value from the perspective of dynamic threshold includes flow burst, flow rate threshold, connection number threshold, packet loss rate threshold, etc.;The preset detection value from the perspective of transmission mode includes abnormal transmission mode, abnormal clustering mode or suspicious communication behavior identified based on machine learning model detection.
[0050] Step S4, intercept the original message of the predetermined time window between the first predetermined time before the current alarm event and the second predetermined time after the current alarm event, as key event message.
[0051] In this step, the predetermined time window can use fixed time or custom time, such as ±60 seconds, 120 seconds or user setting.
[0052] Step S5: Generate a unique ID for the current alarm event and bind it with the critical event message to generate an integrated "event-message" document.
[0053] In this step, an event ID is generated and bound to the original file, which facilitates the long-term storage and management of the document.
[0054] Step S6: Store the integrated "event-message" document independently and configure a long-term storage mode as needed.
[0055] In this step, the independent storage is no longer affected by the First-In-First-Out (FIFO) strategy. Message files are stored in PCAP format for easier analysis using tools like Wireshark. Independent storage records information such as event ID, time range, storage path, and the number of associated messages. This independent storage is in contrast to normal message storage. Typically, all monitored traffic needs to be stored, but normal storage is short-lived, lacks specificity, and makes retrieval and searching difficult. The integrated "event-message" document supports on-demand configuration; users can save critical event messages long-term (e.g., 6 months, 1 year) or delete them after analysis, thereby improving retrieval and searching efficiency.
[0056] Step S7: The user manages and applies the stored integrated "event-message" document through the alarm interface. Preferably, when managing the integrated "event-message" document, the strategy includes:
[0057] Default save period: e.g., automatically deleted after 30 days;
[0058] User-defined marking: If the user marks the event as "important", the message will be moved to the long-term storage area;
[0059] Compliance archiving: Supports exporting to external storage (NAS, archive server) to meet security and compliance requirements;
[0060] Deleting on demand: Users can proactively delete the event message after analysis.
[0061] In this step, preferably, when applying the integrated "event-message" document, clicking on an event displays the communication objects, session data, and original message details related to that event, while also supporting the download of the original message data packets; the alarm interface displays basic alarm information (time, source IP, destination IP, protocol type, trigger rule); users can expand to view the original message data (session statistics, 5-tuple information, data packet details, etc.).
[0062] Step S8: Perform normal storage of the current full network traffic.
[0063] Based on the same idea, the embodiment of the present application also provides an original message intelligent storage system based on an alarm event, which comprises a traffic collection module, an information analysis module, an alarm event judging module, a message interception module, a binding module, a storage module, an application module and a normal storage module, wherein,
[0064] The traffic collection module is used for deploying a bypass collection device at a network outlet to mirror and collect full network traffic.
[0065] The information analysis module is used for analyzing the current full network traffic to extract static rules, dynamic thresholds and transmission mode related information.
[0066] The alarm event judging module is used for detecting alarm events of the current full network traffic according to the analyzed related information, and the detection content comprises static rule detection, dynamic threshold detection and / or learning model detection; when the detection result finds an alarm event, the message interception module is started; otherwise, the normal storage module is started.
[0067] The message interception module is used for intercepting original messages in a predetermined time window from a first predetermined time before the current alarm event to a second predetermined time after the current alarm event as key event messages.
[0068] The binding module is used for generating a unique ID for the current alarm event and binding the unique ID with the key event messages to generate an "event-message" integrated document.
[0069] The storage module is used for independently storing the "event-message" integrated document and configuring a long-term storage mode as required.
[0070] The application module is used for a user to manage and apply the stored "event-message" integrated document through an alarm interface.
[0071] The normal storage module is used for normally storing the current full network traffic.
[0072] The modules in the embodiment are implemented by a processor, and a memory is appropriately increased when storage is required. The processor can be, but is not limited to, a microprocessor (MPU), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.
[0073] In the above embodiment, all or part of the embodiment can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiment can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function described in the embodiment of the present application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).
[0074] In addition, it should be noted that the event-based raw message intelligent storage system and the event-based raw message intelligent storage method described in the embodiment are corresponding. The description and limitation of the method are also applicable to the system, which will not be described here.
[0075] As can be seen from the above technical solutions, the event-based raw message intelligent storage method and system provided by the embodiment of the present application avoids long-term storage of full flow, reduces storage space occupation by more than 80%; ensures the traceability of events, quickly locates the source of failure or attack; supports flexible storage strategy, and takes into account the operation and maintenance and compliance requirements; improves the traceability efficiency and shortens the event response time.
[0076] The above description is only the preferred embodiment of the present application and the explanation of the technical principles applied, and is not intended to limit the scope of the application claimed, but only represents the preferred embodiment of the present application. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the inventive concept. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.
Claims
1. A method for intelligent storage of raw messages based on alarm events, characterized in that, The method includes the following steps: Step S1: Deploy a bypass acquisition device at the network egress point to mirror and collect all network traffic. Step S2: Analyze the current full network traffic and extract information related to static rules, dynamic thresholds, and transmission modes; Step S3: Based on the parsed relevant information, perform alarm event detection on the current full network traffic. The detection content includes static rule detection, dynamic threshold detection, and / or learning model detection. When the detection results detect an alarm event, proceed to step S4; If no alarm event is found, proceed to step S8; Step S4: Capture the original message within the predetermined time window from the first predetermined time before the current alarm event to the second predetermined time after the current alarm event, and use it as the critical event message; Step S5: Generate a unique ID for the current alarm event and bind it with the key event message to generate an integrated "event-message" document; Step S6: Store the integrated "event-message" document independently and configure a long-term storage mode as needed; Step S7: The user manages and applies the stored "event-message" integrated document through the alarm interface; Step S8: Perform normal storage of the current full network traffic.
2. The method according to claim 1, characterized in that, Step 2 requires static rule-related information, including port information and IP address; dynamic threshold-related information, including traffic volume, traffic rate, number of connections, and packet loss rate; and transmission mode-related information, including communication behavior patterns and clustering models.
3. The method according to claim 1, characterized in that, When detecting alarm events in step S3, preset detection values are set based on static rules, dynamic thresholds, transmission modes, etc. When the relevant information being parsed contains the preset detection values, it is determined that an alarm event has been detected.
4. The method according to claim 3, characterized in that, Preset detection values from the perspective of static rules include: malicious IP and port rules; preset detection values from the perspective of dynamic thresholds include traffic surge, traffic rate threshold, connection number threshold, and packet loss rate threshold; preset detection values from the perspective of transmission patterns include abnormal transmission patterns, abnormal clustering patterns, or suspicious communication behaviors detected and identified based on machine learning models.
5. The method according to claim 1, characterized in that, In step S4, the time window is predetermined, using either a fixed time or a custom time.
6. The method according to claim 1, characterized in that, The predetermined time window in step S4 is ±60 seconds or ±120 seconds.
7. The method according to claim 1, characterized in that, When storing independently in step S6, the message file is stored in PCAP format; when storing independently, the event ID, time range, storage path, and number of associated messages are recorded.
8. The method according to claim 1, characterized in that, When managing the integrated "event-message" document in step S7, the strategies include: default retention period; manual marking by users; compliant archiving when security and compliance requirements are met; and deletion as needed.
9. The method according to claim 1, characterized in that, Step S7 involves applying the integrated "event-message" document, including: When a specific event is clicked, the system displays the communication objects, session data, and raw message details related to that event, and also supports downloading the raw message data packets. The alarm interface displays basic alarm information, including time, source IP, destination IP, protocol type, and triggering rule; Users can expand to view the raw message data, including session statistics, 5-tuple information, and packet details.
10. A raw message intelligent storage system based on alarm events, characterized in that, The system includes: a traffic acquisition module, an information parsing module, an alarm event determination module, a packet interception module, a binding module, a storage module, an application module, and a normal storage module; among which, The traffic acquisition module is used to deploy a bypass acquisition device at the network egress point to mirror the acquisition of all network traffic. The information parsing module is used to parse the current full network traffic and extract information related to static rules, dynamic thresholds, and transmission modes. The alarm event determination module is used to detect alarm events on the current full network traffic based on the parsed relevant information. The detection content includes static rule detection, dynamic threshold detection and / or learning model detection. When the detection result finds an alarm event, the packet interception module is activated; otherwise, the normal storage module is activated. The message interception module is used to intercept the original messages within a predetermined time window from the first predetermined time before the current alarm event to the second predetermined time after the current alarm event, and use them as critical event messages. The binding module is used to generate a unique ID for the current alarm event and bind it with the key event message to generate an integrated "event-message" document; The storage module is used to independently store the integrated "event-message" document and configure a long-term storage mode as needed; The application module is used by users to manage and apply the stored integrated "event-message" document through the alarm interface; The normal storage module is used to store the current full network traffic in a normal state.