Network message discrete grabbing and dynamic archiving method and system

Abstract

The application discloses a network message discrete capturing and dynamic archiving method and system, and the method comprises the following steps: setting network message capturing parameters; dividing the network message capturing process into multiple discrete capturing periods, screening the network messages meeting preset filtering conditions in each period according to the preset filtering conditions; meanwhile, recording the starting time in each period and dynamically calculating the time difference, automatically exiting and saving the period file when the running time reaches the set duration of the single period, and entering an interval waiting state, starting the next capturing period after the set interval time ends; setting a file size threshold, naming the file according to the time node for query, and controlling the total time length of the saved network messages through the cyclic storage mode, and dynamically archiving the stored network messages. The method is suitable for message capturing in different time periods, and can realize screening and file saving and archiving according to specific IP addresses, specific ports and special message types.

Description

Technical Field

[0001] This invention relates to a method and system for discrete network packet capture and dynamic archiving, belonging to the field of power system automation technology. Background Technology

[0002] In current power automation systems, intelligent communication gateways serve as the core hub between the process layer and the station control layer, undertaking the forwarding and processing of massive amounts of real-time messages such as IEC61850, GOOSE, and SV. Their operational stability directly impacts the reliability of power grid monitoring, protection, and control functions. When abnormal events such as network storms, communication interruptions, or device malfunctions occur, maintenance personnel must rely on the communication messages stored locally on the gateway for accurate fault tracing and cause analysis.

[0003] However, existing embedded gateway packet capture schemes mainly suffer from the following technical bottlenecks: First, they generally adopt a continuous recording mode of "capture first, then store," which causes the packet files to expand rapidly over time. However, the embedded storage resources of the gateway are extremely limited, which can easily lead to the rapid exhaustion of storage space and even affect the normal operation of the system. Second, they lack a dynamic control mechanism for capture time slots and file size, and cannot adaptively adjust the acquisition strategy according to changes in network load, which may result in the loss of packets during critical periods due to storage overflow. Third, traditional schemes only implement simple time or size segmentation and fail to establish an effective archive index structure, resulting in low efficiency in historical packet querying and making it difficult to meet the urgent need for long-term backtracking in power fault analysis.

[0004] Therefore, there is an urgent need for an intelligent packet capture method that can adapt to the resource-constrained environment of embedded gateways. While ensuring the integrity of packets during critical periods, this method can extend the query period of historical packets and improve the operation and maintenance capabilities of power communication networks through discretization capture and automated archiving mechanisms. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for discrete network packet capture and dynamic archiving. By pre-screening packets through a kernel filter to reduce data copying, and then using raw sockets to capture raw data packets at the link layer, the method bypasses the protocol stack to directly parse the packets and finally displays the protocol information or saves them as a PCAP file. Bypass packet capture does not affect normal communication and solves the problem of long-term, traceable packet retention within limited embedded storage resources.

[0006] To achieve the above objectives, the present invention is implemented using the following technical solution.

[0007] In a first aspect, the present invention provides a method for discrete network packet capture and dynamic archiving, comprising:

[0008] Configure network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time.

[0009] The network packet capture process is divided into multiple discrete capture cycles. Within each cycle, network packets that meet the preset filtering conditions are filtered. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the cycle is automatically exited and the cycle file is saved. Then, the cycle enters an interval waiting state and starts the next capture cycle after the set interval time ends.

[0010] Set a file size threshold, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method to dynamically archive stored network packets.

[0011] Furthermore, the network packet capture parameters include the physical network card, IP address, and port number.

[0012] Furthermore, the network packet capture process includes:

[0013] A packet capture request is sent, and the unified interface receives the preset filtering conditions and readable text rules, compiles them into BPF bytecode, and loads them into the kernel.

[0014] Create a PF_PACKET socket to establish a listener between the kernel's network device driver and the network protocol stack;

[0015] The network card receives network data packets, which then reach the kernel network protocol stack.

[0016] The data packets are matched using a BPF filter: successfully matched data packets enter the kernel buffer and wait to be read; unmatched data packets continue to be processed along the normal kernel network transmission path.

[0017] The packet capture process enters a loop through a callback function, continuously reading the matched data packets from the kernel buffer.

[0018] The method of this invention achieves efficient and flexible packet capture by calling the PF_PACKET socket in the kernel and combining it with the BPF filter.

[0019] Furthermore, the runtime is controlled by a timer to control the packet capture duration. The start time is recorded upon startup, and the time difference is continuously checked during the packet capture loop. Once the set duration is reached, the process automatically exits and saves the file.

[0020] Furthermore, the dynamic calculation process of the time difference includes:

[0021] Call the system function to get the current timestamp as the start time, and open the output file at the same time;

[0022] Enter the main packet capture loop: When a data packet is captured, it is processed and written to a file. Before the end of each loop, the current time is retrieved again, the difference between the current time and the start time is calculated, and it is determined whether the set duration has been reached. If no data packet is captured, it is directly determined whether the set duration has been reached.

[0023] When the runtime reaches the set value, the packet capture loop ends, the output file is closed, and the buffer is flushed.

[0024] The method of this invention uses dynamic calculation of time difference without relying on external timers. Instead, it is based on the comparison between the timestamp of the main packet capture loop and the difference, which enables precise control of packet capture duration.

[0025] Furthermore, the dynamic archiving process includes:

[0026] Before each packet capture begins, a unique filename is generated based on the start time to save the network packets to be captured.

[0027] Capture network packets and continuously write the data to the packet capture file being used;

[0028] Determine whether the size of the captured file reaches the file size threshold:

[0029] If the limit is not exceeded, continue writing network packets;

[0030] If the limit has been exceeded, stop writing and close the packet capture file that has exceeded the limit, and determine whether the memory has reached the set memory threshold:

[0031] If the memory has not reached the set memory threshold, i.e., memory is not tight, generate a new filename according to the current time, create a new packet capture file and start capturing;

[0032] If memory reaches the set memory threshold, i.e. memory is tight, old packet capture files will be deleted in a loop according to the preset strategy until memory is restored to normal, and then the above packet capture process will continue.

[0033] The dynamic archiving mechanism employed in this invention enables automatic fragmentation and memory control of captured files, effectively avoiding the problem of excessively large memory sizes in individual files leading to difficulties in subsequent analysis.

[0034] Secondly, the present invention provides a network packet discrete capture and dynamic archiving system, comprising:

[0035] The parameter setting module is used to set network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time.

[0036] The discrete capture module is used to divide the network packet capture process into multiple discrete capture cycles. Within each cycle, the network packets that meet the preset filtering conditions are filtered. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the module automatically exits and saves the cycle file, and enters an interval waiting state. After the set interval time ends, the next capture cycle is started.

[0037] The dynamic archiving module is used to set file size thresholds, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method, thus dynamically archiving stored network packets.

[0038] Thirdly, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the network packet discrete capture and dynamic archiving method described in any of the first aspects.

[0039] Fourthly, the present invention provides a computer device, comprising:

[0040] Memory, used to store computer programs / instructions;

[0041] A processor for executing the computer program / instructions to implement the steps of the network packet discrete capture and dynamic archiving method described in any of the first aspects.

[0042] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0043] 1. The network packet discrete capture and dynamic archiving method provided by this invention accurately controls the packet capture duration by recording the start time and dynamically calculating the time difference in the packet capture loop, thereby achieving precise discrete capture of network packets; it compiles user-input text rules into BPF bytecode and loads them into the kernel, matching them before network packets enter the protocol stack, achieving efficient BPF filtering, significantly reducing the overhead of uploading and processing irrelevant packets, and improving overall packet capture performance; at the same time, it can automatically fragment according to file size thresholds and cyclically delete old files when memory is tight, completing dynamic archiving;

[0044] The method of this invention enables stable and controllable network packet collection in resource-constrained environments, effectively avoiding the problems of excessively large single files and memory overload, and further improving the flexibility of packet capture.

[0045] 2. The computer-readable storage medium and computer device provided by the present invention can execute the steps of the network packet discrete capture and dynamic archiving method provided by the present invention. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the structure of the network packet discrete capture and dynamic archiving method provided in an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating the network packet data capture process of the network packet discrete capture and dynamic archiving method provided in an embodiment of the present invention.

[0048] Figure 3 This is a flowchart of the time-dynamic processing of the network packet discrete capture and dynamic archiving method provided in an embodiment of the present invention;

[0049] Figure 4 This is a flowchart illustrating the dynamic archiving process of the network packet discrete capture and dynamic archiving method provided in an embodiment of the present invention. Detailed Implementation

[0050] It should be noted that:

[0051] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0052] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0053] Example 1

[0054] like Figure 1 As shown in the figure, this embodiment introduces a method for discrete network packet capture and dynamic archiving, including:

[0055] Configure network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time.

[0056] The network packet capture process is divided into multiple discrete capture cycles. Within each cycle, network packets that meet the preset filtering conditions are filtered. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the cycle is automatically exited and the cycle file is saved. Then, the cycle enters an interval waiting state and starts the next capture cycle after the set interval time ends.

[0057] Set a file size threshold, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method to dynamically archive stored network packets.

[0058] In summary, the network packet discrete capture and dynamic archiving method in this embodiment divides the continuous packet capture process into multiple independent discrete capture cycles. Within each cycle, the runtime is precisely controlled using the start time and dynamic time difference. After the cycle ends, the file is automatically saved and enters a set interval waiting state, thus achieving long-term continuous monitoring of network packets. At the same time, by adopting a time-based file naming method and setting a file size threshold and implementing dynamic archiving through cyclic storage, the memory size of a single file is limited, avoiding memory pressure caused by excessively large files.

[0059] Preferably, the network packet capture parameters also include the physical network card, IP address, and port number.

[0060] In this embodiment, the system identifies all available physical network cards on the current machine through the network card interface provided by the system, and allows the user to select a specific physical network card for capture based on the actual network topology;

[0061] By setting the target address parameter, the packet capture range is limited to the packets communicating between a specific host and the outside world, thereby effectively isolating the network behavior of the specific host and eliminating communication interference from other unrelated devices in the network;

[0062] By setting port numbers, traffic can be refined to the specific application protocol level, enabling the separate collection and analysis of service request and response messages using external packet parsing tools, and allowing for more detailed querying of service status.

[0063] Preferred, such as Figure 2 As shown, the network packet capture process includes:

[0064] Using the unified interface provided by the upper-layer application service, network data packets are captured, and a packet capture request is sent. The unified interface receives the preset filtering conditions and readable text rules, compiles them into BPF bytecode, and loads them into the kernel.

[0065] Create a special PF_PACKET socket to capture packets, establish a listener between the kernel's network device driver and the network protocol stack, and achieve early capture of network packets;

[0066] The network card receives network data packets, which then reach the kernel network protocol stack.

[0067] The BPF filter is used to match the data packets: successfully matched data packets enter the kernel buffer and wait to be read; unmatched data packets continue to be processed along the normal kernel network transmission path; this effectively avoids the overhead of uploading irrelevant network packets.

[0068] The packet capture process enters a loop through a callback function, continuously reading the matched data packets from the kernel buffer. The network packet capture method used in this embodiment not only ensures the accuracy of packet capture but also reduces the impact on system performance, providing an efficient and low-interference data acquisition foundation for subsequent discrete periodic control and dynamic archiving.

[0069] Preferably, the runtime is controlled by a timer to control the packet capture duration. The start time is recorded at startup, the time difference is continuously checked during the packet capture loop, and the process automatically exits and saves the file after the set duration is reached.

[0070] Preferred, such as Figure 3 As shown, the dynamic calculation process of the time difference includes:

[0071] Call the system time function to get the current timestamp as the start time, and open the output file to prepare to write the message;

[0072] Enter the main packet capture loop: In the main loop, when a data packet is successfully captured through the interface, it is processed and written to a file, and then written to the output file; after writing is completed, before the end of this loop, the system function is called again to obtain the current time, calculate the difference between it and the start time, and compare it with the set duration. If the set duration is reached, the loop exits; otherwise, the loop continues to the next loop.

[0073] If no data packet is captured in the main loop, the difference between the current time and the start time is directly calculated to determine whether the set duration has been reached. The loop method used in this embodiment effectively avoids the problem of not being able to exit in time when waiting in vain.

[0074] When the runtime reaches the set value, the packet capture loop is immediately terminated, the output file is closed, and the interface is called to refresh the buffer, ensuring that all captured network packets are completely saved to the disk.

[0075] Preferred, such as Figure 4 As shown, the dynamic archiving process includes:

[0076] Before each packet capture begins, a unique filename is generated based on the start time to save the network packets to be captured.

[0077] Capture network packets and continuously write the data to the packet capture file being used;

[0078] Determine whether the size of the captured file reaches the file size threshold:

[0079] If the limit is not exceeded, continue writing network packets;

[0080] If the limit has been exceeded, stop writing and close the packet capture file that has exceeded the limit, and determine whether the memory has reached the set memory threshold:

[0081] If the memory has not reached the set memory threshold, i.e., memory is not tight, generate a new filename according to the current time, create a new packet capture file and start capturing;

[0082] If memory reaches the set memory threshold, i.e. memory is tight, old packet capture files will be deleted in a loop according to the preset strategy until memory is restored to normal, and then the above packet capture process will continue.

[0083] The dynamic archiving method used in this embodiment prevents abnormal packet capture exits due to disk fullness, and also enables long-term packet capture operation in resource-constrained environments.

[0084] Example 2

[0085] Based on the network packet discrete capture and dynamic archiving method described in Embodiment 1, this embodiment introduces a network packet discrete capture and dynamic archiving system, including:

[0086] The parameter setting module is used to set network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time.

[0087] The discrete capture module is used to divide the network packet capture process into multiple discrete capture cycles. Within each cycle, the network packets that meet the preset filtering conditions are filtered. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the module automatically exits and saves the cycle file, and enters an interval waiting state. After the set interval time ends, the next capture cycle is started.

[0088] The dynamic archiving module is used to set file size thresholds, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method, thus dynamically archiving stored network packets.

[0089] Example 3

[0090] Based on the network packet discrete capture and dynamic archiving method described in Embodiment 1, this embodiment introduces a computer-readable storage medium storing a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the steps of the network packet discrete capture and dynamic archiving method as described in any of Embodiment 1.

[0091] Example 4

[0092] Based on the network packet discrete capture and dynamic archiving method described in Embodiment 1, this embodiment provides a computer device, including:

[0093] Memory, used to store computer programs / instructions;

[0094] A processor is configured to execute the computer program / instructions to implement the steps of the network packet discrete capture and dynamic archiving method as described in any one of Embodiment 1.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for discrete network packet capture and dynamic archiving, characterized in that, include: Configure network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time. The network packet capture process is divided into multiple discrete capture cycles. In each capture cycle, network packets are filtered according to the preset filtering conditions to obtain network packets that meet the preset filtering conditions. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the cycle is automatically exited and the cycle file is saved. Then, the cycle enters an interval waiting state and starts the next capture cycle after the set interval time ends. Set a file size threshold, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method to dynamically archive stored network packets.

2. The method for discrete network packet capture and dynamic archiving according to claim 1, characterized in that, The network packet capture parameters also include the physical network card, IP address, and port number.

3. The method for discrete network packet capture and dynamic archiving according to claim 1, characterized in that, The network packet capture process includes: The unified interface receives the preset filtering conditions and readable text rules, compiles them into BPF bytecode, and loads them into the kernel; Create a PF_PACKET socket to establish a listener between the kernel driver and the network protocol stack; The network card receives network data packets and sends them to the kernel network protocol stack; The BPF filter is used to match the data packets: matching packets enter the kernel buffer and wait to be read; non-matching packets continue to be processed along the normal transmission path. The callback function continuously reads the matched data packets from the kernel buffer in a loop.

4. The method for discrete network packet capture and dynamic archiving according to claim 1, characterized in that, The runtime is controlled by a timer.

5. The method for discrete network packet capture and dynamic archiving according to claim 1, characterized in that, The dynamic calculation process of the time difference includes: Call the system function to get the current timestamp as the start time, and open the output file at the same time; Enter the main packet capture loop: When a data packet is captured, it is processed and written to a file. Before the end of each loop, the current time is retrieved again, the difference between the current time and the start time is calculated, and it is determined whether the set duration has been reached. If no data packet is captured, it is directly determined whether the set duration has been reached. When the runtime reaches the set value, the packet capture loop ends, the output file is closed, and the buffer is flushed.

6. The method for discrete network packet capture and dynamic archiving according to claim 1, characterized in that, The dynamic archiving process includes: Before each packet capture begins, a unique filename is generated based on the start time to save the network packets to be captured. Capture network packets and continuously write the data to the packet capture file being used; Determine whether the size of the captured file reaches the file size threshold: If the limit is not exceeded, continue writing network packets; If the limit has been exceeded, stop writing and close the packet capture file that has exceeded the limit, and determine whether the memory has reached the set memory threshold: If the memory has not reached the set memory threshold, i.e., memory is not tight, generate a new filename according to the current time, create a new packet capture file and start capturing; If memory reaches the set memory threshold, i.e. memory is tight, old packet capture files will be deleted in a loop according to the preset strategy until memory is restored to normal, and then the above packet capture process will continue.

7. A system for discrete network packet capture and dynamic archiving, characterized in that, include: The parameter setting module is used to set network packet capture parameters, which include preset filtering conditions, the duration of a single period, and the set interval time. The discrete capture module is used to divide the network packet capture process into multiple discrete capture cycles. Within each cycle, the network packets that meet the preset filtering conditions are filtered. At the same time, the start time of each cycle is recorded and the time difference is dynamically calculated. When the running time reaches the duration set for a single cycle, the module automatically exits and saves the cycle file, and enters an interval waiting state. After the set interval time ends, the next capture cycle is started. The dynamic archiving module is used to set file size thresholds, name files by time nodes for easy querying, and control the total retention time of network packets through a circular storage method, thus dynamically archiving stored network packets.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the network packet discrete capture and dynamic archiving method as described in any one of claims 1 to 6.

9. A computer device, characterized in that, include: Memory, used to store computer programs / instructions; A processor for executing the computer program / instructions to implement the steps of the network packet discrete capture and dynamic archiving method according to any one of claims 1 to 6.

Images