Shared memory-based network flow collector multipath distribution method and system

Abstract

The invention provides a shared memory-based network flow collector multi-path distribution method, which is characterized by comprising the following steps of: S1, initializing a shared memory; s2, receiving and storing a data packet into the shared memory; and S3, reading the data packet, releasing the shared memory, and re-executing the step S2 until the task is finished. According to the invention, the data packet is stored at one time through the shared memory mechanism and the network flow collector, and a plurality of processes can directly read the data from the shared memory, so that the repeated transmission of the data packet in the traditional method is avoided, and the frequent switching between the kernel mode and the user mode is reduced, thereby obviously improving the data transmission efficiency and improving the user experience. And the performance overhead of the system is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular, relates to a multi-channel distribution method and system for a network flow collector based on shared memory. Background Art

[0002] With the rapid development of the Internet and the popularization of network applications, the scale and complexity of network traffic have exploded. For scenarios with increasing demands for network monitoring, data analysis, and traffic control, network traffic collectors, as basic tools, undertake the task of real-time collection and processing of data packets.

[0003] At present, in practical applications, network traffic collectors usually need to distribute the same collected data packet to multiple processes for further processing and analysis by different processes. Traditional network traffic collector multi-channel distribution methods rely on mechanisms such as network sockets, pipes, or message queues to transfer data packets between multiple processes so that each process receives the required network traffic data. However, these methods have some significant shortcomings and limitations.

[0004] First, these multiplex distribution methods often require frequent switching between kernel mode and user mode, which increases the overhead of data transmission. Especially when data packets need to be sent to multiple processes, repeated transmission will lead to unnecessary waste of system resources, further affecting the efficiency of data processing.

[0005] Secondly, when multiple processes need to obtain data packets from the traffic collector, if the method of sending them one by one is adopted, the traffic collector needs to create an independent data transmission channel for each process, resulting in a large amount of system resources such as CPU and memory being occupied, and the overall performance of the system is affected. In some high-real-time scenarios, the traffic collector needs to ensure that multiple processes can receive the same data packet simultaneously and quickly. Traditional methods have latency problems in this regard and cannot meet the synchronization requirements of multiple processes.

[0006] Finally, as network traffic and processing requirements increase, traffic collectors need to interact with more processes. Whenever a new process needs to be added, a new data transmission channel usually needs to be established for the process, increasing the complexity and development cost of the system. Especially in the scenario of large-scale concurrent processing, system scalability faces great challenges and it is difficult to flexibly adapt to changing needs. This expansion difficulty limits the scalability and flexibility of the system, resulting in the inefficiency of traditional methods when processing larger-scale traffic.

[0007] To sum up, when the network traffic collector needs to distribute data packets to multiple different processes, there are obvious performance problems and expansion difficulties, namely, low network traffic collection efficiency, high resource consumption, high system expansion complexity and difficulty in multiple processes accessing the same data packet simultaneously, which need to be solved urgently. Summary of the invention

[0008] In view of the defects in the prior art, the object of the present invention is to provide a method and system for multi-channel distribution of network traffic collectors based on shared memory.

[0009] A shared memory-based network traffic collector multi-channel distribution method according to the present invention includes:

[0010] Step S1: Initialize shared memory;

[0011] Step S2: receiving and storing data packets into the shared memory;

[0012] Step S3: read the data packet, release the shared memory, and re-execute step S2 until the task is completed.

[0013] Preferably, in the step S1, identity authentication is performed according to the identifier to determine whether the identity authentication is passed. If the result is yes, the qualification of shared memory access permission is obtained, and then the shared memory is initialized; if the result is no, initialization of the shared memory is prohibited;

[0014] The identification includes: a process ID or a key.

[0015] Preferably, the step S2 includes:

[0016] Step S2.1: Collect data packets and traverse the index table; according to the index table, search for free storage units in the shared memory;

[0017] Step S2.2: Acquire the write lock of the idle storage unit, and store the data packet in the idle storage unit;

[0018] Step S2.3: Release the write lock to allow other processes to read the contents of the storage unit.

[0019] Preferably, in step S3, a read lock of an index table of a shared memory is acquired, and the data packet is read according to the index table of the shared memory; the index table records metadata information of all the data packets; the metadata information includes storage location, storage size and reference count; the reference count is zero, indicating that all the data packets have been read.

[0020] A network traffic collector multi-channel distribution system based on shared memory provided by the present invention includes:

[0021] Module M1: Initialize shared memory;

[0022] Module M2: receiving and storing data packets into the shared memory;

[0023] Module M3: reads the data packet, releases the shared memory, and re-triggers module M2 until the task is completed.

[0024] Preferably, in the module M1, identity authentication is performed according to the identifier to determine whether the identity authentication is passed. If the result is yes, the qualification of shared memory access permission is obtained, and then the shared memory is initialized; if the result is no, initialization of the shared memory is prohibited;

[0025] The identification includes: a process ID or a key.

[0026] Preferably, the module M2 includes:

[0027] Module M2.1: collect data packets, traverse the index table; according to the index table, search for free storage units in the shared memory;

[0028] Module M2.2: Acquire the write lock of the idle storage unit, and store the data packet in the idle storage unit;

[0029] Module M2.3: Release the write lock, allowing other processes to read the contents of the storage unit.

[0030] Preferably, in the module M3, the read lock of the index table of the shared memory is collected, and the data packet is read according to the index table of the shared memory; the index table records the metadata information of all the data packets; the metadata information includes the storage location, storage size and reference count; the reference count is zero, indicating that all the data packets have been read.

[0031] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the multi-channel distribution method of the network traffic collector based on shared memory are implemented.

[0032] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-channel distribution method of the network traffic collector based on shared memory are implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention uses a shared memory mechanism, and the network traffic collector stores data packets at one time. Multiple processes can directly read data from the shared memory, avoiding repeated transmission of data packets in traditional methods and reducing frequent switching between kernel state and user state, thereby significantly improving data transmission efficiency and reducing system performance overhead.

[0035] 2. The present invention does not need to create a new data transmission channel for a new process when expanding it. The new process can directly access the shared memory, thereby simplifying the expansion process and significantly reducing the complexity of system expansion. In other words, the present invention can maintain high scalability and flexibility when facing more concurrent process requirements.

[0036] 3. The present invention uses a shared memory mechanism so that all processes can read data packets from the same memory area at the same time, ensuring that data packets can be accessed synchronously between multiple processes, avoiding the delay problem in traditional systems, ensuring that each process can obtain the latest network data packets in real time, and improving the real-time performance of the system.

[0037] 4. The present invention uses a memory allocation and recovery mechanism to effectively avoid the waste of memory resources and ensure the efficient use of shared memory; and the present invention dynamically releases memory according to the usage of data packets, reduces the memory fragmentation problem, and ensures the continuous and stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 A schematic diagram of the process provided by the present invention;

[0040] Figure 2 A schematic diagram of the framework provided by the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0042] The present invention uses a shared memory mechanism to implement network traffic distribution, reducing the data transmission overhead between processes and avoiding repeated data packet transmission. Specifically, the traffic collector stores the collected network data packets in a shared memory area, and all processes that need to receive data packets can directly read data from the memory area. The shared memory mechanism can significantly reduce the frequent switching between kernel state and user state, and improve system performance.

[0043] In the use of shared memory, in order to prevent excessive occupation of shared memory, the present invention will adopt a memory allocation and recycling mechanism to dynamically allocate memory space according to actual needs, and release the unused memory area in time after the data packet is read by all processes. In order to further improve the system performance, a cache mechanism is introduced, that is, the cache is used to store the recently frequently used data packets, avoiding repeated memory allocation and release operations, and reducing the memory fragmentation of the system.

[0044] Traditional systems need to create independent data transmission channels for each process when adding a new process, which increases system complexity and development costs. The present invention uses shared memory to allow new processes to directly read data packets in the shared memory without the need to establish additional transmission channels. By designing a unified shared memory access interface, the process joining and exiting operations are simplified, thereby reducing the complexity of system expansion.

[0045] The traditional network traffic collector system will encounter large delays when distributing multi-process data, especially in high-concurrency scenarios, where synchronization and communication between processes often become bottlenecks. To solve this problem, the present invention implements an efficient memory access and data scheduling strategy, which reduces data copying and transmission time through pre-allocated shared memory, ensuring that data packets can be delivered to each process with the lowest delay. In addition, the present invention uses memory recycling and caching strategies to ensure that the traffic collector can stably process network traffic data.

[0046] In summary, the technical problems to be solved by the present invention are embodied in the following aspects:

[0047] 1. When the current traffic collector system performs multi-channel distribution, additional performance overhead is caused due to frequent switching between kernel state and user state, as well as repeated transmission of data packets. The present invention realizes one-time storage and multi-process sharing of data packets through a shared memory mechanism, reduces the system's context switching and repeated transmission, and solves the performance bottleneck problem.

[0048] 2. The current traffic collector system needs to establish an independent data transmission channel for each process when communicating between processes, which leads to high consumption of CPU and memory. The present invention avoids the need to create an independent transmission channel for each process by adopting a shared memory approach, greatly reduces the occupation of system resources, and achieves more efficient resource utilization.

[0049] 3. In the current traffic collector system, the need for multiple processes to receive the same data packet at the same time often encounters delay and synchronization problems. The present invention uses a shared memory method so that all processes can read data packets from the memory at the same time, ensuring the synchronization of data distribution and greatly improving real-time performance.

[0050] 4. When the current flow collector is expanded, each time a new process is added, a new data transmission channel needs to be established, which increases the complexity of the system. The present invention uses shared memory to enable new processes to directly access shared data without the need to establish additional independent channels, thereby simplifying the expansion process and significantly reducing the development difficulty and cost of system expansion.

[0051] According to a shared memory-based network traffic collector multi-channel distribution method provided by the present invention, the execution steps of the software system thereof include:

[0052] Step S1: Initialize shared memory;

[0053] Step S2: Acquire and store the data packet into the shared memory;

[0054] Step S3: read the data packet, release the shared memory, and re-execute step S2 until the task is completed.

[0055] Specifically, in step S1, the permission to access the shared memory is allocated to each process, which is specifically implemented as follows: when the system starts the initialization shared memory phase, the identity authentication function is used. When a process attempts to access the shared memory, first, identity authentication is required, and the process needs to provide a pre-configured unique identifier; the identifier includes a process ID or a key; during the authentication phase, it is compared and verified with a list of legal process identifiers stored in the system. Only processes that pass the identity authentication are eligible to further obtain shared memory access rights.

[0056] Specifically, in step S2, combined with the initialization content of step S1, the data packets are sequentially stored in the shared memory as follows: when the network traffic collector captures the data packet, first, the index table is traversed to find the storage unit with an idle state. Starting from the starting position of the index table, the search is carried out in sequence until an idle storage unit is found. This search process is implemented by checking the storage status field of the index table entry.

[0057] After finding a free storage unit, the network traffic collector applies to the memory lock mechanism for a write lock on the memory area where the storage unit is located. After obtaining the memory write lock, the collector stores the data packet in the selected free storage unit, then updates the status information of the storage unit and the metadata associated with it, and finally releases the write lock so that other processes can confirm whether to read the content of the storage unit;

[0058] Specifically, in step S3, combined with the metadata obtained in step S2, the data packets are obtained in an orderly and consistent manner, which is specifically achieved as follows: In order to enable multiple processes to obtain data packets in an orderly and consistent manner, an index table is maintained in the shared memory in step S1. The index table records the metadata information of all data packets, including the storage location, storage size and current reference count of the data packets. The index table is organized in the order in which the data packets are stored, so that the order will not be disordered when the process reads the data packets, and the reading order is guaranteed to be consistent with the capture and storage order of the data packets.

[0059] Each process can identify unprocessed data packets by consulting the index table, so that the process can obtain the next data packet to be read, avoiding omission or duplication of processing. To prevent multiple processes from modifying the index table or accessing the same data packet at the same time, the system uses read-write locks to control access to the index table and data packets. When a process needs to read a data packet, it first obtains a read lock on the index table to safely read the metadata. This ensures that the index table will not be modified by other processes during the reading process.

[0060] After reading is completed, the process will update the reference count and reduce it by 1. When the reference count drops to zero, it means that all processes have read the data packet and the system can reclaim the corresponding memory space.

[0061] The above mechanism ensures the orderliness and consistency of data packet reading. In addition, the combination of reference counting and read-write locks ensures data security and effective use of memory resources in a multi-process environment.

[0062] Specifically, in step S3, the new process can directly access the shared memory, which is specifically achieved by initializing the access rights of the shared memory, allowing the new process to access the data packets of the traffic collector through the shared memory interface.

[0063] During the system initialization phase, i.e., step S1, the system sets a unified access interface and permission management for the shared memory area. After determining the access rights, the subsequently joined processes find the identifier, i.e., the shared memory identifier, through the provided interface. Specifically, the new process maps the shared memory area to its own address space without establishing a new data transmission channel. It can access the data packets stored in the shared memory by the network traffic collector simply through the defined shared memory access interface.

[0064] Specifically, regarding exception handling and fault recovery, the solution is as follows: when a process has an exception, such as no response, crash or abnormal exit, or a shared memory failure, such as lock or data packet not released, the system's exception monitoring module will detect the abnormal state and use the process heartbeat mechanism to detect whether the process is running normally, or monitor whether the memory lock has not been released for a long time through the status flag in the shared memory control structure.

[0065] Then, the system quickly isolates the abnormal process or memory area to prevent the fault from spreading. For abnormal processes, the system will actively terminate their access to shared memory, remove them from the access permission table, and release the memory locks they occupy; for unreleased memory blocks, the system determines whether they can be safely recycled based on reference counts, avoiding system blockages caused by resource occupation and ensuring stability and continuity in a multi-process environment.

[0066] Specifically, step 1: system initialization; when the system starts, first initialize the shared memory, allocate a piece of memory for all processes to use, ensure that multiple processes can exchange and access data through the shared memory area, and set up index tables, memory locks and reference counting control mechanisms to manage data access and storage;

[0067] The network traffic collector stores the data packets at one time;

[0068] In other words, the shared memory initialization module is triggered; when the system starts, a shared memory area is created for use by multiple processes. This module is responsible for allocating a fixed-size shared memory area and using index tables, reference counts, and memory lock initialization to manage the control structure of the shared memory. The shared memory initialization module also allocates permissions for each process to access the shared memory to ensure subsequent concurrent access.

[0069] Step 2: Traffic collector: The network traffic collector captures network traffic, continuously obtains data packets from the network, and temporarily stores the data packets in the shared memory. The data packets are stored in the shared memory in order, and the metadata of each data packet is recorded to prepare for subsequent access; the metadata includes: storage location, data size.

[0070] In other words, the packet storage module is triggered; the network traffic collector stores the captured network packets in the shared memory. This module is responsible for writing the network packets to a specific location in the shared memory, updating the corresponding index information in the memory control structure, and recording the storage location, size, and process access of the packets.

[0071] Step 3: Process read preparation; multiple processes in the system start reading data packets stored in shared memory according to demand. In order to ensure the consistency of data access between processes, a synchronization mechanism is used to control the order of reading, which can ensure that each process can obtain data packets in an orderly and consistent manner without competition or data corruption;

[0072] In other words, the process synchronization module is triggered; when multiple processes read data packets from the shared memory at the same time, in order to ensure data consistency and synchronization, the module adopts a synchronization mechanism to ensure that only one process can modify the shared memory control structure at any time.

[0073] Step 4: Data packet processing and memory recycling: When a process successfully reads a data packet and completes processing, the system will track which processes have read the data packet. Through the reference counting mechanism, when all processes have finished reading the data packet, the memory recycling module will mark this part of the memory as available, freeing up space for new data packets to store, avoiding memory waste;

[0074] In other words, the packet reading module is triggered; each process reads the network packet from the shared memory according to its own needs. The module first finds the storage location of the packet through the index table of the shared memory, and then reads the data from that location. In order to avoid data competition, the reading module can control the access order of the process through the locking mechanism to ensure that each process can successfully read the data;

[0075] The memory recycling module is triggered; when all processes have finished reading the data packet, the module is responsible for recycling the corresponding shared memory space and marking it as free for new data packet storage. The module automatically releases the relevant memory area when the reference count of the data packet drops to zero, ensuring efficient use of memory resources.

[0076] Step 5: New process joins; if a new process needs to join to process data packets, the new process can quickly join the data processing flow after finding the shared memory through the shared memory access interface and directly access the existing shared memory;

[0077] In other words, the extended process management module is triggered; when a new process joins the system, the module is responsible for initializing the new process's access rights to the shared memory. The extended process does not need to establish a new data transmission channel, and can directly access the data packets of the traffic collector through the shared memory interface. This module manages concurrent access of multiple processes through a unified memory access interface to achieve system scalability and flexibility.

[0078] Step 6: Exception handling and fault recovery: During the operation of the system, if a process has an exception or crashes, the exception handling module will respond quickly, isolate the problem process, prevent it from affecting the normal operation of other processes, and ensure that the data in the shared memory can still be read by other normal running processes;

[0079] In other words, the exception handling and fault recovery module is triggered; this module is responsible for monitoring the status of shared memory and processes during system operation. When a process is abnormal or crashes, the exception handling module isolates the impact of the process to prevent data packet loss or memory lock problems. The fault recovery module monitors the status of shared memory to ensure that other processes in the system can continue to read data normally, and attempts to restart or recover the operation of the abnormal process.

[0080] Through this, the present invention improves the efficiency of network traffic collection, reduces resource consumption, solves the need for multiple processes to synchronously access the same data packet, reduces the complexity of system expansion, improves data transmission efficiency, and reduces system resource overhead.

[0081] The present invention also provides a shared memory-based network traffic collector multi-channel distribution system, which can be implemented by executing the process steps of the shared memory-based network traffic collector multi-channel distribution method, that is, those skilled in the art can understand the shared memory-based network traffic collector multi-channel distribution method as a preferred implementation of the shared memory-based network traffic collector multi-channel distribution system.

[0082] A network traffic collector multi-channel distribution system based on shared memory provided by the present invention includes:

[0083] Module M1: Initialize shared memory;

[0084] Module M2: receiving and storing data packets into the shared memory;

[0085] Module M3: reads the data packet, releases the shared memory, and re-triggers module M2 until the task is completed.

[0086] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the multi-channel distribution method of the network traffic collector based on shared memory are implemented.

[0087] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the multi-channel distribution method of the network traffic collector based on shared memory are implemented.

[0088] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0089] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A network traffic collector multi-channel distribution method based on shared memory, characterized in that: include: Step S1: Initialize shared memory; Step S2: receiving and storing data packets into the shared memory; Step S3: read the data packet, release the shared memory, and re-execute step S2 until the task is completed.

2. The shared memory-based network traffic collector multi-channel distribution method according to claim 1 is characterized in that: In the step S1, identity authentication is performed according to the identifier to determine whether the identity authentication is passed, and if the result is yes, the qualification of shared memory access permission is obtained, and then the shared memory is initialized; If the result is no, initialization of shared memory is prohibited; The identification includes: a process ID or a key.

3. The shared memory-based network traffic collector multi-channel distribution method according to claim 1, characterized in that: In the step S2, it includes: Step S2.1: Collect data packets and traverse the index table; according to the index table, search for free storage units in the shared memory; Step S2.2: Acquire the write lock of the idle storage unit, and store the data packet in the idle storage unit; Step S2.3: Release the write lock to allow other processes to read the contents of the storage unit.

4. The shared memory-based network traffic collector multi-channel distribution method according to claim 3 is characterized in that: In the step S3, a read lock of an index table of a shared memory is acquired, and the data packet is read according to the index table of the shared memory; the index table records metadata information of all the data packets; The metadata information includes storage location, storage size and reference count; The reference count is zero, indicating that all the data packets have been read.

5. A network traffic collector multi-channel distribution system based on shared memory, characterized in that: include: Module M1: Initialize shared memory; Module M2: receiving and storing data packets into the shared memory; Module M3: reads the data packet, releases the shared memory, and re-triggers module M2 until the task is completed.

6. The network traffic collector multi-channel distribution system based on shared memory according to claim 5, characterized in that: In the module M1, identity authentication is performed according to the identifier to determine whether the identity authentication is passed. If the result is yes, the qualification of shared memory access permission is obtained, and then the shared memory is initialized; If the result is no, initialization of shared memory is prohibited; The identification includes: a process ID or a key.

7. The network traffic collector multi-channel distribution system based on shared memory according to claim 6, characterized in that: The module M2 includes: Module M2.1: collect data packets, traverse the index table; according to the index table, search for free storage units in the shared memory; Module M2.2: Acquire the write lock of the idle storage unit, and store the data packet in the idle storage unit; Module M2.3: Release the write lock, allowing other processes to read the contents of the storage unit.

8. The network traffic collector multi-channel distribution system based on shared memory according to claim 7, characterized in that: In the module M3, a read lock of an index table of a shared memory is acquired, and the data packet is read according to the index table of the shared memory; the index table records metadata information of all the data packets; The metadata information includes storage location, storage size and reference count; The reference count is zero, indicating that all the data packets have been read.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the shared memory-based network traffic collector multi-channel distribution method described in any one of claims 1 to 4 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the shared memory-based network traffic collector multi-channel distribution method described in any one of claims 1 to 4 are implemented.

Images