Network and signal information collection and processing small file aggregation storage method and system

Abstract

The application discloses a network and signal information collection processing small file aggregation storage method and system, the method comprises the following steps: setting a buffer in an application layer, and aggregating multiple small files to the buffer; when the data volume of the buffer reaches a preset threshold, sequentially writing the aggregated data into a temporary aggregation file of an underlying file system at one time; creating an independent file object for each small file, and mapping the data of the small file to a corresponding data segment in the temporary aggregation file by using a cross-file data mapping interface of the file system, so that the small files share the physical data block of the temporary aggregation file; and the system is used for implementing the method. By intelligent buffering and aggregation of the application layer, the application converts massive small file writing into a small amount of continuous block writing, and realizes multi-file sharing of physical data by using the data mapping function of the file system, so that the performance, storage efficiency and compatibility are considered on the premise of not modifying the underlying file system and not changing the upper application interface.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for aggregating and storing small files in network and signal intelligence acquisition and processing. Background Technology

[0002] In network and signals intelligence gathering systems, front-end detection equipment (such as network traffic probes and radio monitoring stations) typically generate massive amounts of raw data fragments at extremely high frequencies and with very fine time granularity. These data fragments include, but are not limited to: raw network protocol packet capture files (such as pcap and pcapng formats), protocol session slices segmented according to specific rules, radio frequency signal fragment files at specific time windows or frequency points, and intermediate result files generated during modulation and demodulation. A common characteristic of these files is that individual files are small in size (ranging from a few KB to several hundred KB), but the cumulative number of files is extremely large, constituting a typical small file-intensive business scenario.

[0003] With the expansion of the monitoring scope, the improvement of data collection accuracy, and the requirement for continuous 24 / 7 operation, the number of small files generated by the system daily is growing exponentially. This data characteristic presents significant challenges to the backend storage and processing system, specifically in the following aspects: When traditional file systems handle the continuous writing of massive numbers of small files, each file write operation involves an independent I / O request. This results in a large number of random small block write operations, leading to huge disk addressing overhead. Furthermore, each write operation requires executing a complete file system call process, causing the number of system calls to increase exponentially. Frequent system context switching and metadata operations cause a sharp drop in overall I / O throughput and a significant increase in write latency. For scenarios with stringent real-time requirements, such as network and signals intelligence processing, write performance bottlenecks directly lead to data backlog, affecting the timeliness of subsequent protocol parsing, signal reconstruction, and analysis, making it difficult to meet the real-time processing requirements of the intelligence gathering chain.

[0004] The dispersed storage of massive numbers of small files can lead to severe storage fragmentation. Numerous discontinuous small data blocks scattered across the physical storage medium violate the principle of locality of reference. This not only impacts efficiency during writes but, more seriously, affects subsequent intelligence processing. When upper-layer protocol decoders, signal reconstruction modules, or analysis tools need to sequentially read, scan, or correlate these files, the disk head must frequently jump, degrading what could have been a sequential read into an inefficient random read, severely hindering overall processing capabilities. Simultaneously, the file system needs to maintain independent inodes, directory entries, and other metadata for each small file. The sheer volume of small files causes the metadata table to swell dramatically, consuming significant amounts of memory and disk space. Metadata operations themselves become performance bottlenecks; for example, operations such as listing directories and querying file status (stat) become extremely slow, further reducing system access efficiency.

[0005] While the industry has made some optimization attempts to address the small file issue, these approaches all have significant limitations and are difficult to directly apply to intelligence gathering systems. One approach is application-layer encapsulation, such as packaging multiple small files into custom container formats like TFRecord or SequenceFile. While this method aggregates I / O, it requires both the reader and writer to use dedicated APIs or parsing libraries, breaking the standard POSIX file system interface contract. Many existing toolchains in intelligence processing systems (such as Wireshark, tcpdump, and dedicated protocol analysis software) rely on standard system calls like open, read, and write. Modifying these tools to adapt to dedicated formats is costly and sometimes impractical. Another approach involves modifying the underlying file system, such as optimizing the inode structure for small files, implementing internal log structuring, or appending and merging. While this approach can achieve better performance, it is extremely complex and heavily reliant on specific kernel versions or storage architectures. It struggles to achieve unified and convenient deployment and maintenance across heterogeneous, multi-model front-end acquisition devices and diverse deployment platforms (such as physical machines, virtual machines, and containers), resulting in poor universality.

[0006] Therefore, without modifying the underlying file system or changing the access methods of upper-layer applications, researching an aggregated storage method that can effectively improve small file write performance, reduce storage fragmentation, reduce metadata overhead, and seamlessly integrate into existing intelligence processing workflows has significant practical application value and is also a key technical need that urgently needs to be addressed in this field. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and system for small file aggregation and storage in network and signal intelligence collection and processing, which can significantly improve the small file read and write performance, reduce storage fragmentation and metadata overhead, and maintain full transparent compatibility with standard file system interfaces.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the method of the present invention includes the following steps: S1, Set up a buffer at the application layer to aggregate the data content of multiple small files to be stored into the buffer; S2, when the amount of data in the buffer reaches a preset threshold, the aggregated data is sequentially written to a temporary aggregate file of the underlying file system in one go; S3, create an independent file object for each small file, and use the cross-file data mapping interface provided by the file system to map the data range of each small file to the corresponding data segment in the temporary aggregate file, so that multiple small files share the physical data block of the temporary aggregate file.

[0009] As a further improvement to the method of the present invention, in step S1, the small file includes a network protocol raw data file, a protocol slice file, or a signal fragment file.

[0010] As a further improvement to the method of the present invention, in step S2, the aggregated data is written to the temporary aggregated file through the standard open and write file system interface.

[0011] As a further improvement to the method of the present invention, in step S3, the independent file object is created by using the open and O_CREAT flags.

[0012] As a further improvement to the method of the present invention, in step S3, the cross-file data mapping interface is the copy_file_range system call, which does not perform actual data copying during its execution process, but only establishes a reference relationship from the small file to the specified data range in the temporary aggregate file at the metadata layer.

[0013] As a further improvement to the method of the present invention, after step S3, step S4 is also included: deleting the temporary aggregate file; wherein, the file system in step S3 supports a reference counting and delayed cleanup mechanism to ensure that after the temporary aggregate file is deleted, the physical data blocks referenced by it continue to provide data access for each small file.

[0014] As a further improvement to the method of the present invention, after the small file is mapped, it exists in the file system as a real file with complete POSIX semantics, and the application can access and manage it through the standard open, read, write, stat, and unlink interfaces.

[0015] The present invention also provides a small file aggregation and storage system for network and signal intelligence acquisition and processing, comprising a processor and a computer-readable storage medium interconnected with each other, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the above-described method for small file aggregation and storage in network and signal intelligence acquisition and processing.

[0016] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for aggregating and storing small files in network and signal intelligence acquisition and processing.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-described method for aggregating and storing small files in network and signal intelligence acquisition and processing.

[0018] Compared with the prior art, the advantages of the present invention are as follows: This invention transforms massive random small file writes into a small number of continuous large block writes through intelligent buffering and aggregation at the application layer, and enables multiple files to share the same physical data through the data mapping function of the file system. Thus, without modifying the underlying file system or changing the upper-layer application interface, it perfectly balances high performance, high storage efficiency and excellent compatibility. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0021] The technical solution adopted in this embodiment is as follows: Figure 1 As shown, it includes the following steps: S1, Set up a buffer at the application layer to aggregate the data content of multiple small files to be stored into the buffer; S2, when the amount of data in the buffer reaches a preset threshold, the aggregated data is sequentially written to a temporary aggregate file of the underlying file system in one go; S3, create an independent file object for each small file, and use the cross-file data mapping interface provided by the file system to map the data range of each small file to the corresponding data segment in the temporary aggregate file, so that multiple small files share the physical data block of the temporary aggregate file.

[0022] In specific implementation applications, in step S1, the small file includes network protocol raw data file, protocol slice file, or signal fragment file.

[0023] In specific implementation applications, in step S2, the aggregated data is written to the temporary aggregated file through the standard open and write file system interfaces.

[0024] In specific implementation applications, in step S3, the independent file object is created by using the open and O_CREAT flags.

[0025] In specific implementation applications, in step S3, the cross-file data mapping interface is the copy_file_range system call. Its execution process does not perform actual data copying, but only establishes a reference relationship from the small file to the specified data range in the temporary aggregate file at the metadata layer.

[0026] In specific implementation applications, after step S3, step S4 is also included: deleting the temporary aggregate file; wherein, the file system in step S3 supports reference counting and delayed cleanup mechanisms to ensure that after the temporary aggregate file is deleted, the physical data blocks referenced by it continue to provide data access for each small file.

[0027] In practical applications, after the small file is mapped, it exists in the file system as a real file with complete POSIX semantics. The application can access and manage it through the standard open, read, write, stat, and unlink interfaces.

[0028] The specific implementation process of this embodiment is as follows: In this embodiment, the front-end probe generates thousands of small pcap slice files per second (average size approximately 50KB). To efficiently store these files and provide them directly for use by subsequent protocol analysis tools (such as Wireshark and Bro / Zeek), an aggregation storage component called "FileAggregator" is deployed at the application layer. This component runs in user space and intercepts file write requests from the application.

[0029] Step S1: Buffer Aggregation The "FileAggregator" component initializes a fixed-size circular buffer of 4MB in memory (this size can be adjusted based on physical memory and typical file size, serving as a "preset threshold"). Simultaneously, a temporary file is pre-created in the JuiceFS file system to store the aggregated data, for example, at the path / data / aggregation_pool / aggregate_ <timestamp>.tmp.

[0030] When an application calls the write system call in an attempt to write a small pcap slice file (such as flow_172.16.1.100_20231027_120100.pcap), the call is intercepted by the "FileAggregator" component.

[0031] The component does not immediately write the file data to disk. Instead, it appends the file's data content, its predetermined final storage path (e.g., / captures / flow_172.16.1.100_20231027_120100.pcap), size, and other metadata to a memory buffer. The buffer maintains a metadata table that records the offset, length, and target path of each small file within the buffer.

[0032] This process is repeated, with subsequent small file data being continuously appended to the same buffer. The buffer management module continuously checks the used space.

[0033] Step S2: Batch writing A batch write operation is triggered when the amount of data accumulated in the buffer reaches the 4MB threshold, or when the component receives an explicit refresh instruction (such as when the processing process is closed normally).

[0034] The "FileAggregator" component uses standard open and write system calls to write the entire 4MB buffer of data as a contiguous block to a pre-prepared temporary aggregation file (aggregate_) in a single, sequential manner. <timestamp>In the .tmp file. This write is a large, continuous I / O operation, which reduces the number of system calls and disk addressing overhead to a very low level compared to directly writing thousands of independent small files.

[0035] After the write operation is complete, you can optionally call fsync to ensure the data is written to disk. At this point, there is only one large file on the disk containing data from multiple smaller files, achieving aggregation in physical storage.

[0036] Step S3: Create files and establish mappings Creating an independent file object: For each small file recorded in the buffer metadata table, the "FileAggregator" component calls the open system call with the O_CREAT flag at its final target path (e.g., / captures / flow_172.16.1.100_20231027_120100.pcap) to create an empty file of size 0. This operation allocates an independent inode, directory entry, and other metadata structure for the small file in the file system, giving it a complete file identity.

[0037] Performing zero-copy data mapping: This is crucial to this approach. Next, the component iterates through each created empty small file. For each small file, the component calls the `copy_file_range` system call, performing the following operations: Source file descriptor: points to the temporary aggregate file aggregate_ <timestamp>.tmp.

[0038] Source offset: Based on the buffer metadata table, locate the starting byte offset of the small file data in the temporary aggregate file (for example, the second file may start at offset 1.5MB).

[0039] Target file descriptor: points to the newly created empty small file (e.g., flow_172.16.1.100_20231027_120100.pcap).

[0040] Copy length: The size of the small file (e.g., 50KB).

[0041] On file systems that support this feature (such as JuiceFS), this `copy_file_range` call does not perform actual data block copying. Its semantics are "establish a data reference relationship from a specified range in the source file to a specified range in the destination file." The system only points the logical data block pointers of small files to the corresponding physical data blocks in the temporary aggregate file at the metadata level. The metadata of multiple small files (File A, B, C...) will point to different data segments within the same temporary aggregate file, achieving physical data sharing.

[0042] Step S4: Clean up temporary containers and enable transparent access Deleting temporary files: After all small files have successfully completed data mapping, the "FileAggregator" component calls the unlink system call to delete the temporary aggregation file aggregate_. <timestamp>.tmp.

[0043] Reference counting guarantee: Because the underlying JuiceFS file system supports reference counting and copy-on-write (CoW) or delayed cleanup mechanisms, when a temporary aggregate file is deleted, the system only removes its directory entry. The underlying data blocks are not immediately released because they are still referenced by the metadata of each smaller file (reference count greater than 0). This ensures data security and continued accessibility.

[0044] Transparent Access: At this point, the entire aggregation storage process is completely transparent to upper-layer applications. At the file system level, each small file (e.g., / captures / flow_172.16.1.100_20231027_120100.pcap) is an independent, POSIX-compliant regular file. Subsequent protocol analysis tools can directly access or manage it through standard interfaces such as open, read, lseek, stat, and unlink, without needing to modify or be aware that its underlying data is shared storage. When reading via read, the file system automatically locates the corresponding data block in the original temporary aggregation file based on its metadata pointer and returns it to the application.

[0045] Through the examples, it can be seen that the advantages of this technical solution are specific and comprehensive, mainly reflected in the following three aspects: First, in terms of performance improvement, the solution sets up a buffer at the application layer to aggregate scattered small file data and write it sequentially at once. This fundamentally transforms a large number of high-overhead random I / O operations into efficient sequential I / O operations, significantly reducing the number of system calls and disk seek time, greatly improving write throughput and reducing latency, thus meeting the real-time requirements of intelligence processing. Second, in terms of storage optimization, the solution uses interfaces such as copy_file_range to implement cross-file data mapping, allowing multiple logically independent small files to share the same physical data block of the temporary aggregated file, avoiding duplicate data storage and saving space. At the same time, the continuous data layout formed by aggregated writing effectively reduces disk fragmentation, improves the efficiency of subsequent sequential reads, and because multiple files share data blocks, the pressure on metadata such as inodes that the file system needs to maintain is also significantly alleviated. Finally, and crucially, compatibility and ease of use are paramount. The entire solution is implemented through standard POSIX file system interfaces (such as open, read, and write), ultimately presenting applications with perfectly normal, independent files. All existing protocol analysis tools and data processing flows at the upper layer can be used seamlessly without any modifications, achieving performance optimization that is completely transparent to the application layer and greatly reducing deployment and integration costs.

[0046] Optimizations and variants: Buffer management can be further optimized, for example, by using double buffers or circular queues, so that while one buffer is performing batch write and mapping operations, the other buffer can continue to receive newly arriving small files, thus achieving pipelined processing and further improving throughput.

[0047] The "preset threshold" is not limited to a fixed size; it can also be dynamically triggered based on a time window (such as refreshing every 100 milliseconds) or the number of files (such as accumulating 1000 files).

[0048] The method in this embodiment is also applicable to other file systems that support similar copy_file_range zero-copy semantics or internal data block sharing mechanisms (such as Btrfs, ZFS, etc.).

[0049] This embodiment also includes a small file aggregation and storage system for network and signal intelligence acquisition and processing, comprising a processor and a computer-readable storage medium interconnected with each other. The computer-readable storage medium stores a computer program, which is executed by the processor to implement the steps of the above-described method for small file aggregation and storage in network and signal intelligence acquisition and processing.

[0050] This embodiment also includes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for aggregating and storing small files in network and signal intelligence acquisition and processing.

[0051] This embodiment also includes a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the above-described method for aggregating and storing small files in network and signal intelligence acquisition and processing.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0053] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.< / timestamp> < / timestamp> < / timestamp> < / timestamp>

Claims

1. A method for aggregating and storing small files in network and signal intelligence acquisition and processing, characterized in that, include: S1, Set up a buffer at the application layer to aggregate the data content of multiple small files to be stored into the buffer; S2, when the amount of data in the buffer reaches a preset threshold, the aggregated data is sequentially written to a temporary aggregate file of the underlying file system in one go; S3. Create an independent file object for each small file, and use the cross-file data mapping interface provided by the file system to map the data range of each small file to the corresponding data segment in the temporary aggregate file, so that multiple small files share the physical data block of the temporary aggregate file. The cross-file data mapping interface is the copy_file_range system call, which does not actually copy data during its execution process, but only establishes a reference relationship from the small file to the specified data range in the temporary aggregate file at the metadata layer.

2. The method for aggregating and storing small files in network and signal intelligence acquisition and processing according to claim 1, characterized in that, In step S1, the small file includes network protocol raw data file, protocol slice file, or signal fragment file.

3. The method for aggregating and storing small files in network and signal intelligence acquisition and processing according to claim 1, characterized in that, In step S2, the aggregated data is written to the temporary aggregated file through the standard open and write file system interfaces.

4. The method for aggregating and storing small files in network and signal intelligence acquisition and processing according to claim 1, characterized in that, In step S3, the independent file object is created using the open and O_CREAT flags.

5. The method for aggregating and storing small files in network and signal intelligence acquisition and processing according to claim 1, characterized in that, Following step S3, step S4 is also included: deleting the temporary aggregate file; wherein, the file system in step S3 supports reference counting and delayed cleanup mechanisms to ensure that after the temporary aggregate file is deleted, the physical data blocks referenced by it continue to provide data access for each small file.

6. The method for aggregating and storing small files in network and signal intelligence acquisition and processing according to claim 1, characterized in that, After the small file is mapped, it exists in the file system as a real file with complete POSIX semantics. The application can access and manage it through the standard open, read, write, stat, and unlink interfaces.

7. A small file aggregation and storage system for network and signal intelligence acquisition and processing, characterized in that, The method includes an interconnected processor and a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the steps of the small file aggregation and storage method for network and signals intelligence acquisition and processing as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the small file aggregation and storage method for network and signal intelligence acquisition and processing as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the steps of the small file aggregation and storage method for network and signal intelligence acquisition and processing as described in any one of claims 1 to 6.

Images