One-way file transfer method based on disk dynamic IO technology

Abstract

The application relates to the technical field of digital information transmission and electric digital data processing, in particular to a one-way file transmission method based on a disk dynamic IO technology. A one-way external terminal encapsulates file data by using a customized protocol, and sends the file data to a one-way internal terminal through a one-way channel, and neither the internal terminal nor the external terminal triggers local disk read-write operation. The one-way internal terminal pushes the data to an internal network receiving server. The receiving server sequentially writes the data into a large cache file according to the optimal write performance block size of the disk, monitors the disk input and output load parameters in real time, and dynamically adjusts the release rate of the final file parsed and restored from the large cache file. The application eliminates the disk input and output bottleneck of the intermediate nodes in the transmission link, reduces the read-write call frequency in the small file scene, dynamically adjusts the release rate according to the load, avoids data loss caused by the lag of the receiving end processing, and guarantees the reliability of the one-way file transmission.

Description

Technical Field

[0001] This invention relates to the field of digital information transmission and electronic digital data processing technology, specifically to a one-way file transfer method based on disk dynamic I / O technology. Background Technology

[0002] Currently, to achieve one-way file transfer in a one-way isolation environment, the conventional approach uses standard network file transfer protocols (such as File Transfer Protocol, Secure Copy Protocol, Secure File Transfer Protocol, and Server Message Block Protocol). The specific implementation process is as follows: The external machine located outside the one-way isolation device retrieves the file from the source file server using the aforementioned standard protocols and caches the file data on its local disk. Subsequently, the one-way isolation device physically delivers the file data from the external machine to the internal machine via a one-way transmission channel. After receiving the data, the internal machine reassembles the file data on its local disk and then pushes the reassembled file to the receiving server on the internal network using the standard file transfer protocol, completing the entire one-way transfer process.

[0003] Based on the aforementioned conventional approach, file data must undergo three disk I / O operations during transmission: one on the external machine's local disk, one on the internal machine's local disk, and finally on the receiving server's disk. Unidirectional isolation devices, acting as intermediate transmission channels, are typically limited by hardware architecture and lack large-capacity storage and independent redundant disk array environments. When faced with scenarios involving the transmission of numerous small or mixed files, the sheer volume of data necessitates extremely frequent disk I / O operations. This severely restricts file transmission performance due to the limited disk I / O processing capacity of the intermediate nodes, leading to a sharp decline in transmission performance. Disk I / O operations thus become the core bottleneck restricting the efficiency of unidirectional file transmission. Summary of the Invention

[0004] The purpose of this invention is to provide a one-way file transfer method based on disk dynamic I / O technology, which can effectively solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] One-way file transfer methods based on disk dynamic I / O technology include:

[0007] The one-way external terminal uses a customized file transfer protocol to encapsulate the file data to be transmitted, and sends the encapsulated file protocol data to the one-way internal terminal via the one-way transmission channel. Neither the one-way external terminal nor the one-way internal terminal triggers local disk read / write operations during the transmission process.

[0008] The one-way internal terminal will directly push the received file protocol data to the intranet file receiving server via TCP protocol;

[0009] After receiving the file protocol data, the intranet file receiving server divides the file protocol data into blocks according to the preset optimal disk write performance block size, and writes the divided file protocol data sequentially into a pre-created large cache file.

[0010] The intranet file receiving server monitors the IO load parameters of the local disk in real time, and dynamically adjusts the release rate of reading the file protocol data blocks from the large cache file and parsing and restoring the final file according to the IO load parameters.

[0011] Preferably, the step of encapsulating the file data to be transmitted using a customized file transfer protocol includes:

[0012] For each original file to be transmitted, extract the file metadata and file entity data, construct a fixed-length encapsulation header from the file metadata, and divide the file entity data into several entity data slices;

[0013] According to the temporal relationship between the fixed-length encapsulation header and the entity data slice in the original file, the fixed-length encapsulation header and the entity data slice of multiple consecutive original files are concatenated end to end to generate a continuous byte stream structure, and a stream end identifier is appended to the end of the byte stream structure to form the file protocol data. The fixed-length encapsulation header contains an offset indicator field and a data length field corresponding to the original file.

[0014] Preferably, the step of sequentially writing the segmented file protocol data into a pre-created large cache file includes:

[0015] When the intranet file receiving server receives the first file protocol data, it requests the allocation of a large cache file with a fixed total capacity from the local disk and initializes the sequential write pointer in memory.

[0016] When the accumulated amount of the received file protocol data in the memory buffer reaches the optimal write performance block size of the disk, a disk write operation is triggered, and the data block corresponding to the accumulated data amount is appended to the end of the current file of the large cache file through the sequential write pointer;

[0017] After each disk write operation is completed, the sequential write pointer is moved backward by an offset equal to the optimal write performance block size of the disk, and the memory buffer is cleared to receive the next round of accumulated data.

[0018] Preferably, the real-time monitoring of local disk IO load parameters includes:

[0019] A sliding time window is constructed in the intranet file receiving server, and the disk queue depth, read / write operations per second, and average IO wait time of the local disk within the sliding time window are collected according to a fixed sampling period.

[0020] The disk queue depth, read / write operations per second, and average IO wait time of multiple sampling points collected within the sliding time window are calculated by mean and variance respectively. The calculated mean and variance are combined to generate the IO load parameters. At the end of each sampling period, the oldest sampling point data in the sliding time window is replaced with the latest sampling point data.

[0021] Preferably, the step of dynamically adjusting the release rate of reading the file protocol data blocks from the large cache file and parsing and restoring the final file according to the IO load parameters includes:

[0022] A preset set of IO load threshold intervals, which includes light load intervals, medium load intervals, and heavy load intervals;

[0023] The IO load parameters are matched with the IO load threshold range set. When the match is in the light load range, the release rate is increased to a first preset rate. When the match is in the medium load range, the current release rate is maintained. When the match is in the heavy load range, the current release rate is reduced to a second preset rate according to a preset attenuation factor.

[0024] The cache parsing thread in the intranet file receiving server reads the file protocol data block from the large cache file according to the adjusted release rate, and restores the read file protocol data block as an independent file to disk.

[0025] Preferably, neither the unidirectional external terminal nor the unidirectional internal terminal triggers local disk read / write operations during transmission, including:

[0026] The one-way external terminal receives the file protocol data through a kernel-mode socket and uses direct memory access technology to map the buffer associated with the socket to the physical memory of the sending end of the one-way transmission channel.

[0027] The one-way internal terminal directly obtains the file protocol data through the physical memory of the receiving end of the one-way transmission channel, and uses the direct memory access technology to map the data in the physical memory of the receiving end to the transmission queue buffer of the network card used to send data to the intranet file receiving server.

[0028] Throughout the entire process of the file protocol data flowing from the one-way external terminal to the one-way internal terminal, the data flow path is entirely between physical memory and network card buffer, bypassing the file system layer and disk driver layer.

[0029] Preferably, after dividing the file entity data into several entity data slices, the method further includes:

[0030] Extract the file type features of the file data to be transmitted, and match the corresponding target compression dictionary from a preset compression dictionary library based on the file type features;

[0031] The entity data slice is losslessly compressed using the target compression dictionary to obtain the compressed slice data, and a compression flag and a length field before compression are added to the fixed-length encapsulation header.

[0032] The compressed slice data replaces the original entity data slices, and the operation of splicing the first and last parts according to the temporal relationship is performed. The compressed dictionary library stores differentiated dictionary tree structures for text files, image files and database files.

[0033] Preferably, when requesting allocation of a large cache file with a fixed total capacity from the local disk, the method further includes:

[0034] The allocated large cache file is logically divided into a first cache partition and a second cache partition, the size of which is equal to half of the fixed total capacity.

[0035] When the sequential write pointer points to the first cache segment and the first cache segment is full, the sequential write pointer is switched to the beginning position of the second cache segment to continue the append write operation. At the same time, an independent thread is started to perform read and parsing restoration operations on the full first cache segment.

[0036] When the second cache segment is full, the sequential write pointer is switched back to the starting position of the first cache segment whose space has been released. At the same time, an independent thread is started to perform read and parsing restoration operations on the full second cache segment, forming an alternating write and read ping-pong cache mechanism.

[0037] Preferably, after combining the calculated mean and variance to generate the IO load parameters, the method further includes:

[0038] The IO load parameters of the current sampling period and the previous multiple sampling periods are input into a pre-deployed recurrent neural network model in a time series, and the recurrent neural network model outputs the predicted IO load parameters for the next sampling period.

[0039] The predicted IO load parameters are weighted and fused with the actual collected current IO load parameters. The fused parameters are then used to replace the original IO load parameters to perform a matching operation with the IO load threshold range set.

[0040] After every 1000 file restoration operations, the actual disk write time and the prediction time of the recurrent neural network model are extracted, the prediction error loss function value is calculated, and the weight matrix in the recurrent neural network model is updated using the backpropagation algorithm.

[0041] Preferably, when the data flow path is entirely between physical memory and the network card buffer, it also includes:

[0042] After the kernel-mode socket of the one-way external machine receives the file protocol data, it calls the hardware encryption engine to perform symmetric encryption on the file protocol data and stores the encrypted data in the physical memory of the sending end.

[0043] After the data is acquired from the physical memory of the receiving end of the one-way internal terminal, and before the data is mapped to the transmit queue buffer of the network card, the hardware decryption engine paired with the hardware encryption engine is invoked to perform symmetric decryption processing on the data.

[0044] During the symmetric encryption and decryption processes, a message authentication code is appended to each data block. The one-way internal terminal verifies the message authentication code before decryption. If the verification fails, the current data block is discarded and the packet loss sequence number is reported back to the one-way external terminal.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1. This invention eliminates local disk read / write operations between internal and external terminals of a unidirectional isolation device. When file data flows between internal and external terminals, it is transmitted via direct memory access technology, mapping between physical memory and the network card buffer, bypassing the file system layer and disk driver layer. File data is directly pushed to the receiving server on the internal network. The receiving server writes the received data into a pre-created large cache file according to the optimal disk write performance block size. This processing method reduces the original three disk read / write operations in the transmission link to a single disk write operation on the final receiving server, eliminating disk I / O bottlenecks at intermediate nodes and reducing the frequency of disk read / write calls in scenarios with a large number of small files.

[0047] 2. The receiving server extracts disk input / output load parameters through a sliding time window and combines this with a recurrent neural network model for prediction and weighted fusion. Based on the load range, it dynamically adjusts the release rate of data read from the large cache file and restored to independent files. This mechanism ensures that disk write operations during file restoration remain within a reasonable range that the current hardware can handle, avoiding data loss due to the receiving server's disk write rate being lower than the unidirectional channel transmission rate. A ping-pong caching mechanism is used to achieve parallel isolation between writing and reading. Hardware-based encryption / decryption and message authentication code verification are introduced during data transfer to physical memory. Combined with lossless compression processing using a compression dictionary matched to file type, this ensures the security and integrity of unidirectional data transmission while reducing the actual data transfer volume in the unidirectional channel. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the overall process of unidirectional file transfer based on disk dynamic I / O technology according to the present invention.

[0049] Figure 2 This is a flowchart illustrating the fixed-length encapsulation header construction and byte stream splicing process of the present invention.

[0050] Figure 3 This is a flowchart of the direct memory access mapping and hardware encryption / decryption process of the present invention;

[0051] Figure 4 This is a flowchart of the large cache file block sequential writing process of the present invention;

[0052] Figure 5 The flowchart for the alternating write and read operation of file type matching compression and ping-pong caching in this invention is shown below;

[0053] Figure 6 This is a flowchart of the LSTM-based IO load prediction and dynamic rate adjustment process of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Please refer to Figure 1This embodiment provides a one-way file transfer method based on disk dynamic I / O technology. The one-way external terminal and the one-way internal terminal are deployed on opposite sides of a one-way isolation device. The one-way external terminal and the external source file server are in the same network domain, and the one-way internal terminal and the internal network file receiving server are in the same internal network domain. The one-way transmission channel is a physical one-way transmission link, which only supports the one-way flow of data from the one-way external terminal to the one-way internal terminal and does not support the reverse transmission of control messages and data messages.

[0056] Specifically, the one-way external terminal obtains the file data to be transmitted from the source file server and encapsulates the data using a customized file transfer protocol. For each original file to be transmitted, the one-way external terminal extracts file metadata and file entity data. The file metadata includes the filename, creation time, modification time, user permissions, and total file length. The extracted metadata is used to construct a fixed-length encapsulation header, with a fixed total length of 256 bytes. This ensures that the header parsing process does not require dynamic length calculation, improving parsing efficiency. Simultaneously, the one-way external terminal divides the file entity data into several entity data slices, each with a fixed length of 4096 bytes. For the last entity data slice of the original file, if its actual length is less than 4096 bytes, it is padded with 0x00 bytes to reach 4096 bytes, and the length of the padded bytes is recorded in the fixed-length encapsulation header.

[0057] In this embodiment, the fixed-length encapsulation header has a total length of 256 bytes. The definitions of each field are shown in the table below. By using fixed-length fields, the parsing process does not require dynamic length calculation, thus improving parsing efficiency.

[0058] Table 1. Definition of Fields in Fixed-Length Encapsulation Head

[0059] 0-15 16 Unique identifier for files A globally unique identifier for the original file, using the UUID format, used to distinguish different original files. 16-31 16 Stream end identifier matching field Used to verify the boundaries of the byte stream, consistent with the first 16 bytes of the stream end identifier. 32-47 16 file name The original file name is padded with 0x00 for any part that is less than 16 bytes. 48-55 8 Total file length Total length of the original file's entity data in bytes, excluding padding bytes. 56-63 8 Entity data starting offset The starting offset of the entity data in the original file within the byte stream structure, i.e., the offset indicator field. 64-71 8 Total length of entity data The total length of the entity data in the byte stream corresponding to the original file, i.e., the data length field. 72-79 8 File creation timestamp The creation time of the original file is displayed in Unix timestamp format, in milliseconds. 80-87 8 File modification timestamp The last modification time of the original file, in Unix timestamp format, is in milliseconds. 88-95 8 Number of slices Total number of entity data slices corresponding to the original file 96-97 2 Compression flag and padding length The high 1 bit is the compression flag, and the low 15 bits are the total length of the padding bytes for the entity data slice. 98-105 8 Total length before compression The total byte length of the entity data before compression is only effective when the compression flag is set to 1. 106-255 150 Reserved fields Used for future feature expansion; default value is 0x00.

[0060] refer to Figure 2 The one-way external terminal, based on the temporal relationship between the fixed-length encapsulation header and the entity data slices in the original file, concatenates the fixed-length encapsulation headers and entity data slices of multiple consecutive original files end-to-end to generate a continuous byte stream structure. A stream end identifier is then appended to the end of the byte stream structure to form the file protocol data. The byte stream concatenation logic of this process is quantified and described by the following formula:

[0061]

[0062] in, This is the continuous byte stream structure corresponding to the final generated file protocol data. This represents the total number of original files currently awaiting transmission. For the first The fixed-length encapsulation header corresponding to each original file For the first The total number of entity data slices corresponding to each original file. For the first The first original file Each entity data slice An end-of-stream identifier, appended to the end of the byte stream structure, is a fixed 16-byte sequence used to identify the end boundary of the current byte stream. The offset indicator field in the fixed-length encapsulation header indicates the starting offset of the entity data of the original file corresponding to the current fixed-length encapsulation header within the entire byte stream structure. The data length field indicates the total byte length of the entity data of the current original file, excluding padding bytes, ensuring accurate extraction of the complete entity data of the corresponding original file and removal of padding bytes during subsequent parsing.

[0063] The one-way external terminal sends the encapsulated file protocol data to the one-way internal terminal via a one-way transmission channel. Neither the one-way external nor the one-way internal terminal triggers local disk read / write operations during the transmission process. The entire processing of the file protocol data on the one-way external terminal is completed in physical memory, without calling the file system layer or writing any data to the local disk. The file protocol data received by the one-way internal terminal is directly stored in the pre-allocated physical memory space, without performing any local disk read or write operations, thus bypassing the disk driver layer throughout the entire process.

[0064] The one-way internal terminal directly pushes the received file protocol data to the internal network file receiving server via TCP. After receiving the file protocol data, the one-way internal terminal directly constructs a TCP packet in memory, uses the file protocol data as the payload of the TCP packet, and sends it to the corresponding TCP listening port of the internal network file receiving server through the internal network Ethernet link. Throughout the entire process, the one-way internal terminal does not perform any local disk write or read operations on the received data; all data processing is completed in physical memory.

[0065] refer to Figure 4After receiving file protocol data, the intranet file receiving server divides the data into blocks according to a preset optimal disk write performance block size and sequentially writes the block data into a pre-created large cache file. The preset optimal disk write performance block size is the optimal write block size obtained through sequential write performance testing, specifically tailored to the local disk hardware characteristics of the intranet file receiving server. In this embodiment, the optimal disk write performance block size is set to 128KB. This block size matches the physical sector configuration of the disk and the file system block size configuration, avoiding disk seek overhead caused by small block writes. When the intranet file receiving server receives the first file protocol data, it requests a fixed total capacity large cache file from the local disk. The fixed total capacity is determined based on the peak transmission rate of the unidirectional transmission channel and the maximum cache duration of the intranet file receiving server. In this embodiment, the fixed total capacity of the large cache file is set to 10GB. During the allocation process, a contiguous disk block allocation method is used to allocate physically contiguous disk sectors to the large cache file, avoiding write performance degradation caused by file fragmentation.

[0066] At the same time, the intranet file receiving server initializes the sequential write pointer in memory, and the initial value of the sequential write pointer points to the starting byte position of the large cache file.

[0067] When the accumulated file protocol data in the memory buffer reaches the optimal disk write performance block size, a disk write operation is triggered. The data block corresponding to the accumulated data is appended to the end of the current file in the large cache file via a sequential write pointer. The memory buffer is a kernel-mode circular buffer with a total capacity four times the optimal disk write performance block size. It is used to cache file protocol data received from the TCP socket, preventing data loss due to network jitter. After each disk write operation, the sequential write pointer is moved forward by an offset equal to the optimal disk write performance block size, and the storage space corresponding to the written data in the memory buffer is cleared to receive the next round of accumulated data.

[0068] The intranet file receiving server monitors the local disk's I / O load parameters in real time and dynamically adjusts the rate at which it reads file protocol data blocks from large cache files and releases the final parsed and reconstructed files based on these parameters. Specifically, a sliding time window is constructed within the intranet file receiving server, with a length of 100 sampling points and a fixed sampling period of 100ms, meaning the total duration covered by the sliding time window is 10s. The intranet file receiving server collects data on the local disk's disk queue depth, read / write operations per second, and average I / O latency within the sliding time window according to the fixed sampling period.

[0069] The disk queue depth is the number of I / O requests waiting to be processed in the disk driver layer at the sampling time. The number of reads and writes per second is the total number of read and write operations completed by the disk during the sampling period. The average I / O wait time is the average time taken for all I / O requests to go from entering the queue to being processed during the sampling period.

[0070] The intranet file receiving server calculates the mean and variance of disk queue depth, read / write operations per second, and average IO wait time for multiple sampling points collected within a sliding time window. The calculated mean and variance are combined to generate IO load parameters. At the end of each sampling period, the oldest sampling point data in the sliding time window is replaced with the latest data, achieving a rolling update of the sliding time window and ensuring that the IO load parameters reflect the current disk load status in real time. The logic of the mean calculation is quantified by the following formula:

[0071]

[0072] in, For the first The average value of parameters corresponding to each sampling period, The length of the sliding time window, i.e., the total number of sampling points. For the first time window within the sliding time window The parameter sample values ​​corresponding to each sampling point include disk queue depth, read / write operations per second, and average IO wait time.

[0073] The logic of variance calculation is quantitatively described by the following formula:

[0074]

[0075] in, For the first The parameter variance corresponding to each sampling period is used to characterize the degree of fluctuation of the parameter sampling values ​​within the sliding time window.

[0076] The internal network file receiving server has a preset set of IO load threshold ranges, including light load, medium load, and heavy load ranges. For the average disk queue depth, the light load range is 0-4, the medium load range is 4-12, and the heavy load range is greater than 12. For the average read / write operations per second, the light load range is 0-500, the medium load range is 500-1500, and the heavy load range is greater than 1500. For the average IO wait time, the light load range is 0-5ms, the medium load range is 5-20ms, and the heavy load range is greater than 20ms. The range matching rule is as follows: when the average values ​​of the three parameters all fall within the same range, it is considered a match for that range; when the average values ​​of the three parameters fall within different ranges, the range with the highest load is taken as the matching result.

[0077] The intranet file receiving server matches IO load parameters with a set of IO load threshold ranges. When a match is made in a light load range, the release rate is increased to a first preset rate; when a match is made in a medium load range, the current release rate remains unchanged; when a match is made in a heavy load range, the current release rate is reduced to a second preset rate according to a preset attenuation factor. The adjustment logic of the release rate is quantified by the following formula:

[0078] When matching to a lightly loaded region When matching reaches the mid-load interval, When matching an overloaded region, .

[0079] in, For the first Release rate corresponding to each sampling period The first preset rate is the maximum release rate supported by the system. For the first The current release rate corresponding to each sampling period The preset attenuation factor has a value range of [value range missing]. In this embodiment, Set to 0.7, The lower limit is the second preset rate, which is the minimum release rate supported by the system, to avoid the file restoration process from stalling due to the release rate dropping to 0.

[0080] The cache parsing thread in the intranet file receiving server reads file protocol data blocks from the large cache file according to the adjusted release rate, and then restores the read file protocol data blocks into independent files and writes them to disk. The release rate is controlled by adjusting the amount of data read per operation and the reading time interval of the cache parsing thread. When the release rate increases, the amount of data read per operation increases and the reading time interval shortens; when the release rate decreases, the amount of data read per operation decreases and the reading time interval lengthens. After reading data blocks from the large cache file, the cache parsing thread first parses the fixed-length encapsulation header in the byte stream structure. Based on the offset indicator field and data length field in the fixed-length encapsulation header, it extracts the corresponding entity data slices of the original file, removes padding bytes, and concatenates them to obtain the complete file entity data. Combining this with the file metadata in the fixed-length encapsulation header, it restores the original file and writes the restored original file to the specified storage path on the local disk, completing the one-way file transfer process.

[0081] In this embodiment, no local disk read / write operations are triggered during the one-way transmission of file data between the external and internal terminals. Only one disk write and restore operation is performed on the internal network file receiving server, reducing the number of disk I / O operations in the transmission link and mitigating the impact of disk I / O bottlenecks on transmission performance. Simultaneously, the fixed-length encapsulation header and continuous byte stream encapsulation method simplify the data parsing process. Combined with dynamic release rate adjustment based on disk I / O load parameters, the disk I / O load during the file restoration process is maintained within a reasonable range, avoiding data loss due to insufficient disk write capacity.

[0082] In another embodiment, reference Figure 3 Both the one-way external and one-way internal terminals are equipped with network interface cards (NICs) supporting Direct Memory Access (DMI). These NICs support PCIe bus interfaces, enabling direct data transfer between physical memory and the NIC buffer without CPU intervention. The one-way external terminal receives file protocol data to be transferred via kernel-mode sockets. The buffer associated with the kernel-mode socket is a contiguous physical memory page in kernel space, avoiding address mapping overhead caused by memory paging. The one-way external terminal utilizes DMI to map the socket-associated buffer to the sending end's physical memory in the one-way transmission channel. During the mapping process, only the address mapping relationship of the memory page table is modified; no data copying operation is performed, achieving zero-copy data transfer. The address mapping logic is quantified using the following formula:

[0083]

[0084] in, This is the target starting address of the physical memory of the sending end in a one-way transmission channel. The physical starting address of the socket associated buffer. This is the offset of the data block in the socket buffer. This formula enables a linear address mapping between the socket buffer and the physical memory of the sending end, ensuring a complete mapping of the data block.

[0085] In this embodiment, the core parameters of the direct memory access mapping path between the one-way external terminal and the one-way internal terminal are shown in the table below. By configuring continuous physical memory, the stability and efficiency of zero-copy transmission are ensured.

[0086] Table 2 Parameter Table for Unidirectional Transfer Memory Mapping Path

[0087] Kernel-mode socket buffer size 16MB The total capacity of the circular buffer associated with the one-way external socket kernel mode is used to buffer protocol data to be transmitted. Total physical memory size of the sending end 32MB The total contiguous physical memory capacity bound to the sending end of the unidirectional transmission channel is allocated using 2MB of large page memory. Total physical memory size of the receiving end 32MB The total contiguous physical memory capacity bound to the receiver of the unidirectional transmission channel is allocated using 2MB of large page memory. Internal network sending NIC queue buffer size 8MB The total capacity of the network card's data transmission queue for one-way internal machine sending data to the internal network is divided into 8 independent sub-queues. Memory-mapped page size 2MB Direct memory access mapping (DMR) uses smaller memory page sizes, reducing the number of page table entries and lowering address translation overhead. DMA channel number 2 The PCIeDMA channel number bound to the direct memory access operation corresponds to the hardware channel of the network interface card (NIC) device. Address mapping timeout 100ms The timeout threshold for memory page table mapping operations; if this threshold is exceeded, the current mapping operation is terminated, and exception handling is triggered. Maximum Mapped Data Block Length 4MB The maximum data block length supported by a single direct memory access mapping is used to avoid excessive memory resource consumption in a single mapping.

[0088] The one-way internal terminal directly obtains file protocol data through the receiving end physical memory of the one-way transmission channel. The receiving end physical memory consists of pre-allocated contiguous physical memory pages within the one-way internal terminal, directly bound to the hardware receiving port of the one-way transmission channel. Utilizing direct memory access (DMI) technology, the one-way internal terminal maps data in the receiving end physical memory to the transmission queue buffer of the network interface card (NIC) used to send data to the intranet file receiving server. This mapping process also employs a zero-copy approach, modifying only the address mapping relationships in the memory page tables without performing any data copy operations. Throughout the entire process of file protocol data flowing from the one-way external terminal to the one-way internal terminal, the data flow path remains entirely between physical memory and the NIC buffer, bypassing the file system layer and disk driver layer. During this process, the CPUs of both the one-way external and internal terminals only perform memory page table configuration and mapping operations, without executing any file system calls or triggering any local disk read / write instructions, fundamentally eliminating the disk I / O bottleneck at intermediate nodes.

[0089] After the kernel-mode socket of the one-way external machine receives file protocol data, it calls the hardware encryption engine to perform symmetric encryption on the file protocol data. The hardware encryption engine is a hardware cryptographic module integrated in the network interface card (NIC) device, supporting the Chinese national standard SM4 symmetric encryption algorithm. The encryption process is completed in hardware, without the need for central processing unit (CPU) involvement, thus avoiding the performance degradation caused by CPU resource consumption during encryption operations. The block processing logic of symmetric encryption is quantitatively described by the following formula:

[0090]

[0091] in, For the first Each plaintext data block corresponds to a ciphertext data block. To use the key SM4 encryption function, For the first One plaintext data block, For the first The ciphertext data block corresponding to each data block, the initial ciphertext The initial vector is a preset value with a length of 16 bytes. This encryption mode is a ciphertext block chaining mode, which ensures that the same plaintext data block is encrypted in different positions to obtain different ciphertext data blocks, thereby improving encryption security.

[0092] The one-way external terminal stores the encrypted data in the sending end's physical memory and sends it to the one-way internal terminal via the one-way transmission channel. After the one-way internal terminal retrieves the data from the receiving end's physical memory, and before mapping the data to the network card's transmit queue buffer, it calls the hardware decryption engine paired with the hardware encryption engine to perform symmetric decryption processing on the data. The hardware decryption engine is also a hardware cryptographic module integrated in the one-way internal terminal's network card device. Using the same key and initialization vector as the encryption process, it performs the decryption operation in reverse order of the encryption process to restore the original file protocol data.

[0093] During symmetric encryption and decryption, a message authentication code is appended to each data block. This message authentication code is generated using the SM3 cryptographic hash algorithm. The generation logic of the message authentication code is quantified using the following formula:

[0094]

[0095] in, For the first The message authentication code corresponding to each data block This is the SM3 hash algorithm function. Generate a key for the message authentication code. This is a byte concatenation operation. For the first The plaintext content of each data block. For the first Each data block has a corresponding sequence number, which increments in the order the data blocks were sent to ensure that the message authentication code for each data block is unique and to prevent replay attacks.

[0096] Before decryption, the one-way internal terminal verifies the message authentication code. Specifically, it extracts the message authentication code from the received data block, recalculates it based on the received data block content, sequence number, and shared key, and compares the recalculated code with the received code. If they match perfectly, the verification passes, and the subsequent decryption operation is performed. If the verification fails, the current data block is discarded, and a packet loss sequence number is sent back to the one-way external terminal. This sequence number feedback is achieved through an out-of-band reverse control channel reserved in the one-way isolation device. This channel only supports the transmission of control messages such as packet loss sequence numbers and does not support the reverse transmission of any file data, ensuring that the security of one-way isolation remains unaffected. After receiving the packet loss sequence number, the one-way external terminal retransmits the corresponding data block, ensuring the complete transmission of the file data.

[0097] In this embodiment, direct memory access technology enables zero-copy transmission of file data between the one-way external and one-way internal machines, bypassing the file system and disk driver layers throughout the process, completely eliminating disk I / O overhead at intermediate nodes and improving data transmission efficiency. Simultaneously, by using a hardware encryption and decryption engine integrated into the network card to perform symmetric encryption and decryption operations, combined with message authentication code integrity verification, the security and integrity of data during one-way transmission are ensured without affecting transmission performance, avoiding the risk of data tampering or theft.

[0098] In a preferred embodiment, reference Figure 5 After dividing the file entity data into several entity data slices, the one-way external terminal extracts the file type features of the file data to be transmitted. The file type features are determined by the original file name suffix and the magic number field in the file header, ensuring the accuracy of file type identification and avoiding type identification errors caused by file name suffix tampering. Based on the identified file type features, the one-way external terminal matches the corresponding target compression dictionary from a preset compression dictionary library. The compression dictionary library stores differentiated dictionary tree structures for text files, image files, and database files. Different types of dictionary tree structures are optimized for the data features of the corresponding file types, which can improve the compression ratio and compression efficiency.

[0099] In this embodiment, the compression dictionary library is configured with differentiated dictionary parameters for different file types. The core configurations for each type are shown in the table below. Through targeted parameter optimization, the compression efficiency and compression ratio of the corresponding file types are improved.

[0100] Table 3. Compression dictionary configuration table for different file types

[0101] Text files 16 64KB 32KB 3 bytes Text data such as log files, code files, and document files Image files 12 32KB 16KB 4 bytes Uncompressed image files such as BMP and TIFF, and RAW format images. database file 20 128KB 64KB 8 bytes Structured data such as database backup files and data warehouse export files General type 14 48KB 24KB 3 bytes Unidentified file type, general compression scenario

[0102] The one-way external terminal uses the matched target compression dictionary to perform lossless compression on the entity data slices, obtaining the compressed slice data. The lossless compression process employs the LZ77 algorithm combined with Huffman coding. The target compression dictionary is a static dictionary pre-trained based on a large amount of sample data for the corresponding file type, avoiding the overhead of transmitting dictionary information with a dynamic dictionary and improving compression efficiency. The quantification of compression performance is described by the following formula:

[0103]

[0104] in, The compression ratio corresponding to the entity data slice. The length in bytes of the entity data slice before compression. This represents the byte length of the compressed sliced ​​data. A compression ratio greater than 1 indicates that the data has been compressed. The higher the compression ratio, the greater the reduction in data size.

[0105] The one-way external terminal adds a compression flag and a pre-compression length field to the fixed-length encapsulation header. The compression flag is a 1-bit binary data; when it is 1, it indicates that the entity data slice corresponding to the current fixed-length encapsulation header has been compressed; when it is 0, it indicates that the entity data slice has not been compressed. The pre-compression length field records the total byte length of the entity data before compression, ensuring that the intranet file receiving server can accurately restore the original length of entity data during decompression. The one-way external terminal replaces the original entity data slice with the compressed slice data and performs a head-to-tail concatenation operation according to the timing relationship to generate a continuous byte stream structure. When parsing file protocol data, the intranet file receiving server first reads the compression flag in the fixed-length encapsulation header. If the compression flag is 1, it decompresses the compressed slice data based on the corresponding compression dictionary to restore the original entity data slice, and then performs subsequent file restoration operations.

[0106] When the intranet file receiving server requests a fixed total capacity cache file from the local disk, it logically divides the allocated cache file into a first cache fragment and a second cache fragment. The size of both the first and second cache fragments is equal to half the fixed total capacity; that is, when the fixed total capacity of the cache file is 10GB, the size of both the first and second cache fragments is 5GB. The intranet file receiving server configures independent write and read pointers for the first and second cache fragments respectively, ensuring that write and read operations on the two fragments are isolated and do not affect each other. The switching logic of the write pointer is quantified using the following formula:

[0107]

[0108] in, Write pointers to the currently effective order. This is the pointer to the starting address of the first cache segment. This is the pointer to the end address of the first cached fragment. This is the write pointer corresponding to the first cache slice. This is the pointer to the starting address of the second cache segment. This is the pointer to the end address of the second cache fragment. This is the write pointer corresponding to the second cache slice.

[0109] When the sequential write pointer points to the first cache slice and the first cache slice is full, the intranet file receiving server switches the sequential write pointer to the beginning of the second cache slice and continues to perform append write operations. Simultaneously, it starts an independent cache parsing thread to perform read and parsing / restoration operations on the full first cache slice. When the second cache slice is full, the intranet file receiving server switches the sequential write pointer back to the beginning of the first cache slice, which has completed parsing / restoration and released space. It also starts an independent cache parsing thread to perform read and parsing / restoration operations on the full second cache slice, forming an alternating write-read ping-pong caching mechanism. In the ping-pong caching mechanism, write operations and read / parsing operations are performed on different cache slices. At any given time, only one cache slice is performing write operations while the other is performing read / parsing operations. This avoids disk seek overhead and lock contention issues caused by concurrent read and write operations on the same file, improving disk I / O efficiency. Furthermore, the independent cache parsing thread and write thread are bound to different CPU cores, avoiding context switching overhead between threads and further improving the parallelism of data processing.

[0110] In this embodiment, lossless compression of entity data slices is performed using a static compression dictionary based on file type matching, reducing the actual data transmission volume in the unidirectional transmission channel and increasing the effective transmission bandwidth of the unidirectional transmission channel. Simultaneously, by dividing the large cache file into two cache fragments and constructing an alternating write-read ping-pong caching mechanism, parallel isolation between write operations and read parsing operations is achieved, avoiding disk I / O performance degradation caused by concurrent reads and writes and improving the read / write efficiency of large cache files.

[0111] In yet another preferred embodiment, reference is made to Figure 6 After the intranet file receiving server generates IO load parameters by combining the calculated mean and variance, it inputs the IO load parameters of the current sampling period and the previous multiple sampling periods into a pre-deployed recurrent neural network model in a time series. The recurrent neural network model adopts a long short-term memory network structure, which can effectively capture the time series characteristics of IO load parameters, avoid the gradient vanishing problem of traditional recurrent neural networks, and improve the accuracy of load prediction.

[0112] In this embodiment, the recurrent neural network used for IO load prediction adopts a two-layer long short-term memory network structure. The core parameters of each network layer are shown in the table below. Through the multi-layer structure design, the long and short-term dependency characteristics of the IO load time series are effectively captured.

[0113] Table 4. Recurrent Neural Network Model Structure Parameters

[0114] Input layer - - 6 6 0 LSTM Hidden Layer 1 64 tanh 6 64 0.2 LSTM Hidden Layer 2 32 tanh 64 32 0.2 Fully connected layer 16 ReLU 32 16 0.1 Output layer 6 Linear activation 16 6 0

[0115] Input dimension 6 corresponds to six IO load parameters: mean disk queue depth, variance disk queue depth, mean read / write operations per second, variance read / write operations per second, mean average IO latency, and variance average IO latency. Output dimension 6 corresponds to the predicted values ​​of these six IO load parameters for the next sampling period. The update logic of the hidden layer states and cell states in the Long Short-Term Memory network is quantified using the following formula:

[0116]

[0117] in, for The output of the forget gate at any moment, for Input gate output at any time, for The state of candidate cells at any given time. for Cellular state at any given moment for The output gate outputs at any time. for Hidden layer output at any given time, for The model input at any given time, i.e., the IO load parameters for the current sampling period. for Hidden layer output at any given time, for Cellular state at any given moment This is the weight matrix corresponding to the gating. This is the bias vector corresponding to the gating. It is the sigmoid activation function. The hyperbolic tangent activation function is used. This is for the Hadamard product operation.

[0118] The output layer of the recurrent neural network model is based on the hidden layer output of the last time step. The predicted IO load parameters for the next sampling period are obtained by mapping through a fully connected layer. The predicted IO load parameters include the disk queue depth, read / write operations per second, and the predicted mean and variance of the average IO wait time for the next sampling period.

[0119] The intranet file receiving server performs a weighted fusion of the predicted IO load parameters and the actual collected current IO load parameters. The fused parameters are then used to replace the original IO load parameters in the matching operation against the IO load threshold range set. The logic of the weighted fusion is quantified by the following formula:

[0120]

[0121] in, For the first IO load parameters after fusion of sampling periods The preset prediction weight coefficients have a range of values. In this embodiment, Set to 0.6, For the first Predicted IO load parameters for each sampling period For the first The actual IO load parameters collected in each sampling period. After completing every 1000 file restoration operations, the intranet file receiving server extracts the actual disk write time and the prediction time of the recurrent neural network model, calculates the prediction error loss function value, and updates the weight matrix in the recurrent neural network model using the backpropagation algorithm. The calculation logic of the loss function is quantitatively described by the following formula:

[0122]

[0123] in, The value of the mean squared error loss function. The total number of samples used in the loss calculation. For the first The actual disk I / O load value corresponding to each sample For the first The model predicts the load value for each sample.

[0124] Specifically, the intranet file receiving server will construct a training dataset by collecting all IO load parameter samples and corresponding predicted values ​​during 1000 file restoration operations. An adaptive moment estimation optimizer will be used, and a backpropagation algorithm will be executed based on the loss function value to calculate the gradient of the loss function with respect to all weight matrices and bias vectors in the model. The values ​​of the weight matrix and bias vectors will be updated according to a preset learning rate, completing the online iterative update of the model. A dynamic decay strategy is adopted for the learning rate. The initial learning rate is set to 0.001, and every 10 model updates, the learning rate decays to 0.9 times the original value to avoid oscillations and non-convergence issues. Simultaneously, the intranet file receiving server sets a gradient pruning threshold for the model's weight updates. When the norm of the gradient exceeds the preset threshold, the gradient is normalized to avoid gradient explosion and ensure the stability of model updates. By predicting the IO load for the next sampling period using a recurrent neural network model and combining it with the fusion parameters obtained from weighted fusion, the server can detect the changing trend of disk IO load in advance and adjust the release rate accordingly, avoiding adjustment lag issues caused by sudden increases in disk load.

[0125] In this embodiment, a recurrent neural network model with a long short-term memory (LSTM) structure is used to learn the time-series features of the IO load parameters, outputting the predicted IO load parameters for the next sampling period. By combining the weighted fusion of the predicted and actual values, the release rate is adjusted in advance, solving the lag problem of traditional load adjustment methods. Simultaneously, the model is iteratively updated using online sample data, enabling it to adapt to dynamic changes in disk IO performance and maintain high prediction accuracy, further optimizing the disk IO control effect during file restoration.

Claims

1. A one-way file transfer method based on disk dynamic I / O technology, characterized in that, include: The one-way external terminal uses a customized file transfer protocol to encapsulate the file data to be transmitted, and sends the encapsulated file protocol data to the one-way internal terminal via the one-way transmission channel. Neither the one-way external terminal nor the one-way internal terminal triggers local disk read / write operations during the transmission process. The one-way internal terminal pushes the received file protocol data to the intranet file receiving server via TCP protocol; After receiving the file protocol data, the intranet file receiving server divides the file protocol data into blocks according to the preset optimal disk write performance block size, and writes the divided file protocol data sequentially into a pre-created large cache file. The intranet file receiving server monitors the IO load parameters of the local disk in real time, and dynamically adjusts the release rate of reading the file protocol data blocks from the large cache file and parsing and restoring the final file according to the IO load parameters.

2. The one-way file transfer method based on disk dynamic I / O technology according to claim 1, characterized in that, The process of encapsulating the file data to be transmitted using a custom file transfer protocol includes: For each original file to be transmitted, extract the file metadata and file entity data, construct a fixed-length encapsulation header from the file metadata, and divide the file entity data into several entity data slices; According to the temporal relationship between the fixed-length encapsulation header and the entity data slice in the original file, the fixed-length encapsulation header and the entity data slice of multiple consecutive original files are concatenated end to end to generate a continuous byte stream structure, and a stream end identifier is appended to the end of the byte stream structure to form the file protocol data. The fixed-length encapsulation header contains an offset indicator field and a data length field corresponding to the original file.

3. The one-way file transfer method based on disk dynamic I / O technology according to claim 1, characterized in that, The step of sequentially writing the segmented file protocol data into a pre-created large cache file includes: When the intranet file receiving server receives the first file protocol data, it requests the allocation of a large cache file with a fixed total capacity from the local disk and initializes the sequential write pointer in memory. When the accumulated amount of the received file protocol data in the memory buffer reaches the optimal write performance block size of the disk, a disk write operation is triggered, and the data block corresponding to the accumulated data amount is appended to the end of the current file of the large cache file through the sequential write pointer; After each disk write operation is completed, the sequential write pointer is moved backward by an offset equal to the optimal write performance block size of the disk, and the memory buffer is cleared to receive the next round of accumulated data.

4. The one-way file transfer method based on disk dynamic I / O technology according to claim 1, characterized in that, The real-time monitoring of local disk IO load parameters includes: A sliding time window is constructed in the intranet file receiving server, and the disk queue depth, read / write operations per second, and average IO wait time of the local disk within the sliding time window are collected according to a fixed sampling period. The disk queue depth, read / write operations per second, and average IO wait time of multiple sampling points collected within the sliding time window are calculated by mean and variance respectively. The calculated mean and variance are combined to generate the IO load parameters. At the end of each sampling period, the oldest sampling point data in the sliding time window is replaced with the latest sampling point data.

5. The one-way file transfer method based on disk dynamic I / O technology according to claim 1, characterized in that, Dynamically adjust the release rate of reading file protocol data blocks from the large cache file and parsing and restoring the final file based on the IO load parameters, including: A preset set of IO load threshold intervals, which includes light load intervals, medium load intervals, and heavy load intervals; The IO load parameters are matched with the IO load threshold range set. When the match is in the light load range, the release rate is increased to a first preset rate. When the match is in the medium load range, the current release rate is maintained. When the match is in the heavy load range, the current release rate is reduced to a second preset rate according to a preset attenuation factor. The cache parsing thread in the intranet file receiving server reads the file protocol data block from the large cache file according to the adjusted release rate, and restores the read file protocol data block as an independent file to disk.

6. The one-way file transfer method based on disk dynamic I / O technology according to claim 1, characterized in that, Neither the unidirectional external terminal nor the unidirectional internal terminal triggers local disk read / write operations during transmission, including: The one-way external terminal receives the file protocol data through a kernel-mode socket and uses direct memory access technology to map the buffer associated with the socket to the physical memory of the sending end of the one-way transmission channel. The one-way internal terminal directly obtains the file protocol data through the physical memory of the receiving end of the one-way transmission channel, and uses the direct memory access technology to map the data in the physical memory of the receiving end to the transmission queue buffer of the network card used to send data to the intranet file receiving server. Throughout the entire process of the file protocol data flowing from the one-way external terminal to the one-way internal terminal, the data flow path is entirely between physical memory and network card buffer, bypassing the file system layer and disk driver layer.

7. The one-way file transfer method based on disk dynamic I / O technology according to claim 2, characterized in that, After dividing the file entity data into several entity data slices, the method further includes: Extract the file type features of the file data to be transmitted, and match the corresponding target compression dictionary from a preset compression dictionary library based on the file type features; The entity data slice is losslessly compressed using the target compression dictionary to obtain the compressed slice data, and a compression flag and a length field before compression are added to the fixed-length encapsulation header. The compressed slice data replaces the original entity data slices, and the operation of splicing the first and last parts according to the temporal relationship is performed. The compressed dictionary library stores differentiated dictionary tree structures for text files, image files and database files.

8. The one-way file transfer method based on disk dynamic I / O technology according to claim 3, characterized in that, When requesting allocation of a fixed total capacity for the large cache file from the local disk, the following is also included: The allocated large cache file is logically divided into a first cache partition and a second cache partition, the size of which is equal to half of the fixed total capacity. When the sequential write pointer points to the first cache segment and the first cache segment is full, the sequential write pointer is switched to the beginning position of the second cache segment to continue the append write operation. At the same time, an independent thread is started to perform read and parsing restoration operations on the full first cache segment. When the second cache segment is full, the sequential write pointer is switched back to the starting position of the first cache segment whose space has been released. At the same time, an independent thread is started to perform read and parsing restoration operations on the full second cache segment, forming an alternating write and read ping-pong cache mechanism.

9. The one-way file transfer method based on disk dynamic I / O technology according to claim 4, characterized in that, After combining the calculated mean and variance to generate the IO load parameters, the method further includes: The IO load parameters of the current sampling period and the previous multiple sampling periods are input into a pre-deployed recurrent neural network model in a time series, and the recurrent neural network model outputs the predicted IO load parameters for the next sampling period. The predicted IO load parameters are weighted and fused with the actual collected current IO load parameters. The fused parameters are then used to replace the original IO load parameters to perform a matching operation with the IO load threshold range set. After every 1000 file restoration operations, the actual disk write time and the prediction time of the recurrent neural network model are extracted, the prediction error loss function value is calculated, and the weight matrix in the recurrent neural network model is updated using the backpropagation algorithm.

10. The one-way file transfer method based on disk dynamic I / O technology according to claim 6, characterized in that, When the data flow path is entirely between physical memory and the network card buffer, it also includes: After the kernel-mode socket of the one-way external machine receives the file protocol data, it calls the hardware encryption engine to perform symmetric encryption on the file protocol data and stores the encrypted data in the physical memory of the sending end. After the data is acquired from the physical memory of the receiving end of the one-way internal terminal, and before the data is mapped to the transmit queue buffer of the network card, the hardware decryption engine paired with the hardware encryption engine is invoked to perform symmetric decryption processing on the data. During the symmetric encryption and decryption processes, a message authentication code is appended to each data block. The one-way internal terminal verifies the message authentication code before decryption. If the verification fails, the current data block is discarded and the packet loss sequence number is reported back to the one-way external terminal.

Images