Method for realizing multitask file ferrying and disk falling based on optical shutter system

Abstract

The invention discloses a method for realizing multitask file ferrying and disk falling based on an optical shutter system, which comprises the following steps of: respectively customizing a configuration file and starting a service program at a sending end and a receiving end, and then issuing a dynamic task or monitoring and confirming whether sent data arrives or not; and finally, confirming a sending result or confirming a file receiving result through the log file. According to the method, non-blocking processing of multiple tasks is carried out, the files are placed in a memory file system or a hard disk, file ferrying takes IO reading and writing as a source and a purpose, the size of a single IO buffer area is based on the size of an MTU, concurrent processing of the multiple tasks of file ferrying is achieved, and meanwhile the transmission rate of a whole link is guaranteed.

Description

Technical Field

[0001] The invention belongs to the field of data management and data synchronization, and in particular relates to a method for realizing multi-task file ferrying and disk placement based on an optical gate system. Background Art

[0002] With the popularization of computer network applications and the improvement of network security awareness, more and more units divide their internal departments into different network security domains according to different confidentiality levels. High-level network security domains cannot flow confidential information to low-level network security domains. In the process of file inflow and outflow, security operations such as antivirus and auditing are often required. At this time, file ferrying and storage have become the core functions of security ferrying. The file ferrying requirements that support multi-task file ferrying and storage and whose transmission rate cannot be seriously affected by file IO need to be solved. Summary of the invention

[0003] The purpose of the present invention is to propose a non-blocking multi-task processing method to address the above problems. The file is stored in a memory file system or a hard disk. The file ferry takes IO reading and writing as the source and destination. The size of a single IO buffer is based on an MTU (Maximum Transmission Unit) size. This not only solves the concurrent processing of file ferrying multi-tasks, but also ensures the transmission rate of the entire link.

[0004] To achieve the above-mentioned purpose, the technical solution proposed by the present invention is a method for realizing multi-task file transfer based on an optical gate system, which includes the following corresponding steps at the sending end and the receiving end:

[0005] Sending side:

[0006] Step 11: Customize the configuration file, configuration file and configuration items. The configuration items include working mode, ferry source, destination address and port, local IO buffer size, work queue, task waiting queue and sending interval delay. The working mode sender is set to "active", and the ferry source and destination address are configured according to the actual network scenario.

[0007] Step 12: Start the service program;

[0008] Step 13: Start task delivery through the manage program + json configuration file included in the service program;

[0009] Step 14: Confirm the sending result through the log file.

[0010] Receiver:

[0011] Step 21: Customize the configuration file, configuration file and configuration items. The configuration items include working mode, ferry source, destination address and port, local IO buffer size, out-of-order sorting method, maximum out-of-order tolerance interval, maximum transmission queue, default save directory, and same file processing strategy. The working mode receiving end is "positive", and the ferry source and destination address are configured according to the actual network scenario.

[0012] Step 22: Start the service program;

[0013] Step 23: Start to listen to the UDP socket of the local address and port. If the sent data arrives, the service of receiving and persisting it to the specified file directory is triggered;

[0014] Step 24: Confirm the file reception result through the log file or the actual persistence directory.

[0015] In the above step 11, the size of the local IO buffer needs to be configured according to the actual processing performance and memory size of the computer.

[0016] The above-mentioned work queue and task waiting queue are the key to the threshold and processing performance of multi-task issuance. The size of the two queues must be configured according to the actual processing performance of the computer. The sending interval delay is one of the means to control the sending rate, which can directly control the transmission rate to ensure the success rate of file ferrying.

[0017] The default work queue size is 8 and the task waiting queue size is 1024.

[0018] In step 21, the size of the local IO buffer needs to be configured according to the actual processing performance and memory size of the computer to ensure consistency with the sending end.

[0019] In step 21, the above-mentioned disorder sorting method needs to be determined according to the actual disorder situation of the computer network. If the probability of disorder is very small, select the "dynamic" mode; if the probability of disorder is very large, select the "static" mode.

[0020] In step 21, when setting the maximum out-of-order tolerance interval, it is necessary to consider that when the out-of-order quantity is greater than the configured number, it is determined that packet loss occurs, and it is necessary to adjust the sending interval delay of the sending end or adjust the system parameters.

[0021] In step 21, when the sending end does not specify a persistent directory for receiving end files, the default storage directory is valid.

[0022] In step 21, the same file processing strategy is to append "thread number-timestamp" by default to avoid file duplication; if the "cover" mode is specified, it means overwriting the original file.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] 1. During the file transfer process, the program parameters can be flexibly configured according to the computer configuration and performance, ensuring the performance and stability of the transmission while supporting multi-tasking concurrency.

[0025] 2. During the file transfer process, both the sender and the receiver involve disk IO. However, by adjusting the size of the IO local buffer, various problems caused by CPU processing IO interrupts, such as packet loss and low transmission rate, can be effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the sending end of the present invention;

[0027] Figure 2 It is a flow chart of the receiving end of the present invention;

[0028] Figure 3 The default configuration file for the sender program startup;

[0029] Figure 4 It is the JSON configuration file template for the manage program;

[0030] Figure 5 This is an example of logging on the sender side;

[0031] Figure 6 The default configuration file for the receiving program to start;

[0032] Figure 7 Example of logging for the receiving end. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0034] The present invention aims to solve the stability and transmission performance of multi-task file one-way ferrying and disk storage in the optical gate scenario. To this end, a method for realizing non-blocking processing of multi-tasks and disk storage to reduce IO interruption based on the optical gate system is proposed, which includes two parts of code implementation, the sending end and the receiving end. The processes of the sending end and the receiving end are as follows: Figure 1 and 2 shown.

[0035] The program of the present invention involves two ends, a sending end and a receiving end. The following describes how to use it from both ends:

[0036] Sending side:

[0037] Step 1: Customize the configuration file, configuration file and configuration items. The configuration items include working mode (Mode), ferry source, destination address and port (Lip, Lport, Dip, Dport), local IO buffer size (PKbLen), work queue (WorkQlen), task waiting queue (WaitQlen) and sending interval delay (UdpPause). First, the working mode sender is "active". The ferry source and destination are configured according to the actual network scenario. The local IO buffer size is one of the important indicators that affect the transmission rate. It needs to be configured according to the actual processing performance and memory size of the computer. The work queue and task waiting queue are the thresholds and processing performance of multi-task delivery. The size of the two queues must be configured according to the actual processing performance of the computer. The default work queue size is 8 and the task waiting queue size is 1024. The sending interval delay is one of the means to control the sending rate. It can directly control the transmission rate to ensure the success rate of file ferrying.

[0038] Step 2: After the configuration is completed, the service program can be started. Because the program in the present invention is developed in the form of a dynamic library, a main program needs to call its start() function to wake up the service program.

[0039] Step 3: After the program service is started, the sender can start the actual business work. Users can use the manage program + json configuration file included in the program to implement simple task distribution. json configuration file, such as Figure 4 As shown, the configuration items include the source file directory (inFileUrl), the destination file directory (outFileUrl), the ferry file name (filename), and whether the source file is deleted after the ferry is completed (del). Execute the "manage--json--index 15--pathtest.json" command to issue the file ferry task. "Index" is the unique identifier inside the bus and is fixed. "Path" points to the json configuration file. After the task is issued, if the command console returns "[code=0][success]", it means that the task is successfully issued. If the tasks issued in batches for multiple times exceed the length of the task waiting queue, then "[code=10000][failed]" will be returned to indicate that the task failed to be issued.

[0040] Step 4: You can view the task sending results through the log / cnfs.log. The logs starting with "[ERROR]" are the logs of sending failures, and the logs starting with "[INFO]" are the key record logs.

[0041] Receiver:

[0042] Step 1: Customize the configuration file, configuration file and configuration items. The configuration items include working mode (Mode), ferry source, destination address and port (Lip, Lport, Dip, Dport), local IO buffer size (PKbLen), out-of-order sorting method (SortType), out-of-order maximum tolerance interval (SortCount), maximum transmission queue (FileMonitor), default save directory (StoragePath), and same file processing strategy (SameFileModify). First, the receiving end of the working mode is "positive". The ferry source and destination are configured according to the actual network scenario. The local IO buffer size is one of the important indicators that affect the transmission rate. It needs to be configured according to the actual processing performance and memory size of the computer (consistent with the sending end). The out-of-order sorting method needs to be determined according to the actual network disorder of the computer. If the probability of disorder is very small, it is recommended to use the "dynamic" mode. If the probability of disorder is very high, it is recommended to use the "static" mode. The maximum tolerance interval of disorder, when the number of disorder is greater than the configured number, it is considered that packet loss has occurred, and it is necessary to adjust the sending interval delay of the sending end or adjust the system parameters. The default save directory is valid when the sender does not specify the file persistence directory on the receiver. The default strategy for handling identical files is to append "thread number-timestamp" to avoid file duplication, but if the "cover" mode is specified, it means overwriting the original file.

[0043] Step 2: After the configuration is completed, the service program can be started. Because the program of the present invention is developed in the form of a dynamic library, a main program needs to call its start() function to wake up the service program.

[0044] Step 3: After the program service is started, it starts to listen to the UDP socket of the local address and port. If the sent data arrives, it triggers the service of receiving and persisting it to the specified file directory.

[0045] Step 4: You can view the file receiving result through the log / cnfs.log, which indicates that the receiving is successful. You can also view the file in the actual persistence directory.

[0046] The present invention provides different services in the form of configuration files for different business scenarios (sending or receiving). Figure 3 and Figure 6For the default sender and receiver configuration files, there are two corresponding functions, active_conf_init() and positive_conf_init(), which are used to parse the configuration files and initialize according to the configuration. Similarly, different services also have their own processing functions, active_loop() and positive(). For logging, different services share a function print_debug_cnfs(fd,level,fmt,...).

[0047] Regarding the meaning of the parameters of the logging function, the first parameter "fd" represents the file to which the log is output. The actual parameter of this parameter is obtained by calling the system function open(); the second parameter "level" represents the level identifier at the beginning of the log record, such as Figure 5 and Figure 7 As shown, "[Error]" and "[Info]" are both level expressions. The actual parameters are based on the defined enumeration. Currently supported enumeration values ​​are "SWS_LOG_DEBUG", "SWS_LOG_INFO", "SWS_LOG_IMPT", "SWS_LOG_WARN" and "SWS_LOG_ERROR". Error logs generally use "SWS_LOG_ERROR" as the actual parameter of "level", so that the file will record logs with "[Error]" as the header identifier; the third parameter "fmt" represents the format string, that is, the actual content of the log record, which follows the formatted output style of c; the fourth parameter "..." represents a variable parameter, and the actual parameter is the actual number or character that needs to be formatted to replace the placeholder in "fmt";

[0048] For the initialization and processing functions, the following will elaborate on the specific implementation of the code from two business perspectives.

[0049] The specific implementation steps of the sender are as follows:

[0050] 1.1 In active_conf_init(), first parse the configuration file and get the parameters in the configuration file, "Lip", "Lport", "Dip", "Dport", "PKbLe", "WorkQlen", "WaitQlen" and "UdpPause".

[0051] 1.2 Then, start initialization, the initialization content is as follows.

[0052] 1.3 According to the data parsed by "Lip", "Lport", "Dip" and "Dport", the UDP socket is initialized to forward the file data, the socket is set to non-blocking mode, and the SO_SNDBUF type of setsockopt() is used to set the send buffer to avoid UDP packet loss at the sending end.

[0053] 1.4 According to the data parsed from "WorkQlen" and "WaitQlen", the work queue and task waiting queue are initialized, and malloc() and cnfs_task_queue_new() are used to create the work queue and task waiting queue respectively. The actual data structure of the work queue is an array (the length is the value of "WorkQlen"), while the task waiting queue (the maximum length is "WaitQlen") is a single-item list linked list that maintains the head pointer and tail pointer.

[0054] 1.5 So far, the configuration file parsing and initialization of the sender are completed. Next, we will start to explain the specific implementation of the business function active_loop().

[0055] 1.6active_loop() is mainly a while infinite loop, which is used to continuously receive the ferrying tasks issued by the bus. Once the request for file ferrying is received from the bus message, the file ferrying task is immediately added to the task waiting queue through the cnfs_task_queue_put() function. If the current waiting queue size exceeds the set threshold (the value of "WaitQlen"), the task addition fails. If no task is received, the task that has been queued to the task waiting queue begins to be processed. For the tasks that have been queued to the task waiting queue, the work queue is looped every time to see if there are extra positions to be queued. If not, the tasks in the current work queue are processed; if so, the tasks are taken out from the task waiting queue by calling the cnfs_task_queue_get() function and placed in the work queue, so that the next loop can start processing the tasks that have been queued to the work queue.

[0056] 1.7 When the work queue is working, it sends file data to the receiving end by calling the cnfs_udp_send() function. Each time, it traverses the task queue in sequence, reads the part of the source file that needs to be ferryed (local IO buffer, "PKblen" mentioned in the configuration file of the sender of the invention), and sends it to the receiving end using the udp socket.

[0057] 1.8 After the cnfs_udp_send() function, the linux_cpu_pause(sws_int_t n) function is called, and the input parameter is the "UdpPause" mentioned in the configuration file of the sender of the invention, which is used to control the sending rate of the sender to avoid sending too fast and causing packet loss at the receiving end.

[0058] The specific implementation steps of the receiving end are as follows:

[0059] 2.1 In positive_conf_init(), first parse the configuration file and obtain the parameters in the configuration file, "Lip", "Lport", "Dip", "Dport", "PKbLe", "SortType",

[0060] "SortCount", "MainCpu", "RcvCpu", "StoragePath" and

[0061] "SameFileModify".

[0062] 2.2 Then start initialization, the initialization content is as follows.

[0063] 2.3 Based on the data obtained from the analysis of "Lip", "Lport", "Dip" and "Dport", the initial

[0064] The UDP socket is used to forward file data, the socket is set to non-blocking mode, and the send buffer is set using the SO_SNDBUF type of setsockopt() to avoid UDP packet loss at the sending end.

[0065] 2.4 Initialize the UDP out-of-order processing function according to the value of "SortType".

[0066] "dynamic", the udp receiving thread needs to initialize a red-black tree container and a session. The session mainly includes three values, namely the old index value, the starting index value and the ending index value, which are used to process out-of-order data. The udp receiving thread receives the udp data in a loop, and enters out-of-order processing after receiving valid data: if the starting index of the session is 0 and the index of the current data is equal to the old index of the session plus one, it is considered normal data and the data can be queued. If the above conditions are not met, if the starting index is 0, the starting index is set to the old index plus one, and the ending index is the index of the current data. Then, if the current index is greater than the old index, the old index is updated to the current index value, and then memory is applied to save this data and insert it into the red-black tree container. The red-black tree container compares the value according to the value. When inserting a new node, the initial state is set to red, and then it is recolored according to the parent node of the node or the sibling node of the parent node. Then it is rotated and recolored according to the color of the parent node and the uncle node. Finally, after the whole tree is balanced and satisfies all the properties of the red-black tree, the in-order traversal of the whole tree will be an ordered sequence. After insertion, determine whether the starting index is equal to the current data index. If not, exit and re-receive. If they are equal, perform in-order traversal. Whenever the data index taken out of the container is subtracted from the old index by 1, it is sequential data, put into the queue, and increment the old index. When the data index is equal to the end index, clear the values ​​of the starting index and the end index, and exit. If the data index is not subtracted from the old index by 1, the starting index is equal to the old index plus 1. The final result is that if the udp data is not lost, the data in the queue will be ordered data. When a certain number of udp reception errors occur (refer to the "SortCount" mentioned in the configuration file of the receiving end of the invention), the udp receiving thread will also exit. If "SortType" is "static", initialize an array of a certain size (refer to the "SortCount" mentioned in the configuration file of the receiving end of the invention). The out-of-order processing flow refers to the red-black tree processing, but the memory does not need to be dynamically applied, and only needs to be entered and exited in the corresponding array subscript each time.

[0067] 2.5 Based on "RcvCpu", initialize a transceiver cache queue to cache the received data in this queue. At the same time, initialize a child thread, which is bound to the cpu (the value of "RcvCpu") to execute the reception and cache the data into the queue. By calling the cnfs_udp_recv() function to receive data, a fixed length of data will be read from the kernel buffer each time (local IO buffer, "PKblen" mentioned in the configuration file of the sender of the invention). Because the data received by udp may be out of order, it is necessary to call the out-of-order processing function mentioned in the previous step. The data after out-of-order processing is finally cached in the transceiver cache queue by calling the queue_put() function.

[0068] 2.6 At this point, the configuration file parsing and initialization of the receiving end are completed. Next, we will start to explain the specific implementation of the business function positive_loop().

[0069] 2.7 First, bind the CPU (the value of "MainCpu") of the main thread execution according to "MainCpu". Then start a while infinite loop to continuously fetch data from the send and receive cache queue. Because the data in the queue has been processed by the disorder processing function, it can be directly persisted to the hard disk. Finally, the path for data persistence and the processing method of the same file name are determined according to "StoragePath" and "SameFileModify". If it is "cover", it means overwriting, and if it is "rename", it means renaming the duplicate file. The renaming rule is "file name + timestamp".

[0070] The above contents are further detailed descriptions of the present invention in combination with specific preferred technical solutions, and the specific implementation of the present invention cannot be considered to be limited to these descriptions. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention, and these improvements should also be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for implementing multi-task file transfer based on a light gate system, characterized in that Contains the following steps on the sending and receiving ends: Sending side: Step 11: Customize the configuration file, configuration file and configuration items. The configuration items include working mode, ferry source, destination address and port, local IO buffer size, work queue, task waiting queue and sending interval delay. The working mode sender is set to "active", and the ferry source and destination address are configured according to the actual network scenario. Step 12: Start the service program; Step 13: Start task delivery through the manage program + json configuration file included in the service program; Step 14: Confirm the sending result through the log file; Receiver: Step 21: Customize the configuration file, configuration file and configuration items. The configuration items include working mode, ferry source, destination address and port, local IO buffer size, out-of-order sorting method, maximum out-of-order tolerance interval, maximum transmission queue, default save directory, and same file processing strategy. The working mode receiving end is "positive", and the ferry source and destination address are configured according to the actual network scenario. Step 22: Start the service program; Step 23: Start to listen to the UDP socket of the local address and port. If the sent data arrives, the service of receiving and persisting it to the specified file directory is triggered; Step 24: Confirm the file reception result through the log file or the actual persistence directory.

2. The method for implementing multi-task file transfer based on the optical shutter system according to claim 1, characterized in that: In step 11, the size of the local IO buffer needs to be configured according to the actual processing performance and memory size of the computer.

3. The method for implementing multi-task file transfer based on the optical shutter system as claimed in claim 1, characterized in that: The work queue and task waiting queue described in step 11 are the thresholds for multi-task delivery and the key to processing performance. The sizes of the two queues must be configured based on the actual processing performance of the computer. The sending interval delay is one of the means to control the sending rate, which can directly control the transmission rate to ensure the success rate of file ferrying.

4. The method for implementing multi-task file transfer based on the optical shutter system according to claim 3, characterized in that The default work queue size is 8, and the task waiting queue size is 1024.

5. The method for implementing multi-task file transfer based on the optical shutter system according to claim 1, characterized in that: In step 21, the size of the local IO buffer needs to be configured according to the actual processing performance and memory size of the computer to ensure consistency with the sending end.

6. The method for implementing multi-task file transfer based on the optical shutter system as claimed in claim 1, characterized in that: In step 21, the out-of-order sorting method needs to be determined according to the actual out-of-order situation of the computer network. If the probability of out-of-order is very small, select the "dynamic" mode; if the probability of out-of-order is very large, select the "static" mode.

7. The method for implementing multi-task file transfer based on the optical shutter system according to claim 1, characterized in that: When setting the maximum out-of-order tolerance interval in step 21, it is necessary to consider that when the out-of-order quantity is greater than the configured number, it is determined that packet loss occurs, and it is necessary to adjust the sending interval delay of the sending end or adjust the system parameters.

8. The method for implementing multi-task file transfer based on the optical shutter system as claimed in claim 1, characterized in that: In step 21, when the sending end does not specify a persistent directory for receiving end files, the default storage directory is valid.

9. The method for implementing multi-task file transfer based on the optical shutter system according to claim 1, characterized in that: In step 21, the same file processing strategy defaults to appending "thread number-timestamp" to avoid file duplication; if the "cover" mode is specified, it means overwriting the original file.

Images