File synchronization system based on kernel injection tracking

Abstract

The invention discloses a file synchronization system based on kernel injection tracking. The file synchronization system comprises an operation and maintenance management subsystem, a kernel injection subsystem, a change capture subsystem, a data transmission subsystem, a data updating subsystem and a file initialization subsystem, the file synchronization method of the file synchronization system comprises the following steps: step 1, when a process calls a modification function to write a synchronization file managed by an operation and maintenance management subsystem, a kernel injection subsystem sends call information to a change capture subsystem; step 2, the change capture subsystem receives the call information and packages the call information into change data, finds a corresponding change data queue according to a file name, adds the change data to the tail of the queue, and then notifies a data transmission subsystem; step 3, the data transmission subsystem constructs a file synchronization message and transmits the message to the data updating subsystem; 4, the data updating subsystem writes the file updating data into the corresponding file, and finally the file synchronization process is completed.

Description

Technical Field

[0001] This invention relates to the field of data synchronization, and in particular to a file synchronization system based on kernel injection tracing. Background Technology

[0002] With the rapid development of the internet, more and more business systems and tasks require file synchronization capabilities. File synchronization has several important functions:

[0003] (1) Improve work efficiency: It can automatically synchronize files from the source path to the target path, eliminating the tedious manual copying and pasting operations and greatly improving work efficiency. Whether on the same computer or on different devices, as long as it is connected to the Internet, files can be automatically synchronized and updated in real time.

[0004] (2) Ensure data security: Through automatic synchronization, files will be backed up to multiple devices or cloud storage. Even if a device malfunctions or a file is accidentally deleted, the file can still be recovered from other devices or the cloud to avoid data loss.

[0005] (3) Promote team collaboration: For cross-departmental and cross-regional team projects, file synchronization and sharing are the foundation for achieving efficient collaboration. Team members can jointly edit and modify files on the enterprise cloud drive, and update content in real time to achieve seamless collaboration. For example, software development teams can synchronize code and documents through the enterprise cloud drive, and members can understand the project progress and the work results of others in a timely manner, quickly solve problems and integrate resources, shorten the project development cycle, and improve the overall team collaboration efficiency.

[0006] Currently available file synchronization software either achieves synchronization by copying the entire file or by comparing and calculating file changes to achieve incremental file synchronization. Both of these methods are extremely bandwidth and CPU-intensive when synchronizing large files with minimal changes. Summary of the Invention

[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide a file synchronization system based on kernel injection tracing, which addresses the shortcomings of the existing technology.

[0008] To address the aforementioned technical problems, this invention discloses a file synchronization system based on kernel injection tracing, comprising: an operation and maintenance management subsystem, a kernel injection subsystem, a change capture subsystem, a data transmission subsystem, a data update subsystem, and a file initialization subsystem;

[0009] The operation and maintenance management subsystem manages file synchronization relationships, displays the synchronization topology, and provides real-time updates on synchronization status.

[0010] The kernel injection subsystem injects into all processes in the system using Linux ptrace technology, tracks and modifies the modification functions called by these processes. When a process calls a modification function, the kernel injection subsystem injects the process to trace the call information, including the file name, file location, length of data written, and file truncation information. If the file to which the data is written is the file that needs to be synchronized, the kernel injection subsystem sends the call information to the change capture subsystem.

[0011] The change capture subsystem stores, manages, and maintains data change information for all files that need to be synchronized. Business processes may continuously write to files, so file change data is different at different points in time. The change capture subsystem maintains a change data queue for each file that needs to be synchronized. Each element in the queue represents a file data change at a certain point in time. The queue elements are sorted according to the change time. When the change capture subsystem receives the call information sent by the kernel injection subsystem, it encapsulates the call information into change data, finds the corresponding change data queue according to the file name, adds the change data to the end of the queue, and then notifies the data transmission subsystem.

[0012] The data transmission subsystem is responsible for transmitting file change data on multiple servers that need to synchronize files. The data transmission subsystem scans the file synchronization relationship in real time, constructs a file synchronization message for the node that needs to be synchronized, and sends the message to the peer node. The peer node's data transmission subsystem receives the file synchronization message and passes it to the data update subsystem.

[0013] The data update subsystem receives file update data from the data transmission subsystem, writes the file update data into the corresponding file, and finally completes the file synchronization process.

[0014] The file initialization subsystem and data initialization subsystem first equalize the file data on both sides before file synchronization. (When establishing the initial file synchronization relationship, the source node will first copy the entire file to the destination node to ensure that the file data on the two nodes is consistent in the initial state; when the file data on the source node changes subsequently, the incremental data can be synchronized to the destination node in real time.)

[0015] The file synchronization steps based on the above kernel injection tracing file synchronization system include:

[0016] Step 1: When a process calls a modification function to write to a synchronized file managed by the operation and maintenance management subsystem, the kernel injection subsystem sends the call information to the change capture subsystem.

[0017] Step 2: Upon receiving the call information, the change capture subsystem encapsulates it into change data, locates the corresponding change data queue based on the filename, adds the change data to the end of the queue, and then notifies the data transmission subsystem.

[0018] Step 3: The data transmission subsystem constructs a file synchronization message and transmits the message to the data update subsystem;

[0019] Step 4: The data update subsystem writes the file update data to the corresponding file, thus completing the file synchronization process.

[0020] The operation and maintenance management subsystem performs synchronization relationship configuration management, specifically including:

[0021] Each server is treated as a synchronization node, and the file to be synchronized is treated as a synchronization object. Therefore, the synchronization relationship consists of a synchronization node and a synchronization object. Users configure the file synchronization relationship through the operation and maintenance management subsystem. The data structure sync_node represents the synchronization node, the data structure sync_file represents the file object to be synchronized, and the data structure sync_relation represents the synchronization relationship. When users need to synchronize files, they first create a synchronization node, then select the synchronization object, and finally establish the synchronization relationship. The operation and maintenance management subsystem creates the corresponding sync_node object, sync_file object, and sync_relation object.

[0022] The synchronization relationship configuration includes the following steps:

[0023] Step a1: The user selects the server where the file to be synchronized is located and configures the server information as a sync_node object.

[0024] In step a2, the user configures each file to be synchronized as an independent sync_file object.

[0025] In step a3, the user configures the sync_node object and the sync_file object into a sync_relation object based on the synchronization relationship.

[0026] Step a4: The user initiates the file initialization process, copying the source synchronization file to the destination synchronization file according to the synchronization direction, so that the files on both sides are initially aligned.

[0027] Step a5: Start file synchronization and begin incremental file synchronization.

[0028] The operation and maintenance management subsystem provides real-time display of the synchronization topology and synchronization status, specifically including: file synchronization supports one-to-many synchronization relationships and cascading synchronization relationships;

[0029] For example, node A synchronizes files to nodes B and C, node B synchronizes files to node D, and node D synchronizes files to node E. This combination of one-to-many and cascading synchronization relationships creates complex synchronization topologies. The real-time synchronization topology status display module graphically illustrates these relationships.

[0030] The synchronization action start / stop module can start or stop file synchronization between nodes.

[0031] The sync_node structure includes the following attributes:

[0032] node_id: Integer, node ID, uniquely identifies a synchronization node, and the node_id of a synchronization node is unique.

[0033] node_ip: String, the IP address of the node.

[0034] node_info: String, node description information.

[0035] sync_relations: An array of sync_relation objects, representing a set of synchronization relationships. A synchronization node can establish synchronization relationships with multiple other nodes.

[0036] The sync_file structure includes the following attributes:

[0037] file_id: Integer, file ID, uniquely identifies a synchronized file, different files have different node_id.

[0038] file_path: String, the path to the synchronized file.

[0039] file_info: String, information about the synchronized file.

[0040] The sync_relation structure includes the following attributes:

[0041] remote_node_id: Integer, the ID of the peer node.

[0042] file_id: Integer, the ID of the file to be synchronized.

[0043] `sync_direction`: Integer, indicating the synchronization direction: 1. Synchronize from local to peer; 2. Synchronize from peer to local. This file synchronization system does not support bidirectional file synchronization, only unidirectional synchronization.

[0044] status: An integer representing the synchronization status: 0, initialization incomplete, synchronization not yet started; 1, synchronization in progress, normal; 2, synchronization paused; 3, synchronization terminated. The synchronization relationship terminates when the user deletes it.

[0045] The operation and maintenance management subsystem can start or stop the synchronization action, which can start or stop file synchronization between nodes.

[0046] The sync_node structure includes the following attributes: int node_id (node ​​ID, which uniquely identifies a synchronization node; the node_id of a synchronization node is unique), char* node_ip (node ​​IP address), char* node_info (node ​​information), and sync_relation[] sync_relations (a collection of synchronization relationships; a node can establish synchronization relationships with multiple nodes and can synchronize multiple files simultaneously, so a node can contain multiple synchronization relationships at the same time).

[0047] The sync_file structure includes the following attributes: int file_id (file ID, which uniquely identifies a synchronization file; different files have different file_ids), char* file_path (synchronization file path), and char* file_info (synchronization file information).

[0048] The `sync_relation` structure includes the following attributes: `int remote_node_id` (peer node ID), `int file_id` (synchronization file ID), `int sync_direction` (synchronization direction: 1, synchronize from local to peer; 2, synchronize from peer to local. Bidirectional file synchronization is not supported, only one-way synchronization is supported), and `int status` (synchronization status: 0, file initialization not complete; 1, synchronization in progress, normal; 2, synchronization paused; 3, synchronization terminated. The synchronization relationship terminated when the synchronized file was deleted by the user).

[0049] The kernel injection subsystem includes:

[0050] The process scanning module uses the structure os_proc to represent an operating system process and establishes a process queue os_proc_list to represent the set of operating system processes. The process injection module scans the operating system processes, constructs an os_proc object for each process, and puts it into os_proc_list. At the same time, for objects that exist in os_proc_list but no longer have a corresponding process in the operating system, they are marked as invalid.

[0051] Process injection module: The process injection module scans the queue os_proc_list and injects into any valid process. It tracks the process's open, write, close, lseek, fsync, fdatasync, and sync system calls. When a business process modifies a file, it obtains the file modification content, constructs a file_mod object, and sends it to the change capture subsystem.

[0052] File scanning module: Scans for files that need to be synchronized. When a file is found to be deleted, it constructs a file_mod object and sends it to the change capture subsystem.

[0053] The os_proc object includes the following attributes:

[0054] pid: Integer, process ID.

[0055] process_name: String, process name.

[0056] process_path: String, the executable path of the process.

[0057] is_valid: Boolean value indicating whether the process is valid. If the process no longer exists in the operating system, then the process is invalid.

[0058] is_inject: Boolean value, indicating whether the injection into the process has been completed.

[0059] The file_mod object contains the following attributes:

[0060] syn_file_id: An integer representing the sync_file object ID corresponding to the file that needs to be synchronized.

[0061] flag: Integer, indicating the file modification method: 1. File write. 2. File truncation. 3. File deletion. 4. File creation.

[0062] `pos`: An integer representing the starting position of file writing. For file truncation, it represents the ending position. For file deletion, `pos` is 0, meaningless. For file creation, `pos` is always 0.

[0063] `length`: An integer representing the length written to a file. For truncation, it is 0 (meaningless). For file deletion, it is 0 (meaningless). For file creation, it represents the length of the newly created file.

[0064] For each modification to a file, the change capture subsystem constructs a corresponding `file_mod_cache` object representing that modification; it also constructs a queue `file_mod_list` to store all modifications to a file; and it constructs a file modification cache map.<file_id, file_mod_list> In file_mod_map, file_id is the ID of the sync_file object corresponding to the file to be synchronized, and file_mod_list is the file content modification queue corresponding to the file. The file change data caching module receives file change data file_mod objects sent by the kernel injection subsystem, finds the corresponding file_mod_list from file_mod_map, adds the file_mod to it, and then notifies the data transmission subsystem.

[0065] The file_mod_cache object contains the following attributes:

[0066] version: Integer, the file modification version number, initially 0, and incremented by 1 for each modification to the file.

[0067] is_valid: A boolean value indicating whether the modified data is valid. For example, if previously written data has been subsequently deleted, then the modification is invalid.

[0068] flag: Integer, indicating the file modification method: 1. File write. 2. File truncation. 3. File deletion. 4. File creation.

[0069] `pos`: An integer representing the starting position of file writing. For file truncation, it represents the ending position. For file deletion, `pos` is 0, meaningless. For file creation, `pos` is always 0.

[0070] `length`: An integer representing the length written to a file. For truncation, it is 0 (meaningless). For file deletion, it is 0 (meaningless). For file creation, it represents the length of the newly created file.

[0071] The data transmission subsystem includes the following functions:

[0072] Node data sending: For a specific file, construct a send cache queue file_send_list; construct a send cache map.<sync_node_id,file_send_list[]> send_cache; sync_node_id is the peer node ID, and the file_send_list array is a collection of file cache queues that need to be sent to this node; because it may be necessary to synchronize multiple files to a node at the same time, the value is the queue collection, and the number of elements in the queue collection is the number of files that need to be synchronized; for each node, a thread is started to send the corresponding data in send_cache.

[0073] Node data reception: Construct a receive cache map<sync_node_id,file_recv_list[]> recv_cache; sync_node_id is the sender node ID, and the file_recv_list array is a collection of file data sent by the peer node that needs to be synchronized locally. Each element in the collection corresponds to the data to be synchronized for a file.

[0074] The file_send_list contains the following attributes: int sync_file_id (the file ID to be synchronized), file_mod_cache[] send_list (the file modification data to be synchronized to the peer), and int remote_recv_id (the latest file_mod_cache ID that the peer has indicated it has received and sent locally).

[0075] file_recv_list contains the following attributes: int sync_file_id (the file ID that needs to be synchronized locally) and file_data[] recv_list (the file data that needs to be synchronized locally).

[0076] The file synchronization system based on kernel injection tracing as described in claim 1 is characterized in that, in step (42), file_data contains the following attributes: int version (file modification version number, the corresponding version will increase by 1 for each file modification), int flag (file modification method: 1, file write; 2, file truncation; 3, file deletion; 4, file creation), int pos (for file write, it indicates the starting position of the file write; for file truncation, it indicates the file truncation position; for file deletion, pos is 0, meaningless; for file creation, pos is fixed at 0), char[] data (file data to be synchronized; for file write and creation, it indicates the file data written; for truncation and file deletion, there is no content), int length (for file write, it indicates the writing length; for truncation, the content is 0, meaningless; for file deletion, the content is 0, meaningless; for file creation, it indicates the content length of the newly created file).

[0077] The file_send_list contains the following attributes:

[0078] sync_file_id: An integer representing the ID of the file to be synchronized.

[0079] send_list: A collection of file_mod_cache files that need to be synchronized to the other end.

[0080] remote_recv_version: An integer representing the version of file_mod_cache that the peer has received and sent locally.

[0081] The file_recv_list contains the following attributes:

[0082] sync_file_id: An integer representing the file ID that needs to be synchronized locally.

[0083] recv_list: A collection of file_data objects representing the file data that needs to be synchronized locally.

[0084] The file_data contains the following attributes:

[0085] version: an integer, the file's version number. Each time the file is modified, the corresponding version number increases by 1.

[0086] flag: Integer, indicating the file modification method: 1. File write. 2. File truncation. 3. File deletion. 4. File creation.

[0087] `pos`: An integer representing the starting position of file writing. For file truncation, it represents the ending position. For file deletion, `pos` is 0, meaningless. For file creation, `pos` is always 0.

[0088] `data`: A character array representing the file data to be synchronized. For file writing and creation, it represents the data written to the file. For truncation and file deletion, there is no content.

[0089] `length`: An integer representing the length written to a file. For truncation, it is 0 (meaningless). For file deletion, it is 0 (meaningless). For file creation, it represents the length of the newly created file.

[0090] The data update subsystem reads the receive cache recv_cache of the data transmission subsystem, updates the data corresponding to each file_recv_list to the corresponding file system, and then sends an acknowledgment to the sender to indicate that the corresponding data modification has been synchronized.

[0091] Before synchronizing files, the data initialization subsystem first levels the file data on both sides: The data initialization subsystem traverses all synchronization nodes sync_node, and for all synchronization relationships sync_relation, if the sync_relation value is 0, the source file is sent completely to the destination to achieve initial file leveling, and then the sync_relation value is changed to 1.

[0092] The modified functions include open, write, close, lseek, fsync, fdatasync, and sync.

[0093] Beneficial effects:

[0094] This invention injects and tracks all processes in the operating system, capturing real-time modifications to file data by business processes, storing the changed data in the peer's file system, and ultimately completing file data synchronization. This system avoids repeated scanning and comparison of the file system, efficiently capturing file change data and achieving efficient file system synchronization with low disk I / O and CPU utilization. Attached Figure Description

[0095] Figure 1 This is a diagram illustrating a typical use case for file synchronization software.

[0096] Figure 2 This is a schematic diagram of the overall structure of the file synchronization system.

[0097] Figure 3 This is a schematic diagram of the operation and maintenance management subsystem.

[0098] Figure 4 This is a schematic diagram of the kernel injection subsystem structure.

[0099] Figure 5 This is a schematic diagram of the change capture subsystem structure.

[0100] Figure 6 This is a schematic diagram of the change capture cache optimization process.

[0101] Figure 7 This is a schematic diagram of the data transmission submodule of the data transmission subsystem.

[0102] Figure 8 This is a schematic diagram of the data receiving submodule of the data transmission subsystem.

[0103] Figure 9 This is a schematic diagram of the data update subsystem structure.

[0104] Figure 10 This is a schematic diagram of the file initialization subsystem structure. Detailed Implementation

[0105] File synchronization software is typically used to ensure the consistency of business data files in a business software cluster deployment. The scenario in this embodiment is as follows: Figure 1 As shown, a certain business communication software, for the purpose of business disaster recovery and resilience, supports a 3-machine cluster deployment. The communication software will keep the communication information configured by the user in the address book. In order to realize the cluster capability of the communication software, the user operates as follows:

[0106] 1. The user deploys the communication software and file synchronization system on 3 machines.

[0107] 2. User configuration file synchronization relationship: The synchronization nodes include the three deployed machines, the synchronization object is the address book file, and the synchronization relationship is that the address book files of the three machines are synchronized with each other.

[0108] 3. When a user updates the address book on the master node through the communication software interface, the file synchronization software kernel injection subsystem captures the changes in the address book file data and sends the updated address book data information to the file synchronization software change capture subsystem.

[0109] 4. The change capture subsystem encapsulates contact change information into change data, finds the corresponding change data queue based on the file name, adds the change data to the end of the queue, and then notifies the file synchronization software data transmission subsystem.

[0110] 5. The file synchronization software data transmission subsystem constructs a file synchronization message and sends the file synchronization message to the file synchronization software on the other two nodes.

[0111] 6. After receiving the changed data, the data update subsystems of the other two node file synchronization software update the corresponding address book files, so that the address book files of the three machines are consistent.

[0112] 7. During the switchover of the communication software, the software can continue to work normally because the contents of the address book file are the same.

[0113] like Figure 2 As shown, a file synchronization system based on kernel injection tracing includes: an operation and maintenance management subsystem, a kernel injection subsystem, a change capture subsystem, a data transmission subsystem, a data update subsystem, and a file initialization subsystem;

[0114] The operation and maintenance management subsystem manages file synchronization relationships, displays the synchronization topology, and provides real-time updates on synchronization status.

[0115] The kernel injection subsystem injects itself into all processes in the system using Linux ptrace technology, tracing and modifying the modification functions called by these processes. When a process calls a modification function, the kernel injection subsystem injects the process's traced call information, including the filename of the data being written, the file location, the length of the data being written, and file truncation information. If the file being written is the same file that needs to be synchronized, the kernel injection subsystem sends the call information to the change capture subsystem. The modification functions include open, write, close, lseek, fsync, fdatasync, and sync.

[0116] The change capture subsystem stores, manages, and maintains data change information for all files that need to be synchronized. The change capture subsystem maintains a change data queue for each file that needs to be synchronized. Each element in the queue represents a file data change at a certain point in time. The queue elements are sorted according to the change time. When the change capture subsystem receives the call information sent by the kernel injection subsystem, it encapsulates the call information into change data, finds the corresponding change data queue according to the file name, adds the change data to the end of the queue, and then notifies the data transmission subsystem.

[0117] The data transmission subsystem is responsible for transmitting file change data on multiple servers that need to synchronize files. The data transmission subsystem scans the file synchronization relationship in real time, constructs a file synchronization message for the node that needs to be synchronized, and sends the message to the peer node. The peer node's data transmission subsystem receives the file synchronization message and passes it to the data update subsystem.

[0118] The data update subsystem receives file update data from the data transmission subsystem, writes the file update data into the corresponding file, and finally completes the file synchronization process.

[0119] The file initialization subsystem and the data initialization subsystem first level the file data on both sides before file synchronization; (when establishing the initial file synchronization relationship, the source node will first copy the entire file to the destination node to ensure that the file data of the two nodes are consistent in the initial state; when the file data of the source node changes subsequently, the incremental data can be synchronized to the destination node in real time.)

[0120] The file synchronization steps based on the above kernel injection tracing file synchronization system include:

[0121] Step 1: When a process calls a modification function to write to a synchronized file managed by the operation and maintenance management subsystem, the kernel injection subsystem sends the call information to the change capture subsystem.

[0122] Step 2: Upon receiving the call information, the change capture subsystem encapsulates it into change data, locates the corresponding change data queue based on the filename, adds the change data to the end of the queue, and then notifies the data transmission subsystem.

[0123] Step 3: The data transmission subsystem constructs a file synchronization message and transmits the message to the data update subsystem;

[0124] Step 4: The data update subsystem writes the file update data to the corresponding file, thus completing the file synchronization process.

[0125] The operation and maintenance management subsystem performs synchronization relationship configuration management, specifically including:

[0126] Each server is treated as a synchronization node, and the file to be synchronized is treated as a synchronization object. Therefore, the synchronization relationship consists of a synchronization node and a synchronization object. Users configure the file synchronization relationship through the operation and maintenance management subsystem. The data structure sync_node represents the synchronization node, the data structure sync_file represents the file object to be synchronized, and the data structure sync_relation represents the synchronization relationship. When users need to synchronize files, they first create a synchronization node, then select the synchronization object, and finally establish the synchronization relationship. The operation and maintenance management subsystem creates the corresponding sync_node object, sync_file object, and sync_relation object.

[0127] like Figure 3As shown, to implement the operation and maintenance management subsystem, three modules are required: a synchronization relationship configuration module, a synchronization topology display module, and a synchronization start / stop module. The operation and maintenance management subsystem maintains global synchronization relationship configuration information `sync_node[] g_sync_node` and synchronization file information `g_sync_file`. `g_sync_node` is an array of multiple `sync_node` objects, and `g_sync_file` is an array of multiple `sync_file` objects. The synchronization relationship configuration management module generates synchronization configuration information, constructs `sync_node` and `sync_file` object instances, and adds them to `g_sync_nodes` and `g_sync_file`. The real-time synchronization topology status display module reads the synchronization configuration information from the `sync_node` object instances and displays the synchronization topology relationship. The synchronization start / stop module starts or stops the data synchronization relationship based on the synchronization configuration information stored in `g_sync_nodes`.

[0128] The operation and maintenance management subsystem provides real-time display of the synchronization topology and synchronization status, including: file synchronization supports one-to-many synchronization relationships and cascading synchronization relationships. For example, node A can simultaneously synchronize file data to nodes B and C, and also synchronize file data to node D, and then node D synchronizes file data to node E; combinations of one-to-many synchronization relationships and cascading synchronization relationships can construct complex synchronization topologies, and the operation and maintenance management subsystem supports graphical display of synchronization topologies;

[0129] The operation and maintenance management subsystem can start or stop the synchronization action, which can start or stop file synchronization between nodes.

[0130] like Figure 4As shown, to implement the kernel injection subsystem, three modules are required: a process scanning module, a process injection module, and a file scanning module. The kernel injection subsystem maintains a global variable `os_proc[] g_os_proc`, which stores all current operating system information. The process scanning module scans all operating system processes in real time. When a new process is created, it constructs an `os_proc` object instance, adds it to `g_os_proc`, and notifies the process injection module to inject into the process. When a process exits, it removes the corresponding `os_proc` object instance from `g_os_proc` and notifies the process injection module to stop injection. The process injection module scans `g_os_proc`. For processes not yet injected, it uses Linux ptrace technology to trace the process's `open`, `write`, `close`, `lseek`, `fsync`, `fdatasync`, and `sync` function calls. When a process calls these functions, the process injection module obtains the call information, constructs a `file_mod` object, and sends the object to the change capture submodule. The file scanning module scans all synchronized files in real time. When it detects the creation or deletion of a file, it constructs a `file_mod` object and sends the object to the change capture submodule.

[0131] For each modification to a file, the change capture subsystem constructs a corresponding `file_mod_cache` object representing that modification; it also constructs a queue `file_mod_list` to store all modifications to a file; and it constructs a file modification cache map.<file_id, file_mod_list> In file_mod_map, file_id is the ID of the sync_file object corresponding to the file to be synchronized, and file_mod_list is the file content modification queue corresponding to the file. The file change data caching module receives file change data file_mod objects sent by the kernel injection subsystem, finds the corresponding file_mod_list from file_mod_map, adds the file_mod to it, and then notifies the data transmission subsystem.

[0132] like Figure 5 As shown, to implement the change capture subsystem, two sub-modules are required: a change data storage module and a cache optimization module. The change data storage module constructs a global file change data cache, `file_mod_map` and `g_file_mod_map`, which stores all file data change information. The cache optimization module scans newly generated file change data to assess its impact on older change data. For example, if older file change data includes 100 bytes of data written at a certain location in a file, and subsequent new file changes delete all of this data, then the old written data becomes invalid. The cache optimization process is as follows... Figure 6 As shown.

[0133] To implement the data transmission subsystem, two modules are needed: a data sending submodule and a data receiving submodule. For the sending module, such as... Figure 7 As shown, a send cache (send_cache) is constructed, and a dedicated send thread is created for each send node to be responsible for sending data synchronized to that node. For the receive module, as... Figure 8 As shown, a receive buffer recv_cache is established to store the received messages, and then the data update subsystem is notified.

[0134] Implement a data update subsystem, such as Figure 9 As shown, it receives notifications from the data transmission subsystem, reads the file data to be updated from recv_cache, and updates the data to the file system.

[0135] Implement a file initialization subsystem, such as Figure 10 As shown, upon receiving a notification from the operations and maintenance management subsystem, it reads information about uninitialized nodes from `g_sync_node` and calls the `scp` command to complete the initialization. Each file's initialization is handled by a separate thread that calls the `scp` command; the thread terminates upon successful execution.

[0136] The data transmission subsystem includes the following functions:

[0137] Node data sending: For a specific file, construct a send cache queue file_send_list; construct a send cache map.<sync_node_id,file_send_list[]> send_cache; sync_node_id is the peer node ID, the file_send_list array is the collection of file cache queues to be sent to this node; value is the queue collection, the number of elements in the queue collection is the number of files to be synchronized; for each node, start a thread to send the corresponding data in send_cache.

[0138] Node data reception: Construct a receive cache map<sync_node_id,file_recv_list[]> recv_cache; sync_node_id is the sender node ID, and the file_recv_list array is a collection of file data sent by the peer node that needs to be synchronized locally. Each element in the collection corresponds to the data to be synchronized for a file.

[0139] The data update subsystem reads the receive cache recv_cache of the data transmission subsystem, updates the data corresponding to each file_recv_list to the corresponding file system, and then sends an acknowledgment to the sender to indicate that the corresponding data modification has been synchronized.

[0140] Before synchronizing files, the data initialization subsystem first levels the file data on both sides: The data initialization subsystem traverses all synchronization nodes sync_node, and for all synchronization relationships sync_relation, if the sync_relation value is 0, the source file is sent completely to the destination to achieve initial file leveling, and then the sync_relation value is changed to 1.

[0141] This invention provides a file synchronization system based on kernel injection tracing. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A file synchronization system based on kernel injection tracing, characterized in that, include: Operation and maintenance management subsystem, kernel injection subsystem, change capture subsystem, data transmission subsystem, data update subsystem, and file initialization subsystem; The operation and maintenance management subsystem manages file synchronization relationships, displays the synchronization topology, and provides real-time updates on synchronization status. The kernel injection subsystem injects into all processes in the system, tracks and modifies the modification functions called by these processes. When a process calls a modification function, the kernel injection subsystem injects the process to track the call information, including the file name of the data to be written, the file location, the length of the data to be written, and the file truncation information. If the file to which the data is written is the file that needs to be synchronized, the kernel injection subsystem sends the call information to the change capture subsystem. The change capture subsystem stores, manages, and maintains data change information for all files that need to be synchronized. The change capture subsystem maintains a change data queue for each file that needs to be synchronized. Each element in the queue represents a file data change at a certain point in time. The queue elements are sorted according to the change time. When the change capture subsystem receives the call information sent by the kernel injection subsystem, it encapsulates the call information into change data, finds the corresponding change data queue according to the file name, adds the change data to the end of the queue, and then notifies the data transmission subsystem. The data transmission subsystem is responsible for transmitting file change data on multiple servers that need to synchronize files. The data transmission subsystem scans the file synchronization relationship in real time, constructs a file synchronization message for the node that needs to be synchronized, and sends the message to the peer node. The peer node's data transmission subsystem receives the file synchronization message and passes it to the data update subsystem. The data update subsystem receives file update data from the data transmission subsystem, writes the file update data into the corresponding file, and finally completes the file synchronization process. The file initialization subsystem and the data initialization subsystem first equalize the file data on both sides before file synchronization.

2. The file synchronization system based on kernel injection tracing according to claim 1, characterized in that, The file synchronization steps based on the above kernel injection tracing file synchronization system include: Step 1: When a process calls a modification function to write to a synchronized file managed by the operation and maintenance management subsystem, the kernel injection subsystem sends the call information to the change capture subsystem. Step 2: Upon receiving the call information, the change capture subsystem encapsulates it into change data, locates the corresponding change data queue based on the filename, adds the change data to the end of the queue, and then notifies the data transmission subsystem. Step 3: The data transmission subsystem constructs a file synchronization message and transmits the message to the data update subsystem; Step 4: The data update subsystem writes the file update data to the corresponding file, thus completing the file synchronization process.

3. A file synchronization system based on kernel injection tracing according to claim 1, characterized in that, The operation and maintenance management subsystem performs synchronization relationship configuration management, specifically including: Each server is treated as a synchronization node, and the file to be synchronized is treated as a synchronization object. Therefore, the synchronization relationship consists of a synchronization node and a synchronization object. Users configure the file synchronization relationship through the operation and maintenance management subsystem. The data structure sync_node represents the synchronization node, the data structure sync_file represents the file object to be synchronized, and the data structure sync_relation represents the synchronization relationship. When users need to synchronize files, they first create a synchronization node, then select the synchronization object, and finally establish the synchronization relationship. The operation and maintenance management subsystem creates the corresponding sync_node object, sync_file object, and sync_relation object.

4. A file synchronization system based on kernel injection tracing according to claim 3, characterized in that, The operation and maintenance management subsystem provides real-time display of the synchronization topology and synchronization status, including: file synchronization supports one-to-many synchronization relationships and cascading synchronization relationships. Combinations of one-to-many synchronization relationships and cascading synchronization relationships can construct complex synchronization topology relationships. The operation and maintenance management subsystem supports graphical display of synchronization topology relationships. The operation and maintenance management subsystem can start or stop the synchronization action, which can start or stop file synchronization between nodes.

5. A file synchronization system based on kernel injection tracing according to claim 1, characterized in that, The kernel injection subsystem includes: The process scanning module uses the structure os_proc to represent an operating system process and establishes a process queue os_proc_list to represent the set of operating system processes. The process injection module scans the operating system processes, constructs an os_proc object for each process, and puts it into the process queue os_proc_list. At the same time, for objects that exist in the process queue os_proc_list but no longer have a corresponding process in the operating system, they are marked as invalid. Process injection module: The process injection module scans the process queue os_proc_list. As long as a process is valid, it injects into it, tracks the process system calls, and when the business process modifies a file, it obtains the file modification content, constructs a file_mod object, and sends it to the change capture subsystem. File scanning module: Scans for files that need to be synchronized. When a file is found to be deleted, it constructs a file_mod object and sends it to the change capture subsystem.

6. A file synchronization system based on kernel injection tracing according to claim 2, characterized in that, For each modification to a file, the change capture subsystem constructs a corresponding `file_mod_cache` object representing that modification; it also constructs a queue `file_mod_list` to store all modifications to a file; and it constructs a file modification cache map.<file_id, file_mod_list> In file_mod_map, file_id is the ID of the sync_file object corresponding to the file to be synchronized, and file_mod_list is the file content modification queue corresponding to the file. The file change data cache module receives the file change data file_mod object sent by the kernel injection subsystem, finds the corresponding column file_mod_list from the file modification cache file_mod_map, adds the file change data file_mod to it, and then notifies the data transmission subsystem.

7. A file synchronization system based on kernel injection tracing according to claim 1, characterized in that, The data transmission subsystem includes the following functions: Node data sending: For a specific file, construct a send cache queue file_send_list; construct a send cache map.<sync_node_id,file_send_list[]> send_cache; the key is the peer node ID, and the value is the set of file cache queues to be sent to that node; the value is the queue set, and the number of elements in the queue set is the number of files to be synchronized; for each node, a thread is started to send the corresponding data in the send_cache; Node data reception: Construct a receive cache map<sync_node_id,file_recv_list[]> recv_cache; The key is the sender node ID, and the value is a collection of file data sent by the peer node that needs to be synchronized locally. Each element in the collection corresponds to the data to be synchronized for a file.

8. A file synchronization system based on kernel injection tracing according to claim 1, characterized in that, The data update subsystem reads the receive cache recv_cache of the data transmission subsystem, updates the data corresponding to each file_recv_list to the corresponding file system, and then sends an acknowledgment to the sender to indicate that the corresponding data modification has been synchronized.

9. A file synchronization system based on kernel injection tracing as described in claim 1, characterized in that, Before synchronizing files, the data initialization subsystem first levels the file data on both sides: The data initialization subsystem traverses all synchronization nodes sync_node, and for all synchronization relationships sync_relation, if the sync_relation value is 0, the source file is sent completely to the destination to achieve initial file leveling, and then the sync_relation value is changed to 1.

10. A file synchronization system based on kernel injection tracing as described in claim 1, characterized in that, The modified functions include the open function, write function, close function, lseek function, fsync function, fdatasync function, and sync function.

Images