A file synchronization method of a bidirectional gateway and a bidirectional gateway

Abstract

The application provides a file synchronization method and a two-way gateway, historical file information in a target directory of a source file server is stored into a file information database of a sending end, current file information in the target directory is obtained through a query thread pool, and file changes of the source file server can be accurately detected according to the current file information and the historical file information; once the changes are detected, the change information and the target file are immediately sent to a receiving end, so that the synchronization accuracy of the file is ensured. Furthermore, concurrent processing can be performed by using the query thread pool, a first task thread pool and a second task thread pool, and a large number of file synchronization tasks can be efficiently processed. When file information is saved in the memory, only relevant information in the target directory of the source file server is stored, and the whole file information does not need to be stored, so that the memory occupation is reduced, and the system performance is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, in particular to a file synchronization method of a two-way gateway and the two-way gateway. BACKGROUND

[0002] The two-way gateway is deployed between networks of different security levels, and realizes data transmission across networks and domains through a physical logical isolation card. The hardware architecture of the two-way gateway usually adopts a 2+1 architecture, that is, composed of a sending host, a receiving host and a two-way isolation component, and the two-way isolation component is composed of two isolation cards and an optical fiber connection. The network proxy program and the security engine running on the sending host and the receiving host form a software architecture, and the security host is formed through the hardware architecture and the software architecture, and finally the security host provides data transmission service across networks and domains.

[0003] The sending host and the receiving host of the two-way gateway adopt proxy technology to receive and send network data on both sides of the equipment, unload data below the network layer, extract application protocols and data according to the application protocol filtering function, perform security check on the data, encapsulate the safe data by using a private protocol, and realize data transmission across networks and domains through the isolation component between the two hosts.

[0004] The file synchronization function is an important function of the two-way gateway, and the file synchronization module of the equipment runs in the system of the sending end and the receiving end of the equipment respectively. The file synchronization module of the sending end queries the file state of the file server through the file protocol such as Ftp, queries the addition, modification and deletion actions of the file through the file protocol, downloads the file, encapsulates the file data by using a private protocol, transmits the file data to the file synchronization module of the receiving host through the isolation component, and finally uploads the file by using the file protocol to realize the file synchronization function.

[0005] However, the file synchronization function module of the existing two-way gateway occupies a large amount of memory when saving file information, which reduces the system performance. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a file synchronization method of a two-way gateway and the two-way gateway, so as to solve the problem that the file synchronization function module of the existing two-way gateway occupies a large amount of memory when saving file information, which reduces the system performance.

[0007] The file synchronization method of the two-way gateway provided by the embodiments of the present application, the two-way gateway includes a sending end and a receiving end, the sending end includes a query thread pool, a first task thread pool and a file information database; wherein the file information database includes historical file information of a source file server;

[0008] The file synchronization method includes:

[0009] The current file information in the target directory of the source file server is obtained by using the query thread pool, the current file information is compared with the historical file information, if the current file information is inconsistent with the corresponding file information in the file information database, file change information is obtained, the file change information is sent to the first task thread pool, and the current file information is updated in the file information database; wherein the file change information is used to indicate an added file, a deleted file or a changed file.

[0010] The file change information is obtained by using the first task thread pool, if the file change information indicates an added file or a changed file, the added or changed target file is downloaded, and the file change information and the target file are sent to the receiving end.

[0011] The file change information and the target file are received by using the second task thread pool of the receiving end, and the file change information and the target file are sent to the target file server.

[0012] In the above technical solution, the historical file information in the target directory of the source file server is stored in the file information database of the sending end, the current file information in the target directory is obtained by using the query thread pool, and the file change of the source file server can be accurately detected according to the current file information and the historical file information. Once the change is detected, the change information and the target file are immediately sent to the receiving end, ensuring the synchronization accuracy of the file. At the same time, in the two-way gate, the internal network and the external network can be isolated, effectively preventing network attacks and data leakage. Moreover, the query thread pool, the first task thread pool and the second task thread pool can be used for concurrent processing, and a large number of file synchronization tasks can be efficiently processed. The method of the embodiment only stores the related information in the target directory of the source file server when using the memory to save the file information, without storing the entire file information, thereby reducing the memory occupation and improving the system performance.

[0013] In some optional embodiments, after obtaining the file change information, the method further includes:

[0014] If the file change information indicates a deleted file, the file change information is sent to the receiving end.

[0015] In the above technical solution, the deleted file situation can also be accurately detected and processed. Once the information of the deleted file is detected, the file change information is immediately sent to the receiving end, ensuring that the deletion operation of the file is accurately and correctly performed.

[0016] In some optional embodiments, the query thread pool includes a query thread and a record thread.

[0017] Obtaining current file information in a target directory of a source file server, comparing the current file information with historical file information, if the current file information is inconsistent with corresponding file information in a file information database, obtaining file change information, and sending the file change information to a first task thread pool, comprising:

[0018] Using a query thread, obtaining current file information in a target directory of a source file server, storing the current file information in a file information structure, and setting a file state of each file in the file information structure to an initial state;

[0019] Using a record thread, comparing the current file information in the file information structure with historical file information, and obtaining a comparison result;

[0020] According to the comparison result, updating the file state of each file in the file information structure;

[0021] Using the query thread, obtaining file change information according to the file state of each file, and sending the file change information to the first task thread pool.

[0022] In the above technical solution, the separation of the query thread and the record thread enables the file query and comparison operations to be performed in parallel. The query thread is responsible for obtaining and storing the current file information, while the record thread is responsible for comparing the current and historical file information, which greatly improves the overall file synchronization efficiency. By setting the initial state of each file in the file information structure and updating the file state based on the comparison result, the accuracy and completeness of the file change information can be ensured, which can avoid missing or false detection of file changes. By dividing the complex file synchronization process into multiple independent sub-tasks, each sub-task is processed by a dedicated thread, which can reduce the complexity of the entire system and improve the stability and reliability of the system. Even if a thread fails, it will not affect the normal operation of other threads, and the system can be easily expanded to handle more source file servers by simply increasing the number of query threads and record threads to handle more file synchronization tasks.

[0023] In some optional embodiments, the sending end further includes a hash table pool and a linked list pool; the hash table pool and the linked list pool both adopt a lock-free circular queue, the hash table in the hash table pool is in an initialization state, and the linked list in the linked list pool is in an initialization state;

[0024] The file information structure includes: a file name storage, a file size storage, a file modification time storage, a parent directory node storage, a hash table pointer, a linked list pointer, and a file state; wherein the file state includes an initial state, an added state, a modified state, a deleted state, or an unchanged state.

[0025] In the technical solution, the hash table pool and the linked list pool are designed as lock-free circular queues, and the query thread, the record thread and the task thread can operate the hash table pool and the linked list pool simultaneously, thereby achieving high parallelism, which helps to improve the overall file synchronization efficiency. The design of the lock-free circular queue has high stability and reliability, and can avoid the deadlock and performance problems caused by lock competition.

[0026] In some optional embodiments, the current file information in the target directory of the source file server is acquired, the current file information is stored in a file information structure, and the file state of each file in the file information structure is set to an initial state. The method further includes:

[0027] From the hash table pool and the linked list pool, available hash tables and linked lists are acquired.

[0028] The file information structure is linked to the linked list, and the file information structure is added to the hash table.

[0029] The query message, the hash table and the linked list are sent to the record thread. The query message includes a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing the query.

[0030] In the technical solution, two query methods of hash table search and linked list search are provided. The hash table is a data structure for direct access according to a key value, and the key value is mapped to a position in the table to access the record, so as to speed up the search. The linked list is a commonly used data structure, and its advantages include that the size of the linked list is dynamic, and the node can be added or reduced at any time according to the requirement, unlike the array which needs to pre-allocate fixed memory space. In the linked list, the insertion and deletion of the node only need to modify the pointing of the pointer, and do not need to move a large amount of data, so the insertion and deletion are efficient. The elements in the linked list are stored in the order of insertion, and the nodes can be accessed sequentially, which is suitable for the case of traversing the linked list. Each node in the linked list contains data and a pointer to the next node, so compared with the array, the memory space can be more fully utilized.

[0031] In some optional embodiments, the current file information of the file information structure is compared with the historical file information to obtain a comparison result, which includes:

[0032] The query message is read to acquire the file names of all files in the target directory in the file information database, and the hash table is queried according to the file name. The file state is updated according to the result of querying the hash table.

[0033] In some optional embodiments, the file state is updated according to the result of querying the hash table, which includes:

[0034] If the first file in the target directory in the file information database is found in the hash table, and the file size or the file modification time of the first file recorded in the file information database and the file information structure body are inconsistent, the file state of the first file in the file information structure body is set to the change state;

[0035] If the second file in the target directory in the file information database is found in the hash table, and the file size and the file modification time of the second file recorded in the file information database and the file information structure body are consistent, the file state of the second file in the file information structure body is set to the unchanged state.

[0036] If the third file in the target directory in the file information database is not found in the hash table, the file information corresponding to the third file is stored in the file information structure body, and is added to the hash table and the linked list, and the file state of the third file in the file information structure body is set to the deleted state.

[0037] In the above technical solution, the file information is obtained by querying the hash table to update the file state, and compared with directly traversing the file system, the operation complexity of the hash table query is O(1), which has high efficiency. At the same time, the file retrieval is also faster according to the hash table query based on the file name.

[0038] In some optional embodiments, after updating the file state according to the result of querying the hash table, the method further comprises:

[0039] The first result message, the hash table and the linked list are sent to the query thread by using the record thread; wherein the first result message comprises: the hash table pointer file size, the linked list pointer, the storage directory information of the hash table and the linked list, and the information representing the end of the hash table query.

[0040] The first result message is received by using the query thread, and the file state is updated by traversing the linked list; wherein the file state is updated by traversing the linked list, comprising: if the file state of the fourth file in the file information structure body is the initial state, the file state of the fourth file is set to the new state.

[0041] In the above technical solution, when the file state is updated according to the result of querying the hash table, the file state of the changed file has been updated from the initial state to the change state, the file state of the unchanged file has been updated from the initial state to the unchanged state, and the file state of the deleted file has been updated from the initial state to the deleted state. Then, the query thread is used to traverse the linked list, find the file whose file state is still in the initial state, and update the file state of these files to the new state.

[0042] In some optional embodiments, after traversing the linked list to update the file state, the method further comprises:

[0043] The query thread is used to send a processing message, a hash table and a linked list to the first task thread pool; wherein the processing message includes: a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing task processing;

[0044] The first task thread pool is used to receive the processing message, download the target file and send file change information to the receiving end, or send the file change information to the receiving end; the second result message, the hash table and the linked list are fed back to the query thread; wherein the second result message includes: a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing the end of linked list traversal;

[0045] The query thread is used to receive the second result message, and send a storage message, a hash table and a linked list to the record thread; wherein the storage message includes: a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing storage to the file information database;

[0046] The record thread is used to update the historical file information in the file information database to the current file information, release the occupied memory of the file information structure, return the linked list to the linked list pool, and return the hash table to the hash table pool.

[0047] In the above technical solution, the concurrent processing capabilities of the first task thread pool and the query thread are utilized to handle multiple tasks simultaneously, such as downloading files and sending file change information. This helps to improve the overall processing speed and efficiency. The asynchronous operation design is adopted, and the execution and feedback of tasks are performed separately, which can avoid blocking the query thread and other critical tasks, improving the response speed and throughput of the system. The inter-thread communication mechanism is used in processing tasks, so error handling and recovery can be achieved to some extent. For example, when the first task thread pool encounters an error during task processing, the query thread can resend messages or perform other error recovery operations. By sending a storage message to the record thread, the historical file information in the file information database is updated to the current file information, which enables the system to dynamically manage file information and maintain the real-time and accuracy of the database. By releasing the occupied memory of the file information structure, this solution can effectively manage memory resources and avoid memory leaks. At the same time, returning the linked list and hash table to the corresponding pool can also achieve resource recycling, improving the performance and efficiency of the system.

[0048] The bidirectional gateway provided by the embodiment of the application includes a sending end and a receiving end, and the sending end includes a query thread pool, a first task thread pool and a file information database;

[0049] The file information database is used to store historical file information of a source file server;

[0050] The query thread pool is configured to: obtain current file information in a target directory of a source file server, compare the current file information with historical file information, obtain file change information if the current file information is inconsistent with corresponding file information in a file information database, send the file change information to the first task thread pool, and update the current file information in the file information database; wherein the file change information is configured to indicate an added file, a deleted file, or a changed file.

[0051] The first task thread pool is configured to: obtain the file change information, download a target file that is added or changed if the file change information indicates that the target file is added or changed, and send the file change information and the target file to a receiving end.

[0052] The receiving end is configured to: receive the file change information and the target file, and send the file change information and the target file to the target file server.

[0053] In some optional embodiments, the first task thread pool is further configured to: send the file change information to the receiving end if the file change information indicates that a file is deleted after the file change information is obtained.

[0054] In some optional embodiments, the query thread pool includes a query thread and a record thread.

[0055] The query thread is configured to: obtain current file information in a target directory of a source file server, store the current file information in a file information structure, and set a file state of each file in the file information structure to an initial state.

[0056] The record thread is configured to: compare the current file information with historical file information of the file information structure to obtain a comparison result, and update the file state of each file in the file information structure according to the comparison result.

[0057] The query thread is configured to: obtain file change information according to the file state of each file, and send the file change information to the first task thread pool.

[0058] The sending end further includes a hash table pool and a linked list pool; the hash table pool and the linked list pool are both in a lock-free circular queue, a hash table in the hash table pool is in an initialization state, and a linked list in the linked list pool is in an initialization state; the file information structure includes: a storage file name, a storage file size, a storage file modification time, a storage parent directory node, a hash table pointer, a linked list pointer, and a file state; and the file state includes an initial state, an added state, a modified state, a deleted state, or an unchanged state.

[0059] In some optional embodiments, the query thread is configured to:

[0060] Obtaining available hash table and linked list from hash table pool and linked list pool;

[0061] Linking the file information structure to the linked list and adding the file information structure to the hash table;

[0062] Sending a query message, the hash table and the linked list to the recording thread; the query message comprises a hash table pointer file size, a linked list pointer, directory information stored in the hash table and the linked list, and information representing that the query is performed.

[0063] In some optional embodiments, the recording thread is configured to:

[0064] Reading the query message, obtaining file names of all files in the target directory in the file information database, and querying the hash table according to the file names, and updating the file status according to the result of querying the hash table, specifically comprising:

[0065] If a first file in the target directory in the file information database is found in the hash table, and the file size or the file modification time of the first file recorded in the file information database and the file information structure is inconsistent, setting the file status of the first file in the file information structure as a change status;

[0066] If a second file in the target directory in the file information database is found in the hash table, and the file size and the file modification time of the second file recorded in the file information database and the file information structure are consistent, setting the file status of the second file in the file information structure as an unchanged status;

[0067] If a third file in the target directory in the file information database is not found in the hash table, storing the file information corresponding to the third file to the file information structure, adding the third file to the hash table and the linked list, and setting the file status of the third file in the file information structure as a deletion status.

[0068] In some optional embodiments, the recording thread is configured to: sending a first result message, the hash table and the linked list to the query thread; the first result message comprises a hash table pointer file size, a linked list pointer, directory information stored in the hash table and the linked list, and information representing that the hash table query is completed.

[0069] The query thread is configured to: receiving the first result message, and updating the file status by traversing the linked list; wherein updating the file status by traversing the linked list comprises: if a file status of a fourth file in the file information structure is an initial status, setting the file status of the fourth file as an added status.

[0070] In some optional embodiments, the query thread is configured to send the processing message, the hash table and the linked list to the first task thread pool; wherein the processing message comprises: a hash table pointer file size, a linked list pointer, directory information stored by the hash table and the linked list, and information representing task processing.

[0071] The first task thread pool is configured to receive the processing message, download the target file and send file change information to the receiving end, or send the file change information to the receiving end; and feed back a second result message, the hash table and the linked list to the query thread; wherein the second result message comprises: a hash table pointer file size, a linked list pointer, directory information stored by the hash table and the linked list, and information representing the end of linked list traversal.

[0072] The query thread is further configured to receive the second result message, and send a storage message, the hash table and the linked list to the record thread; wherein the storage message comprises: a hash table pointer file size, a linked list pointer, directory information stored by the hash table and the linked list, and information representing storage into a file information database.

[0073] The record thread is configured to update historical file information in the file information database to current file information, release occupied memory of the file information structure, return the linked list to the linked list pool, and return the hash table to the hash table pool.

[0074] The electronic device provided by the embodiments of the present application comprises a processor and a memory, the memory stores machine readable instructions executable by the processor, and the machine readable instructions are executed by the processor to perform the method according to any one of the above.

[0075] The computer readable storage medium provided by the embodiments of the present application stores a computer program, and the computer program is executed by a processor to perform the method according to any one of the above. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0077] Figure 1 A file synchronization method flow chart of a bidirectional gateway provided by the embodiments of the present application;

[0078] Figure 2 A working flow diagram of a bidirectional gateway provided by the embodiments of the present application;

[0079] Figure 3A workflow schematic diagram of the query thread pool provided by the embodiment of the present application is provided.

[0080] Figure 4 A thread workflow schematic diagram of the sending end provided by the embodiment of the present application is provided.

[0081] Figure 5 A possible structure schematic diagram of the electronic device provided by the embodiment of the present application is provided.

[0082] Icon: 110-communication unit, 120-memory, 130-input unit, 131-touch sensitive surface, 132-other input device, 140-display unit, 141-display panel, 150-sensor, 160-audio circuit, 170-wireless communication unit, 180-processor, 190-power supply. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0084] Please refer to Figure 1 , Figure 1 A file synchronization method step flowchart of a bidirectional gateway provided by the embodiment of the present application is provided. The bidirectional gateway of the embodiment includes a sending end and a receiving end, the sending end and the receiving end receive and send network data on both sides of the bidirectional gateway by using proxy technology, unload data below the network layer, extract application protocols and data according to the application protocol filtering function, perform security check on the data, encapsulate the safe data by using a private protocol, and realize data transmission across networks and domains through an isolation component between the sending end and the receiving end.

[0085] The sending end includes a query thread pool, a first task thread pool and a file information database; the file information database includes historical file information of a source file server.

[0086] Specifically, the file synchronization method of the embodiment includes:

[0087] Step 100, the current file information in the target directory of the source file server is obtained by using the query thread pool, the current file information is compared with the historical file information, if the current file information is inconsistent with the corresponding file information in the file information database, the file change information is obtained, the file change information is sent to the first task thread pool, and the current file information is updated in the file information database; wherein the file change information is used to indicate the new file, the deleted file or the changed file.

[0088] Specifically, a thread in the query thread pool is used to obtain current file information about files in the target directory of the source file server, and the current file information includes file name, modification time, file size, etc. The historical file information stored in the file information database also includes file name, modification time, file size, etc. The specific content of the comparison between the current file information and the historical file information is: the current modification time and the historical modification time corresponding to the files with the same file name are compared for consistency, and the current file size and the historical file size are compared for consistency, to determine the changed files or unchanged files. According to the file name, if the file name appears in the current file information but is not found in the historical file information, it is determined that the file is a newly added file. According to the file name, if the file name is not found in the current file information but appears in the historical file information, it is determined that the file is a deleted file.

[0089] Step 200: Using the first task thread pool, obtain file change information. If the file change information indicates a newly added file or a changed file, download the target file that is newly added or changed, and send the file change information and the target file to the receiving end.

[0090] Specifically, the file synchronization between the sending end and the receiving end is realized by transmission through the isolation component. Here, a thread in the first task thread pool is used for task processing. If a file is newly added or changed in the source file server, the target file that is newly added or changed is downloaded, and the target file and file change information indicating the newly added or changed file are sent to the receiving end.

[0091] Step 300: Using the second task thread pool of the receiving end, receive the file change information and the target file, and send the file change information and the target file to the target file server.

[0092] Specifically, the receiving end receives the target file and the file change information transmitted by the sending end through the isolation component, and uses a thread in the second task thread pool to upload the target file and the file change information to the target file server through a file protocol.

[0093] In the embodiment of the present application, the historical file information in the target directory of the source file server is stored into the file information database of the sending end, the current file information in the target directory is obtained through the query thread pool, and the file change of the source file server can be accurately detected according to the current file information and the historical file information. Once the change is detected, the change information and the target file are immediately sent to the receiving end, so as to ensure the synchronization accuracy of the file. Meanwhile, in the two-way gate, the internal network and the external network can be isolated, so as to effectively prevent network attacks and data leakage. In addition, the query thread pool, the first task thread pool and the second task thread pool can be used for concurrent processing, so as to efficiently process a large number of file synchronization tasks. In the method of the embodiment, when the memory is used to save the file information, only the related information in the target directory of the source file server is stored, and the whole file information does not need to be stored, so that the memory occupation is reduced, and the system performance is improved.

[0094] In some optional embodiments, after the file change information is obtained in step 200, the method further includes: if the file change information indicates that the file is deleted, sending the file change information to the receiving end.

[0095] In the embodiment of the present application, the deletion of the file can also be accurately detected and processed. Once the information of the deleted file is detected, the file change information is immediately sent to the receiving end, so as to ensure that the deletion operation of the file is accurately and correctly performed.

[0096] In some optional embodiments, the query thread pool includes a query thread and a record thread; in step 100, the current file information in the target directory of the source file server is obtained, the current file information is compared with the historical file information, if the current file information is inconsistent with the corresponding file information in the file information database, the file change information is obtained, and the file change information is sent to the first task thread pool, including:

[0097] In step 110, the current file information in the target directory of the source file server is obtained by using the query thread, the current file information is stored into the file information structure, and the file state of each file in the file information structure is set to an initial state;

[0098] In step 120, the current file information of the file information structure is compared with the historical file information by using the record thread, and a comparison result is obtained;

[0099] In step 130, the file state of each file of the file information structure is updated according to the comparison result;

[0100] In step 140, the file change information is obtained according to the file state of each file by using the query thread, and the file change information is sent to the first task thread pool.

[0101] In the embodiments of the present application, the separation of the query thread and the record thread enables the query and comparison operations of the files to be performed in parallel. The query thread is responsible for obtaining and storing the current file information, and the record thread is responsible for comparing the current and historical file information, which greatly improves the overall file synchronization efficiency. By setting the initial state of each file in the file information structure and updating the file state based on the comparison result, the accuracy and integrity of the file change information can be ensured, which can avoid missing or false detection of file changes. By decomposing the complex file synchronization process into multiple independent sub-tasks, each sub-task is processed by a dedicated thread, which can reduce the complexity of the entire system and improve the stability and reliability of the system. Even if a thread fails, it will not affect the normal operation of other threads, and the system can be easily expanded to handle more source file servers by simply increasing the number of query threads and record threads to handle more file synchronization tasks.

[0102] The sending end further includes a hash table pool and a linked list pool. The hash table pool and the linked list pool both adopt a lock-free circular queue. The hash table in the hash table pool is in an initial state, and the linked list in the linked list pool is in an initial state. The file information structure includes a file name, a file size, a file modification time, a parent directory node, a hash table pointer, a linked list pointer, and a file state. The file state includes an initial state, an added state, a modified state, a deleted state, or an unchanged state.

[0103] In the embodiments of the present application, the hash table pool and the linked list pool both adopt a lock-free circular queue design. The query thread, the record thread, and the task thread can simultaneously operate on the hash table pool and the linked list pool, achieving high parallelism, which helps to improve the overall file synchronization efficiency. Moreover, the lock-free circular queue design has high stability and reliability, which can avoid deadlocks and performance problems caused by lock competition.

[0104] In some optional embodiments, in step 110, the current file information in the target directory of the source file server is obtained, the current file information is stored in the file information structure, and the file state of each file in the file information structure is set to an initial state. The method further includes:

[0105] In step 111, available hash tables and linked lists are obtained from the hash table pool and the linked list pool.

[0106] In step 112, the file information structure is linked to the linked list, and the file information structure is added to the hash table.

[0107] Step 113, sending a query message, a hash table and a linked list to a record thread; wherein the query message includes: a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing the query.

[0108] In the embodiments of the present application, two query methods of hash table search and linked list search are provided, wherein the hash table is a data structure for direct access according to key value, and the record is accessed by mapping the key value to a position in the table to speed up the search. The linked list is a commonly used data structure, and its advantages include: the size of the linked list is dynamic, and nodes can be added or reduced at any time as needed, unlike arrays that need to pre-allocate fixed memory space. Inserting and deleting nodes in the linked list only needs to modify the pointing of the pointer, and does not need to move a large amount of data, so the insertion and deletion efficiency is high. The elements in the linked list are stored in the order of insertion, and the nodes can be accessed sequentially, which is suitable for situations where the linked list needs to be traversed. Each node in the linked list contains data and a pointer to the next node, so compared with arrays, memory space can be more fully utilized.

[0109] In some optional embodiments, step 120, comparing the current file information of the file information structure body with the historical file information to obtain a comparison result, including:

[0110] Step 121, reading the query message, obtaining the file names of all files in the target directory in the file information database, and querying the hash table according to the file name, updating the file state according to the result of querying the hash table, and the updating of the file state includes the following three cases:

[0111] The first case, if the first file in the target directory in the file information database is found in the hash table, and the file size or the file modification time of the first file recorded in the file information database and the file information structure body is inconsistent, the file state of the first file in the file information structure body is set to the change state;

[0112] The second case, if the second file in the target directory in the file information database is found in the hash table, and the file size and the file modification time of the second file recorded in the file information database and the file information structure body are consistent, the file state of the second file in the file information structure body is set to the unchanged state;

[0113] The third case, if the third file in the target directory in the file information database is not found in the hash table, the file information corresponding to the third file is stored in the file information structure body, and is added to the hash table and the linked list, and the file state of the third file in the file information structure body is set to the deletion state.

[0114] In the embodiments of the present application, the file information is obtained by querying the hash table to update the file status, and the operation complexity of the hash table query is O(1) compared with directly traversing the file system, which has high efficiency. Meanwhile, the file searching is faster according to the hash table query.

[0115] In some optional embodiments, after the file status is updated according to the result of the hash table query in step 121, the method further comprises:

[0116] In step 122, the first result message, the hash table and the linked list are sent to the query thread by using the record thread, wherein the first result message comprises: the hash table pointer file size, the linked list pointer, the directory information stored in the hash table and the linked list, and the information representing the end of the hash table query.

[0117] In step 123, the first result message is received by using the query thread, and the file status is updated by traversing the linked list, wherein the file status is updated by traversing the linked list, comprising: if the file status of the fourth file in the file information structure is the initial state, the file status of the fourth file is set to the new state.

[0118] In the embodiments of the present application, when the file status is updated according to the result of the hash table query, the file status of the changed file has been updated from the initial state to the changed state, the file status of the unchanged file has been updated from the initial state to the unchanged state, and the file status of the deleted file has been updated from the initial state to the deleted state. Then, the query thread is used to traverse the linked list, find the file whose file status is still in the initial state, and update the file status of these files to the new state.

[0119] In some optional embodiments, after the file status is updated by traversing the linked list in step 123, the method further comprises:

[0120] In step 124, the processing message, the hash table and the linked list are sent to the first task thread pool by using the query thread, wherein the processing message comprises: the hash table pointer file size, the linked list pointer, the directory information stored in the hash table and the linked list, and the information representing the task processing.

[0121] After step 124, the embodiment enters the step process of step 200, and the step process specifically comprises the following implementation steps in the embodiment:

[0122] In step 210, the processing message is received by using the first task thread pool, the target file is downloaded and the file change information is sent to the receiving end, or the file change information is sent to the receiving end, the second result message, the hash table and the linked list are fed back to the query thread, wherein the second result message comprises: the hash table pointer file size, the linked list pointer, the directory information stored in the hash table and the linked list, and the information representing the end of the linked list traversal.

[0123] Step 220, using the query thread, receiving a second result message, sending a storage message to the record thread, a hash table and a linked list; wherein the storage message includes: hash table pointer file size, linked list pointer, hash table and linked list storage directory information, and information representing storage to the file information database;

[0124] Step 230, using the record thread, updating the historical file information in the file information database to the current file information, releasing the occupied memory of the file information structure, returning the linked list to the linked list pool, and returning the hash table to the hash table pool.

[0125] In the embodiment of the application, the concurrent processing capability of the first task thread pool and the query thread is utilized, and multiple tasks such as downloading files and sending file change information can be processed simultaneously. This helps to improve the overall processing speed and efficiency. The design of asynchronous operation is adopted, and the execution and feedback of the task are performed separately, which can avoid blocking the query thread and other critical tasks, and improve the response speed and throughput of the system. The inter-thread communication mechanism is adopted when processing the task, so that error handling and recovery can be achieved to some extent. For example, when an error occurs in the first task thread pool when processing the task, the query thread can resend the message or perform other error recovery operations. By sending the storage message to the record thread, the historical file information in the file information database is updated to the current file information, which enables the system to dynamically manage file information and maintain the real-time and accuracy of the database. By releasing the occupied memory of the file information structure, the scheme can effectively manage memory resources and avoid memory leakage and other problems. At the same time, returning the linked list and hash table to the corresponding pool can also realize the recycling of resources and improve the performance and efficiency of the system.

[0126] All the optional technical solutions of the above-mentioned embodiment of the application can be combined in any combination to form optional technical solutions of the application. The following provides specific embodiments for illustration:

[0127] Please refer to Figure 2 , Figure 2 The working flowchart of the bidirectional gate provided in the embodiment of the application. The embodiment adopts the design method of queue, hash table pool, linked list pool, database storage, multi-thread asynchronous parallel processing and dynamic storage of current directory (referring to the directory in which the thread is working) file information, which can reduce memory occupancy and improve file synchronization transmission file performance.

[0128] The query thread pool is composed of multiple threads, and the file information of the source file server is remotely queried and recorded according to the information of the file synchronization strategy (such as IP, username, directory, etc.) through Ftp or other protocols, and according to the recorded file information and state, the file name, size and time are compared to find whether the file has changed, and if the file has changed (added, deleted or modified), the information is synchronized to the first task thread pool.

[0129] The first task thread pool is composed of multiple threads, and is responsible for receiving the added, deleted or modified information of the file of the query thread pool, and downloading the file according to the information type, and sending the file information and data or sending the file deletion information to the file synchronization of the receiving end host.

[0130] The receiving end is mainly composed of the second task thread pool, and is responsible for receiving the file information and data or file deletion information sent by the sending end host file synchronization. According to the information of the synchronization strategy (such as IP, username, directory, etc.), the file information and data or the file deletion information is remotely uploaded to the target server through Ftp or other protocols. Finally, the target file server completes the file storage or file deletion.

[0131] Specifically, the query thread pool is divided into two types of threads, namely query threads and record threads, and the two types of threads are used in pairs to perform different functions, and each synchronization strategy is allocated to a pair of threads to complete the synchronization task in stages. For details, please refer to Figure 3 , Figure 3 The working flowchart of the query thread pool provided by the embodiment of the application is shown.

[0132] The query thread communicates with the source file server through Ftp or other protocols, queries the directory information and file information of the synchronization strategy, and in the execution process, an available hash table is obtained from the "hash table pool", an available linked list is obtained from the "linked list pool", then the file or directory information under the current directory is stored in the memory of the file information structure body, and this block of memory is added to the hash table and linked to the linked list. The key fields of the file information structure body are defined as shown in Table 1.

[0133] Table 1, file information key field table

[0134]

[0135] When the information in a directory is queried (or a certain number is reached), the hash table, linked list and message structure (msg=1 query message) are stored in a block of memory, and the memory is added to the query queue, and the memory block is read by the record thread and the corresponding processing is completed. The key fields of the message structure are shown in Table 2.

[0136] Table 2, message structure key field table

[0137]

[0138] The query thread reads the result msg=4 sent by the record thread from the query queue, judges and processes the file state according to the information in the message, and sends the file state operation information to the task thread pool through the task queue (msg=3). The judgment and processing of the file state are as follows:

[0139] fst=0, indicating that there is no record in the database, but there is such a file on the server, set fst=1;

[0140] fst=2, indicating that there is no record in the database and the file on the server is different, set fst=2;

[0141] fst=3, indicating that there is a record in the database, but there is no such file on the server, set fst=3;

[0142] When the task thread pool finishes processing the file task, it feeds back the result msg=4 through the task queue, and the query thread forwards the result to the record thread through the message structure and the query queue, updates the database record data and releases the related resources.

[0143] After the record thread reads the message structure in the query queue, it processes according to the scene and type:

[0144] Send all information in the current directory scene: query message msg=1: get the information of the first node in the linked list (the first node is the current directory information), query the database record related to the current directory in the database, and use the file name in the hash table to find the file. When found, compare the file size and time, and set fst=2 if not equal, and set fst=4 if equal. When not found, store the information of the database record in a new file information structure, mount it to the linked list, and set fst=3. After processing all the data, feedback the result (msg=2) to the query thread through the query queue, and the query thread sends the data to the record thread through msg=2.

[0145] The record thread receives msg=2 (stores information), updates the state information of the newly added, modified and deleted in the linked list to the database, releases the file information structure memory, and finally returns the linked list and hash table to the linked list pool and hash table pool respectively.

[0146] Batch send current directory information scene: if the query thread finds that the directory information is large, the nmax value is set, for example, when it reaches 10,000, 10,000 file information structure bodies are stored, and then sent to the record thread. The data record processing thread queries the database according to the file name in the linked list, and when found, compares the file size and time, sets fst=2 when not equal, and sets fst=4 when equal, and marks the database record as having been operated. When not found, a new record is inserted in the database.

[0147] When nmax=0 is received, it indicates that the file or directory query of the current directory is complete, the record thread queries the database for records that are not marked as having been operated, and stores the related information into the file information structure body, adds it to the hash table and the linked list, and sets fst=3. If there are many records, for example, more than 10,000, the message is sent to the query thread in batches, and the related database records are deleted. Then the query thread forwards the message to the task thread to implement the deletion task.

[0148] Among them, the query queue and the task queue: a lock-free bidirectional queue design is used to realize the communication of the query thread, the record thread and the task thread. The hash table pool: a lock-free circular queue design is used, and each hash table is in an initialized state. Each hash table is used to store file information structure body pointers, and is used by the query thread, the record thread and the task thread. The linked list pool: a lock-free circular queue design is used, and each linked list is in an initialized state. It is used to store file information structure body pointers, and is used by the query thread, the record thread and the task thread.

[0149] Please refer to Figure 4 , Figure 4 The thread work flow diagram of the sending end provided by the embodiment of the application, specifically includes the following steps:

[0150] 1) The query thread communicates with the source server through Ftp or the like, and queries the directory information and file information of the service according to the synchronization strategy.

[0151] 2) An available hash table is obtained from the hash table pool.

[0152] 3) An available linked list is obtained from the linked list pool.

[0153] 4) The subordinate file or directory information of the current directory is queried and stored in the file information structure body.

[0154] 5) The file information structure body is linked to the linked list.

[0155] 6) The file information structure body is added to the hash table.

[0156] 7) The record thread is sent msg=1, the hash table and the linked list.

[0157] 8) record thread reads the message.

[0158] 8.1) According to the directory information of the first node of the chain table, query all subfiles and directories of this directory.

[0159] 8.2) According to the file name recorded in the file information database, query the hash table.

[0160] 8.3) If found, set fst=2 if ftime is not equal or fsize is not equal, and set fst=4 if ftime and fsize are both equal respectively.

[0161] 8.4) If not found, store to the file information structure, add to the chain table and the hash table, and set fst=3.

[0162] 9) Record thread returns the result msg=4, hash table, and chain table.

[0163] 10) Query thread traverses the chain table.

[0164] 10.1) fst=0, indicating that there is no record in the database, but there is such a file on the server, set fst=1.

[0165] 10.2) fst=2, indicating that the recorded information in the database is different from the file information on the server, set fst=2.

[0166] 10.3) fst=3, indicating that there is a record in the database, but there is no such file on the server, set fst=3.

[0167] 11) Query thread sends the processing message msg=3, hash table, and chain table to the first task thread.

[0168] 12) Download the file and send it to the receiving end or send the file deletion information to the receiving end.

[0169] 13) First task thread feeds back the result msg=4, hash table, and chain table to the query thread.

[0170] 14) Query thread sends the storage message msg=2, hash table, and chain table to the record thread.

[0171] 15) Record thread updates the relevant information in the database and releases the file information memory.

[0172] 16) Record thread returns the chain table to the chain table pool.

[0173] 17) Record thread returns the hash table to the hash table pool.

[0174] 18) Set the current directory to one of its subdirectories, and start processing the information of the current directory from the beginning of "2)" until all the files and directories under the directory and the directory itself are processed.

[0175] The receiving end synchronization software, the second task thread pool of the receiving end synchronization software, after receiving, sends file information or deletes file information to the target server according to the synchronization strategy.

[0176] In the above process, if the query thread finds that the number of files under the directory is large, the nmax value will be set to realize batch processing, which specifically includes:

[0177] 1) For example, when 10,000 file information structure bodies are stored, nmax = 10,000 is set in the message and sent to the record thread.

[0178] 2) The record thread queries the database according to the file name in the linked list, and when found, compares the file size and time, sets fst = 2 when the file size and time are different, and sets fst = 4 when the file size and time are equal, and marks the database record as having been operated. When not found, a new record is inserted in the database.

[0179] 3) When nmax = 0 is received, it means that the files or directories under the current directory have been queried, the record thread queries the records in the database that have not been marked as having been operated, and stores the related information in the file information structure body, adds it to the hash table and linked list, and sets fst = 3. If there are too many records, for example, more than 10,000, messages will be sent to the query thread in batches.

[0180] 4) Before planning resources for the hash table pool and the linked list pool, the record thread updates the relevant record status in the database (including clearing the operation mark).

[0181] 5) The query thread forwards the message to the first task thread pool to execute the deletion task.

[0182] In summary, the embodiment adopts queue-based, hash table pool, linked list pool, database storage, multi-thread asynchronous parallel processing, and dynamic storage of current directory file information, which has the following advantages:

[0183] Reduces memory usage, only stores the relevant information of the currently queried directory, and no longer stores all file information. If the current directory stores a large number of files and directories, the number limit can be set to implement batch query and synchronization. The memory can be controlled within hundreds of megabytes according to design needs. Low-end and mid-end devices can support synchronization of tens of millions of files, with good competitive parameters.

[0184] The three role threads of the query thread, the record thread and the task processing thread asynchronously process the same directory or the same batch of files, the data between the threads is processed through the design of lock-free and phase exclusive data, high-speed asynchronous parallel processing is realized, the file parallel transmission performance is improved, and the application in a gigabit device is applicable.

[0185] The hash table pool and the linked list pool realized through the lock-free circular queue provide the memory-based storage and searching functions for each current directory or a certain number of files, the memory amount for storing the file information is controlled through controlling the number of the query directory or the files under the same directory, and the large file information storage and searching functions are realized in combination with the method for storing the file information of other directories in a database.

[0186] The bidirectional gateway provided in the embodiment of the application comprises a sending end and a receiving end, the sending end comprises a query thread pool, a first task thread pool and a file information database;

[0187] The file information database is used for storing the historical file information of the source file server.

[0188] The query thread pool is used for obtaining the current file information in a target directory of the source file server, comparing the current file information with the historical file information, obtaining file change information if the current file information is inconsistent with the corresponding file information in the file information database, sending the file change information to the first task thread pool, and updating the current file information in the file information database; wherein the file change information is used for indicating an added file, a deleted file or a changed file.

[0189] The first task thread pool is used for obtaining the file change information, downloading the added or changed target file if the file change information indicates an added file or a changed file, and sending the file change information and the target file to the receiving end.

[0190] The receiving end is used for receiving the file change information and the target file, and sending the file change information and the target file to the target file server.

[0191] The bidirectional gateway of the embodiment also realizes the file synchronization method through the above module, and the implementation mechanism is the same as that of the above related method embodiment, and details can be referred to the description of the above embodiment, which will not be repeated here.

[0192] The bidirectional gateway of the embodiment can be arranged at the browser client side and used as an engine device of the browser. The bidirectional gateway can also be arranged in an electronic device and separately perform its function.

[0193] Figure 5 is a schematic structural diagram of an electronic device provided in the embodiment of the application. Referring to Figure 5The electronic device can be used to implement the file synchronization method provided in the above embodiments. Specifically:

[0194] The electronic device can include a communication unit 110, a memory 120 including one or more computer readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a WiFi (wireless fidelity) module 170, a processor 180 including one or more processing cores, and a power supply 190, etc. Those skilled in the art can understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them: Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them:

[0195] The communication unit 110 can be used for receiving and transmitting signals in the process of information or communication. The communication unit 110 can be a network communication device such as an RF circuit, a router, a modem, etc. In particular, when the communication unit 110 is an RF circuit, the communication unit 110 receives downlink information from a base station and transmits data related to uplink to the base station. Generally, the RF circuit as the communication unit includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an Low Noise Amplifier (LNA), a duplexer, etc. In addition, the communication unit 110 can communicate with a network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to the Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc. The memory 120 can be used to store software programs and modules, and the processor 180 can execute various functions and data processing by running the software programs and modules stored in the memory 120. The memory 120 can include a program storage area and a data storage area. The program storage area can store operating systems, application programs required for at least one function (such as a sound play function, an image play function, etc.), etc. The data storage area can store data created according to the use of the electronic device (such as audio data, a phonebook, etc.), etc. In addition, the memory 120 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid state memory device. Accordingly, the memory 120 can further include a memory controller to provide access to the memory 120 for the processor 180 and the input unit 130.

[0196] The input unit 130 can be used to receive inputted digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls. Specifically, the input unit 130 can include a touch-sensitive surface 131 and other input devices 132. The touch-sensitive surface 131, also known as a touch display or touchpad, can collect touch operations (such as a user's operation on or near the touch-sensitive surface 131 using a finger, a stylus, or any suitable object or accessory) of the user thereon or there near, and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 131 can include two parts of a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, and converts it into touch coordinates, and then sends it to the processor 180, and can receive the command from the processor 180 and execute it. In addition, the touch-sensitive surface 131 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131, the input unit 130 can also include other input devices 132. Specifically, the other input devices 132 can include one or more of a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc.

[0197] The display unit 140 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device, which can be composed of graphics, text, icons, video, and any combination thereof. The display unit 140 can include a display panel 141, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 131 can cover the display panel 141, and when the touch-sensitive surface 131 detects a touch operation thereon or there near, it transmits to the processor 180 to determine the type of touch event, and then the processor 180 provides corresponding visual output on the display panel 141 according to the type of touch event. Although in the Figure 5 embodiments, the touch-sensitive surface 131 and the display panel 141 are implemented as two independent components to realize input and output functions, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize input and output functions.

[0198] The electronic device may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 141 according to the ambient light level, and the proximity sensor can turn off the display panel 141 and / or the backlight when the electronic device is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the electronic device, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0199] Audio circuitry 160, a speaker, and a microphone provide an audio interface between the user and the electronic device. Audio circuitry 160 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 160, converted back into audio data, and output to processor 180 for processing. The audio data is then transmitted via RF circuitry 110 to, for example, another electronic device, or output to memory 120 for further processing. Audio circuitry 160 may also include an earphone jack to facilitate communication between external headphones and the electronic device.

[0200] To achieve wireless communication, the electronic device can be equipped with a wireless communication unit 170, which can be a WiFi module. WiFi is a short-range wireless transmission technology. Through the wireless communication unit 170, the electronic device can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 5 The wireless communication unit 170 is shown, but it is understood that it is not a necessary component of the electronic device and can be omitted as needed without changing the nature of the invention.

[0201] The processor 180 is the control center of the electronic device, connects all parts of the mobile phone through various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 120 and calling data stored in the memory 120, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 can include one or more processing cores; preferably, the processor 180 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 180.

[0202] The electronic device also includes a power supply 190 (such as a battery) for supplying power to various components, and preferably, the power supply can be logically connected to the processor 180 through a power management system, so as to realize functions such as management of charging, discharging and power consumption management through the power management system. The power supply 190 can also include one or more than one direct or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator and any other components.

[0203] Although not shown, the electronic device can also include a camera, a Bluetooth module, etc., which will not be described here. In particular, in the present embodiment, the display unit of the electronic device is a touch screen display, and the electronic device further includes a memory and one or more than one program, wherein the one or more than one program is stored in the memory and is configured to be executed by the one or more than one processor, and the one or more than one program includes instructions for: obtaining current file information in a target directory of a source file server by using a query thread pool, comparing the current file information with historical file information, if the current file information is inconsistent with corresponding file information in a file information database, obtaining file change information, sending the file change information to a first task thread pool, and updating the current file information in the file information database; wherein the file change information is used to indicate an added file, a deleted file or a changed file; obtaining the file change information by using the first task thread pool, if the file change information indicates an added file or a changed file, downloading the added or changed target file, and sending the file change information and the target file to a receiving end; receiving the file change information and the target file by using a second task thread pool of the receiving end, and sending the file change information and the target file to the target file server.

[0204] Optionally, the memory is also used to store instructions for: if the file change information indicates a deleted file, sending the file change information to the receiving end.

[0205] Optionally, the memory is further configured to store instructions for: obtaining, by the query thread, current file information in the target directory of the source file server, storing the current file information into a file information structure, and setting a file state of each file in the file information structure as an initial state; obtaining, by the record thread, a comparison result by comparing the current file information with historical file information of the file information structure; updating the file state of each file in the file information structure according to the comparison result; and obtaining, by the query thread, file change information according to the file state of each file, and sending the file change information to the first task thread pool.

[0206] Optionally, the memory is further configured to store instructions for: obtaining available hash tables and linked lists from the hash table pool and the linked list pool; linking the file information structure into the linked list, and adding the file information structure into the hash table; and sending a query message, the hash table and the linked list to the record thread, wherein the query message comprises a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing that the query is performed.

[0207] Optionally, the memory is further configured to store instructions for: reading the query message, obtaining file names of all files in the target directory in the file information database, and querying the hash table according to the file names, and updating the file state according to a result of querying the hash table.

[0208] Optionally, the memory is further configured to store instructions for: if a first file in the target directory in the file information database is found in the hash table, and file sizes or file modification times of the first file recorded in the file information database and the file information structure are inconsistent, setting the file state of the first file in the file information structure as a change state; if a second file in the target directory in the file information database is found in the hash table, and file sizes and file modification times of the second file recorded in the file information database and the file information structure are consistent, setting the file state of the second file in the file information structure as an unchanged state; and if a third file in the target directory in the file information database is not found in the hash table, storing file information corresponding to the third file into the file information structure, adding the third file into the hash table and the linked list, and setting the file state of the third file in the file information structure as a deletion state.

[0209] Optionally, the memory is further configured to store instructions for sending, by the record thread, the first result message, the hash table and the linked list to the query thread; wherein the first result message comprises: a hash table pointer file size, a linked list pointer, directory information of the hash table and the linked list, and information indicating that the hash table query is completed; and receiving, by the query thread, the first result message, and traversing the linked list to update the file state; wherein traversing the linked list to update the file state comprises: if the fourth file in the file information structure is in the initial state, setting the fourth file to be in the new state.

[0210] Optionally, the memory is further configured to store instructions for sending, by the query thread, the processing message, the hash table and the linked list to the first task thread pool; wherein the processing message comprises: a hash table pointer file size, a linked list pointer, directory information of the hash table and the linked list, and information indicating that the task is processed. Receiving, by the first task thread pool, the processing message, downloading the target file and sending the file change information to the receiving end, or sending the file change information to the receiving end; feeding back, by the first task thread pool, the second result message, the hash table and the linked list to the query thread; wherein the second result message comprises: a hash table pointer file size, a linked list pointer, directory information of the hash table and the linked list, and information indicating that the linked list traversal is completed. Receiving, by the query thread, the second result message, and sending, by the query thread, the storage message, the hash table and the linked list to the record thread; wherein the storage message comprises: a hash table pointer file size, a linked list pointer, directory information of the hash table and the linked list, and information indicating that the storage is completed in the file information database. Updating, by the record thread, the historical file information in the file information database to the current file information, releasing the occupied memory of the file information structure, returning the linked list to the linked list pool, and returning the hash table to the hash table pool.

[0211] In the embodiments of the present application, it should be understood that the disclosed device and method can be implemented by other means. The device embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, and for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, which can be electrical, mechanical or other forms.

[0212] In addition, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0213] Furthermore, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0214] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0215] The above description is merely illustrative of the application, and not in limitation of the principles of the application. Any modification and change, which can be made by those skilled in the art, without departing from the spirit and principle of the application, shall be included in the scope of protection of the application.

Claims

1. A file synchronization method of a bidirectional gateway, characterized by, The bidirectional gateway comprises a sending end and a receiving end, the sending end comprises a query thread pool, a first task thread pool and a file information database; wherein the file information database comprises historical file information of a source file server; The method comprises: Using the query thread pool, current file information in a target directory of the source file server is obtained, the current file information is compared with the historical file information, if the current file information is inconsistent with corresponding file information in the file information database, file change information is obtained, the file change information is sent to the first task thread pool, and the current file information is updated in the file information database; wherein the file change information is used to indicate an added file, a deleted file or a changed file; Using the first task thread pool, the file change information is obtained, if the file change information indicates an added file or a changed file, the added or changed target file is downloaded, and the file change information and the target file are sent to the receiving end; Using the second task thread pool of the receiving end, the file change information and the target file are received, and the file change information and the target file are sent to the target file server; The query thread pool comprises a query thread and a record thread; The current file information in the target directory of the source file server is obtained, the current file information is compared with the historical file information, if the current file information is inconsistent with corresponding file information in the file information database, file change information is obtained, the file change information is sent to the first task thread pool, and the current file information is updated in the file information database, comprising: Using the query thread, current file information in a target directory of the source file server is obtained, the current file information is stored in a file information structure body, and the file state of each file in the file information structure body is set to an initial state; Using the record thread, the current file information of the file information structure body is compared with the historical file information to obtain a comparison result; According to the comparison result, the file state of each file in the file information structure body is updated; Using the query thread, the file change information is obtained according to the file state of each file, and the file change information is sent to the first task thread pool.

2. The method of claim 1, wherein, After the file change information is obtained, the method further comprises: If the file change information indicates a deleted file, the file change information is sent to the receiving end.

3. The method of claim 1, wherein, The sending end further comprises a hash table pool and a linked list pool; the hash table pool and the linked list pool both adopt a lock-free circular queue, the hash table in the hash table pool is in an initialization state, and the linked list in the linked list pool is in an initialization state; The file information structure body comprises a storage file name, a storage file size, a storage file modification time, a storage parent directory node, a hash table pointer, a linked list pointer and a file state; wherein the file state comprises an initial state, an added state, a modified state, a deleted state or an unchanged state.

4. The method of claim 3, wherein, The method further comprises: Obtaining available hash table and linked list from the hash table pool and the linked list pool; Linking the file information structure to the linked list and adding the file information structure to the hash table; Sending a query message, the hash table and the linked list to the recording thread, wherein the query message comprises a hash table pointer, a file size, a linked list pointer, hash table and linked list storage directory information, and information indicating that the query is being performed.

5. The method of claim 4, wherein, The method further comprises: Reading the query message, obtaining the file names of all files in the target directory in the file information database, and querying the hash table according to the file names and updating the file state according to the query result of the hash table.

6. The method of claim 5, wherein, The method further comprises: If the first file in the target directory in the file information database is found in the hash table and the file size or the file modification time of the first file recorded in the file information database and the file information structure is inconsistent, setting the file state of the first file in the file information structure to the changed state; If the second file in the target directory in the file information database is found in the hash table and the file size and the file modification time of the second file recorded in the file information database and the file information structure are consistent, setting the file state of the second file in the file information structure to the unchanged state; If the third file in the target directory in the file information database is not found in the hash table, storing the file information corresponding to the third file in the file information structure and adding the third file to the hash table and the linked list, and setting the file state of the third file in the file information structure to the deleted state.

7. The method of claim 6, wherein, The method further comprises: Using the recording thread to send a first result message, the hash table and the linked list to the query thread, wherein the first result message comprises a hash table pointer, a file size, a linked list pointer, hash table and linked list storage directory information, and information indicating that the hash table query is completed; Using the query thread to receive the first result message and update the file state by traversing the linked list, wherein the file state of the fourth file in the file information structure is set to the added state if the file state of the fourth file is the initial state.

8. The method of claim 7, wherein, The method further comprises: Using the query thread to send a processing message, the hash table and the linked list to the first task thread pool, wherein the processing message comprises a hash table pointer, a file size, a linked list pointer, hash table and linked list storage directory information, and information indicating that the task is being processed. The first task thread pool is used to receive the processing message, download a target file, send the file change information to the receiving end, or send the file change information to the receiving end; the second result message, the hash table and the linked list are fed back to the query thread; wherein the second result message comprises a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing the end of linked list traversal; The query thread is used to receive the second result message, send a storage message, the hash table and the linked list to the recording thread; wherein the storage message comprises a hash table pointer file size, a linked list pointer, hash table and linked list storage directory information, and information representing storage to a file information database; The recording thread is used to update the historical file information in the file information database to the current file information, release the occupied memory of the file information structure, return the linked list to the linked list pool, and return the hash table to the hash table pool.

9. A bidirectional gateway, characterized by The bidirectional gateway comprises a sending end and a receiving end, and the sending end comprises a query thread pool, a first task thread pool and a file information database; The file information database is used to store historical file information of a source file server; The query thread pool is used to obtain current file information in a target directory of the source file server, compare the current file information with the historical file information, obtain file change information if the current file information is inconsistent with corresponding file information in the file information database, send the file change information to the first task thread pool, and update the current file information in the file information database; wherein the file change information is used to indicate an added file, a deleted file or a changed file; The query thread pool comprises a query thread and a recording thread; The obtaining of the current file information in the target directory of the source file server, the comparison of the current file information with the historical file information, the obtaining of the file change information if the current file information is inconsistent with corresponding file information in the file information database, and the sending of the file change information to the first task thread pool, comprise: The query thread is used to obtain the current file information in the target directory of the source file server, store the current file information in a file information structure, and set a file state of each file in the file information structure to an initial state; The recording thread is used to compare the current file information of the file information structure with the historical file information, and obtain a comparison result; According to the comparison result, the file state of each file in the file information structure is updated; The query thread is used to obtain the file change information according to the file state of each file, and send the file change information to the first task thread pool; The first task thread pool is used to obtain the file change information, download an added or changed target file if the file change information indicates an added file or a changed file, and send the file change information and the target file to the receiving end. The receiving end is configured to receive the file change information and the target file, and send the file change information and the target file to a target file server.

10. An electronic device, comprising: Comprising: A processor and a memory, the memory storing machine readable instructions executable by the processor, the machine readable instructions, when executed by the processor, performing the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program, when executed by a processor, performs the method of any one of claims 1-8.

Images