File system synchronization method and system based on vectorization digest algorithm

Abstract

The invention relates to a file system synchronization method and system based on a vectorization digest algorithm, and belongs to the technical field of file system processing. The method comprises the following steps of: (1) for a file directory needing to be synchronized, constructing an abstract tree for a composition structure of the file directory by using an abstract algorithm; (2) storing the file catalog with the generated abstract tree and subordinate files of the file catalog on a remote server, and sending a synchronization instruction to a terminal; (3) the terminal performs file directory synchronization; and (4) daily maintenance of the synchronous directory. According to the method, a set of vectorized abstract algorithm is self-defined, the abstract tree is constructed by using the vectorized abstract algorithm, the vectorized abstract algorithm is applied to the synchronization process of the file system, calculation is carried out by converting data into vectors, the parallel processing capacity of a modern computer can be fully utilized, and the abstract generation speed is remarkably increased.

Description

Technical Field

[0001] The invention relates to a file system synchronization method and system based on a vectorizable digest algorithm, belonging to the technical field of file system processing. Background Art

[0002] A digest algorithm, also known as a hash algorithm or a hash algorithm, is a method that converts input data of any length into a return value of a fixed length through a series of calculation methods and rules. This value is called a hash value. The main features of a digest algorithm include compressibility, easy calculation, resistance to modification, strong resistance to collision, and irreversibility. It is often used in data integrity verification, digital signatures, etc.

[0003] Vectorization is a method of accelerating computing through parallel processing technology. Modern processors have widely supported vectorized accelerated instruction sets, such as SSE and AVX of the x86 architecture, NEON and SVE of the ARM architecture, and Vector extension instructions of the RISC-V architecture. Through vectorized instruction sets, multiple data can be operated on at the same time, thereby improving computing efficiency and accelerating algorithm operation.

[0004] In computer science, directory structure is an important part of the file system, which determines how files are organized on the disk. The tree directory structure is a common directory structure, which imitates the tree structure in real life and has the advantages of clear hierarchy, easy management, and high access efficiency. The tree directory structure, also known as a hierarchical structure or a tree-like structure, is a directory structure that starts from the root directory and the subdirectories at all levels are organized in a branching manner. In this structure, each directory (including subdirectories) can contain files and other subdirectories. The biggest advantage of the tree directory structure is that it has good scalability and flexibility, and can easily add, delete, move or rename files and directories.

[0005] A summary tree is a tree structure formed by combining the directory structure of a file system with the summary values ​​of files in the directory. The organization of the tree structure is consistent with the directory structure of the file system. The difference is that the files in the directory are represented by the summary values ​​of the files in the summary tree. For a directory, a summary file is created in it to record the summary values ​​of all files in the directory. The summary value of this summary file will be counted as the summary value of the directory by the upper directory. However, there is a lack of effective file system synchronization methods in the prior art, and the summary generation speed is slow. For this reason, the present invention is proposed. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a file system synchronization method and system based on a vectorizable digest algorithm.

[0007] The technical solution of the present invention is as follows:

[0008] A file system synchronization method based on a vectorizable digest algorithm, the steps are as follows:

[0009] (1) For the file directory to be synchronized, use the digest algorithm to construct a digest tree for the composition structure of the file directory;

[0010] (2) storing the file directory of the generated summary tree and its subordinate files on the remote server, and sending a synchronization instruction to the terminal;

[0011] (3) The terminal performs file directory synchronization;

[0012] (4) Daily maintenance of synchronization directory.

[0013] Through the above process, the directories in the file system can be synchronized and updated safely and reliably at a low cost. The consistency of the digest values ​​of local files and server-side files can ensure the consistency of files. At the same time, the consistency of the digest trees of local directories and server directories can ensure the consistency of the two directories. Therefore, the directories that need to be synchronized in the file system can also be reliably guaranteed to be consistent.

[0014] According to a preferred embodiment of the present invention, in step (1), the summary tree construction process includes file directory processing and file processing in the directory, which is specifically as follows:

[0015] File directory: Each file directory contains a summary file, which is used to describe the summary values ​​of the files in the directory and the next-level directory. The summary value of the summary file is used as the summary value of the file directory.

[0016] File: Use the digest algorithm to calculate the digest value of the file, and save the file name and the calculated digest value to the digest file in the directory where the file is located;

[0017] Summary tree: A tree structure of summary values ​​is formed by combining the summary values ​​of each subdirectory / file in a folder with the tree structure of the directory in the file system.

[0018] Preferably, according to the present invention, in step (1), the digest algorithm steps are:

[0019] ① First, select 8 linearly independent key vectors K1, K2, ... K8, and divide the original data into independent data blocks from D1, D2, ... Dn in units of 8 bytes (the part less than 8 bytes at the end is padded with 0xFF, 0xFF is a hexadecimal number, 1 byte is represented by 8 bits, 8 bits are all 1 in binary representation is 11111111, converted to hexadecimal is 0xFF);

[0020] ② Treat the independent data blocks of D1, D2...Dn as vectors and perform vector products with the secret key vector K1 to obtain F1K1, F2K1...FnK1, and aggregate the obtained vector products to obtain MK1 (accumulate, discard the high-order data of each dimension, and only keep one byte). Similarly, perform vector products with the secret key vectors K2, K3...K8 and aggregate the results to obtain MK2, MK3...MK8;

[0021] The high-order data is the data that overflows when doing the product, such as 200*200=40000, which is expressed in unsigned hexadecimal as 0xC8*0xC8=0x9C40, which is 1-byte data multiplied by 1-byte data and the result becomes 2-byte data. However, since the final result requires 1-byte data, the high-order 9C of 0x9C40 is discarded, and only 0x40 is saved as the result;

[0022] ③Use accumulation, keep only the low byte, and aggregate MK1, MK2...MK8 to obtain summary M;

[0023] ④ For a single file in the file system, use steps ①-③ to get the summary value. For a directory in the file system, you need to create a summary file, use the summary file to record the names and summary values ​​of all the files it contains, and use the summary value of the summary file as the summary value of the entire directory.

[0024] Preferably, according to the present invention, in step (3), the synchronization process is:

[0025] Add the file directory that needs to be synchronized to the directory synchronization queue;

[0026] If the synchronization queue is empty, it means that the synchronization process is over. If the synchronization queue is not empty, a directory synchronization task is taken out of the queue. For this directory synchronization task, the summary file A of the file directory in the server is downloaded and compared with the summary file B in the local directory. The comparison content is a single data element, that is, the file / directory name and the summary value:

[0027] For a single data element, if it exists in summary file A but not in summary file B, it means that the item is newly added. If the item is a file, it can be downloaded directly. If the item is a directory, it will be added to the directory synchronization queue and wait for processing.

[0028] For a single data element, if it does not exist in summary file A but exists in summary file B: it means that the item is a deleted item, and the corresponding item in the terminal directory can be directly deleted;

[0029] For a single data element, if it exists in summary file A and also exists in summary file B, the summary values ​​of the item are compared. If the summary values ​​are the same, it means that the item has not changed. If the summary values ​​are different, if the item is a file, the file is downloaded again and replaced. If the item is a directory, the directory is added to the directory synchronization queue and waits for processing.

[0030] If all items in the current directory summary file have been processed, the current item will be deleted from the synchronization queue, and the synchronization requests in the synchronization queue will continue to be processed until the synchronization queue is empty. At this point, the synchronization task processing of the entire file directory is completed.

[0031] Preferably, in step (4), since the contents of the summary files in the directory based on the summary tree are the same, it can be ensured that the files in the two directories are the same. Therefore, during daily maintenance, it is only necessary to check whether the summary value of the summary file under the server synchronization directory is consistent with the summary value of the summary file under the local synchronization directory to determine whether the file under the synchronization directory is updated, which greatly reduces the network overhead and work overhead of daily maintenance.

[0032] Since the above calculation processes are independent of each other, they can be easily divided into small calculation units, which is convenient for using multiple cores for parallel processing. Moreover, since the calculations used are mostly multiplication and addition, it is more friendly to optimize using vectorized instructions in various architectures.

[0033] A file system synchronization system based on a vectorizable digest algorithm, comprising:

[0034] A summary tree construction module is used to construct a summary tree for the composition structure of the file directory to be synchronized using a summary algorithm;

[0035] The instruction sending module is used to store the file directory of the generated summary tree and its subordinate files on the remote server and send a synchronization instruction to the terminal;

[0036] Synchronization module, used to synchronize the file directory of the terminal;

[0037] Maintenance module, used for daily maintenance of synchronization directories.

[0038] The beneficial effects of the present invention are:

[0039] The present invention customizes a set of vectorizable summary algorithms and uses them to construct summary trees, which are applied in the synchronization process of the file system. For the calculation process of the summary, the advantage is that it can be vectorized. By converting data into vectors for calculation, the parallel processing capabilities of modern computers can be fully utilized, and the speed of summary generation can be significantly improved. At the same time, the content that needs to be synchronized can be quickly identified and the corresponding operations can be performed. For tasks that require large-scale periodic file system synchronization, the unity of correctness and speed can be achieved efficiently, greatly improving the efficiency of file system synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an example diagram of the summary algorithm of the present invention.

[0041] Figure 2 This is a diagram of the file system summary tree structure of the present invention.

[0042] Figure 3 It is the summary tree logic diagram of the present invention.

[0043] Figure 4 It is a schematic diagram of the use of the directory summary file of the present invention during synchronization.

[0044] Figure 5 This is a flowchart of file system directory synchronization based on the abstract tree structure of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0046] Embodiment 1:

[0047] This embodiment provides a file system synchronization method based on a vectorizable digest algorithm, and the steps are as follows:

[0048] (1) For the file directory to be synchronized, use the digest algorithm to construct a digest tree for the composition structure of the file directory;

[0049] The summary tree construction process includes file directory processing and file processing in the directory, as follows:

[0050] File directory: Each file directory contains a summary file, which is used to describe the summary values ​​of the files in the directory and the next-level directory. The summary value of the summary file is used as the summary value of the file directory.

[0051] File: Use the digest algorithm to calculate the digest value of the file, and save the file name and the calculated digest value to the digest file in the directory where the file is located;

[0052] The summary algorithm steps are:

[0053] ① First, select 8 linearly independent key vectors K1, K2, ... K8, and divide the original data into independent data blocks from D1, D2, ... Dn in units of 8 bytes (the part less than 8 bytes at the end is padded with 0xFF, 0xFF is a hexadecimal number, 1 byte is represented by 8 bits, 8 bits are all 1 in binary representation is 11111111, converted to hexadecimal is 0xFF);

[0054] ② Treat the independent data blocks of D1, D2...Dn as vectors and perform vector products with the secret key vector K1 to obtain F1K1, F2K1...FnK1, and aggregate the obtained vector products to obtain MK1 (accumulate, discard the high-order data of each dimension, and only keep one byte). Similarly, perform vector products with the secret key vectors K2, K3...K8 and aggregate the results to obtain MK2, MK3...MK8;

[0055] The high-order data is the data that overflows when doing the product, such as 200*200=40000, which is expressed in unsigned hexadecimal as 0xC8*0xC8=0x9C40, which is 1-byte data multiplied by 1-byte data and the result becomes 2-byte data. However, since the final result requires 1-byte data, the high-order 9C of 0x9C40 is discarded, and only 0x40 is saved as the result;

[0056] ③Use accumulation, keep only the low byte, and aggregate MK1, MK2...MK8 to obtain summary M;

[0057] ④ For a single file in the file system, use steps ①-③ to get the summary value. For a directory in the file system, you need to create a summary file, use the summary file to record the names and summary values ​​of all the files it contains, and use the summary value of the summary file as the summary value of the entire directory.

[0058] Summary tree: The summary values ​​of each subdirectory / file in the folder are combined with the tree structure of the directory in the file system to form a tree structure of summary values. For details, see Figure 2 , Figure 3 The contrast, Figure 2 That is, the digest value of each subdirectory / file is calculated using the digest algorithm. Figure 3 That is, the tree structure of summary values ​​formed by combining the tree structure of the directory is called the summary tree.

[0059] (2) storing the file directory of the generated summary tree and its subordinate files on the remote server, and sending a synchronization instruction to the terminal;

[0060] (3) The terminal performs file directory synchronization. The synchronization process is as follows:

[0061] Add the file directory that needs to be synchronized to the directory synchronization queue;

[0062] If the synchronization queue is empty, it means that the synchronization process is over. If the synchronization queue is not empty, a directory synchronization task is taken out of the queue. For this directory synchronization task, the summary file A of the file directory in the server is downloaded and compared with the summary file B in the local directory. The comparison content is a single data element, that is, the file / directory name and the summary value. Figure 4 As shown:

[0063] For a single data element, if it is in summary file A but not in summary file B (e.g. Figure 4 File 1): indicates that the item is a new item. If the item is a file, it can be downloaded directly. If the item is a directory, it will be added to the directory synchronization queue and wait for processing;

[0064] For a single data element, if it is not in summary file A but is in summary file B (e.g. Figure 4 File 2): This indicates that the item is a deletion item. Simply delete the corresponding item in the terminal directory.

[0065] For a single data element, if it exists in summary file A and also exists in summary file B, the summary values ​​of the item are compared. If the summary values ​​are the same (such as Figure 4 3), then the item has not changed. If the digest value is different (such as Figure 4 4), if the item is a file, re-download the file and replace it; if the item is a directory, add the directory to the directory synchronization queue and wait for processing;

[0066] If all items in the current directory summary file have been processed, the current item will be deleted from the synchronization queue, and the synchronization requests in the synchronization queue will continue to be processed until the synchronization queue is empty. At this point, the synchronization task processing of the entire file directory is completed.

[0067] (4) Daily maintenance of the synchronization directory. Specifically, since the contents of the summary files in the directory based on the summary tree are the same, the files in the two directories can be guaranteed to be the same. Therefore, during daily maintenance, it is only necessary to check whether the summary value of the summary file under the server synchronization directory is consistent with the summary value of the summary file under the local synchronization directory to determine whether the file under the synchronization directory is updated, which greatly reduces the network overhead and work overhead of daily maintenance.

[0068] Embodiment 2:

[0069] This embodiment provides a file system synchronization system based on a vectorizable digest algorithm, including:

[0070] A summary tree construction module is used to construct a summary tree for the composition structure of the file directory to be synchronized using a summary algorithm;

[0071] The instruction sending module is used to store the file directory of the generated summary tree and its subordinate files on the remote server and send a synchronization instruction to the terminal;

[0072] Synchronization module, used to synchronize the file directory of the terminal;

[0073] Maintenance module, used for daily maintenance of synchronization directories.

[0074] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A file system synchronization method based on a vectorizable digest algorithm, characterized in that: Here are the steps: (1) For the file directory to be synchronized, use the digest algorithm to construct a digest tree for the composition structure of the file directory; (2) storing the file directory of the generated summary tree and its subordinate files on the remote server, and sending a synchronization instruction to the terminal; (3) The terminal performs file directory synchronization; (4) Daily maintenance of synchronization directory.

2. The file system synchronization method based on the vectorizable digest algorithm as claimed in claim 1, characterized in that: In step (1), the summary tree construction process includes file directory processing and file processing in the directory, as follows: File directory: Each file directory contains a summary file, which is used to describe the summary values ​​of the files in the directory and the next-level directory. The summary value of the summary file is used as the summary value of the file directory. File: Use the digest algorithm to calculate the digest value of the file, and save the file name and the calculated digest value to the digest file in the directory where the file is located; Summary tree: A tree structure of summary values ​​is formed by combining the summary values ​​of each subdirectory / file in a folder with the tree structure of the directory in the file system.

3. The file system synchronization method based on the vectorizable digest algorithm as claimed in claim 2, characterized in that: In step (1), the digest algorithm steps are: ① First, select 8 linearly independent key vectors K1, K2...K8, and divide the original data into independent data blocks from D1, D2...Dn in units of 8 bytes; ② Treat the independent data blocks of D1, D2...Dn as vectors and perform vector products with the secret key vector K1 to obtain F1K1, F2K1...FnK1, and aggregate the obtained vector products to obtain MK1. Similarly, perform vector products with the secret key vectors K2, K3...K8 and aggregate the results to obtain MK2, MK3...MK8; ③Use accumulation, keep only the low byte, and aggregate MK1, MK2...MK8 to obtain summary M; ④ For a single file in the file system, use steps ①-③ to get the summary value. For a directory in the file system, you need to create a summary file, use the summary file to record the names and summary values ​​of all the files it contains, and use the summary value of the summary file as the summary value of the entire directory.

4. The file system synchronization method based on the vectorizable digest algorithm as claimed in claim 3, characterized in that: In step (3), the synchronization process is: Add the file directory that needs to be synchronized to the directory synchronization queue; If the synchronization queue is empty, it means that the synchronization process is over. If the synchronization queue is not empty, a directory synchronization task is taken out of the queue. For this directory synchronization task, the summary file A of the file directory in the server is downloaded and compared with the summary file B in the local directory. The comparison content is a single data element, that is, the file / directory name and the summary value: For a single data element, if it exists in summary file A but not in summary file B, it means that the item is newly added. If the item is a file, it can be downloaded directly. If the item is a directory, it will be added to the directory synchronization queue and wait for processing. For a single data element, if it does not exist in summary file A but exists in summary file B: it means that the item is a deleted item, and the corresponding item in the terminal directory can be directly deleted; For a single data element, if it exists in summary file A and also exists in summary file B, the summary values ​​of the item are compared. If the summary values ​​are the same, it means that the item has not changed. If the summary values ​​are different, if the item is a file, the file is downloaded again and replaced. If the item is a directory, the directory is added to the directory synchronization queue and waits for processing. If all items in the current directory summary file have been processed, the current item will be deleted from the synchronization queue, and the synchronization requests in the synchronization queue will continue to be processed until the synchronization queue is empty. At this point, the synchronization task processing of the entire file directory is completed.

5. The file system synchronization method based on the vectorizable digest algorithm as claimed in claim 4, characterized in that: In step (4), specifically, since the contents of the summary files in the directory based on the summary tree are the same, the files in the two directories can be guaranteed to be the same. Therefore, during daily maintenance, it is only necessary to check whether the summary value of the summary file in the server synchronization directory is consistent with the summary value of the summary file in the local synchronization directory to determine whether the files in the synchronization directory are updated.

6. A file system synchronization system based on a vectorizable digest algorithm, characterized in that: include: A summary tree construction module is used to construct a summary tree for the composition structure of the file directory to be synchronized using a summary algorithm; The instruction sending module is used to store the file directory of the generated summary tree and its subordinate files on the remote server and send a synchronization instruction to the terminal; Synchronization module, used to synchronize the file directory of the terminal; Maintenance module, used for daily maintenance of synchronization directories.

Images